mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
10ad9c9392
Backend images publish an immutable `sha-<commit>-<suffix>` tag alongside mutable `master-*`, `latest-*` and `v<version>-*` tags. Nothing checked whether an incoming build was newer than what a mutable tag already pointed at, so last writer won. That is not theoretical. Backend CI queues run hours deep (measured 259.7 min average queue wait against 18.6 min average execution, peak concurrency 19) and master pushes get a concurrency group keyed by github.sha, so no run supersedes another and completion order does not track commit order. On 19 Jul 2026: 10:41 UTC build of626ae4d51(contains the cuDNN packaging fix from #10946) advanced master-nvidia-l4t-cuda-13-arm64-longcat-video 15:40 UTC build of10211948b, a commit from 06:45 UTC that predates the fix, overwrote the same tag Every consumer pulling `master-*` after that got a pre-fix image, which bundles a partial 4-of-8 cuDNN library set at 9.24.0 against a venv shipping a complete set at 9.20.0.48 and fails at inference with CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH. The fix was built correctly and on time, then un-shipped by a straggler, silently, for two days. Add an ordering guard. `scripts/tag-guard.mjs` reads the tag list docker/metadata-action produced, resolves what each mutable tag currently points at via its `org.opencontainers.image.revision` label, asks the GitHub compare API how the incoming commit relates to it, and emits only the tags this build may publish. Immutable `sha-*` tags always publish: pinning one is how this incident was worked around and that must keep working. Wired into both push paths that write mutable tags: - backend_merge.yml, quay and Docker Hub `imagetools create` steps. This is the multi-arch manifest merge and also the tagging step for single-arch backends, since backend_build.yml pushes by digest only. - backend_build_darwin.yml publish job. Darwin images are `crane push`ed from a raw OCI tarball and carried no labels at all, so the job now stamps the revision with `crane mutate` after pushing; until each darwin tag has gone through that once the guard fails open with a warning. Blocked tags are logged, annotated with `::warning::` and listed in the job summary. The whole failure mode being fixed here is silence, so a quiet skip would be the same bug wearing a different hat. Unknown or unresolvable revisions and registry/API errors fail open with a warning rather than blocking: a guard that failed closed on an API blip would itself stop shipping merged fixes. Release `v*` tags go through the guard too. A fresh version tag has never been published so it takes the "tag does not exist yet" path and always publishes; the guard costs nothing there and does protect `latest-*` if an old release build is ever re-run. Logic lives in scripts/lib/tag-guard.mjs with unit tests in scripts/lib/tag-guard_test.mjs, run by `make test-ci-scripts` from lint.yml. Registry and GitHub lookups take an injected fetch so the decision table is tested without network access. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]