From 8521af145f8e640a8390adb341b5d8a8ab595a60 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 12 May 2026 22:35:55 +0000 Subject: [PATCH] ci(merge): source per-arch digests from ci-cache, not local-ai-backends MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to PR #9781. v4.2.2 (run 25745181433) showed the keepalive anchor in ci-cache wasn't enough on its own: 19 of 37 multiarch merges still failed with "manifest not found" for the same digests we'd just anchored. Quay's manifest GC is per-repository. The anchor tag in ci-cache protects the manifest copy that lives in ci-cache, but the same digest in local-ai-backends is independently tracked and gets reaped because nothing in local-ai-backends references it (push-by-digest=true leaves it untagged). The merge then asks `local-ai-backends@sha256:` and quay correctly says "not found" in that repo, even though `ci-cache@sha256:` is alive and well. Empirical confirmation against a live failed digest from v4.2.2: $ docker buildx imagetools inspect quay.io/go-skynet/ci-cache@sha256:05377fe6... Name: quay.io/go-skynet/ci-cache@sha256:05377fe6... MediaType: application/vnd.docker.distribution.manifest.v2+json $ docker buildx imagetools inspect quay.io/go-skynet/local-ai-backends@sha256:05377fe6... ERROR: ... not found Switch the source of the quay merge step to ci-cache. The blobs the manifest references are already accessible from local-ai-backends (verified via direct registry HEAD: HTTP 200 from both repos — the original push cross-mounted blobs at content-addressable storage time and they outlive the per-repo manifest GC). buildx imagetools create republishes the manifest into local-ai-backends, then writes the user-facing manifest list pointing at it. End state is self-contained: the published manifest list references child manifests by digest only, no embedded reference to ci-cache. Dockerhub merge step is unchanged. Dockerhub's GC isn't aggressive enough to reap untagged manifests at the timescales we operate on (verified: localai/localai-backends@ still resolves cleanly after >24h). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto --- .github/workflows/backend_merge.yml | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/.github/workflows/backend_merge.yml b/.github/workflows/backend_merge.yml index 0490cc6b3..896ac86fc 100644 --- a/.github/workflows/backend_merge.yml +++ b/.github/workflows/backend_merge.yml @@ -88,6 +88,25 @@ jobs: latest=${{ inputs.tag-latest }} suffix=${{ inputs.tag-suffix }},onlatest=true + # Source from ci-cache, not local-ai-backends. + # + # The build job pushes per-arch manifests to local-ai-backends with + # push-by-digest=true (no tag), then anchors a tagged copy into + # ci-cache so the manifest can be retrieved hours later when this + # merge runs. Quay's manifest GC, however, is per-repository: the + # anchor tag in ci-cache protects the manifest there, but the same + # digest in local-ai-backends has no tag in *that* repo and gets + # reaped independently. Sourcing local-ai-backends@ here + # then fails with "manifest not found" — exactly the regression + # we hit on v4.2.2 (19/37 multiarch merges failed). + # + # ci-cache@ resolves because we anchored it there. buildx + # imagetools create copies the manifest into local-ai-backends + # (cross-repo within the same registry, blobs already cross-mounted + # from the original push so no transfer needed) and publishes the + # manifest list with the user-facing tags. The resulting manifest + # list is fully self-contained in local-ai-backends — child digests + # only, no embedded references to ci-cache. - name: Create manifest list and push (quay) if: github.event_name != 'pull_request' working-directory: /tmp/digests @@ -104,7 +123,7 @@ jobs: else # shellcheck disable=SC2086 docker buildx imagetools create $tags \ - $(printf 'quay.io/go-skynet/local-ai-backends@sha256:%s ' *) + $(printf 'quay.io/go-skynet/ci-cache@sha256:%s ' *) fi - name: Create manifest list and push (dockerhub)