From 9689e4acea77d515db7a6865fd9c6d1fe6865313 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 30 Jul 2026 13:02:04 +0000 Subject: [PATCH] fix(ci): skip the master image rebuild for commits no image can see On 2026-07-30, 12 of the 23 queued runs of this workflow were commits like "add 1 new model to gallery" or a docs fix, each rebuilding all 18 container images. That was roughly 216 queued jobs producing byte-identical output, in a queue holding 1071 jobs with an oldest entry two days old. Verified against the shipped Dockerfile before assuming it: the final stage copies only entrypoint.sh, healthcheck.sh and the local-ai binary, there is no go:embed of gallery/ or docs/, and the gallery is fetched at runtime from github:mudler/LocalAI/gallery/index.yaml@master. A gallery-only commit produces an identical image, and the gallery change reaches users through GitHub whether or not an image is rebuilt, so nothing is delayed by skipping. Add a `changes` job that decides once whether the push can affect an image; the other 11 jobs take `needs: changes` and an `if:` on its output. A job gate rather than paths-ignore on the trigger, for two reasons that both fail silently if got wrong: - paths-ignore on `push` also applies to tag pushes, and a tag created on an existing commit carries an empty commits list. That would skip the release image build with no failure anywhere. The gate short-circuits to build for refs/tags/*, and for a base commit that is missing, zero or unresolvable -- the same run-everything posture the backend matrix filter takes for a truncated diff. - the merge jobs use `if: ${{ !cancelled() && ... }}`, and !cancelled() is true when a dependency is skipped, so they need the gate named explicitly or they would try to merge manifest lists for images never built. Checked the decision logic against real commits from the queue: the two gallery/docs commits resolve to build=false, the two code commits to build=true, and all three fallback paths (tag, zero base, unresolvable base) to build=true. Signed-off-by: Ettore Di Giacinto Assisted-by: Claude:opus-5 [claude-code] --- .agents/ci-caching.md | 11 ++++- .github/workflows/image.yml | 85 ++++++++++++++++++++++++++++--------- 2 files changed, 76 insertions(+), 20 deletions(-) diff --git a/.agents/ci-caching.md b/.agents/ci-caching.md index cc0e7c5d0..a79199c80 100644 --- a/.agents/ci-caching.md +++ b/.agents/ci-caching.md @@ -173,7 +173,16 @@ Two properties this relies on: - `paths-ignore` skips a run only when **every** changed file matches, so a PR touching the gallery *and* Go code still runs everything. That is what makes the exclusion safe rather than a hole. - `master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these four entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported". -Deliberately **not** filtered: `image.yml` on master push still rebuilds all 18 container images for a gallery-only commit. Skipping it would mean the `master` and `latest` tags are not republished for that commit, which is a publishing-semantics decision rather than a pure cost one. +### `image.yml` on master push is gated too, by a job rather than a path filter + +The same reasoning applies to master pushes, and the volume is larger there: on 2026-07-30, **12 of the 23 queued `image.yml` runs** were commits like "add 1 new model to gallery" or a docs fix, each rebuilding all 18 container images. + +`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `github:mudler/LocalAI/gallery/index.yaml@master`. A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users through GitHub immediately whether or not an image is rebuilt. + +Two properties to preserve if you touch it: + +- **It is a job gate, not `paths-ignore`.** `paths-ignore` on `push` also applies to tag pushes, and a tag created on an existing commit carries an empty commits list, which would silently skip the release image build. The gate short-circuits to "build" for `refs/tags/*`, and for any push whose base commit is missing, zero, or unresolvable. +- **The merge jobs must name the gate explicitly.** They use `if: ${{ !cancelled() && ... }}`, and `!cancelled()` is true when a dependency is *skipped*, so without the extra condition they would run and try to merge manifest lists for images that were never built. ## The `DEPS_REFRESH` cache-buster (Python backends) diff --git a/.github/workflows/image.yml b/.github/workflows/image.yml index 655ef1b43..cd90e849b 100644 --- a/.github/workflows/image.yml +++ b/.github/workflows/image.yml @@ -13,8 +13,53 @@ cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: - hipblas-jobs: + # Decide once whether this push can change any image. Gallery metadata is + # fetched at runtime and never baked into an image, and docs/markdown never + # enter one, so a push confined to those paths produces byte-identical + # images. On 2026-07-30, 12 of the 23 queued runs of this workflow were + # commits like "add 1 new model to gallery" or a docs fix, each rebuilding + # all 18 images. + # + # A job-level gate rather than `paths-ignore` on the trigger: paths-ignore + # would also apply to tag pushes, and a tag created on an existing commit + # carries an empty commits list, which would silently skip the release image + # build. Tags short-circuit to "build" below, as does a push whose base + # commit cannot be resolved -- the same run-everything posture the backend + # matrix filter takes for a truncated diff. + changes: if: github.repository == 'mudler/LocalAI' + runs-on: ubuntu-latest + outputs: + build: ${{ steps.decide.outputs.build }} + steps: + - uses: actions/checkout@v7 + with: + fetch-depth: 0 + - id: decide + env: + BEFORE: ${{ github.event.before }} + AFTER: ${{ github.sha }} + run: | + set -euo pipefail + emit() { echo "$2"; echo "build=$1" >> "$GITHUB_OUTPUT"; exit 0; } + case "${GITHUB_REF}" in + refs/tags/*) emit true "tag push: building every image" ;; + esac + if [ -z "${BEFORE:-}" ] || [ "${BEFORE}" = "0000000000000000000000000000000000000000" ] \ + || ! git cat-file -e "${BEFORE}^{commit}" 2>/dev/null; then + emit true "no resolvable base commit: building every image" + fi + files="$(git diff --name-only "${BEFORE}" "${AFTER}")" + echo "changed files:"; echo "${files:-}" + [ -z "${files}" ] && emit true "empty diff: building every image" + if echo "${files}" | grep -qvE '^(gallery/|docs/|examples/)|\.md$'; then + emit true "push touches image-visible content: building" + fi + emit false "only gallery/docs/markdown changed: images identical, skipping" + + hipblas-jobs: + needs: changes + if: github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' uses: ./.github/workflows/image_build.yml with: tag-latest: ${{ matrix.tag-latest }} @@ -47,7 +92,8 @@ ubuntu-codename: 'noble' core-image-build: - if: github.repository == 'mudler/LocalAI' + needs: changes + if: github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' uses: ./.github/workflows/image_build.yml with: tag-latest: ${{ matrix.tag-latest }} @@ -155,8 +201,8 @@ # merge whenever any matrix cell of the parent build fails or is # cancelled. Same fix as backend.yml's merge jobs — we still want to # publish the manifest list for tag-suffixes whose legs all succeeded. - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: core-image-build + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, core-image-build] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -168,8 +214,8 @@ quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} gpu-vulkan-image-merge: - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: core-image-build + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, core-image-build] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -187,8 +233,8 @@ # Each merge job needs only its parent build matrix and is filtered by # tag-suffix in image_merge.yml's artifact-download pattern. gpu-nvidia-cuda-12-image-merge: - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: core-image-build + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, core-image-build] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -200,8 +246,8 @@ quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} gpu-nvidia-cuda-13-image-merge: - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: core-image-build + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, core-image-build] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -213,8 +259,8 @@ quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} gpu-intel-image-merge: - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: core-image-build + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, core-image-build] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -226,8 +272,8 @@ quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} gpu-hipblas-image-merge: - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: hipblas-jobs + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, hipblas-jobs] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -239,8 +285,8 @@ quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} nvidia-l4t-arm64-image-merge: - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: gh-runner + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, gh-runner] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -252,8 +298,8 @@ quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} nvidia-l4t-arm64-cuda-13-image-merge: - if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }} - needs: gh-runner + if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' }} + needs: [changes, gh-runner] uses: ./.github/workflows/image_merge.yml with: tag-latest: 'auto' @@ -265,7 +311,8 @@ quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} gh-runner: - if: github.repository == 'mudler/LocalAI' + needs: changes + if: github.repository == 'mudler/LocalAI' && needs.changes.outputs.build == 'true' uses: ./.github/workflows/image_build.yml with: tag-latest: ${{ matrix.tag-latest }}