mirror of
https://github.com/mudler/LocalAI.git
synced 2026-05-18 05:33:09 -04:00
* ci: close the GC race + cascade-skip + darwin grpc gaps from v4.2.1
v4.2.1's backend.yml run (#25701862853) exposed three independent issues
on top of the singletons fix shipped in ea001995. Address all three plus
two related cleanups:
1. quay GC race in backend-merge-jobs-multiarch (12/37 merges failed with
"manifest not found"). Even after PR #9746 split multi/single-arch
merges, the multiarch matrix itself takes ~2h to drain at
max-parallel: 8, and the earliest per-arch digests (push-by-digest,
no tag) get reaped by quay's GC before the merge runs. The split
bounded the race for multiarch; it doesn't eliminate it. Anchor each
per-arch digest immediately to a tag in the internal ci-cache image
(`keepalive-<run_id><tag-suffix>-<platform-tag>`). Quay won't GC
tagged manifests. backend_merge.yml deletes the keepalive tags via
quay REST API after publishing the user-facing manifest list.
Cleanup is best-effort: if the quay token is not OAuth-scoped the
merge does NOT fail, the orphan tags just persist.
2. cascade-skip on backend-merge-jobs-singlearch. v4.2.1 had 2 failed
and 2 cancelled singlearch builds (out of 199); GHA's default
`needs:` semantics cascade-skipped the entire singlearch merge
matrix, so zero singleton tags were applied even though 197
singletons built successfully. Wrap the merge `if:` in
`!cancelled() && ...` for both multi and single arch in backend.yml
and backend_pr.yml so partial build failures publish the successful
tag-suffixes.
3. Darwin llama-cpp grpc-server build fails with `find_package(absl)`
not found. Same shape as the ccache/blake3/fmt/hiredis/xxhash/zstd
fix already in `Dependencies`: a brew cache hit restores
`/opt/homebrew/Cellar/grpc` so `brew install grpc` no-ops, but
abseil isn't in our Cellar cache list and never gets installed
alongside, leaving grpc's CMake unable to resolve it. Mirror the
`brew reinstall ccache` line with `brew reinstall grpc` to
re-validate grpc's full transitive dep closure on every cache-hit
run.
4. Move the four heaviest CUDA cpp builds back to bigger-runner. v4.2.1
wall-clock: -gpu-nvidia-cuda-12-llama-cpp 5h36m,
-gpu-nvidia-cuda-12-turboquant 6h05m,
-gpu-nvidia-cuda-13-llama-cpp 5h37m,
-gpu-nvidia-cuda-13-turboquant 6h05m. The cuda-12 turboquant and
cuda-13 turboquant entries are over GHA's 6h job timeout. Phase 5.3
of the free-tier migration (PR #9730) had explicitly flagged this
batch as 'highest-risk' with a per-entry revert path. All other
matrix entries (vulkan-llama-cpp ~47m, ROCm hipblas-llama-cpp ~2h,
intel sycl-f32 ~1h49m) stay on free-tier ubuntu-latest.
Verified locally: all six edited workflow YAMLs parse cleanly. Real
verification has to come from the next tag release run.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ci: extract keepalive anchor + cleanup into .github/scripts/
The two inline shell blocks from the previous commit are long enough to
hurt readability of the workflow YAML and benefit from their own files
with self-contained docs. Move them to .github/scripts/:
anchor-digest-in-cache.sh backend_build.yml's keepalive anchor
cleanup-keepalive-tags.sh backend_merge.yml's best-effort cleanup
Workflow steps reduce to a single `run:` invocation each, with all the
parameter plumbing handled by env vars on the step. backend_merge.yml
also gains a sparse `actions/checkout@v6` step (sparse to .github/scripts
only) so the cleanup script is available on the runner — backend_build
already checks out for the docker build.
Net workflow diff: -36 lines across the two files. Script logic and
behavior are byte-identical to the inline version.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
156 lines
5.3 KiB
YAML
156 lines
5.3 KiB
YAML
---
|
|
name: 'merge backend manifest list (reusable)'
|
|
|
|
# Reusable workflow that joins per-arch digest artifacts (uploaded by
|
|
# backend_build.yml when called with platform-tag) into a single tagged
|
|
# multi-arch manifest list. Called once per backend by backend.yml after
|
|
# both per-arch build jobs succeed.
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
tag-latest:
|
|
description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
|
|
required: false
|
|
type: string
|
|
default: ''
|
|
tag-suffix:
|
|
description: 'Backend tag suffix (e.g. -cpu-faster-whisper). Used to compute the artifact pattern and the final tag suffix.'
|
|
required: true
|
|
type: string
|
|
secrets:
|
|
dockerUsername:
|
|
required: false
|
|
dockerPassword:
|
|
required: false
|
|
quayUsername:
|
|
required: true
|
|
quayPassword:
|
|
required: true
|
|
|
|
jobs:
|
|
merge:
|
|
runs-on: ubuntu-latest
|
|
env:
|
|
quay_username: ${{ secrets.quayUsername }}
|
|
steps:
|
|
# Sparse checkout: the merge job needs `.github/scripts/` (for the
|
|
# keepalive cleanup script) but none of the source tree.
|
|
- name: Checkout (.github/scripts only)
|
|
uses: actions/checkout@v6
|
|
with:
|
|
sparse-checkout: |
|
|
.github/scripts
|
|
sparse-checkout-cone-mode: false
|
|
|
|
# `--` separator anchors the glob so we don't over-match sibling
|
|
# backends whose tag-suffix happens to be a prefix of ours
|
|
# (e.g. -cpu-vllm vs -cpu-vllm-omni). Must stay in sync with the
|
|
# upload-artifact name in backend_build.yml.
|
|
- name: Download digests
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
pattern: digests${{ inputs.tag-suffix }}--*
|
|
merge-multiple: true
|
|
path: /tmp/digests
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@master
|
|
|
|
- name: Login to DockerHub
|
|
if: github.event_name != 'pull_request'
|
|
uses: docker/login-action@v4
|
|
with:
|
|
username: ${{ secrets.dockerUsername }}
|
|
password: ${{ secrets.dockerPassword }}
|
|
|
|
- name: Login to Quay.io
|
|
if: ${{ env.quay_username != '' }}
|
|
uses: docker/login-action@v4
|
|
with:
|
|
registry: quay.io
|
|
username: ${{ secrets.quayUsername }}
|
|
password: ${{ secrets.quayPassword }}
|
|
|
|
- name: Docker meta
|
|
id: meta
|
|
if: github.event_name != 'pull_request'
|
|
uses: docker/metadata-action@v6
|
|
with:
|
|
images: |
|
|
quay.io/go-skynet/local-ai-backends
|
|
localai/localai-backends
|
|
tags: |
|
|
type=ref,event=branch
|
|
type=semver,pattern={{raw}}
|
|
type=sha
|
|
flavor: |
|
|
latest=${{ inputs.tag-latest }}
|
|
suffix=${{ inputs.tag-suffix }},onlatest=true
|
|
|
|
- name: Create manifest list and push (quay)
|
|
if: github.event_name != 'pull_request'
|
|
working-directory: /tmp/digests
|
|
run: |
|
|
set -euo pipefail
|
|
tags=$(jq -cr '
|
|
.tags
|
|
| map(select(startswith("quay.io/")))
|
|
| map("-t " + .)
|
|
| join(" ")
|
|
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
|
if [ -z "$tags" ]; then
|
|
echo "No quay.io tags from docker/metadata-action; skipping quay merge"
|
|
else
|
|
# shellcheck disable=SC2086
|
|
docker buildx imagetools create $tags \
|
|
$(printf 'quay.io/go-skynet/local-ai-backends@sha256:%s ' *)
|
|
fi
|
|
|
|
- name: Create manifest list and push (dockerhub)
|
|
if: github.event_name != 'pull_request'
|
|
working-directory: /tmp/digests
|
|
run: |
|
|
set -euo pipefail
|
|
tags=$(jq -cr '
|
|
.tags
|
|
| map(select(startswith("localai/")))
|
|
| map("-t " + .)
|
|
| join(" ")
|
|
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
|
if [ -z "$tags" ]; then
|
|
echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
|
|
else
|
|
# shellcheck disable=SC2086
|
|
docker buildx imagetools create $tags \
|
|
$(printf 'localai/localai-backends@sha256:%s ' *)
|
|
fi
|
|
|
|
- name: Inspect manifest
|
|
if: github.event_name != 'pull_request'
|
|
run: |
|
|
set -euo pipefail
|
|
first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
|
if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
|
|
docker buildx imagetools inspect "$first_tag"
|
|
fi
|
|
|
|
# See .github/scripts/cleanup-keepalive-tags.sh for why this is
|
|
# best-effort and what the failure modes are.
|
|
- name: Cleanup keepalive tags in ci-cache
|
|
if: github.event_name != 'pull_request' && success()
|
|
env:
|
|
TAG_SUFFIX: ${{ inputs.tag-suffix }}
|
|
QUAY_TOKEN: ${{ secrets.quayPassword }}
|
|
run: .github/scripts/cleanup-keepalive-tags.sh
|
|
|
|
- name: Job summary
|
|
if: github.event_name != 'pull_request'
|
|
run: |
|
|
set -euo pipefail
|
|
echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
|
|
jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
|
|
echo >> "$GITHUB_STEP_SUMMARY"
|
|
echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
|
|
ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"
|