From 2bee7a5ab1b5bf7433ae91db86b0e8756f5f85ad Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 27 Jun 2026 08:29:09 +0000 Subject: [PATCH] ci(paged): add early-warning canary for vendored llama.cpp paged patches The paged backend (backend/cpp/llama-cpp-localai-paged) pins its own verified llama.cpp tip and is excluded from the nightly auto-bumper so a naive bump can never silently break the shipped build. That exclusion also removed the early warning of upstream drift. This restores the signal without touching the pin. Add .github/workflows/llama-cpp-paged-canary.yml (weekly + workflow_dispatch): - apply-check job (ubuntu-latest, toolchain-free): resolve the latest ggml-org/llama.cpp master tip, shallow-checkout it, and apply the full paged series 0001-0030 in order with the build's own git-apply method via the new shared helper .github/scripts/paged-canary-apply.sh. Red on any apply break. - compile job (needs apply-check): on the exact tip it validated, build the paged backend (cublas) inside the same base-grpc-cuda-12 toolchain and the same `make grpc-server` target the shipped build uses, so a red means upstream drift, not toolchain noise. nvcc compiles the kernels with no GPU present. Red here = run a PIN_SYNC (rebase + bit-exact gate + re-export), then bump the paged Makefile pin. The canary is signal-only: it opens no PR and never moves the pin, so the shipped build and the dep-bump PRs stay green regardless. It is fully separate from bump_deps. The lone pre-existing quirk in the series (patch 0019 carries a stray modify hunk against the dev-only doc SSM_DECODE_FIX_RESULTS.md, absent from any clean upstream checkout; git apply is atomic so it rejects the whole patch and cascades to 0021/0022/0026/0028) is handled path-scoped: the helper excludes only that dev-doc and still applies 0019's real code hunks atomically, mirroring prepare.sh's tolerance, so the quirk never false-positives the canary but a genuine code break in 0019 still turns it red. Point the existing pin comments in backend/cpp/llama-cpp-localai-paged/Makefile and .github/workflows/bump_deps.yaml at this canary as the drift signal, and document it in the PIN_SYNC doc: canary red -> do a pin-sync. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto --- .github/scripts/paged-canary-apply.sh | 77 ++++++++ .github/workflows/bump_deps.yaml | 8 + .github/workflows/llama-cpp-paged-canary.yml | 173 ++++++++++++++++++ backend/cpp/llama-cpp-localai-paged/Makefile | 7 + .../patches/paged/PIN_SYNC_9d5d882d.md | 29 +++ 5 files changed, 294 insertions(+) create mode 100755 .github/scripts/paged-canary-apply.sh create mode 100644 .github/workflows/llama-cpp-paged-canary.yml diff --git a/.github/scripts/paged-canary-apply.sh b/.github/scripts/paged-canary-apply.sh new file mode 100755 index 000000000..1060560f2 --- /dev/null +++ b/.github/scripts/paged-canary-apply.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# +# paged-canary-apply.sh - apply the vendored paged-attention patch series +# (backend/cpp/llama-cpp/patches/paged/0001-0030) to a llama.cpp checkout, the +# same way the build does, but tolerating the ONE known-benign pre-existing +# quirk in the series. Used by the early-warning canary +# (.github/workflows/llama-cpp-paged-canary.yml) so it only goes red on a REAL +# upstream break, never on that quirk. +# +# Usage: paged-canary-apply.sh +# is normally backend/cpp/llama-cpp/patches (it holds the +# top-level base series 0*.patch, currently empty, and the paged/ subseries). +# +# Exit 0 = the whole series applied -> patches still fit upstream. +# Exit !=0 = a patch failed to apply = the red signal: an upstream change moved +# the tree out from under the patches, so it is time to run a PIN_SYNC. +# +# Apply method MIRRORS backend/cpp/llama-cpp/Makefile's `llama.cpp` target: +# plain `git apply --verbose`, which natively tolerates @@ line-number offsets +# but NOT context-line changes. Matching the build's method is the point - the +# canary's apply result is exactly what the real build's apply would do. +# +# The ONLY tolerance, and it is path-scoped (not a blanket `|| true`): patch +# 0019 carries a stray *modify* hunk against the dev-only doc +# SSM_DECODE_FIX_RESULTS.md, a file that exists only on the DGX dev tree and is +# absent from any clean upstream checkout. `git apply` is atomic, so that single +# missing-file hunk rejects the whole patch - and because 0021/0022/0026/0028 +# build on 0019's code, the rejection cascades to them too. This is a +# PRE-EXISTING shipped-series defect, present identically on every pin, NOT an +# upstream break (see backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md +# and PIN_BUMP_APPLY_CHECK.md). We exclude ONLY that dev-doc path and still +# apply 0019's real code hunks atomically, so a genuine code-hunk break in 0019 +# still fails the canary. prepare.sh tolerates the same hunk via +# `patch ... || true`; this mirrors that tolerance precisely. + +set -euo pipefail + +CHECKOUT="${1:?usage: paged-canary-apply.sh }" +PATCHES="${2:?usage: paged-canary-apply.sh }" + +# The lone tolerated dev-doc, and the only patch allowed to carry it. +DEVDOC_GLOB='*SSM_DECODE_FIX_RESULTS.md' +DEVDOC_PATCH='0019-qwen35-ssm-decode-fused-gather.patch' + +# Resolve to absolute paths so the apply works after we cd into the checkout. +PATCHES="$(cd "$PATCHES" && pwd)" +cd "$CHECKOUT" + +shopt -s nullglob + +apply_one() { + local p="$1"; shift + echo "paged-canary: applying $(basename "$p")" + if ! git apply --verbose "$@" "$p"; then + echo "::error::paged patch no longer applies to the upstream llama.cpp tip: $(basename "$p")" + echo "::error::upstream drifted past the vendored paged series - run a PIN_SYNC (backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md), do NOT bump the pin blindly" + exit 1 + fi +} + +# Base series first (parity with the build: patches/0*.patch before +# patches/paged/0*.patch). Currently empty; nullglob makes this a no-op. +for p in "$PATCHES"/0*.patch; do + apply_one "$p" +done + +# Paged series, in order. +for p in "$PATCHES"/paged/0*.patch; do + if [ "$(basename "$p")" = "$DEVDOC_PATCH" ]; then + # Apply 0019's real code hunks; exclude ONLY the benign dev-doc hunk. + apply_one "$p" --exclude="$DEVDOC_GLOB" + else + apply_one "$p" + fi +done + +echo "paged-canary: the full paged patch series applied cleanly to the upstream tip" diff --git a/.github/workflows/bump_deps.yaml b/.github/workflows/bump_deps.yaml index 506e2fe44..f3df0750d 100644 --- a/.github/workflows/bump_deps.yaml +++ b/.github/workflows/bump_deps.yaml @@ -18,6 +18,14 @@ jobs: # backend/cpp/llama-cpp-localai-paged/Makefile) and advanced ONLY by the # manual PIN_SYNC process. Do not add it here. (turboquant CAN be # auto-bumped below because its fork branch carries the patches.) + # + # Excluding it from the auto-bumper removed the early warning of upstream + # drift; that signal is restored separately by the dedicated canary + # .github/workflows/llama-cpp-paged-canary.yml, which weekly applies + + # compiles the paged series against the latest llama.cpp tip and goes red + # when upstream breaks it (prompting a PIN_SYNC). The canary is + # signal-only - it never opens a bump PR and never moves the pin - so + # this dep-bump workflow and its PRs stay green regardless. include: - repository: "ggml-org/llama.cpp" variable: "LLAMA_VERSION" diff --git a/.github/workflows/llama-cpp-paged-canary.yml b/.github/workflows/llama-cpp-paged-canary.yml new file mode 100644 index 000000000..1c29eb434 --- /dev/null +++ b/.github/workflows/llama-cpp-paged-canary.yml @@ -0,0 +1,173 @@ +name: 'llama.cpp paged patches: upstream canary' + +# EARLY-WARNING CANARY for the vendored paged-attention patch series +# (backend/cpp/llama-cpp/patches/paged/0001-0030). +# +# WHY THIS EXISTS +# The paged backend (backend/cpp/llama-cpp-localai-paged) pins its OWN verified +# llama.cpp tip (LLAMA_VERSION in backend/cpp/llama-cpp-localai-paged/Makefile) +# and is intentionally EXCLUDED from the nightly auto-bumper +# (.github/workflows/bump_deps.yaml), so a naive upstream bump can never silently +# break the shipped build. The cost of that safety: nobody finds out when +# upstream DRIFTS past the patches. This canary restores that signal WITHOUT +# touching the shipped pin - weekly it tries the patch series + a real compile +# against the LATEST llama.cpp master tip and goes red the moment upstream breaks +# the patches. +# +# RED HERE means: time to run a PIN_SYNC (rebase the patches onto the new tip, +# pass the bit-exact gate on the GPU, re-export the .patch files, THEN advance +# the pin in backend/cpp/llama-cpp-localai-paged/Makefile). See +# backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md. +# +# SIGNAL-ONLY: this workflow moves no pinned version, ships nothing, and is fully +# decoupled from bump_deps - so the main dep-bump PR stays green regardless. A +# green run means "the paged series still applies and compiles on upstream HEAD"; +# a red run means "upstream moved - schedule a pin-sync". + +on: + schedule: + # Weekly (Mondays 06:00 UTC), mirroring the weekly DEPS_REFRESH / bump_deps + # cadence. Offset from bump_deps' nightly 20:00 so the two never pile up. + - cron: '0 6 * * 1' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: llama-cpp-paged-canary + cancel-in-progress: false + +env: + # Upstream source of truth - the same repo/branch bump_deps tracks for the + # stock llama-cpp pin. + LLAMA_UPSTREAM: 'https://github.com/ggml-org/llama.cpp' + +jobs: + apply-check: + # Cheap, fast, toolchain-free early warning: does the series still APPLY to + # the latest upstream tip? A patch no longer applying is by far the most + # common way upstream breaks a vendored series, so this runs first, is + # reliable on a free runner, and feeds the resolved tip to the compile job. + if: github.repository == 'mudler/LocalAI' + runs-on: ubuntu-latest + timeout-minutes: 20 + outputs: + tip: ${{ steps.resolve.outputs.tip }} + steps: + - name: Checkout LocalAI + uses: actions/checkout@v7 + + - name: Resolve latest llama.cpp master tip + id: resolve + run: | + tip="$(git ls-remote "$LLAMA_UPSTREAM" refs/heads/master | cut -f1)" + if [ -z "$tip" ]; then + echo "::error::could not resolve llama.cpp master tip from $LLAMA_UPSTREAM" + exit 1 + fi + pin="$(grep -m1 'LLAMA_VERSION?=' backend/cpp/llama-cpp-localai-paged/Makefile | cut -d= -f2)" + echo "latest llama.cpp master tip: $tip" + echo "shipped paged pin: $pin" + echo "tip=$tip" >> "$GITHUB_OUTPUT" + { + echo "## llama.cpp paged canary" + echo "" + echo "- upstream master tip: \`$tip\`" + echo "- shipped paged pin: \`$pin\`" + } >> "$GITHUB_STEP_SUMMARY" + + - name: Checkout llama.cpp at latest tip (shallow) + run: | + mkdir -p /tmp/llama.cpp + cd /tmp/llama.cpp + git init -q + git remote add origin "$LLAMA_UPSTREAM" + git fetch -q --depth 1 origin "${{ steps.resolve.outputs.tip }}" + git checkout -q FETCH_HEAD + git log --oneline -1 + + - name: Apply paged patch series (build's git-apply method) + run: | + bash .github/scripts/paged-canary-apply.sh \ + /tmp/llama.cpp \ + "$PWD/backend/cpp/llama-cpp/patches" + echo "- apply: full paged series applies to the upstream tip :white_check_mark:" >> "$GITHUB_STEP_SUMMARY" + + compile: + # Proves the patches still COMPILE against the latest tip, using the SAME + # toolchain + build target the shipped paged backend uses (the + # base-grpc-cuda-12 builder base + the Makefile `grpc-server` cublas target), + # so a failure means upstream drift, not toolchain noise. CUDA is compiled + # (nvcc; no GPU required) because most of the paged series is CUDA kernels. + # Runs only if the apply check passed, on the exact tip it validated. + # + # If a full CUDA compile on the hosted runner ever proves too heavy/flaky, + # switch `runs-on` to 'bigger-runner' (the runner class the real paged CUDA + # build uses), or drop to a CPU build (BUILD_TYPE='') which still compiles + # all host + CPU paged code, leaving CUDA-kernel coverage to the apply check + # plus the manual PIN_SYNC GPU gate. + needs: apply-check + if: github.repository == 'mudler/LocalAI' + runs-on: ubuntu-latest + timeout-minutes: 180 + steps: + - name: Checkout LocalAI + uses: actions/checkout@v7 + + - name: Free disk space + uses: ./.github/actions/free-disk-space + with: + mode: hosted + + - name: Login to Quay.io + uses: docker/login-action@v4 + with: + registry: quay.io + username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }} + password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }} + + - name: Compile paged backend against latest tip (cublas) + env: + TIP: ${{ needs.apply-check.outputs.tip }} + BUILDER_BASE_IMAGE: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-12-amd64' + run: | + docker run --rm \ + -v "$PWD":/LocalAI -w /LocalAI \ + -e TIP -e LLAMA_UPSTREAM \ + "$BUILDER_BASE_IMAGE" bash -euxo pipefail -c ' + # Mirror the Dockerfile: gRPC lives at /opt/grpc in the base image; + # copy it to the prefix CMake find_package expects. + cp -a /opt/grpc/. /usr/local/ + + # Pre-populate the llama.cpp checkout at the latest tip with the + # paged series applied via the tolerant canary apply (so the benign + # 0019 dev-doc hunk does not abort the build). Because + # backend/cpp/llama-cpp/llama.cpp now exists, the Makefile + # llama.cpp target (strict clone + git apply) is skipped and + # prepare.sh sees the paged sentinel and skips re-applying - so we + # drive the REAL grpc-server build path on top of our apply. + cd backend/cpp/llama-cpp/ + mkdir -p llama.cpp + cd llama.cpp + git init -q + git remote add origin "$LLAMA_UPSTREAM" + git fetch -q --depth 1 origin "$TIP" + git checkout -q FETCH_HEAD + cd /LocalAI + bash .github/scripts/paged-canary-apply.sh \ + backend/cpp/llama-cpp/llama.cpp \ + "$PWD/backend/cpp/llama-cpp/patches" + + # Cheapest real CUDA build that proves the patches compile: one + # CUDA arch, cublas, paged on. CMAKE_ARGS is passed via the + # environment (not as a make arg) so the Makefile += flags are + # still appended, exactly like .docker/llama-cpp-localai-paged-compile.sh. + cd backend/cpp/llama-cpp/ + BUILD_TYPE=cublas \ + LLAMA_PAGED=on \ + CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=80" \ + make grpc-server + test -x grpc-server + ' + echo "- compile: paged series builds (cublas) against the upstream tip :white_check_mark:" >> "$GITHUB_STEP_SUMMARY" diff --git a/backend/cpp/llama-cpp-localai-paged/Makefile b/backend/cpp/llama-cpp-localai-paged/Makefile index 6a5a4f41b..fd1e05f70 100644 --- a/backend/cpp/llama-cpp-localai-paged/Makefile +++ b/backend/cpp/llama-cpp-localai-paged/Makefile @@ -28,6 +28,13 @@ # Advance ONLY via the PIN_SYNC process (rebase patches + bit-exact gate + # re-export), then update this value. See: # backend/cpp/llama-cpp/patches/paged/PIN_SYNC_*.md +# +# This pin = the manual, verified sync. The signal telling you WHEN to do the +# next sync is the early-warning canary +# (.github/workflows/llama-cpp-paged-canary.yml): weekly it applies + compiles +# this patch series against the latest upstream llama.cpp tip and goes red the +# moment upstream drifts past the patches. Canary red -> run a PIN_SYNC, then +# bump this value. The canary never touches this pin; it is signal-only. LLAMA_VERSION?=9d5d882d8cd0f0a9283d87ed5e6fe3ee0d925fb1 CMAKE_ARGS?= diff --git a/backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md b/backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md index 3ad2b3dfb..5ba621116 100644 --- a/backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md +++ b/backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md @@ -7,6 +7,35 @@ re-exported from the rebased commits; **4 patch files changed** and are updated in this commit. A quick decode bench confirms the patchset performs the same on the new tip. +## Early-warning canary: when to run the NEXT pin-sync + +The shipped pin (this file's tip, mirrored in +`backend/cpp/llama-cpp-localai-paged/Makefile`) is advanced ONLY by this manual, +GPU-verified PIN_SYNC. Because the paged backend is excluded from the nightly +auto-bumper (`.github/workflows/bump_deps.yaml`), nothing nightly tells you when +upstream has drifted past the patches. That signal comes from a dedicated +scheduled canary: + +- **Workflow:** `.github/workflows/llama-cpp-paged-canary.yml` (weekly, plus + `workflow_dispatch`). It resolves the latest `ggml-org/llama.cpp` master tip, + then in two jobs (a) APPLIES the full series to that tip with the build's own + `git apply` method via `.github/scripts/paged-canary-apply.sh`, and (b) + COMPILES the paged backend (cublas) against it using the same base-grpc-cuda-12 + toolchain + `make grpc-server` target the shipped build uses. +- **Green** = the series still applies and compiles on upstream HEAD; nothing to + do. +- **Red** = upstream moved out from under the patches. **Canary red -> run a + PIN_SYNC** (rebase the patches onto the new tip, pass the bit-exact gate on the + GPU, re-export the `.patch` files, then advance the pin). The canary is + signal-only: it opens no PR and never moves the pin, so the shipped build and + the dep-bump PRs stay green regardless. +- **0019 handling:** the canary apply helper excludes ONLY the stray + `SSM_DECODE_FIX_RESULTS.md` dev-doc hunk (the pre-existing quirk documented in + the "Pre-existing finding" section below and in `PIN_BUMP_APPLY_CHECK.md`), + applying 0019's real code hunks atomically. So that benign quirk never + false-positives the canary, but a genuine code break in 0019 still turns it + red. + ## Upstream jump - OLD LocalAI pin: `8be759e6`