ci(paged): add early-warning canary for vendored llama.cpp paged patches

The paged backend (backend/cpp/llama-cpp-localai-paged) pins its own verified
llama.cpp tip and is excluded from the nightly auto-bumper so a naive bump can
never silently break the shipped build. That exclusion also removed the early
warning of upstream drift. This restores the signal without touching the pin.

Add .github/workflows/llama-cpp-paged-canary.yml (weekly + workflow_dispatch):

- apply-check job (ubuntu-latest, toolchain-free): resolve the latest
  ggml-org/llama.cpp master tip, shallow-checkout it, and apply the full paged
  series 0001-0030 in order with the build's own git-apply method via the new
  shared helper .github/scripts/paged-canary-apply.sh. Red on any apply break.
- compile job (needs apply-check): on the exact tip it validated, build the
  paged backend (cublas) inside the same base-grpc-cuda-12 toolchain and the
  same `make grpc-server` target the shipped build uses, so a red means upstream
  drift, not toolchain noise. nvcc compiles the kernels with no GPU present.

Red here = run a PIN_SYNC (rebase + bit-exact gate + re-export), then bump the
paged Makefile pin. The canary is signal-only: it opens no PR and never moves
the pin, so the shipped build and the dep-bump PRs stay green regardless. It is
fully separate from bump_deps.

The lone pre-existing quirk in the series (patch 0019 carries a stray modify
hunk against the dev-only doc SSM_DECODE_FIX_RESULTS.md, absent from any clean
upstream checkout; git apply is atomic so it rejects the whole patch and
cascades to 0021/0022/0026/0028) is handled path-scoped: the helper excludes
only that dev-doc and still applies 0019's real code hunks atomically, mirroring
prepare.sh's tolerance, so the quirk never false-positives the canary but a
genuine code break in 0019 still turns it red.

Point the existing pin comments in backend/cpp/llama-cpp-localai-paged/Makefile
and .github/workflows/bump_deps.yaml at this canary as the drift signal, and
document it in the PIN_SYNC doc: canary red -> do a pin-sync.

Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.github/scripts/paged-canary-apply.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+#
+# paged-canary-apply.sh - apply the vendored paged-attention patch series
+# (backend/cpp/llama-cpp/patches/paged/0001-0030) to a llama.cpp checkout, the
+# same way the build does, but tolerating the ONE known-benign pre-existing
+# quirk in the series. Used by the early-warning canary
+# (.github/workflows/llama-cpp-paged-canary.yml) so it only goes red on a REAL
+# upstream break, never on that quirk.
+#
+# Usage: paged-canary-apply.sh <llama.cpp-checkout-dir> <patches-dir>
+#   <patches-dir> is normally backend/cpp/llama-cpp/patches (it holds the
+#   top-level base series 0*.patch, currently empty, and the paged/ subseries).
+#
+# Exit 0  = the whole series applied -> patches still fit upstream.
+# Exit !=0 = a patch failed to apply  = the red signal: an upstream change moved
+#            the tree out from under the patches, so it is time to run a PIN_SYNC.
+#
+# Apply method MIRRORS backend/cpp/llama-cpp/Makefile's `llama.cpp` target:
+# plain `git apply --verbose`, which natively tolerates @@ line-number offsets
+# but NOT context-line changes. Matching the build's method is the point - the
+# canary's apply result is exactly what the real build's apply would do.
+#
+# The ONLY tolerance, and it is path-scoped (not a blanket `|| true`): patch
+# 0019 carries a stray *modify* hunk against the dev-only doc
+# SSM_DECODE_FIX_RESULTS.md, a file that exists only on the DGX dev tree and is
+# absent from any clean upstream checkout. `git apply` is atomic, so that single
+# missing-file hunk rejects the whole patch - and because 0021/0022/0026/0028
+# build on 0019's code, the rejection cascades to them too. This is a
+# PRE-EXISTING shipped-series defect, present identically on every pin, NOT an
+# upstream break (see backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md
+# and PIN_BUMP_APPLY_CHECK.md). We exclude ONLY that dev-doc path and still
+# apply 0019's real code hunks atomically, so a genuine code-hunk break in 0019
+# still fails the canary. prepare.sh tolerates the same hunk via
+# `patch ... || true`; this mirrors that tolerance precisely.
+
+set -euo pipefail
+
+CHECKOUT="${1:?usage: paged-canary-apply.sh <llama.cpp-checkout> <patches-dir>}"
+PATCHES="${2:?usage: paged-canary-apply.sh <llama.cpp-checkout> <patches-dir>}"
+
+# The lone tolerated dev-doc, and the only patch allowed to carry it.
+DEVDOC_GLOB='*SSM_DECODE_FIX_RESULTS.md'
+DEVDOC_PATCH='0019-qwen35-ssm-decode-fused-gather.patch'
+
+# Resolve to absolute paths so the apply works after we cd into the checkout.
+PATCHES="$(cd "$PATCHES" && pwd)"
+cd "$CHECKOUT"
+
+shopt -s nullglob
+
+apply_one() {
+  local p="$1"; shift
+  echo "paged-canary: applying $(basename "$p")"
+  if ! git apply --verbose "$@" "$p"; then
+    echo "::error::paged patch no longer applies to the upstream llama.cpp tip: $(basename "$p")"
+    echo "::error::upstream drifted past the vendored paged series - run a PIN_SYNC (backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md), do NOT bump the pin blindly"
+    exit 1
+  fi
+}
+
+# Base series first (parity with the build: patches/0*.patch before
+# patches/paged/0*.patch). Currently empty; nullglob makes this a no-op.
+for p in "$PATCHES"/0*.patch; do
+  apply_one "$p"
+done
+
+# Paged series, in order.
+for p in "$PATCHES"/paged/0*.patch; do
+  if [ "$(basename "$p")" = "$DEVDOC_PATCH" ]; then
+    # Apply 0019's real code hunks; exclude ONLY the benign dev-doc hunk.
+    apply_one "$p" --exclude="$DEVDOC_GLOB"
+  else
+    apply_one "$p"
+  fi
+done
+
+echo "paged-canary: the full paged patch series applied cleanly to the upstream tip"

.github/workflows/bump_deps.yaml

@@ -18,6 +18,14 @@ jobs:
         # backend/cpp/llama-cpp-localai-paged/Makefile) and advanced ONLY by the
         # manual PIN_SYNC process. Do not add it here. (turboquant CAN be
         # auto-bumped below because its fork branch carries the patches.)
+        #
+        # Excluding it from the auto-bumper removed the early warning of upstream
+        # drift; that signal is restored separately by the dedicated canary
+        # .github/workflows/llama-cpp-paged-canary.yml, which weekly applies +
+        # compiles the paged series against the latest llama.cpp tip and goes red
+        # when upstream breaks it (prompting a PIN_SYNC). The canary is
+        # signal-only - it never opens a bump PR and never moves the pin - so
+        # this dep-bump workflow and its PRs stay green regardless.
         include:
           - repository: "ggml-org/llama.cpp"
             variable: "LLAMA_VERSION"

.github/workflows/llama-cpp-paged-canary.yml

@@ -0,0 +1,173 @@
+name: 'llama.cpp paged patches: upstream canary'
+
+# EARLY-WARNING CANARY for the vendored paged-attention patch series
+# (backend/cpp/llama-cpp/patches/paged/0001-0030).
+#
+# WHY THIS EXISTS
+# The paged backend (backend/cpp/llama-cpp-localai-paged) pins its OWN verified
+# llama.cpp tip (LLAMA_VERSION in backend/cpp/llama-cpp-localai-paged/Makefile)
+# and is intentionally EXCLUDED from the nightly auto-bumper
+# (.github/workflows/bump_deps.yaml), so a naive upstream bump can never silently
+# break the shipped build. The cost of that safety: nobody finds out when
+# upstream DRIFTS past the patches. This canary restores that signal WITHOUT
+# touching the shipped pin - weekly it tries the patch series + a real compile
+# against the LATEST llama.cpp master tip and goes red the moment upstream breaks
+# the patches.
+#
+# RED HERE means: time to run a PIN_SYNC (rebase the patches onto the new tip,
+# pass the bit-exact gate on the GPU, re-export the .patch files, THEN advance
+# the pin in backend/cpp/llama-cpp-localai-paged/Makefile). See
+# backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md.
+#
+# SIGNAL-ONLY: this workflow moves no pinned version, ships nothing, and is fully
+# decoupled from bump_deps - so the main dep-bump PR stays green regardless. A
+# green run means "the paged series still applies and compiles on upstream HEAD";
+# a red run means "upstream moved - schedule a pin-sync".
+
+on:
+  schedule:
+    # Weekly (Mondays 06:00 UTC), mirroring the weekly DEPS_REFRESH / bump_deps
+    # cadence. Offset from bump_deps' nightly 20:00 so the two never pile up.
+    - cron: '0 6 * * 1'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: llama-cpp-paged-canary
+  cancel-in-progress: false
+
+env:
+  # Upstream source of truth - the same repo/branch bump_deps tracks for the
+  # stock llama-cpp pin.
+  LLAMA_UPSTREAM: 'https://github.com/ggml-org/llama.cpp'
+
+jobs:
+  apply-check:
+    # Cheap, fast, toolchain-free early warning: does the series still APPLY to
+    # the latest upstream tip? A patch no longer applying is by far the most
+    # common way upstream breaks a vendored series, so this runs first, is
+    # reliable on a free runner, and feeds the resolved tip to the compile job.
+    if: github.repository == 'mudler/LocalAI'
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    outputs:
+      tip: ${{ steps.resolve.outputs.tip }}
+    steps:
+      - name: Checkout LocalAI
+        uses: actions/checkout@v7
+
+      - name: Resolve latest llama.cpp master tip
+        id: resolve
+        run: |
+          tip="$(git ls-remote "$LLAMA_UPSTREAM" refs/heads/master | cut -f1)"
+          if [ -z "$tip" ]; then
+            echo "::error::could not resolve llama.cpp master tip from $LLAMA_UPSTREAM"
+            exit 1
+          fi
+          pin="$(grep -m1 'LLAMA_VERSION?=' backend/cpp/llama-cpp-localai-paged/Makefile | cut -d= -f2)"
+          echo "latest llama.cpp master tip: $tip"
+          echo "shipped paged pin:           $pin"
+          echo "tip=$tip" >> "$GITHUB_OUTPUT"
+          {
+            echo "## llama.cpp paged canary"
+            echo ""
+            echo "- upstream master tip: \`$tip\`"
+            echo "- shipped paged pin:   \`$pin\`"
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Checkout llama.cpp at latest tip (shallow)
+        run: |
+          mkdir -p /tmp/llama.cpp
+          cd /tmp/llama.cpp
+          git init -q
+          git remote add origin "$LLAMA_UPSTREAM"
+          git fetch -q --depth 1 origin "${{ steps.resolve.outputs.tip }}"
+          git checkout -q FETCH_HEAD
+          git log --oneline -1
+
+      - name: Apply paged patch series (build's git-apply method)
+        run: |
+          bash .github/scripts/paged-canary-apply.sh \
+            /tmp/llama.cpp \
+            "$PWD/backend/cpp/llama-cpp/patches"
+          echo "- apply: full paged series applies to the upstream tip :white_check_mark:" >> "$GITHUB_STEP_SUMMARY"
+
+  compile:
+    # Proves the patches still COMPILE against the latest tip, using the SAME
+    # toolchain + build target the shipped paged backend uses (the
+    # base-grpc-cuda-12 builder base + the Makefile `grpc-server` cublas target),
+    # so a failure means upstream drift, not toolchain noise. CUDA is compiled
+    # (nvcc; no GPU required) because most of the paged series is CUDA kernels.
+    # Runs only if the apply check passed, on the exact tip it validated.
+    #
+    # If a full CUDA compile on the hosted runner ever proves too heavy/flaky,
+    # switch `runs-on` to 'bigger-runner' (the runner class the real paged CUDA
+    # build uses), or drop to a CPU build (BUILD_TYPE='') which still compiles
+    # all host + CPU paged code, leaving CUDA-kernel coverage to the apply check
+    # plus the manual PIN_SYNC GPU gate.
+    needs: apply-check
+    if: github.repository == 'mudler/LocalAI'
+    runs-on: ubuntu-latest
+    timeout-minutes: 180
+    steps:
+      - name: Checkout LocalAI
+        uses: actions/checkout@v7
+
+      - name: Free disk space
+        uses: ./.github/actions/free-disk-space
+        with:
+          mode: hosted
+
+      - name: Login to Quay.io
+        uses: docker/login-action@v4
+        with:
+          registry: quay.io
+          username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+          password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+
+      - name: Compile paged backend against latest tip (cublas)
+        env:
+          TIP: ${{ needs.apply-check.outputs.tip }}
+          BUILDER_BASE_IMAGE: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-12-amd64'
+        run: |
+          docker run --rm \
+            -v "$PWD":/LocalAI -w /LocalAI \
+            -e TIP -e LLAMA_UPSTREAM \
+            "$BUILDER_BASE_IMAGE" bash -euxo pipefail -c '
+              # Mirror the Dockerfile: gRPC lives at /opt/grpc in the base image;
+              # copy it to the prefix CMake find_package expects.
+              cp -a /opt/grpc/. /usr/local/
+
+              # Pre-populate the llama.cpp checkout at the latest tip with the
+              # paged series applied via the tolerant canary apply (so the benign
+              # 0019 dev-doc hunk does not abort the build). Because
+              # backend/cpp/llama-cpp/llama.cpp now exists, the Makefile
+              # llama.cpp target (strict clone + git apply) is skipped and
+              # prepare.sh sees the paged sentinel and skips re-applying - so we
+              # drive the REAL grpc-server build path on top of our apply.
+              cd backend/cpp/llama-cpp/
+              mkdir -p llama.cpp
+              cd llama.cpp
+              git init -q
+              git remote add origin "$LLAMA_UPSTREAM"
+              git fetch -q --depth 1 origin "$TIP"
+              git checkout -q FETCH_HEAD
+              cd /LocalAI
+              bash .github/scripts/paged-canary-apply.sh \
+                backend/cpp/llama-cpp/llama.cpp \
+                "$PWD/backend/cpp/llama-cpp/patches"
+
+              # Cheapest real CUDA build that proves the patches compile: one
+              # CUDA arch, cublas, paged on. CMAKE_ARGS is passed via the
+              # environment (not as a make arg) so the Makefile += flags are
+              # still appended, exactly like .docker/llama-cpp-localai-paged-compile.sh.
+              cd backend/cpp/llama-cpp/
+              BUILD_TYPE=cublas \
+              LLAMA_PAGED=on \
+              CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=80" \
+              make grpc-server
+              test -x grpc-server
+            '
+          echo "- compile: paged series builds (cublas) against the upstream tip :white_check_mark:" >> "$GITHUB_STEP_SUMMARY"

backend/cpp/llama-cpp-localai-paged/Makefile

@@ -28,6 +28,13 @@
 # Advance ONLY via the PIN_SYNC process (rebase patches + bit-exact gate +
 # re-export), then update this value. See:
 #   backend/cpp/llama-cpp/patches/paged/PIN_SYNC_*.md
+#
+# This pin = the manual, verified sync. The signal telling you WHEN to do the
+# next sync is the early-warning canary
+# (.github/workflows/llama-cpp-paged-canary.yml): weekly it applies + compiles
+# this patch series against the latest upstream llama.cpp tip and goes red the
+# moment upstream drifts past the patches. Canary red -> run a PIN_SYNC, then
+# bump this value. The canary never touches this pin; it is signal-only.
 LLAMA_VERSION?=9d5d882d8cd0f0a9283d87ed5e6fe3ee0d925fb1
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/patches/paged/PIN_SYNC_9d5d882d.md

@@ -7,6 +7,35 @@ re-exported from the rebased commits; **4 patch files changed** and are updated
 in this commit. A quick decode bench confirms the patchset performs the same on
 the new tip.
 
+## Early-warning canary: when to run the NEXT pin-sync
+
+The shipped pin (this file's tip, mirrored in
+`backend/cpp/llama-cpp-localai-paged/Makefile`) is advanced ONLY by this manual,
+GPU-verified PIN_SYNC. Because the paged backend is excluded from the nightly
+auto-bumper (`.github/workflows/bump_deps.yaml`), nothing nightly tells you when
+upstream has drifted past the patches. That signal comes from a dedicated
+scheduled canary:
+
+- **Workflow:** `.github/workflows/llama-cpp-paged-canary.yml` (weekly, plus
+  `workflow_dispatch`). It resolves the latest `ggml-org/llama.cpp` master tip,
+  then in two jobs (a) APPLIES the full series to that tip with the build's own
+  `git apply` method via `.github/scripts/paged-canary-apply.sh`, and (b)
+  COMPILES the paged backend (cublas) against it using the same base-grpc-cuda-12
+  toolchain + `make grpc-server` target the shipped build uses.
+- **Green** = the series still applies and compiles on upstream HEAD; nothing to
+  do.
+- **Red** = upstream moved out from under the patches. **Canary red -> run a
+  PIN_SYNC** (rebase the patches onto the new tip, pass the bit-exact gate on the
+  GPU, re-export the `.patch` files, then advance the pin). The canary is
+  signal-only: it opens no PR and never moves the pin, so the shipped build and
+  the dep-bump PRs stay green regardless.
+- **0019 handling:** the canary apply helper excludes ONLY the stray
+  `SSM_DECODE_FIX_RESULTS.md` dev-doc hunk (the pre-existing quirk documented in
+  the "Pre-existing finding" section below and in `PIN_BUMP_APPLY_CHECK.md`),
+  applying 0019's real code hunks atomically. So that benign quirk never
+  false-positives the canary, but a genuine code break in 0019 still turns it
+  red.
+
 ## Upstream jump
 
 - OLD LocalAI pin: `8be759e6`