fix(vllm-cpp): map the CUDA 12 capabilities at the CPU build

vllm.cpp's CUDA kernels need the CUDA 13 toolchain: 12.x nvcc lacks
compute_121a and its ptxas rejects the sm_120a NVFP4 MMA kernels, so
backend/go/vllm-cpp/Makefile ships no CUDA 12 variant and the arm64 CUDA
build targets sm_121a (GB10 / DGX Spark) only.

backend/index.yaml declared nvidia-l4t and nvidia-l4t-cuda-13 but no
nvidia-l4t-cuda-12, so a Jetson AGX Orin (sm_87, JetPack 6) fell through
SystemState.Capability's "default" catch-all and silently installed
cpu-vllm-cpp. That is the right build for that host, but it was
indistinguishable from an oversight both to a reader of the index and to
a user wondering why their GPU box pulled a CPU backend.

Map nvidia-cuda-12 and nvidia-l4t-cuda-12 explicitly at the CPU build,
state the Blackwell-only constraint in the gallery description and the
backend README, and add specs that pin the routing. No behaviour change:
these hosts already resolved to the CPU build via the catch-all.

The README also records the image-tag trap behind the same symptom on a
supported host: /run/localai/capability is baked in at image build time,
so a DGX Spark on the CUDA 12 -nvidia-l4t-arm64 image reports
nvidia-l4t-cuda-12 and gets the CPU build; -nvidia-l4t-arm64-cuda-13 (or
LOCALAI_FORCE_META_BACKEND_CAPABILITY) gets the GPU one.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.agents/building-and-testing.md

@@ -28,7 +28,6 @@ The core Go suites (`./pkg`, `./core`, plus the in-process integration suite `./
   - **Build tags (`COVERAGE_TAGS`, passed via `GINKGO_TAGS`):** defaults to `debug auth`. The `auth` tag is required to compile the real (sqlite-backed) auth implementation and its ~150 `//go:build auth` tests — without it those files aren't built, the tests don't run, and the gate scores auth against a stub (~3.7% instead of ~38%). If you add new tag-gated tests, extend `COVERAGE_TAGS` or they won't count (and likely won't run in CI at all).
 - `make test-coverage-check` — runs `test-coverage`, then `scripts/coverage-check.sh` fails the build if total coverage is **below** the committed baseline in `coverage-baseline.txt`. The Linux job in `.github/workflows/test.yml` runs this instead of `make test`.
 - `make test-coverage-baseline` — regenerates and overwrites `coverage-baseline.txt` from the current run.
-- `make install-hooks` — sets `core.hooksPath` to the versioned `.githooks/`, whose `pre-commit` runs checks scoped to what's staged: Go changes → `make lint` + `make test-coverage-check`; `core/http/react-ui/` changes → `make test-ui-coverage-check` (Playwright e2e + UI coverage gate). A commit touching neither is skipped; bypass with `git commit --no-verify`. The hook resolves golangci-lint's new-from base to `upstream/master` → `origin/master` → `master`, so it works from a fork clone where `origin/master` is stale (passed to `make lint` via `LINT_NEW_FROM`).
 
 ### React UI coverage
 
@@ -38,12 +37,11 @@ The React UI (`core/http/react-ui/`) has **no component/unit tests** — its onl
 - **Browser:** the flake dev shell ships `chromium` and exports `PLAYWRIGHT_CHROMIUM_PATH`; `playwright.config.js` uses it via `launchOptions.executablePath`, and the Makefile skips `playwright install` when it's set. This avoids Playwright's downloaded browser, which can't resolve system libs (`libglib-2.0`, …) on NixOS. In CI (no `PLAYWRIGHT_CHROMIUM_PATH`) the Makefile falls back to `playwright install --with-deps chromium`.
 - The app is a React SPA, so coverage accumulates across in-app navigation within a test; a full `page.goto`/reload resets it.
 - `.nycrc.json` uses `all: true`, so **every `src/**` file is in the report**, including 0%-coverage ones — that's how you spot features with no test at all (sort the HTML report or `coverage-summary.json` by line% ascending). 
-- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`) and pre-commit on `core/http/react-ui/` changes.
+- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`).
 - **Why it has a tolerance (unlike the strict Go gate):** UI e2e coverage is *non-deterministic*. Specs that assert on state and end while async/lazy render work is still in flight collect those lines only when the render beats the coverage teardown — so the total drifts with machine speed/load (a fast local box reads higher than a slow CI runner), diffusely across many specs. The tolerance absorbs that drift, so set the baseline *below* the slow-CI floor, never to a fast-local `make test-ui-coverage-baseline` number, or CI flaps.
 - **Raising coverage is cheap:** a *render-smoke* spec (navigate to a route, assert its header renders) mounts a lazy page and runs its full render + initial effects, capturing most of its lines in a few lines of test — see `e2e/page-render-smoke.spec.js`. Auth is disabled in the test server (`isAdmin=true`), so `RequireAdmin`/`RequireFeature` routes render without a mock. The most *deterministic* win is removing a race: make a spec `await` a rendered element before ending (see `e2e/agents.spec.js` → AgentCreate) so its lines count every run.
 
 Rules (both gates):
-- **Install the hooks:** `make install-hooks` once per clone so lint + coverage run pre-commit. Don't lean on CI for what the hook catches.
-- **Don't work around the gate:** never `git commit --no-verify`, and never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
+- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
 - If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
 - The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.

.githooks/pre-commit

@@ -1,72 +0,0 @@
-#!/usr/bin/env sh
-#
-# LocalAI pre-commit hook. Install it (once per clone) with:
-#
-#     make install-hooks
-#
-# Runs only the checks relevant to what's staged:
-#   - Go files          -> make lint + make test-coverage-check
-#   - core/http/react-ui -> make test-ui-coverage-check (Playwright e2e + gate)
-#   - realtime state machines / specs -> make test-realtime-conformance
-#       (respcoord/**, turncoord/**, or formal-verification/** -- a pure .fizz
-#        spec edit must still re-verify the design, detected separately from Go)
-# A commit touching none of these is skipped entirely (other docs/YAML can't
-# change lint findings, Go coverage, the UI, or the realtime conformance gate).
-#
-# To bypass for a single commit (e.g. a WIP checkpoint): git commit --no-verify
-set -eu
-
-repo_root="$(git rev-parse --show-toplevel)"
-cd "$repo_root"
-
-staged="$(git diff --cached --name-only --diff-filter=ACMRD)"
-
-go_changed=0
-ui_changed=0
-rt_changed=0
-if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
-if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
-if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
-
-if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
-	echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
-	exit 0
-fi
-
-if [ "$go_changed" -eq 1 ]; then
-	# Resolve the ref golangci-lint's new-from-merge-base should compare
-	# against. .golangci.yml pins origin/master, which is correct in CI
-	# (origin == the canonical repo) but wrong from a fork clone, where
-	# origin/master lags behind and lint would report the whole upstream
-	# backlog. Prefer upstream/master, then origin/master, then master.
-	lint_base=""
-	for ref in upstream/master origin/master master; do
-		if git rev-parse --verify --quiet "${ref}^{commit}" >/dev/null 2>&1; then
-			lint_base="$ref"
-			break
-		fi
-	done
-
-	echo "pre-commit ▶ golangci-lint (make lint${lint_base:+, new-from $lint_base})"
-	make lint LINT_NEW_FROM="$lint_base"
-
-	echo "pre-commit ▶ coverage gate (make test-coverage-check) — builds and runs the"
-	echo "             pkg/core suites plus tests/e2e; can take a few minutes."
-	make test-coverage-check
-fi
-
-if [ "$ui_changed" -eq 1 ]; then
-	echo "pre-commit ▶ React UI e2e + coverage gate (make test-ui-coverage-check) —"
-	echo "             rebuilds the UI + ui-test-server, runs the Playwright specs, and"
-	echo "             fails if line coverage regressed; can take a couple of minutes."
-	make test-ui-coverage-check
-fi
-
-if [ "$rt_changed" -eq 1 ]; then
-	echo "pre-commit ▶ realtime state-machine conformance (make test-realtime-conformance) —"
-	echo "             Go transition/rapid tests under -race + FizzBee model check of the"
-	echo "             authoritative specs. Fail-closed: needs FizzBee (make install-fizzbee)."
-	make test-realtime-conformance
-fi
-
-echo "pre-commit ✓ all relevant checks passed"

.github/backend-matrix.yml

@@ -871,6 +871,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1935,6 +1948,32 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2000,6 +2039,32 @@ include:
     backend: "moss-tts-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp'
+    base-image: "ubuntu:24.04"
+    ubuntu-version: '2404'
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2963,6 +3028,19 @@ include:
     dockerfile: "./backend/Dockerfile.privacy-filter"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # Vulkan: base-grpc-vulkan-amd64 carries the SDK. arm64 vulkan is a one-line
   # add once amd64 is proven in CI.
   - build-type: 'vulkan'
@@ -4583,6 +4661,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4597,7 +4689,50 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # vllm-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # omnivoice-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-magpie-tts-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4652,6 +4787,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f32'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f32-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4691,6 +4839,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f16'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f16-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4732,6 +4893,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4774,6 +4949,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4814,6 +5003,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2204'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-arm64-magpie-tts-cpp'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2204'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -4853,6 +5055,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-magpie-tts-cpp'
+    base-image: "rocm/dev-ubuntu-24.04:6.4.4"
+    runs-on: 'ubuntu-latest'
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -5774,6 +5989,14 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-moss-tts-cpp"
     build-type: "metal"
     lang: "go"
+  - backend: "magpie-tts-cpp"
+    tag-suffix: "-metal-darwin-arm64-magpie-tts-cpp"
+    build-type: "metal"
+    lang: "go"
+  - backend: "vllm-cpp"
+    tag-suffix: "-metal-darwin-arm64-vllm-cpp"
+    build-type: "metal"
+    lang: "go"
   - backend: "omnivoice-cpp"
     tag-suffix: "-metal-darwin-arm64-omnivoice-cpp"
     build-type: "metal"

.github/bump_deps.sh

@@ -1,5 +1,8 @@
 #!/bin/bash
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1
 BRANCH=$2
 VAR=$3
@@ -9,10 +12,11 @@ if [ -z "$FILE" ]; then
     FILE="Makefile"
 fi
 
-# -L so a renamed/transferred upstream repo (GitHub answers 301) still
-# resolves instead of handing us the redirect body, and -f so an HTTP error
-# aborts the run rather than letting an error page reach sed below.
-LAST_COMMIT=$(curl -sfL -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")
+# gh_curl follows redirects so a renamed/transferred upstream repo (GitHub
+# answers 301) still resolves, and fails on HTTP errors rather than letting an
+# error page reach sed below. `|| true` keeps a failed lookup from aborting the
+# script at exit 22 with no context — the SHA guard below reports it instead.
+LAST_COMMIT=$(gh_curl -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH" || true)
 
 # Guard the sed input: anything that is not a bare 40-hex SHA (an API error
 # body, an empty response) would otherwise be spliced into the Makefile pin —

.github/bump_docs.sh

@@ -1,7 +1,18 @@
 #!/bin/bash
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1
 
-LATEST_TAG=$(curl -s "https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
+LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
+    "https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
+
+# jq prints the string "null" for a missing key, so a throttled or otherwise
+# unexpected API response would otherwise be published as the docs version.
+if [ -z "$LATEST_TAG" ] || [ "$LATEST_TAG" = "null" ]; then
+    echo "Refusing to bump docs version: could not resolve the latest release tag for $REPO." >&2
+    exit 1
+fi
 
 cat <<< $(jq ".version = \"$LATEST_TAG\"" docs/data/version.json) > docs/data/version.json

.github/bump_vllm_metal.sh

@@ -11,6 +11,9 @@
 # darwin build can only use the exact vLLM version vllm-metal supports, so it may
 # lag the Linux pin (requirements-cublas13-after.txt) until vllm-metal catches up.
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1   # vllm-project/vllm-metal
 FILE=$2   # backend/python/vllm/install.sh
 VAR=$3    # VLLM_METAL_VERSION (used for the workflow's output file names)
@@ -22,12 +25,12 @@ fi
 
 # vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
 # /releases/latest returns the newest one (with its cp312 wheel asset).
-LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
+LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
     "https://api.github.com/repos/$REPO/releases/latest" \
     | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
 
 # The coupled vLLM source version lives in vllm-metal's installer at that tag.
-NEW_VLLM_VERSION=$(curl -fsSL \
+NEW_VLLM_VERSION=$(gh_curl \
     "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
     | grep -oE 'vllm_v="[0-9]+\.[0-9]+\.[0-9]+"' | head -1 | cut -d'"' -f2)
 

.github/bump_vllm_wheel.sh

@@ -9,6 +9,9 @@
 # vars in Makefiles; this script handles the two-value rewrite specific to the
 # vLLM requirements file.
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1   # vllm-project/vllm
 FILE=$2   # backend/python/vllm/requirements-cublas13-after.txt
 VAR=$3    # VLLM_VERSION (used for output file names so the workflow can read them)
@@ -19,7 +22,7 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
 fi
 
 # /releases/latest returns the most recent non-prerelease tag.
-LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
+LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
     "https://api.github.com/repos/$REPO/releases/latest" \
     | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
 

.github/ci/apexentries/README.md

@@ -0,0 +1,194 @@
+# apexentries
+
+Generates gallery entries for the `mudler/*-APEX-GGUF` HuggingFace repositories.
+
+Each APEX repo becomes one **family**: one entry per quality rung the repo
+publishes and one per quantization rung its unsloth counterpart publishes, all
+gathered under the **base model's** entry. LocalAI's variant selector then picks
+the build that fits the hardware in front of it.
+
+## The hub is the base model entry, never a generated `*-apex` parent
+
+Somebody looking for `qwen3.6-35b-a3b` must find every build of those weights
+under that one name: the APEX imatrix rungs, the unsloth quant rungs and any
+speculative build. A separate `qwen3.6-35b-a3b-apex` hub competing with the base
+entry would split the family in two and leave whichever half the user did not
+search for effectively invisible.
+
+So the generator resolves the hub by stripping the `-APEX`, `-MTP` and `-TQ`
+markers and looking the result up in the index, trying both the repo-derived and
+the stem-derived candidate the same way `CounterpartCandidates` does. Then:
+
+- **The hub exists** (14 of the 45 repos, resolving to 10 distinct entries).
+  Nothing new is emitted for the family root. A `variants:` block is spliced into
+  the entry that is already there, textually, leaving its description, icon,
+  tags, overrides and files untouched. The line editing is shared with the
+  `variantproposals` job via `.github/ci/galleryedit`.
+- **The hub is absent** (the other 31). A new hub is emitted, named for the base
+  model and never for the APEX repo. It carries one of the discovered builds as
+  its own payload so it is a complete installable entry rather than a bare index,
+  and that payload is what gives it an `overrides.backend`. Without a declared
+  backend the verifier would skip it, so a hub carrying feature tags would escape
+  the tagging check in silence.
+
+Several APEX repos routinely resolve to one base model, so both paths accumulate
+by hub name rather than assuming one family per hub.
+
+Two references are always filtered out of a hub's list: anything the entry
+already declares, and the hub's own name. The self reference is not merely
+redundant. An unsloth rung whose weights the gallery already ships under the base
+name resolves, through the merge, straight back to the hub, and the verifier
+reads a self reference as a variant that declares variants of its own.
+
+The four hand-written `*-apex` entries (`qwen3.6-35b-a3b-apex`,
+`gemma-4-26b-a4b-it-apex`, `qwen3.5-35b-a3b-apex`,
+`nemotron-3-nano-omni-30b-a3b-reasoning-apex`) are **ordinary builds**, not hubs.
+They are referenced from their hub's variants list like any other rung, and are
+never deleted or renamed.
+
+## Flags
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `-index <path>` | `gallery/index.yaml` | Gallery index to dedup against. Read only, unless `-apply` is passed. |
+| `-only <a,b,c>` | (all) | Comma-separated full repo names (`mudler/Foo-APEX-GGUF`) to restrict generation to. A name that matches nothing is reported as a warning, since it is a typo rather than an empty result. |
+| `-out <path>` | (none) | Write the entries to add to this file. Nothing is written to the gallery. |
+| `-apply` | `false` | Splice the variants into `-index` and append the new entries to it. |
+| `-verify <path>` | (none) | Verify a gallery index and exit. Ignores every other flag. |
+
+Either `-out` or `-apply` is required, otherwise the run has nothing to do.
+
+`-apply` splices variant lines into existing entries and **appends** new ones. It
+never re-serialises the index: it is roughly 40,000 lines, and a YAML round trip
+would reflow the whole file, drop the anchors and merge keys the gallery relies
+on, and produce a diff nobody can review. On the three-family sample the splice
+is 24 added lines across 3 hunks with zero deletions.
+
+## Discovery is by filename suffix, never by repo name
+
+Builds come from the files a repo actually publishes. A filename is never
+constructed from a repo name, because the two disagree:
+`mudler/gemma-4-26B-A4B-it-APEX-GGUF` ships `gemma-4-26B-A4B-APEX-*.gguf`, and
+five other repos likewise drop a suffix (`-it`, `-2603`) or a vendor prefix
+(`NVIDIA-`) that the repo name carries. Composing a URL from the repo name would
+produce a 404 for every one of them, and the 404 would only surface after the
+entry shipped.
+
+The quality ladder is matched on the trailing tier marker, `-(I-)?(Quality|
+Balanced|Compact|Mini|Nano).gguf`. The `I-` prefix marks the imatrix ladder. The
+imatrix ladder is emitted when it is non-empty and the plain ladder is used only
+as a fallback, because two of the 45 repos publish no imatrix tiers at all and
+must still contribute. Eleven repos carry a fifth `I-Nano` rung, so nothing
+assumes a fixed number of rungs.
+
+Every run prints, per repo, the counts that discovery accounted for. If the
+number of classified files is short of the number of `.gguf` files the repo
+publishes, the shortfall is printed as `UNCLASSIFIED`. That check is a set
+difference on counts rather than a second pass over filenames: a second matcher
+would duplicate the tier regex and the two copies would drift. The failure it
+catches is quiet. A publishing-script typo that breaks every imatrix filename in
+a repo does not produce a short ladder; it makes the imatrix ladder empty, and
+the fallback then downgrades the whole family to the plain ladder with nothing
+said. A downstream HTTP check cannot catch it either, because it validates the
+URLs that were emitted, and an undiscovered tier emits none.
+
+The same reasoning applies to `UNACCOUNTED QUANT`, printed when the unsloth
+counterpart demonstrably publishes a wanted quant that produced no build. It is
+reported at discovery time because a dropped quant leaves no trace at all in the
+finished gallery file.
+
+## sha256 always comes from the API
+
+Every file stanza takes its `sha256` from the HuggingFace models API
+(`lfs.sha256`). A GGUF the API describes without one is a fatal error for that
+family: the repo is reported by name and the run ends non-zero. It is never
+substituted from another field, because that is exactly how a Xet hash ends up
+masquerading as a content hash.
+
+## The dflash / mtp tagging rule
+
+An entry is tagged `dflash` or `mtp` **if and only if** it configures the
+matching `spec_type:draft-<feature>`. Variant ranking reads tags and nothing
+else, so a tag that does not match the configuration either promotes a build
+that is no faster or hides one that genuinely is.
+
+A repo name is not configuration. `mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF` ships
+weights that carry MTP heads; an entry that does not enable them is not an MTP
+entry and is not tagged as one.
+
+A generated hub inherits the tags of the build it carries as its payload, rather
+than rebuilding them from the base set, so a hub whose payload configures a
+`spec_type` stays tagged consistently with the overrides copied alongside it.
+
+## Reuse reporting: two categories, not one
+
+Generated entries are deduped against the gallery and against the batch itself.
+The run prints the result under two separate headings, because the two cases are
+not equivalent:
+
+- **URI MATCHES** mean the gallery, or an earlier entry in this batch, already
+  ships exactly these weights. Pointing the hub at the existing entry is correct
+  and needs no thought.
+- **NAME COLLISIONS** mean an entry already owns the name but holds different
+  weights. Referencing it would point the hub at a build other than the one
+  generated. Every one of these must be inspected by hand.
+
+The run then prints `HUBS SPLICED`, listing every reference that will be added to
+an entry the gallery already ships along with the line it will be added at, and
+`HUBS CREATED` for the families that get a new hub. The splices are the part a
+review has to read closely, because they modify entries somebody else wrote.
+
+Hubs are deliberately kept out of the merge. A new hub carries the family's top
+rung as its own payload, so URI dedup would fold the hub into that rung and the
+family would lose the very entry point this command exists to create.
+
+## Workflow: sample first, then the full set
+
+Never run the full generation straight into the gallery. Generate a small,
+deliberately awkward sample, have it reviewed, then run the rest.
+
+```bash
+# 1. Sample three families that between them cover the awkward shapes:
+#    a standard four-rung repo, one with the extra I-Nano rung AND a file stem
+#    that differs from its repo name, and one whose unsloth counterpart shards
+#    its quants across subdirectories.
+go run ./.github/ci/apexentries \
+  -index gallery/index.yaml \
+  -only mudler/Qwen3.6-35B-A3B-APEX-GGUF,mudler/gemma-4-26B-A4B-it-APEX-GGUF,mudler/Step-3.7-Flash-APEX-GGUF \
+  -out /tmp/sample.yaml
+
+# 2. Verify the sample against the gallery it would join, splices included. Apply
+#    to a COPY, never to the real index, and check that the diff is only the
+#    intended variant lines. Compare the verifier output to the gallery's own
+#    baseline: what matters is that the sample adds no new problem, not that the
+#    total is zero.
+cp gallery/index.yaml /tmp/index-copy.yaml
+go run ./.github/ci/apexentries -index /tmp/index-copy.yaml -only <same list> -apply
+diff -u gallery/index.yaml /tmp/index-copy.yaml   # expect zero deletions
+
+go run ./.github/ci/apexentries -verify gallery/index.yaml > /tmp/baseline.log 2>&1
+go run ./.github/ci/apexentries -verify /tmp/index-copy.yaml > /tmp/spliced.log 2>&1
+diff /tmp/baseline.log /tmp/spliced.log
+
+# 3. Have a human review /tmp/sample.yaml and every reported name collision.
+
+# 4. Only then, the full set.
+go run ./.github/ci/apexentries -index gallery/index.yaml -apply
+```
+
+## Tests
+
+```bash
+go test ./.github/ci/apexentries/
+```
+
+The shared line editor has its own package:
+
+```bash
+go test ./.github/ci/galleryedit/
+```
+
+`.github/ci/` is invisible to `go list ./...`, so these specs are not covered by
+`make lint` or the repository test run. `.github/workflows/ci-tools-tests.yaml`
+names the package explicitly; keep that workflow in step with any package added
+under `.github/ci/`.

.github/ci/apexentries/discover.go

@@ -0,0 +1,70 @@
+package main
+
+import (
+	"regexp"
+	"strings"
+)
+
+// tierRE matches the tier marker APEX repos put at the end of a weight
+// filename. Discovery is by suffix because the stem is not predictable from
+// the repo name: six of the 45 repos drop a suffix ("-it", "-2603") or a
+// vendor prefix ("NVIDIA-") that the repo name carries.
+var tierRE = regexp.MustCompile(`-(I-)?(Quality|Balanced|Compact|Mini|Nano)\.gguf$`)
+
+// fullPrecisionRE matches the unquantized source weights an APEX repo publishes
+// alongside its ladder, flat (-F16.gguf) or sharded across a numbered set
+// (-F16-00001-of-00010.gguf). bf16 is accepted because some repos publish that
+// instead, and the match is case-insensitive because the casing varies between
+// publishing scripts.
+//
+// These are deliberately not tiers: they are the weights the ladder is quantized
+// FROM, and generation is scoped to the ladder itself.
+var fullPrecisionRE = regexp.MustCompile(`(?i)-b?f16(-\d{5}-of-\d{5})?\.gguf$`)
+
+// IsFullPrecision reports whether a weight filename is an unquantized source.
+func IsFullPrecision(name string) bool {
+	return fullPrecisionRE.MatchString(name)
+}
+
+// Tier is one discovered build of an APEX repo.
+type Tier struct {
+	Label string
+	File  GGUFFile
+}
+
+// DiscoverAPEXTiers splits a repo's weight files into the imatrix ladder and
+// the plain ladder. mmproj files are never tiers.
+func DiscoverAPEXTiers(files []GGUFFile) (imatrix, plain []Tier) {
+	for _, f := range files {
+		if strings.HasPrefix(f.Name, "mmproj") {
+			continue
+		}
+		m := tierRE.FindStringSubmatch(f.Name)
+		if m == nil {
+			continue
+		}
+		if m[1] != "" {
+			imatrix = append(imatrix, Tier{Label: "I-" + m[2], File: f})
+			continue
+		}
+		plain = append(plain, Tier{Label: m[2], File: f})
+	}
+	return imatrix, plain
+}
+
+// DiscoverMMProj returns the repo's projector file, if it publishes one. The
+// name varies across repos (mmproj.gguf, mmproj-F16.gguf,
+// mmproj-step3.7-flash-f16.gguf), so match the prefix rather than a fixed name.
+func DiscoverMMProj(files []GGUFFile) (GGUFFile, bool) {
+	for _, f := range files {
+		if strings.HasPrefix(f.Name, "mmproj") {
+			return f, true
+		}
+	}
+	return GGUFFile{}, false
+}
+
+// FileStem returns a tier's filename with its tier suffix removed.
+func FileStem(t Tier) string {
+	return tierRE.ReplaceAllString(t.File.Name, "")
+}

.github/ci/apexentries/discover_test.go

@@ -0,0 +1,68 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("DiscoverAPEXTiers", func() {
+	It("finds tiers regardless of how the stem relates to the repo name", func() {
+		// This repo is mudler/gemma-4-26B-A4B-it-APEX-GGUF but its files drop "-it".
+		files := []GGUFFile{
+			{Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf", SHA256: "a"},
+			{Name: "gemma-4-26B-A4B-APEX-I-Nano.gguf", SHA256: "b"},
+			{Name: "gemma-4-26B-A4B-APEX-Quality.gguf", SHA256: "c"},
+			{Name: "mmproj-F16.gguf", SHA256: "d"},
+		}
+
+		imatrix, plain := DiscoverAPEXTiers(files)
+
+		Expect(labels(imatrix)).To(ConsistOf("I-Quality", "I-Nano"))
+		Expect(labels(plain)).To(ConsistOf("Quality"))
+	})
+
+	It("excludes mmproj from the tier list", func() {
+		files := []GGUFFile{{Name: "mmproj.gguf", SHA256: "d"}}
+
+		imatrix, plain := DiscoverAPEXTiers(files)
+
+		Expect(imatrix).To(BeEmpty())
+		Expect(plain).To(BeEmpty())
+	})
+})
+
+var _ = Describe("DiscoverMMProj", func() {
+	It("finds an mmproj whatever its suffix", func() {
+		files := []GGUFFile{
+			{Name: "Model-APEX-I-Mini.gguf", SHA256: "a"},
+			{Name: "mmproj-step3.7-flash-f16.gguf", SHA256: "b"},
+		}
+
+		got, ok := DiscoverMMProj(files)
+
+		Expect(ok).To(BeTrue())
+		Expect(got.Name).To(Equal("mmproj-step3.7-flash-f16.gguf"))
+	})
+
+	It("reports absence when the repo ships none", func() {
+		_, ok := DiscoverMMProj([]GGUFFile{{Name: "Model-APEX-Quality.gguf", SHA256: "a"}})
+
+		Expect(ok).To(BeFalse())
+	})
+})
+
+var _ = Describe("FileStem", func() {
+	It("strips the tier suffix", func() {
+		t := Tier{Label: "I-Quality", File: GGUFFile{Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf"}}
+
+		Expect(FileStem(t)).To(Equal("gemma-4-26B-A4B-APEX"))
+	})
+})
+
+func labels(ts []Tier) []string {
+	out := make([]string, 0, len(ts))
+	for _, t := range ts {
+		out = append(out, t.Label)
+	}
+	return out
+}

.github/ci/apexentries/hf.go

@@ -0,0 +1,130 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+
+	"github.com/mudler/LocalAI/pkg/httpclient"
+)
+
+// ErrNoSHA256 marks a GGUF the HuggingFace API describes without an
+// lfs.sha256. Emitting an entry without a hash would ship an unverifiable
+// download, and guessing one from another field is how a Xet hash ends up
+// masquerading as a content hash, so this is fatal rather than skippable.
+var ErrNoSHA256 = errors.New("gguf file has no lfs.sha256")
+
+// GGUFFile is one .gguf sibling of a HuggingFace repo.
+type GGUFFile struct {
+	Name   string
+	Size   int64
+	SHA256 string
+}
+
+type apiSibling struct {
+	RFilename string `json:"rfilename"`
+	Size      int64  `json:"size"`
+	LFS       *struct {
+		SHA256 string `json:"sha256"`
+	} `json:"lfs"`
+}
+
+type apiModel struct {
+	Siblings []apiSibling `json:"siblings"`
+}
+
+// ParseRepoFiles returns every .gguf sibling described by a models API body.
+func ParseRepoFiles(body []byte) ([]GGUFFile, error) {
+	var m apiModel
+	if err := json.Unmarshal(body, &m); err != nil {
+		return nil, fmt.Errorf("decoding model response: %w", err)
+	}
+
+	var out []GGUFFile
+	for _, s := range m.Siblings {
+		if !strings.HasSuffix(s.RFilename, ".gguf") {
+			continue
+		}
+		if s.LFS == nil || s.LFS.SHA256 == "" {
+			return nil, fmt.Errorf("%s: %w", s.RFilename, ErrNoSHA256)
+		}
+		out = append(out, GGUFFile{Name: s.RFilename, Size: s.Size, SHA256: s.LFS.SHA256})
+	}
+	return out, nil
+}
+
+// FetchOptionalRepoFiles asks the models API for a repo the caller can do
+// without, and reports separately whether the repo was merely unreadable.
+//
+// HuggingFace answers 401 Unauthorized, not 404, for a repo that does not exist
+// when the request carries no credentials. Without a token there is therefore no
+// way to tell "this repo was never published" from "this repo is private", so an
+// optional probe has to treat 401 and 403 exactly like 404: whatever the reason,
+// there is nothing here for us to read, so there is no counterpart.
+//
+// The second return value exists because that collapse is lossy in one
+// direction: 401/403 can also mean a real, gated repo whose quants we would
+// genuinely want. The caller reports those repos so a silently dropped
+// counterpart is visible to a human rather than invisible.
+func FetchOptionalRepoFiles(client *http.Client, repo string) ([]GGUFFile, bool, error) {
+	files, status, err := fetchRepoFiles(client, repo)
+	if err != nil && (status == http.StatusUnauthorized || status == http.StatusForbidden) {
+		return nil, true, nil
+	}
+	return files, false, err
+}
+
+// FetchRepoFiles asks the models API for one repo. A 404 yields (nil, nil) so
+// that probing for an optional counterpart repo is not an error. Every other
+// non-200, 401 and 403 included, is an error: for a repo the run REQUIRES there
+// is no benign reading of "we cannot see it".
+func FetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, error) {
+	files, _, err := fetchRepoFiles(client, repo)
+	return files, err
+}
+
+// fetchRepoFiles does the request and returns the HTTP status alongside the
+// result, so the optional and required callers can apply different policies to
+// the same response without duplicating the request.
+func fetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, int, error) {
+	url := fmt.Sprintf("https://huggingface.co/api/models/%s?blobs=true", repo)
+	req, err := http.NewRequest(http.MethodGet, url, nil)
+	if err != nil {
+		return nil, 0, err
+	}
+	req.Header.Set("User-Agent", "localai-apexentries/1.0")
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, 0, err
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusNotFound {
+		return nil, resp.StatusCode, nil
+	}
+	if resp.StatusCode != http.StatusOK {
+		return nil, resp.StatusCode, fmt.Errorf("%s: unexpected status %d", repo, resp.StatusCode)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, resp.StatusCode, err
+	}
+	files, err := ParseRepoFiles(body)
+	return files, resp.StatusCode, err
+}
+
+// newHTTPClient builds the client used against the HuggingFace API. It goes
+// through pkg/httpclient rather than a bare &http.Client{} because the std
+// client follows redirects and forwards custom credential headers to the
+// redirect target on a cross-host hop (GHSA-3mj3-57v2-4636). This caller sends
+// only a User-Agent today, but it talks to an external API that could start
+// redirecting, and an HF_TOKEN header here later would then leak.
+func newHTTPClient() *http.Client {
+	return httpclient.NewWithTimeout(60 * time.Second)
+}

.github/ci/apexentries/hf_test.go

@@ -0,0 +1,142 @@
+package main
+
+import (
+	"bytes"
+	"io"
+	"net/http"
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestApexEntries(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "apexentries")
+}
+
+// stubTransport answers every request with one canned status and body, so the
+// status handling of the fetchers can be exercised without reaching the real
+// HuggingFace API.
+type stubTransport struct {
+	status int
+	body   string
+}
+
+func (t stubTransport) RoundTrip(req *http.Request) (*http.Response, error) {
+	return &http.Response{
+		StatusCode: t.status,
+		Body:       io.NopCloser(bytes.NewBufferString(t.body)),
+		Header:     make(http.Header),
+		Request:    req,
+	}, nil
+}
+
+func stubClient(status int, body string) *http.Client {
+	return &http.Client{Transport: stubTransport{status: status, body: body}}
+}
+
+const oneGGUFBody = `{"siblings":[{"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}}]}`
+
+var _ = Describe("FetchOptionalRepoFiles", func() {
+	// HuggingFace answers 401 rather than 404 for a repo that does not exist
+	// when the client carries no credentials, so an optional probe cannot tell
+	// "absent" from "unauthorized" and must treat both as "no counterpart".
+	It("treats a 401 as an absent repo and flags it as unavailable", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusUnauthorized, ""), "unsloth/Nope-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+		Expect(unavailable).To(BeTrue())
+	})
+
+	It("treats a 403 as an absent repo and flags it as unavailable", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusForbidden, ""), "unsloth/Gated-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+		Expect(unavailable).To(BeTrue())
+	})
+
+	// A clean 404 is an unambiguous absence, so it must NOT be reported as
+	// unavailable: the whole point of the flag is to separate the ambiguous
+	// case a human may need to look at from the settled one.
+	It("treats a 404 as an absent repo without flagging it as unavailable", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusNotFound, ""), "unsloth/Nope-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+		Expect(unavailable).To(BeFalse())
+	})
+
+	It("parses a 200 body as usual", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusOK, oneGGUFBody), "unsloth/Real-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(unavailable).To(BeFalse())
+		Expect(files).To(HaveLen(1))
+		Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf"))
+		Expect(files[0].SHA256).To(Equal("aa"))
+	})
+
+	// Tolerating 401/403 must not widen into tolerating everything: a 500 is a
+	// broken API, not evidence about whether the repo exists.
+	It("still errors on a 500", func() {
+		_, _, err := FetchOptionalRepoFiles(stubClient(http.StatusInternalServerError, ""), "unsloth/Real-GGUF")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("unexpected status 500"))
+	})
+})
+
+var _ = Describe("FetchRepoFiles", func() {
+	// The APEX repo itself is not optional. A 401 there means the repo the run
+	// was asked to publish cannot be read, which is a real failure and must not
+	// be quietly downgraded to "no files".
+	It("errors on a 401 for a required repo", func() {
+		_, err := FetchRepoFiles(stubClient(http.StatusUnauthorized, ""), "mudler/Model-APEX-GGUF")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("unexpected status 401"))
+	})
+
+	It("errors on a 403 for a required repo", func() {
+		_, err := FetchRepoFiles(stubClient(http.StatusForbidden, ""), "mudler/Model-APEX-GGUF")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("unexpected status 403"))
+	})
+
+	It("still treats a 404 as an absent repo", func() {
+		files, err := FetchRepoFiles(stubClient(http.StatusNotFound, ""), "mudler/Model-APEX-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+	})
+})
+
+var _ = Describe("ParseRepoFiles", func() {
+	It("returns gguf siblings with their lfs sha256", func() {
+		body := []byte(`{"siblings":[
+			{"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}},
+			{"rfilename":"README.md"},
+			{"rfilename":"mmproj.gguf","size":5,"lfs":{"sha256":"bb","size":5}}
+		]}`)
+
+		files, err := ParseRepoFiles(body)
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(HaveLen(2))
+		Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf"))
+		Expect(files[0].SHA256).To(Equal("aa"))
+		Expect(files[1].Name).To(Equal("mmproj.gguf"))
+	})
+
+	It("reports a gguf that carries no lfs sha256", func() {
+		body := []byte(`{"siblings":[{"rfilename":"mmproj.gguf","size":5}]}`)
+
+		_, err := ParseRepoFiles(body)
+
+		Expect(err).To(MatchError(ErrNoSHA256))
+	})
+})

.github/ci/apexentries/hub.go

@@ -0,0 +1,143 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
+)
+
+// IndexText is the gallery index seen as text: the entries it declares plus the
+// exact lines each one occupies, which is what splicing a variants block into an
+// entry the gallery already ships requires.
+//
+// It is a second, narrower read of the same file LoadExisting parses. The two
+// answer different questions: LoadExisting answers "do these weights already
+// exist anywhere", this one answers "where in the file does this entry live".
+type IndexText struct {
+	Lines   []string
+	Entries []*indexEntry
+
+	byName map[string]*indexEntry
+}
+
+// indexEntry is one entry of the index: its name, the variants it already
+// declares, and its coordinates in the file.
+type indexEntry struct {
+	Name     string       `yaml:"name"`
+	Variants []VariantRef `yaml:"variants"`
+
+	Pos galleryedit.Entry `yaml:"-"`
+}
+
+// LoadIndexText reads the gallery index for editing.
+func LoadIndexText(path string) (*IndexText, error) {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return nil, err
+	}
+	return ParseIndexText(string(raw))
+}
+
+// ParseIndexText pairs the decoded entries with the top level list items the
+// text actually contains.
+//
+// If the two views disagree on how many entries there are then every line number
+// a splice would compute is suspect, and the failure mode is writing a variants
+// block into the wrong model. The parse refuses instead.
+func ParseIndexText(text string) (*IndexText, error) {
+	var entries []*indexEntry
+	if err := yaml.Unmarshal([]byte(text), &entries); err != nil {
+		return nil, fmt.Errorf("decoding gallery index: %w", err)
+	}
+
+	lines, starts := galleryedit.Scan(text)
+	if len(starts) != len(entries) {
+		return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number",
+			len(entries), len(starts))
+	}
+
+	ix := &IndexText{Lines: lines, Entries: entries, byName: map[string]*indexEntry{}}
+	for i, e := range entries {
+		if e == nil {
+			return nil, fmt.Errorf("gallery index list item %d is empty; refusing to edit by line number", i)
+		}
+		end := len(lines)
+		if i+1 < len(starts) {
+			end = starts[i+1]
+		}
+		e.Pos = galleryedit.Entry{Name: e.Name, StartLine: starts[i], EndLine: end}
+
+		// First occurrence wins, matching the gallery's own resolution.
+		key := strings.ToLower(e.Name)
+		if _, seen := ix.byName[key]; !seen {
+			ix.byName[key] = e
+		}
+	}
+	return ix, nil
+}
+
+// Find looks an entry up by name, case insensitively.
+func (ix *IndexText) Find(name string) *indexEntry {
+	return ix.byName[strings.ToLower(name)]
+}
+
+// ResolveHub returns the gallery name of a family's hub and whether the gallery
+// already ships an entry under it.
+//
+// The hub is the BASE model entry, never a generated *-apex parent. Somebody
+// looking for qwen3.6-35b-a3b has to find every build of those weights under
+// that one name: the APEX imatrix rungs, the unsloth quant rungs and any
+// speculative build. A separate qwen3.6-35b-a3b-apex hub competing with the base
+// entry would split the family in two and leave whichever half the user did not
+// search for invisible.
+//
+// Both candidates are tried for the same reason CounterpartCandidates tries
+// both. The repo name and the published file stem disagree for several of these
+// repos, and either one may be what the base entry was named after.
+func ResolveHub(ix *IndexText, repoBase, stem string) (name string, exists bool) {
+	candidates := CounterpartCandidates(repoBase, stem)
+	for _, c := range candidates {
+		if n := slug(c); ix.Find(n) != nil {
+			return n, true
+		}
+	}
+	// Nothing matched, so the family needs a hub of its own under the repo
+	// derived name, which is the more reliable of the two.
+	return slug(candidates[0]), false
+}
+
+// HubLabel is the human-cased base model name, for prose rather than lookup.
+func HubLabel(repoBase, stem string) string {
+	return CounterpartCandidates(repoBase, stem)[0]
+}
+
+// filterVariants drops the references a hub must not carry: itself, and anything
+// it already lists.
+//
+// The self reference is not merely redundant. A hub that names itself makes the
+// verifier resolve the reference back to the hub, see that the hub declares
+// variants, and report a variant that declares variants of its own. It arises
+// for real rather than in theory: an unsloth rung whose weights the gallery
+// already ships under the base model name resolves, through Merge, straight back
+// to the hub that is about to reference it.
+func filterVariants(hub string, already []VariantRef, want []string) []string {
+	seen := map[string]bool{strings.ToLower(hub): true}
+	for _, v := range already {
+		seen[strings.ToLower(v.Model)] = true
+	}
+
+	var out []string
+	for _, w := range want {
+		key := strings.ToLower(w)
+		if seen[key] {
+			continue
+		}
+		seen[key] = true
+		out = append(out, w)
+	}
+	return out
+}

.github/ci/apexentries/main.go

@@ -0,0 +1,795 @@
+// Command apexentries generates gallery entries for the mudler APEX GGUF
+// repositories: one entry per imatrix tier and per unsloth quant rung, all
+// gathered under the BASE model's entry. Builds off a *-APEX-MTP-GGUF repo turn
+// speculative decoding on, because those weights retain the model's MTP heads
+// and are only worth their extra size with the heads in use.
+//
+// The base model entry is the hub. Somebody looking for qwen3.6-35b-a3b must
+// find every build of those weights under that one name, so when the gallery
+// already ships the base entry this command splices a variants block into it
+// rather than emitting a competing *-apex parent beside it. Only a family whose
+// base model the gallery does not ship at all gets a new hub entry, and that one
+// is still named for the base model.
+//
+// Builds are discovered by inspecting the filenames a repo actually publishes.
+// Repo names do not reliably predict them: mudler/gemma-4-26B-A4B-it-APEX-GGUF
+// ships gemma-4-26B-A4B-APEX-*.gguf, and six of the 45 repos drop a suffix or a
+// vendor prefix in the same way.
+package main
+
+import (
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path"
+	"sort"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
+)
+
+const (
+	// entryTemplate carries no backend and no parameters of its own, which is
+	// why RenderChild states everything inline.
+	entryTemplate = "virtual.yaml"
+	unslothOwner  = "unsloth"
+	authorListURL = "https://huggingface.co/api/models?author=mudler&limit=300"
+)
+
+// rungRank orders the quality ladder from best to smallest. The HuggingFace API
+// returns siblings alphabetically and DiscoverAPEXTiers preserves that order, so
+// an unsorted variants list reads I-Balanced, I-Compact, I-Mini, I-Nano,
+// I-Quality. Selection ignores authored order, so this is purely so the file a
+// human reviews scans in a meaningful sequence.
+var rungRank = map[string]int{
+	"I-Quality": 0, "I-Balanced": 1, "I-Compact": 2, "I-Mini": 3, "I-Nano": 4,
+	"Quality": 5, "Balanced": 6, "Compact": 7, "Mini": 8, "Nano": 9,
+}
+
+// baseTags are the tags every generated entry carries. dflash and mtp are never
+// among them: RenderChild adds those if and only if the entry configures the
+// matching spec_type.
+var baseTags = []string{"llm", "gguf", "cpu", "gpu"}
+
+// childBuild pairs a rendered entry with its position on the quality ladder, so
+// the parent's variants list can be sorted without re-parsing entry names.
+type childBuild struct {
+	entry GalleryEntry
+	rank  int
+}
+
+// family is one APEX repo's full generated output.
+type family struct {
+	repo      string
+	repoBase  string
+	stem      string
+	hasMMProj bool
+	children  []childBuild
+
+	// skippedRepos are counterpart candidates HuggingFace would not describe.
+	// Carried on the family rather than printed and forgotten so the run can
+	// summarize them next to everything else a reviewer has to eyeball.
+	skippedRepos []string
+	census       fileCensus
+	unaccounted  int
+}
+
+// fileCensus splits the files discovery emitted nothing for into the ones a
+// reviewer must chase and the ones that are deliberately out of scope.
+//
+// Full-precision sources are the second kind: they are the unquantized weights
+// the ladder is derived FROM, not a rung of it. Folding them into the
+// unclassified total would leave a permanent benign baseline, and a permanent
+// baseline is exactly what hides the one file that ever genuinely matters.
+type fileCensus struct {
+	unclassified  int
+	fullPrecision int
+}
+
+// add accumulates one repo's census into a running total.
+func (c *fileCensus) add(o fileCensus) {
+	c.unclassified += o.unclassified
+	c.fullPrecision += o.fullPrecision
+}
+
+// sortedChildren returns the family's builds in ladder order, best first.
+func (f *family) sortedChildren() []childBuild {
+	sorted := append([]childBuild{}, f.children...)
+	sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].rank < sorted[j].rank })
+	return sorted
+}
+
+func main() {
+	verify := flag.String("verify", "", "verify a gallery index and exit")
+	index := flag.String("index", "gallery/index.yaml", "gallery index to dedup against")
+	only := flag.String("only", "", "comma-separated repo names to restrict generation to")
+	out := flag.String("out", "", "write the entries to add to this file")
+	apply := flag.Bool("apply", false, "append the entries to add to -index")
+	flag.Parse()
+
+	if *verify != "" {
+		problems := Verify(*verify)
+		for _, p := range problems {
+			fmt.Fprintln(os.Stderr, p)
+		}
+		if len(problems) > 0 {
+			fmt.Fprintf(os.Stderr, "%d problem(s)\n", len(problems))
+			os.Exit(1)
+		}
+		fmt.Println("index is sound")
+		return
+	}
+
+	if err := generate(*index, *only, *out, *apply); err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		os.Exit(1)
+	}
+}
+
+func generate(indexPath, only, outPath string, apply bool) error {
+	if outPath == "" && !apply {
+		return fmt.Errorf("nothing to do: pass -out <file> or -apply")
+	}
+
+	client := newHTTPClient()
+
+	repos, err := listAPEXRepos(client)
+	if err != nil {
+		return err
+	}
+	if only != "" {
+		repos = restrict(repos, only)
+	}
+	if len(repos) == 0 {
+		return fmt.Errorf("no APEX repos selected")
+	}
+	fmt.Printf("repos selected: %d\n", len(repos))
+
+	var families []family
+	var failed []string
+
+	for _, repo := range repos {
+		f, err := buildFamily(client, repo)
+		if err != nil {
+			// A missing sha256 is fatal for the family rather than skippable: an
+			// entry without one ships an unverifiable download. Report which repo
+			// and keep going, so one bad repo does not hide the state of the rest.
+			fmt.Fprintf(os.Stderr, "FAILED %s: %v\n", repo, err)
+			failed = append(failed, repo)
+			continue
+		}
+		families = append(families, *f)
+	}
+
+	existing, err := LoadExisting(indexPath)
+	if err != nil {
+		return err
+	}
+	ixText, err := LoadIndexText(indexPath)
+	if err != nil {
+		return err
+	}
+	fmt.Printf("existing index: %d names, %d weight URIs, %d lines\n",
+		len(existing.ByName), len(existing.ByURI), len(ixText.Lines))
+
+	// Only the builds go through Merge. A hub is deliberately kept out of it: a
+	// new hub carries the family's top rung as its own payload, so Merge's URI
+	// dedup would fold the hub into that rung and the family would lose the very
+	// entry point this command exists to create. Hub names are checked against
+	// the index directly, by ResolveHub.
+	var generated []GalleryEntry
+	for _, f := range families {
+		for _, c := range f.children {
+			generated = append(generated, c.entry)
+		}
+	}
+
+	add, reused := Merge(existing, generated)
+	reportReuse(existing, generated, reused)
+
+	// Variant references are resolved from `reused`, never used to decide what to
+	// emit: on a within-batch name collision Merge records reused[name] = name
+	// while the first entry of that name is still in `add`, so treating presence
+	// in `reused` as "dropped" would silently emit nothing for it.
+	added := map[string]bool{}
+	for _, e := range add {
+		added[e.Name] = true
+	}
+
+	inserts, newHubs, err := planHubs(families, ixText, reused, added)
+	if err != nil {
+		return err
+	}
+	reportHubs(ixText, inserts, newHubs)
+
+	skipped, census, fullPrecisionRepos, unaccounted := reportSkipped(families)
+
+	add = append(add, newHubs...)
+
+	fmt.Printf("\nentries generated: %d\nentries to add:    %d\nentries reused:    %d\nhubs spliced:      %d\nhubs created:      %d\nrepos skipped:     %d\nexcluded (full precision): %d files across %d repos\nunclassified:      %d\nunaccounted:       %d\n",
+		len(generated), len(add), len(reused), len(inserts), len(newHubs), len(skipped),
+		census.fullPrecision, fullPrecisionRepos, census.unclassified, unaccounted)
+
+	lines, err := galleryedit.Apply(ixText.Lines, inserts)
+	if err != nil {
+		return err
+	}
+
+	if err := writeEntries(add, lines, outPath, apply, indexPath); err != nil {
+		return err
+	}
+
+	if len(failed) > 0 {
+		return fmt.Errorf("%d repo(s) failed: %s", len(failed), strings.Join(failed, ", "))
+	}
+	return nil
+}
+
+// resolveVariant maps a generated child name onto whatever entry actually stands
+// for it after the merge. `added` is consulted first because a within-batch name
+// collision puts a name in BOTH add and reused, and the entry that was emitted
+// is the one the parent must reference.
+func resolveVariant(name string, reused map[string]string, added map[string]bool) string {
+	if added[name] {
+		return name
+	}
+	if target, ok := reused[name]; ok {
+		return target
+	}
+	return name
+}
+
+// SpecTypeForRepo reports the speculative decoding mechanism a repo's builds can
+// turn on with no extra download.
+//
+// The *-APEX-MTP-GGUF repos republish the base weights with the model's own MTP
+// heads retained, so those builds are only worth their extra size if the heads
+// are actually used. Every other APEX repo drops them, and switching MTP on
+// there would name a mechanism the weights cannot serve.
+//
+// The suffix is read off the repo the FILES come from, so nothing downstream has
+// to infer a capability from an entry name.
+func SpecTypeForRepo(repo string) string {
+	if strings.HasSuffix(path.Base(repo), "-APEX-MTP-GGUF") {
+		return "draft-mtp"
+	}
+	return ""
+}
+
+// buildFamily discovers everything one APEX repo and its unsloth counterpart
+// publish, and renders it.
+func buildFamily(client *http.Client, repo string) (*family, error) {
+	files, err := FetchRepoFiles(client, repo)
+	if err != nil {
+		return nil, err
+	}
+	if len(files) == 0 {
+		return nil, fmt.Errorf("no gguf files")
+	}
+
+	imatrix, plain := DiscoverAPEXTiers(files)
+	mmproj, hasMMProj := DiscoverMMProj(files)
+
+	census := reportUnclassified(repo, files, imatrix, plain)
+
+	// The imatrix ladder is preferred, but two of the 45 repos publish no
+	// imatrix tiers at all and must still contribute their plain ladder.
+	ladder := imatrix
+	ladderKind := "imatrix"
+	if len(ladder) == 0 {
+		ladder = plain
+		ladderKind = "plain"
+	}
+	if len(ladder) == 0 {
+		return nil, fmt.Errorf("no tiers discovered")
+	}
+
+	sortTiers(ladder)
+
+	var mm *GGUFFile
+	if hasMMProj {
+		mm = &mmproj
+	}
+
+	repoBase := strings.TrimSuffix(path.Base(repo), "-GGUF")
+	f := &family{repo: repo, repoBase: repoBase, hasMMProj: hasMMProj, census: census}
+
+	// Only the APEX ladder can carry MTP heads; the unsloth counterpart quantizes
+	// the plain weights and gets nothing from this.
+	specType := SpecTypeForRepo(repo)
+
+	for _, t := range ladder {
+		f.children = append(f.children, childBuild{
+			rank: rungRank[t.Label],
+			entry: RenderChild(ChildInput{
+				Name:     slug(repoBase) + "-" + slug(t.Label),
+				Repo:     repo,
+				Template: entryTemplate,
+				SpecType: specType,
+				Weights:  []GGUFFile{t.File},
+				MMProj:   mm,
+				BaseTags: baseTags,
+			}),
+		})
+	}
+
+	stem := FileStem(ladder[0])
+	f.stem = stem
+	fmt.Printf("%s: %d %s tier(s) [%s], stem %s, mmproj %v\n",
+		repo, len(ladder), ladderKind, tierLabels(ladder), stem, hasMMProj)
+
+	counterpart, cpFiles, skipped, err := resolveCounterpart(client, repoBase, stem)
+	f.skippedRepos = skipped
+	if err != nil {
+		return nil, err
+	}
+	if counterpart != "" {
+		builds := DiscoverUnslothQuants(cpFiles)
+
+		// Called here rather than inside Verify: a quant dropped at discovery
+		// leaves no trace at all in the finished gallery file, so the only place
+		// the shortfall is still visible is the moment of discovery.
+		unaccounted := UnaccountedQuants(cpFiles, builds)
+		f.unaccounted = len(unaccounted)
+		for _, p := range unaccounted {
+			fmt.Fprintf(os.Stderr, "UNACCOUNTED QUANT %s: %s\n", counterpart, p)
+		}
+
+		cpMMProj, hasCPMMProj := DiscoverMMProj(cpFiles)
+		var cpMM *GGUFFile
+		if hasCPMMProj {
+			cpMM = &cpMMProj
+		}
+		cpBase := strings.TrimSuffix(path.Base(counterpart), "-GGUF")
+
+		for i, b := range builds {
+			f.children = append(f.children, childBuild{
+				rank: 100 + i,
+				entry: RenderChild(ChildInput{
+					Name:     slug(cpBase) + "-" + slug(b.Quant),
+					Repo:     counterpart,
+					Template: entryTemplate,
+					Weights:  b.Files,
+					MMProj:   cpMM,
+					BaseTags: baseTags,
+				}),
+			})
+		}
+		fmt.Printf("%s: counterpart %s, %d quant build(s) %s\n", repo, counterpart, len(builds), quantLabels(builds))
+	} else {
+		fmt.Printf("%s: no unsloth counterpart\n", repo)
+	}
+
+	return f, nil
+}
+
+// planHubs decides, per family, whether the family's builds are spliced into a
+// base model entry the gallery already ships or gathered under a new hub.
+//
+// Splicing is strongly preferred and is the measured majority-adjacent case. The
+// existing entry keeps its description, icon, tags, overrides and files
+// untouched; only variant lines are added to it.
+func planHubs(families []family, ix *IndexText, reused map[string]string, added map[string]bool) ([]galleryedit.Insert, []GalleryEntry, error) {
+	// Several APEX repos can resolve to one base model, so both paths accumulate
+	// by hub name rather than assuming one family per hub.
+	wantByHub := map[string][]string{}
+	var spliceOrder []string
+
+	var newHubs []GalleryEntry
+	hubAt := map[string]int{}
+
+	for i := range families {
+		f := &families[i]
+
+		hubName, exists := ResolveHub(ix, f.repoBase, f.stem)
+		want := hubVariants(f, ix, reused, added)
+
+		if exists {
+			if _, seen := wantByHub[hubName]; !seen {
+				spliceOrder = append(spliceOrder, hubName)
+			}
+			wantByHub[hubName] = append(wantByHub[hubName], want...)
+			continue
+		}
+
+		if at, dup := hubAt[hubName]; dup {
+			for _, v := range filterVariants(hubName, newHubs[at].Variants, want) {
+				newHubs[at].Variants = append(newHubs[at].Variants, VariantRef{Model: v})
+			}
+			continue
+		}
+
+		builds := f.sortedChildren()
+		if len(builds) == 0 {
+			return nil, nil, fmt.Errorf("%s: no builds to hang a hub on", f.repo)
+		}
+		hubAt[hubName] = len(newHubs)
+		newHubs = append(newHubs, renderHub(hubName, f, builds[0], filterVariants(hubName, nil, want)))
+	}
+
+	var inserts []galleryedit.Insert
+	for _, name := range spliceOrder {
+		e := ix.Find(name)
+		items := filterVariants(name, e.Variants, wantByHub[name])
+		if len(items) == 0 {
+			continue
+		}
+		inserts = append(inserts, galleryedit.Insert{Entry: e.Pos, Variants: items})
+	}
+	return inserts, newHubs, nil
+}
+
+// hubVariants is a family's full build list, in ladder order, named as the hub
+// must reference them after the merge.
+func hubVariants(f *family, ix *IndexText, reused map[string]string, added map[string]bool) []string {
+	var out []string
+
+	// A hand-written *-apex entry is an ordinary build of these weights. It is
+	// never deleted, never renamed and never treated as a hub; it is simply
+	// referenced like any other rung.
+	if apex := slug(f.repoBase); ix.Find(apex) != nil {
+		out = append(out, apex)
+	}
+	for _, c := range f.sortedChildren() {
+		out = append(out, resolveVariant(c.entry.Name, reused, added))
+	}
+	return out
+}
+
+// renderHub builds the hub for a family whose base model the gallery does not
+// ship at all. It is named for the BASE model, never for the APEX repo.
+//
+// It carries one of the discovered builds as its own payload so it is a complete
+// installable entry rather than a bare index pointing at other entries. That
+// payload is what supplies overrides.backend, which matters beyond installation:
+// the verifier can only judge the tagging rule for a backend it can read, so a
+// hub carrying feature tags and no backend would escape the check in silence.
+//
+// The payload's own tags are kept rather than rebuilt from baseTags, so a hub
+// whose payload configures a spec_type stays tagged for it and consistent with
+// the overrides copied alongside.
+func renderHub(name string, f *family, payload childBuild, variants []string) GalleryEntry {
+	e := payload.entry
+	e.Name = name
+	e.Description = fmt.Sprintf(
+		"%s. Quality ladder and quantization rungs published by %s and its unsloth counterpart; LocalAI picks the build that fits the hardware.",
+		HubLabel(f.repoBase, f.stem), f.repo)
+
+	e.Tags = append([]string{}, payload.entry.Tags...)
+	if f.hasMMProj && !hasTag(e.Tags, "vision") {
+		e.Tags = append(e.Tags, "vision")
+	}
+
+	e.Variants = nil
+	for _, v := range variants {
+		e.Variants = append(e.Variants, VariantRef{Model: v})
+	}
+	return e
+}
+
+func hasTag(tags []string, want string) bool {
+	for _, t := range tags {
+		if t == want {
+			return true
+		}
+	}
+	return false
+}
+
+// resolveCounterpart probes the unsloth candidates in order and returns the
+// first that publishes files.
+//
+// CounterpartCandidates is handed a BARE repo name: its cleaner does not strip
+// an owner prefix, so passing "mudler/Foo-APEX-GGUF" would yield "mudler/Foo"
+// and compose into the nonsense probe "unsloth/mudler/Foo".
+//
+// It also returns the candidates HuggingFace refused to describe. Those are
+// indistinguishable from absent without credentials, so they are skipped, but
+// they are named rather than dropped: one of them could be a real gated repo
+// whose quants belong in the gallery.
+func resolveCounterpart(client *http.Client, repoBase, stem string) (string, []GGUFFile, []string, error) {
+	var unavailable []string
+	for _, cand := range CounterpartCandidates(repoBase, stem) {
+		repo := unslothOwner + "/" + cand + "-GGUF"
+		files, unreadable, err := FetchOptionalRepoFiles(client, repo)
+		if err != nil {
+			return "", nil, unavailable, fmt.Errorf("probing %s: %w", repo, err)
+		}
+		if unreadable {
+			unavailable = append(unavailable, repo)
+			continue
+		}
+		if len(files) > 0 {
+			return repo, files, unavailable, nil
+		}
+	}
+	return "", nil, unavailable, nil
+}
+
+// reportUnclassified prints the files discovery turned into nothing.
+//
+// It is a set difference on COUNTS, not a re-match of filenames: re-matching
+// would duplicate the tier regex from discover.go and the two copies would
+// drift. The likeliest trigger is a typo or case change from a publishing script
+// rather than a genuine sixth tier, and because generation falls back to the
+// plain ladder when the imatrix one is empty, a repo whose imatrix files all
+// fail to match silently downgrades the whole family instead of erroring. The
+// downstream HTTP check cannot catch that: it validates URLs that were emitted,
+// and an undiscovered tier emits none.
+// It returns the census so the run can total it.
+func reportUnclassified(repo string, files []GGUFFile, imatrix, plain []Tier) fileCensus {
+	mmprojCount, fullPrecision := 0, 0
+	for _, f := range files {
+		// The mmproj test comes first because projectors are themselves often
+		// published at f16 (mmproj-F16.gguf), and counting such a file in both
+		// buckets would understate the unclassified remainder.
+		if strings.HasPrefix(f.Name, "mmproj") {
+			mmprojCount++
+			continue
+		}
+		if IsFullPrecision(f.Name) {
+			fullPrecision++
+		}
+	}
+
+	classified := len(imatrix) + len(plain) + mmprojCount + fullPrecision
+	if classified >= len(files) {
+		return fileCensus{fullPrecision: fullPrecision}
+	}
+	fmt.Fprintf(os.Stderr, "UNCLASSIFIED %s: %d of %d .gguf files classified, %d unaccounted for\n",
+		repo, classified, len(files), len(files)-classified)
+	return fileCensus{unclassified: len(files) - classified, fullPrecision: fullPrecision}
+}
+
+// reportReuse splits Merge's single reused map into the two cases it conflates.
+//
+// A URI match means the gallery already ships exactly these weights, and
+// pointing the parent at the existing entry is correct. A NAME match with a
+// different URI means an unrelated entry happens to own the name, and
+// referencing it would point the parent at different weights than were
+// generated, substituting a build without saying so. Only the first is safe to
+// wave through.
+func reportReuse(existing *ExistingIndex, generated []GalleryEntry, reused map[string]string) {
+	byName := map[string]GalleryEntry{}
+	for _, e := range generated {
+		if _, seen := byName[e.Name]; !seen {
+			byName[e.Name] = e
+		}
+	}
+
+	var nameCollisions, uriMatches []string
+	for name, target := range reused {
+		gen := byName[name]
+		uri := ""
+		if len(gen.Files) > 0 {
+			uri = gen.Files[0].URI
+		}
+
+		switch {
+		case hasName(existing, name):
+			nameCollisions = append(nameCollisions,
+				fmt.Sprintf("  %s -> gallery entry of the same name (generated uri: %s)", name, orNone(uri)))
+		case target == name:
+			nameCollisions = append(nameCollisions,
+				fmt.Sprintf("  %s -> earlier entry of the same name in this batch (generated uri: %s)", name, orNone(uri)))
+		default:
+			uriMatches = append(uriMatches, fmt.Sprintf("  %s -> %s (same weights: %s)", name, target, orNone(uri)))
+		}
+	}
+	sort.Strings(nameCollisions)
+	sort.Strings(uriMatches)
+
+	fmt.Printf("\nNAME COLLISIONS (%d) - inspect each by hand, the target may hold different weights\n", len(nameCollisions))
+	for _, l := range nameCollisions {
+		fmt.Println(l)
+	}
+	fmt.Printf("\nURI MATCHES (%d) - the gallery or this batch already ships these exact weights\n", len(uriMatches))
+	for _, l := range uriMatches {
+		fmt.Println(l)
+	}
+}
+
+// reportHubs prints exactly what will be written where. The splices are the part
+// a human has to read: they modify entries the gallery already ships, so the
+// review needs the target, the line, and every added reference spelled out.
+func reportHubs(ix *IndexText, inserts []galleryedit.Insert, newHubs []GalleryEntry) {
+	fmt.Printf("\nHUBS SPLICED (%d) - variants added to the EXISTING base model entry, nothing else touched\n", len(inserts))
+	for _, in := range inserts {
+		e := ix.Find(in.Entry.Name)
+		fmt.Printf("  %s (line %d, %d variant(s) already declared):\n", in.Entry.Name, in.Entry.StartLine+1, len(e.Variants))
+		for _, v := range in.Variants {
+			fmt.Printf("    + - model: %s\n", galleryedit.QuoteName(v))
+		}
+	}
+
+	fmt.Printf("\nHUBS CREATED (%d) - the gallery ships no base model entry, so one is emitted for it\n", len(newHubs))
+	for _, h := range newHubs {
+		fmt.Printf("  %s:\n", h.Name)
+		for _, v := range h.Variants {
+			fmt.Printf("    - model: %s\n", v.Model)
+		}
+	}
+}
+
+// reportSkipped names the counterpart repos HuggingFace would not describe, and
+// totals the other two silent-shortfall counters alongside them.
+//
+// A skipped repo is not the same as a clean 404. HuggingFace answers 401 for a
+// nonexistent repo to an unauthenticated client, so the overwhelmingly likely
+// reading is "there is no such counterpart", which is the normal case for the
+// community merges. But a private or gated repo answers 401 too, and that one
+// WOULD have quants worth shipping. Printing the list is what keeps that
+// possibility auditable instead of silently discarded.
+func reportSkipped(families []family) ([]string, fileCensus, int, int) {
+	var skipped []string
+	var census fileCensus
+	fullPrecisionRepos, unaccounted := 0, 0
+	for _, f := range families {
+		skipped = append(skipped, f.skippedRepos...)
+		census.add(f.census)
+		if f.census.fullPrecision > 0 {
+			fullPrecisionRepos++
+		}
+		unaccounted += f.unaccounted
+	}
+	sort.Strings(skipped)
+
+	fmt.Printf("\nREPOS SKIPPED AS UNAVAILABLE (%d) - HuggingFace answered 401/403, which is indistinguishable from absent without a token; check none of these is a real gated repo\n", len(skipped))
+	for _, r := range skipped {
+		fmt.Printf("  %s\n", r)
+	}
+	return skipped, census, fullPrecisionRepos, unaccounted
+}
+
+func hasName(ix *ExistingIndex, name string) bool {
+	_, ok := ix.ByName[name]
+	return ok
+}
+
+func orNone(s string) string {
+	if s == "" {
+		return "(no files)"
+	}
+	return s
+}
+
+// writeEntries emits the additions.
+//
+// -apply does two things in one pass: it writes back the spliced lines, which
+// differ from the original only by the variant lines galleryedit inserted, and
+// then appends the new entries. New entries are APPENDED rather than merged into
+// the structure, for the same reason the splice is textual: a YAML round trip
+// over 40,000 lines would reflow the whole file into an unreviewable diff.
+func writeEntries(add []GalleryEntry, lines []string, outPath string, apply bool, indexPath string) error {
+	if apply {
+		if err := os.WriteFile(indexPath, []byte(strings.Join(lines, "\n")), 0o644); err != nil {
+			return err
+		}
+		fmt.Printf("spliced %s\n", indexPath)
+	}
+
+	if len(add) == 0 {
+		fmt.Println("nothing to append")
+		return nil
+	}
+
+	blob, err := yaml.Marshal(add)
+	if err != nil {
+		return err
+	}
+
+	if outPath != "" {
+		if err := os.WriteFile(outPath, blob, 0o644); err != nil {
+			return err
+		}
+		fmt.Printf("wrote %d entries to %s\n", len(add), outPath)
+	}
+
+	if apply {
+		f, err := os.OpenFile(indexPath, os.O_APPEND|os.O_WRONLY, 0o644)
+		if err != nil {
+			return err
+		}
+		defer f.Close()
+		if _, err := f.Write(blob); err != nil {
+			return err
+		}
+		fmt.Printf("appended %d entries to %s\n", len(add), indexPath)
+	}
+	return nil
+}
+
+// listAPEXRepos returns the mudler repos whose name marks them as APEX builds.
+func listAPEXRepos(client *http.Client) ([]string, error) {
+	req, err := http.NewRequest(http.MethodGet, authorListURL, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("User-Agent", "localai-apexentries/1.0")
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("listing models: unexpected status %d", resp.StatusCode)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var models []struct {
+		ID string `json:"id"`
+	}
+	if err := json.Unmarshal(body, &models); err != nil {
+		return nil, fmt.Errorf("decoding model list: %w", err)
+	}
+
+	var out []string
+	for _, m := range models {
+		if strings.Contains(m.ID, "APEX") {
+			out = append(out, m.ID)
+		}
+	}
+	sort.Strings(out)
+	return out, nil
+}
+
+func restrict(repos []string, only string) []string {
+	want := map[string]bool{}
+	for _, r := range strings.Split(only, ",") {
+		if r = strings.TrimSpace(r); r != "" {
+			want[r] = true
+		}
+	}
+
+	var out []string
+	for _, r := range repos {
+		if want[r] {
+			out = append(out, r)
+			delete(want, r)
+		}
+	}
+	// A name in -only that matched nothing is a typo, not an empty result.
+	for r := range want {
+		fmt.Fprintf(os.Stderr, "WARNING: -only names %s, which is not an APEX repo of this author\n", r)
+	}
+	return out
+}
+
+func sortTiers(tiers []Tier) {
+	sort.SliceStable(tiers, func(i, j int) bool { return rungRank[tiers[i].Label] < rungRank[tiers[j].Label] })
+}
+
+func tierLabels(tiers []Tier) string {
+	var out []string
+	for _, t := range tiers {
+		out = append(out, t.Label)
+	}
+	return strings.Join(out, ",")
+}
+
+func quantLabels(builds []QuantBuild) string {
+	var out []string
+	for _, b := range builds {
+		l := b.Quant
+		if b.Sharded {
+			l += fmt.Sprintf("(%d shards)", len(b.Files))
+		}
+		out = append(out, l)
+	}
+	return strings.Join(out, ",")
+}
+
+// slug turns a repo, tier or quant label into a gallery entry name component.
+func slug(s string) string {
+	return strings.ReplaceAll(strings.ToLower(s), "_", "-")
+}

.github/ci/apexentries/main_test.go

@@ -0,0 +1,344 @@
+package main
+
+import (
+	"strings"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
+)
+
+func mustIndex(text string) *IndexText {
+	ix, err := ParseIndexText(text)
+	ExpectWithOffset(1, err).ToNot(HaveOccurred())
+	return ix
+}
+
+// buildOf renders a realistic child so the specs exercise the payload a hub
+// actually inherits rather than a bare name.
+func buildOf(name, repo, file string, rank int) childBuild {
+	return childBuild{
+		rank: rank,
+		entry: RenderChild(ChildInput{
+			Name:     name,
+			Repo:     repo,
+			Template: entryTemplate,
+			Weights:  []GGUFFile{{Name: file, SHA256: "aa"}},
+			BaseTags: baseTags,
+		}),
+	}
+}
+
+var _ = Describe("ResolveHub", func() {
+	It("picks the base model name over the APEX name, even when both are in the gallery", func() {
+		// The hub is the entry a user searches for. If the *-apex entry were
+		// chosen the family would be gathered under a name nobody looks up, and
+		// the base entry would go on advertising only its own build.
+		ix := mustIndex("- name: qwen3.6-35b-a3b\n  url: u\n- name: qwen3.6-35b-a3b-apex\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX", "Qwen3.6-35B-A3B-APEX")
+
+		Expect(name).To(Equal("qwen3.6-35b-a3b"))
+		Expect(exists).To(BeTrue())
+	})
+
+	It("falls back to the stem-derived candidate when the repo-derived one is absent", func() {
+		// gemma's repo says "-it" and its published files do not, so only one of
+		// the two candidates can match whatever the base entry was named after.
+		ix := mustIndex("- name: gemma-4-26b-a4b\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "gemma-4-26B-A4B-it-APEX", "gemma-4-26B-A4B-APEX")
+
+		Expect(name).To(Equal("gemma-4-26b-a4b"))
+		Expect(exists).To(BeTrue())
+	})
+
+	It("reports the base name as absent rather than settling for the APEX entry", func() {
+		ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "Qwen3.5-35B-A3B-APEX", "Qwen3.5-35B-A3B-APEX")
+
+		Expect(name).To(Equal("qwen3.5-35b-a3b"))
+		Expect(exists).To(BeFalse())
+	})
+
+	It("strips the MTP and TQ markers as well as APEX", func() {
+		ix := mustIndex("- name: qwen3.6-35b-a3b\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX-MTP", "Qwen3.6-35B-A3B-APEX-MTP")
+
+		Expect(name).To(Equal("qwen3.6-35b-a3b"))
+		Expect(exists).To(BeTrue())
+	})
+})
+
+var _ = Describe("planHubs", func() {
+	noReuse := map[string]string{}
+	allAdded := func(names ...string) map[string]bool {
+		out := map[string]bool{}
+		for _, n := range names {
+			out[n] = true
+		}
+		return out
+	}
+
+	It("splices into the existing base entry instead of emitting an *-apex parent", func() {
+		ix := mustIndex("- name: step-3.7-flash\n  url: u\n- name: other\n  url: u\n")
+		fams := []family{{
+			repo:     "mudler/Step-3.7-Flash-APEX-GGUF",
+			repoBase: "Step-3.7-Flash-APEX",
+			stem:     "Step-3.7-Flash-APEX",
+			children: []childBuild{buildOf("step-3.7-flash-apex-i-quality", "mudler/Step-3.7-Flash-APEX-GGUF", "a.gguf", 0)},
+		}}
+
+		inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("step-3.7-flash-apex-i-quality"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(newHubs).To(BeEmpty())
+		Expect(inserts).To(HaveLen(1))
+		Expect(inserts[0].Entry.Name).To(Equal("step-3.7-flash"))
+		Expect(inserts[0].Variants).To(Equal([]string{"step-3.7-flash-apex-i-quality"}))
+	})
+
+	It("merges into an entry that already declares variants, without repeating one", func() {
+		// The gallery's qwen3.6-35b-a3b already lists its APEX build. Re-adding it
+		// would put a duplicate key's worth of noise in the diff and a duplicate
+		// reference in the entry.
+		ix := mustIndex("- name: qwen3.6-35b-a3b\n  variants:\n    - model: qwen3.6-35b-a3b-apex\n  url: u\n" +
+			"- name: qwen3.6-35b-a3b-apex\n  url: u\n")
+		fams := []family{{
+			repo:     "mudler/Qwen3.6-35B-A3B-APEX-GGUF",
+			repoBase: "Qwen3.6-35B-A3B-APEX",
+			stem:     "Qwen3.6-35B-A3B-APEX",
+			children: []childBuild{buildOf("qwen3.6-35b-a3b-apex-i-quality", "mudler/Qwen3.6-35B-A3B-APEX-GGUF", "a.gguf", 0)},
+		}}
+
+		inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("qwen3.6-35b-a3b-apex-i-quality"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(newHubs).To(BeEmpty())
+		Expect(inserts[0].Variants).To(Equal([]string{"qwen3.6-35b-a3b-apex-i-quality"}))
+
+		out, err := galleryedit.Apply(ix.Lines, inserts)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(strings.Count(strings.Join(out, "\n"), "variants:")).To(Equal(1))
+		Expect(out).To(HaveLen(len(ix.Lines) + 1))
+	})
+
+	It("never lets the hub reference itself", func() {
+		// An unsloth rung whose weights the gallery already ships under the base
+		// name resolves, through Merge, straight back to the hub. The verifier
+		// reads a self reference as a variant that declares variants of its own.
+		ix := mustIndex("- name: step-3.7-flash\n  url: u\n")
+		fams := []family{{
+			repo:     "mudler/Step-3.7-Flash-APEX-GGUF",
+			repoBase: "Step-3.7-Flash-APEX",
+			stem:     "Step-3.7-Flash-APEX",
+			children: []childBuild{buildOf("step-3.7-flash-ud-q4-k-m", "unsloth/Step-3.7-Flash-GGUF", "a.gguf", 100)},
+		}}
+
+		inserts, _, err := planHubs(fams, ix, map[string]string{"step-3.7-flash-ud-q4-k-m": "step-3.7-flash"}, map[string]bool{})
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(inserts).To(BeEmpty())
+	})
+
+	It("emits a hub named for the base model when the gallery has none", func() {
+		ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n  url: u\n")
+		fams := []family{{
+			repo:      "mudler/Qwen3.5-35B-A3B-APEX-GGUF",
+			repoBase:  "Qwen3.5-35B-A3B-APEX",
+			stem:      "Qwen3.5-35B-A3B-APEX",
+			hasMMProj: true,
+			children: []childBuild{
+				buildOf("qwen3.5-35b-a3b-apex-i-quality", "mudler/Qwen3.5-35B-A3B-APEX-GGUF", "a.gguf", 0),
+				buildOf("qwen3.5-35b-a3b-ud-q6-k", "unsloth/Qwen3.5-35B-A3B-GGUF", "b.gguf", 102),
+			},
+		}}
+
+		inserts, newHubs, err := planHubs(fams, ix, noReuse,
+			allAdded("qwen3.5-35b-a3b-apex-i-quality", "qwen3.5-35b-a3b-ud-q6-k"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(inserts).To(BeEmpty())
+		Expect(newHubs).To(HaveLen(1))
+
+		hub := newHubs[0]
+		Expect(hub.Name).To(Equal("qwen3.5-35b-a3b"))
+		Expect(hub.Name).ToNot(HaveSuffix("-apex"))
+
+		// A hand-written *-apex entry is an ordinary build, referenced like any
+		// other rung and never deleted or renamed.
+		Expect(hub.Variants).To(Equal([]VariantRef{
+			{Model: "qwen3.5-35b-a3b-apex"},
+			{Model: "qwen3.5-35b-a3b-apex-i-quality"},
+			{Model: "qwen3.5-35b-a3b-ud-q6-k"},
+		}))
+
+		// The verifier skips entries with no declared backend, so a hub without
+		// one would escape the tagging check in silence.
+		Expect(hub.Overrides).To(HaveKeyWithValue("backend", "llama-cpp"))
+		Expect(hub.Files).ToNot(BeEmpty())
+		Expect(hub.Tags).To(ContainElement("vision"))
+	})
+
+	It("gathers two APEX repos that share one base model under a single hub", func() {
+		ix := mustIndex("- name: unrelated\n  url: u\n")
+		fams := []family{
+			{
+				repo:     "mudler/Solo-APEX-GGUF",
+				repoBase: "Solo-APEX",
+				stem:     "Solo-APEX",
+				children: []childBuild{buildOf("solo-apex-i-quality", "mudler/Solo-APEX-GGUF", "a.gguf", 0)},
+			},
+			{
+				repo:     "mudler/Solo-APEX-MTP-GGUF",
+				repoBase: "Solo-APEX-MTP",
+				stem:     "Solo-APEX-MTP",
+				children: []childBuild{buildOf("solo-apex-mtp-i-quality", "mudler/Solo-APEX-MTP-GGUF", "b.gguf", 0)},
+			},
+		}
+
+		_, newHubs, err := planHubs(fams, ix, noReuse, allAdded("solo-apex-i-quality", "solo-apex-mtp-i-quality"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(newHubs).To(HaveLen(1))
+		Expect(newHubs[0].Name).To(Equal("solo"))
+		Expect(newHubs[0].Variants).To(Equal([]VariantRef{
+			{Model: "solo-apex-i-quality"},
+			{Model: "solo-apex-mtp-i-quality"},
+		}))
+	})
+})
+
+var _ = Describe("hubVariants", func() {
+	It("orders builds by quality rung rather than discovery order", func() {
+		// DiscoverAPEXTiers preserves input order and the HF API returns siblings
+		// alphabetically, so an unsorted list reads I-Balanced, I-Compact, I-Mini,
+		// I-Nano, I-Quality. Selection ignores authored order; this is for the
+		// human reading the file.
+		f := family{repoBase: "X-APEX", stem: "X-APEX", children: []childBuild{
+			{rank: rungRank["I-Nano"], entry: GalleryEntry{Name: "x-i-nano"}},
+			{rank: 100, entry: GalleryEntry{Name: "x-ud-q4-k-m"}},
+			{rank: rungRank["I-Quality"], entry: GalleryEntry{Name: "x-i-quality"}},
+			{rank: rungRank["I-Compact"], entry: GalleryEntry{Name: "x-i-compact"}},
+		}}
+
+		got := hubVariants(&f, mustIndex("- name: x\n  url: u\n"), map[string]string{}, map[string]bool{})
+
+		Expect(got).To(Equal([]string{"x-i-quality", "x-i-compact", "x-i-nano", "x-ud-q4-k-m"}))
+	})
+})
+
+var _ = Describe("ParseIndexText", func() {
+	It("refuses to edit by line number when the two views of the file disagree", func() {
+		_, err := ParseIndexText("- name: one\n  url: u\n-\n")
+		Expect(err).To(MatchError(ContainSubstring("empty")))
+	})
+
+	It("records the line range of each entry", func() {
+		ix := mustIndex("- name: first\n  url: u\n- name: second\n  url: u\n")
+
+		Expect(ix.Find("FIRST").Pos.StartLine).To(Equal(0))
+		Expect(ix.Find("first").Pos.EndLine).To(Equal(2))
+		Expect(ix.Find("second").Pos.StartLine).To(Equal(2))
+	})
+})
+
+var _ = Describe("resolveVariant", func() {
+	It("keeps an entry that was emitted even when it is also in reused", func() {
+		// A within-batch name collision records reused[name] = name while the
+		// FIRST entry of that name is still in add. Treating presence in reused as
+		// "dropped" would emit nothing for it.
+		added := map[string]bool{"dup": true}
+		reused := map[string]string{"dup": "dup"}
+
+		Expect(resolveVariant("dup", reused, added)).To(Equal("dup"))
+	})
+
+	It("redirects a reused name at the entry that stands in for it", func() {
+		added := map[string]bool{}
+		reused := map[string]string{"generated": "already-in-gallery"}
+
+		Expect(resolveVariant("generated", reused, added)).To(Equal("already-in-gallery"))
+	})
+})
+
+var _ = Describe("slug", func() {
+	It("lowercases and turns quant underscores into hyphens", func() {
+		Expect(slug("UD-Q4_K_M")).To(Equal("ud-q4-k-m"))
+		Expect(slug("gemma-4-26B-A4B-it-APEX")).To(Equal("gemma-4-26b-a4b-it-apex"))
+		Expect(slug("I-Nano")).To(Equal("i-nano"))
+	})
+})
+
+var _ = Describe("sortTiers", func() {
+	It("puts the imatrix ladder in descending quality order", func() {
+		tiers := []Tier{
+			{Label: "I-Balanced"}, {Label: "I-Compact"}, {Label: "I-Mini"},
+			{Label: "I-Nano"}, {Label: "I-Quality"},
+		}
+		sortTiers(tiers)
+		Expect(tierLabels(tiers)).To(Equal("I-Quality,I-Balanced,I-Compact,I-Mini,I-Nano"))
+	})
+})
+
+var _ = Describe("restrict", func() {
+	It("keeps only the named repos", func() {
+		got := restrict([]string{"mudler/A-APEX-GGUF", "mudler/B-APEX-GGUF"}, "mudler/B-APEX-GGUF")
+		Expect(got).To(Equal([]string{"mudler/B-APEX-GGUF"}))
+	})
+
+	It("returns nothing when the filter matches nothing", func() {
+		Expect(restrict([]string{"mudler/A-APEX-GGUF"}, "mudler/typo")).To(BeEmpty())
+	})
+})
+
+var _ = Describe("reportUnclassified", func() {
+	// One real imatrix rung is always present so the specs measure how the
+	// remaining files are bucketed, not an empty-repo edge case.
+	tier := Tier{Label: "I-Quality", File: GGUFFile{Name: "Model-APEX-I-Quality.gguf"}}
+
+	censusOf := func(names ...string) fileCensus {
+		files := []GGUFFile{tier.File}
+		for _, n := range names {
+			files = append(files, GGUFFile{Name: n})
+		}
+		return reportUnclassified("mudler/Model-APEX-GGUF", files, []Tier{tier}, nil)
+	}
+
+	It("counts a flat full-precision source as excluded, not unclassified", func() {
+		got := censusOf("Carnice-MoE-35B-A3B-F16.gguf")
+		Expect(got.fullPrecision).To(Equal(1))
+		Expect(got.unclassified).To(Equal(0))
+	})
+
+	It("counts every shard of a sharded full-precision source as excluded", func() {
+		got := censusOf(
+			"MiniMax-M2.7-APEX-F16-00001-of-00003.gguf",
+			"MiniMax-M2.7-APEX-F16-00002-of-00003.gguf",
+			"MiniMax-M2.7-APEX-F16-00003-of-00003.gguf",
+		)
+		Expect(got.fullPrecision).To(Equal(3))
+		Expect(got.unclassified).To(Equal(0))
+	})
+
+	It("treats bf16 the same as f16, in either case", func() {
+		got := censusOf("Model-APEX-BF16.gguf", "Model-APEX-bf16-00001-of-00002.gguf", "Model-APEX-f16.gguf")
+		Expect(got.fullPrecision).To(Equal(3))
+		Expect(got.unclassified).To(Equal(0))
+	})
+
+	It("still reports a genuinely unknown filename as unclassified", func() {
+		got := censusOf("Model-APEX-Turbo.gguf")
+		Expect(got.unclassified).To(Equal(1))
+		Expect(got.fullPrecision).To(Equal(0))
+	})
+
+	It("separates the two kinds when a repo publishes both", func() {
+		got := censusOf("Model-APEX-F16.gguf", "Model-APEX-Turbo.gguf")
+		Expect(got.fullPrecision).To(Equal(1))
+		Expect(got.unclassified).To(Equal(1))
+	})
+})

.github/ci/apexentries/merge.go

@@ -0,0 +1,143 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+)
+
+const (
+	hfShorthandPrefix = "huggingface://"
+	hfResolvePrefix   = "https://huggingface.co/"
+	hfResolveInfix    = "/resolve/main/"
+)
+
+// canonicalURI reduces the two interchangeable spellings of a HuggingFace file
+// to one key, so a generated resolve/main URI dedups against the shorthand the
+// gallery uses for the majority of its entries.
+//
+// The repo is exactly the first two path segments; everything after is the file
+// path, which may itself contain slashes because sharded quants live in a
+// subdirectory. Anything that is not recognisably one of the two forms is
+// returned unchanged rather than guessed at, so mirrors and other hosts still
+// dedup on their literal string.
+func canonicalURI(uri string) string {
+	switch {
+	case strings.HasPrefix(uri, hfShorthandPrefix):
+		rest := strings.TrimPrefix(uri, hfShorthandPrefix)
+		owner, after, ok := strings.Cut(rest, "/")
+		if !ok {
+			return uri
+		}
+		name, file, ok := strings.Cut(after, "/")
+		if !ok || owner == "" || name == "" || file == "" {
+			return uri
+		}
+		return hfShorthandPrefix + owner + "/" + name + "/" + file
+
+	case strings.HasPrefix(uri, hfResolvePrefix):
+		rest := strings.TrimPrefix(uri, hfResolvePrefix)
+		repo, file, ok := strings.Cut(rest, hfResolveInfix)
+		if !ok || file == "" {
+			return uri
+		}
+		// A repo is owner/name and nothing more; a longer prefix means this is
+		// some other huggingface.co URL that must not be rewritten.
+		owner, name, ok := strings.Cut(repo, "/")
+		if !ok || owner == "" || name == "" || strings.Contains(name, "/") {
+			return uri
+		}
+		return hfShorthandPrefix + repo + "/" + file
+
+	default:
+		return uri
+	}
+}
+
+// ExistingIndex is the lookup built from the current gallery: entry names, and
+// which entry claims each weight URI.
+type ExistingIndex struct {
+	ByName map[string]int
+	ByURI  map[string]string
+}
+
+// LoadExisting reads the gallery index for dedup purposes only. It is
+// deliberately not used to rewrite the file: the index is 40,000 lines, and a
+// YAML round trip would reflow the whole thing into an unreviewable diff.
+func LoadExisting(path string) (*ExistingIndex, error) {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return nil, err
+	}
+
+	var entries []struct {
+		Name  string `yaml:"name"`
+		Files []struct {
+			URI string `yaml:"uri"`
+		} `yaml:"files"`
+	}
+	if err := yaml.Unmarshal(raw, &entries); err != nil {
+		return nil, fmt.Errorf("parsing %s: %w", path, err)
+	}
+
+	ix := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+	for i, e := range entries {
+		ix.ByName[e.Name] = i
+		for _, f := range e.Files {
+			if f.URI != "" {
+				ix.ByURI[canonicalURI(f.URI)] = e.Name
+			}
+		}
+	}
+	return ix, nil
+}
+
+// Merge splits generated entries into those to add and those already covered.
+// reused maps a generated name to the existing entry that stands in for it, so
+// a parent can reference what is already there instead of duplicating weights.
+// Several APEX repos share one base model, so the same counterpart rungs are
+// generated more than once in a batch. The batch has to dedup against itself as
+// well as against the gallery, tracked locally because the caller may reuse the
+// ExistingIndex it passed in.
+func Merge(existing *ExistingIndex, generated []GalleryEntry) (add []GalleryEntry, reused map[string]string) {
+	reused = map[string]string{}
+	batchNames := map[string]string{}
+	batchURIs := map[string]string{}
+
+	// Canonicalized into a local copy rather than in place: an ExistingIndex may
+	// be hand-built or reused by the caller, so Merge must not rewrite it.
+	existingURIs := make(map[string]string, len(existing.ByURI))
+	for uri, owner := range existing.ByURI {
+		existingURIs[canonicalURI(uri)] = owner
+	}
+
+	for _, e := range generated {
+		// Name is checked before URI: a name collision must block the add
+		// whatever the weights say, since duplicate names corrupt the index.
+		if _, clash := existing.ByName[e.Name]; clash {
+			reused[e.Name] = e.Name
+			continue
+		}
+		if claimant, clash := batchNames[e.Name]; clash {
+			reused[e.Name] = claimant
+			continue
+		}
+		if len(e.Files) > 0 {
+			uri := canonicalURI(e.Files[0].URI)
+			if owner, ok := existingURIs[uri]; ok {
+				reused[e.Name] = owner
+				continue
+			}
+			if claimant, ok := batchURIs[uri]; ok {
+				reused[e.Name] = claimant
+				continue
+			}
+			batchURIs[uri] = e.Name
+		}
+		batchNames[e.Name] = e.Name
+		add = append(add, e)
+	}
+	return add, reused
+}

.github/ci/apexentries/merge_test.go

@@ -0,0 +1,183 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Merge", func() {
+	It("drops a generated entry whose weight URI already exists and reports the existing name", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"qwen3.6-35b-a3b-apex": 0},
+			ByURI: map[string]string{
+				"https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf": "qwen3.6-35b-a3b-apex",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "x-apex-i-quality",
+			Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("x-apex-i-quality", "qwen3.6-35b-a3b-apex"))
+	})
+
+	It("keeps a generated entry whose weights are new", func() {
+		existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+		gen := []GalleryEntry{{
+			Name:  "x-apex-i-mini",
+			Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Mini.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(reused).To(BeEmpty())
+	})
+
+	It("refuses to add an entry whose name collides with an existing one", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"x-apex-i-mini": 0},
+			ByURI:  map[string]string{},
+		}
+		gen := []GalleryEntry{{
+			Name:  "x-apex-i-mini",
+			Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/other.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("x-apex-i-mini", "x-apex-i-mini"))
+	})
+
+	// The gallery records most of its URIs in huggingface:// shorthand while
+	// render.go only ever emits the resolve/main form, so without
+	// canonicalization the majority of the file is invisible to the dedup.
+	It("matches a generated https URI against the shorthand form recorded in the gallery", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"foo-gguf-q8-0": 0},
+			ByURI: map[string]string{
+				"huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf": "foo-gguf-q8-0",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "foo-apex-q8-0",
+			Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0"))
+	})
+
+	It("matches a generated shorthand URI against the https form recorded in the gallery", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"foo-gguf-q8-0": 0},
+			ByURI: map[string]string{
+				"https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf": "foo-gguf-q8-0",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "foo-apex-q8-0",
+			Files: []EntryFile{{URI: "huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0"))
+	})
+
+	// Sharded quants live under a subdirectory, so the file path carries slashes
+	// of its own and only the first two segments are the repo.
+	It("matches across both forms when the file path has a subdirectory", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"model-ud-q4-k-m": 0},
+			ByURI: map[string]string{
+				"huggingface://unsloth/Model-GGUF/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf": "model-ud-q4-k-m",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "model-apex-ud-q4-k-m",
+			Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Model-GGUF/resolve/main/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("model-apex-ud-q4-k-m", "model-ud-q4-k-m"))
+	})
+
+	// Several APEX repos share one base model, so the same unsloth rungs are
+	// generated more than once in a single batch.
+	It("adds only the first of two generated entries sharing a name", func() {
+		existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+		gen := []GalleryEntry{
+			{
+				Name:  "shared-rung-q8-0",
+				Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}},
+			},
+			{
+				Name:  "shared-rung-q8-0",
+				Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Other-GGUF/resolve/main/Other-Q8_0.gguf"}},
+			},
+		}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(add[0].Files[0].URI).To(Equal("https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"))
+		Expect(reused).To(HaveKeyWithValue("shared-rung-q8-0", "shared-rung-q8-0"))
+	})
+
+	It("adds only the first of two generated entries sharing a primary URI", func() {
+		existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+		gen := []GalleryEntry{
+			{
+				Name:  "shared-rung-from-apex",
+				Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}},
+			},
+			{
+				Name:  "shared-rung-from-apex-mtp",
+				Files: []EntryFile{{URI: "huggingface://unsloth/Shared-GGUF/Shared-Q8_0.gguf"}},
+			},
+		}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(add[0].Name).To(Equal("shared-rung-from-apex"))
+		Expect(reused).To(HaveKeyWithValue("shared-rung-from-apex-mtp", "shared-rung-from-apex"))
+	})
+
+	// Anything that is not a HuggingFace URI must survive untouched, so an
+	// unrecognised scheme still dedups against the very same string.
+	It("leaves a URI in neither recognised form alone and still dedups it exactly", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"mirrored-model": 0},
+			ByURI: map[string]string{
+				"https://mirror.example.com/weights/Model-Q8_0.gguf": "mirrored-model",
+			},
+		}
+		gen := []GalleryEntry{
+			{
+				Name:  "mirrored-apex",
+				Files: []EntryFile{{URI: "https://mirror.example.com/weights/Model-Q8_0.gguf"}},
+			},
+			{
+				Name:  "elsewhere-apex",
+				Files: []EntryFile{{URI: "https://mirror.example.com/weights/Other-Q8_0.gguf"}},
+			},
+		}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(add[0].Name).To(Equal("elsewhere-apex"))
+		Expect(reused).To(HaveKeyWithValue("mirrored-apex", "mirrored-model"))
+	})
+})

.github/ci/apexentries/render.go

@@ -0,0 +1,175 @@
+package main
+
+import (
+	"fmt"
+	"path"
+	"strings"
+)
+
+// EntryFile is one downloadable file of a gallery entry.
+type EntryFile struct {
+	Filename string `yaml:"filename"`
+	SHA256   string `yaml:"sha256"`
+	URI      string `yaml:"uri"`
+}
+
+// GalleryEntry is the subset of a gallery entry this generator writes.
+//
+// Named GalleryEntry rather than Entry because the test files dot-import
+// Ginkgo, whose table DSL exports an Entry that a package-level Entry would
+// collide with. The yaml tags are what the gallery index sees, so the Go
+// identifier is free to differ.
+type GalleryEntry struct {
+	Name        string         `yaml:"name"`
+	URL         string         `yaml:"url"`
+	Description string         `yaml:"description,omitempty"`
+	Tags        []string       `yaml:"tags,omitempty"`
+	Overrides   map[string]any `yaml:"overrides,omitempty"`
+	Files       []EntryFile    `yaml:"files,omitempty"`
+	Variants    []VariantRef   `yaml:"variants,omitempty"`
+}
+
+// VariantRef mirrors the gallery's variant reference: a name and nothing else.
+type VariantRef struct {
+	Model string `yaml:"model"`
+}
+
+// ChildInput is everything needed to render one non-parent entry.
+type ChildInput struct {
+	Name string
+	Repo string
+	// DraftRepo is the repo publishing the drafter, when it is not the repo
+	// publishing the weights. Speculative pairings routinely cross repos, so
+	// the drafter cannot be assumed to sit next to the weights. Empty means
+	// same-repo, which is how the *-APEX-MTP-GGUF repos ship.
+	DraftRepo string
+	Template  string
+	Weights   []GGUFFile
+	MMProj    *GGUFFile
+	SpecType  string
+	DraftFile *GGUFFile
+	BaseTags  []string
+}
+
+// specTuning is the acceptance-window tuning each spec type ships with, copied
+// from the hand-written entries that already run these two mechanisms rather
+// than invented here. The two differ because the drafters differ: self-drafted
+// MTP heads produce a short, high-confidence proposal (15+ hand-written entries
+// use 6 with a 0.75 floor), while a separate DFlash drafter is cheap enough to
+// run far ahead unconditionally (the five hand-written dflash entries use 15 and
+// set no floor).
+var specTuning = map[string][]string{
+	"draft-mtp":    {"spec_n_max:6", "spec_p_min:0.75"},
+	"draft-dflash": {"spec_n_max:15"},
+}
+
+func hfURI(repo, file string) string {
+	return fmt.Sprintf("https://huggingface.co/%s/resolve/main/%s", repo, file)
+}
+
+// localPath is where a downloaded file lands.
+//
+// The hand-written entries namespace by the repo's BARE name
+// (llama-cpp/models/<repo>/<file>), which is not unique. LiquidAI/LFM2.5-8B-A1B-GGUF
+// and unsloth/LFM2.5-8B-A1B-GGUF share a basename, so both claim
+// llama-cpp/models/LFM2.5-8B-A1B-GGUF/, and installing the second after the first
+// either overwrites weights whose recorded sha256 belongs to the other file or is
+// skipped as already present. Two owners publishing the same model name is the
+// normal case for quantizers, not an edge case, so the owner has to be in the path.
+//
+// The owner becomes its own path segment rather than being folded into the
+// directory name: owner/repo is unique on HuggingFace and "/" cannot occur inside
+// either half, so this is the only form that is collision-proof by construction.
+// It still reads as the hand-written convention with the owner restored, and the
+// extra depth is already present in the index for sharded builds.
+func localPath(kind, repo, file string) string {
+	// path.Dir yields "." for a repo named without an owner, which path.Join
+	// drops, so such a caller keeps the historical two-segment layout.
+	return path.Join("llama-cpp", kind, path.Dir(repo), path.Base(repo), file)
+}
+
+// RenderChild builds one child entry.
+//
+// The dflash/mtp tag is added if and only if this entry sets a spec_type,
+// because variant ranking reads tags and nothing else, and a tag that does not
+// match what the entry configures either promotes a build that is no faster or
+// hides one that is.
+func RenderChild(in ChildInput) GalleryEntry {
+	e := GalleryEntry{
+		Name:      in.Name,
+		URL:       fmt.Sprintf("github:mudler/LocalAI/gallery/%s@master", in.Template),
+		Tags:      append([]string{}, in.BaseTags...),
+		Overrides: map[string]any{},
+	}
+
+	// gallery/virtual.yaml carries no backend, so nothing else would name an
+	// engine for these entries. Matching the hand-written entries on
+	// known_usecases too: LocalAI would fall back to the backend defaults, but
+	// generated entries should not read differently from their neighbours.
+	e.Overrides["backend"] = "llama-cpp"
+	e.Overrides["known_usecases"] = []string{"chat"}
+
+	options := []string{"use_jinja:true"}
+
+	for _, w := range in.Weights {
+		e.Files = append(e.Files, EntryFile{
+			Filename: localPath("models", in.Repo, w.Name),
+			SHA256:   w.SHA256,
+			URI:      hfURI(in.Repo, w.Name),
+		})
+	}
+	e.Overrides["parameters"] = map[string]any{
+		"model": localPath("models", in.Repo, in.Weights[0].Name),
+	}
+
+	if in.MMProj != nil {
+		// An explicit known_usecases SUPPRESSES the backend-default fallback in
+		// core/gallery/models_types.go, so a multimodal entry left at chat-only
+		// never matches FilterGalleryModelsByUsecase(FLAG_VISION) or
+		// FilterGalleryModelsByMultimodal and vanishes from the UI's vision and
+		// multimodal filters. 19 of the 45 APEX repos ship an mmproj.
+		e.Overrides["known_usecases"] = []string{"chat", "vision"}
+		e.Overrides["mmproj"] = localPath("mmproj", in.Repo, in.MMProj.Name)
+		e.Files = append(e.Files, EntryFile{
+			Filename: localPath("mmproj", in.Repo, in.MMProj.Name),
+			SHA256:   in.MMProj.SHA256,
+			URI:      hfURI(in.Repo, in.MMProj.Name),
+		})
+	}
+
+	// A spec type is configured independently of a drafter FILE. Weights that
+	// carry their own MTP heads need no second download, and requiring one left
+	// the *-APEX-MTP-GGUF builds shipping the larger heads-bearing weights with
+	// the heads switched off: a strictly bigger download at the same speed,
+	// ranked identically to the plain rung at the same tier.
+	if in.SpecType != "" {
+		options = append(options, "spec_type:"+in.SpecType)
+		options = append(options, specTuning[in.SpecType]...)
+		// The tag is derived from the spec type this entry sets and from nothing
+		// else. Variant ranking reads tags only, so a tag taken from a repo or
+		// entry NAME would promote a build that is no faster whenever the name
+		// and the configuration disagree.
+		e.Tags = append(e.Tags, strings.TrimPrefix(in.SpecType, "draft-"))
+	}
+
+	if in.SpecType != "" && in.DraftFile != nil {
+		// Fall back to the weights repo so pairings that publish the drafter
+		// alongside the weights keep working without restating the repo.
+		draftRepo := in.DraftRepo
+		if draftRepo == "" {
+			draftRepo = in.Repo
+		}
+		draftPath := localPath("models", draftRepo, in.DraftFile.Name)
+
+		e.Overrides["draft_model"] = draftPath
+		e.Overrides["flash_attention"] = "on"
+		e.Files = append(e.Files, EntryFile{
+			Filename: draftPath,
+			SHA256:   in.DraftFile.SHA256,
+			URI:      hfURI(draftRepo, in.DraftFile.Name),
+		})
+	}
+
+	e.Overrides["options"] = options
+	return e
+}

.github/ci/apexentries/render_test.go

@@ -0,0 +1,249 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("RenderChild", func() {
+	It("tags an entry that configures draft-dflash", func() {
+		e := RenderChild(ChildInput{
+			Name:      "qwen3.5-9b-dflash",
+			Repo:      "mudler/Example-APEX-GGUF",
+			Template:  "virtual.yaml",
+			Weights:   []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			SpecType:  "draft-dflash",
+			DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"},
+			BaseTags:  []string{"llm", "gguf"},
+		})
+
+		Expect(e.Tags).To(ContainElement("dflash"))
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Overrides["options"]).To(ContainElement("spec_type:draft-dflash"))
+		Expect(e.Overrides["draft_model"]).ToNot(BeNil())
+	})
+
+	It("does not tag an MTP-named repo that configures no speculation", func() {
+		// mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF ships MTP-bearing weights. Weights
+		// that carry the heads are not an entry that enables them, and tagging it
+		// would win the feature axis without being any faster.
+		e := RenderChild(ChildInput{
+			Name:     "qwen3.6-35b-a3b-apex-mtp-i-quality",
+			Repo:     "mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Qwen3.6-35B-A3B-APEX-MTP-I-Quality.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Tags).ToNot(ContainElement("dflash"))
+		Expect(e.Overrides).ToNot(HaveKey("draft_model"))
+	})
+
+	It("lists every shard of a sharded build and points the model at the first", func() {
+		e := RenderChild(ChildInput{
+			Name:     "step-3.7-flash-ud-q4-k-m",
+			Repo:     "unsloth/Step-3.7-Flash-GGUF",
+			Template: "virtual.yaml",
+			Weights: []GGUFFile{
+				{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"},
+				{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"},
+			},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Files).To(HaveLen(2))
+		params, ok := e.Overrides["parameters"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		Expect(params["model"]).To(HaveSuffix("00001-of-00002.gguf"))
+		Expect(e.Files[0].URI).To(Equal(
+			"https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf"))
+	})
+
+	It("wires mmproj when the repo publishes one", func() {
+		e := RenderChild(ChildInput{
+			Name:     "example-i-mini",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}},
+			MMProj:   &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["mmproj"]).ToNot(BeNil())
+		Expect(e.Files).To(HaveLen(2))
+	})
+
+	It("names the engine and the usecases the hand-written entries name", func() {
+		// gallery/virtual.yaml supplies no backend, so an entry that omits one
+		// names no engine at all and cannot load.
+		e := RenderChild(ChildInput{
+			Name:     "example-i-mini",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["backend"]).To(Equal("llama-cpp"))
+		Expect(e.Overrides["known_usecases"]).To(ContainElement("chat"))
+	})
+
+	It("draws the drafter from DraftRepo when the pairing spans two repos", func() {
+		// unsloth/Qwen3-4B-GGUF pairs with a drafter published separately by
+		// AtomicChat, so a drafter URI built from the weights repo 404s.
+		e := RenderChild(ChildInput{
+			Name:      "qwen3-4b-dflash",
+			Repo:      "unsloth/Qwen3-4B-GGUF",
+			DraftRepo: "AtomicChat/Qwen3-4B-DFlash-GGUF",
+			Template:  "virtual.yaml",
+			Weights:   []GGUFFile{{Name: "Qwen3-4B-Q4_K_M.gguf", SHA256: "a"}},
+			SpecType:  "draft-dflash",
+			DraftFile: &GGUFFile{Name: "Qwen3-4B-DFlash.Q8_0.gguf", SHA256: "b"},
+			BaseTags:  []string{"llm", "gguf"},
+		})
+
+		Expect(e.Files[0].URI).To(Equal(
+			"https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf"))
+		Expect(e.Files[1].URI).To(Equal(
+			"https://huggingface.co/AtomicChat/Qwen3-4B-DFlash-GGUF/resolve/main/Qwen3-4B-DFlash.Q8_0.gguf"))
+		Expect(e.Files[1].Filename).To(Equal(
+			"llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf"))
+		Expect(e.Overrides["draft_model"]).To(Equal(
+			"llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf"))
+	})
+
+	It("falls back to the weights repo for the drafter when DraftRepo is empty", func() {
+		// The *-APEX-MTP-GGUF repos ship the drafter alongside the weights.
+		e := RenderChild(ChildInput{
+			Name:      "example-apex-dflash",
+			Repo:      "mudler/Example-APEX-GGUF",
+			Template:  "virtual.yaml",
+			Weights:   []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			SpecType:  "draft-dflash",
+			DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"},
+			BaseTags:  []string{"llm", "gguf"},
+		})
+
+		Expect(e.Files[1].URI).To(Equal(
+			"https://huggingface.co/mudler/Example-APEX-GGUF/resolve/main/Example-DFlash.Q8_0.gguf"))
+		Expect(e.Files[1].Filename).To(Equal(
+			"llama-cpp/models/mudler/Example-APEX-GGUF/Example-DFlash.Q8_0.gguf"))
+	})
+})
+
+var _ = Describe("RenderChild known_usecases", func() {
+	It("declares vision alongside chat when the entry carries an mmproj", func() {
+		// An explicit known_usecases suppresses the backend-default fallback, so a
+		// chat-only multimodal entry disappears from the UI's vision filter.
+		e := RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			MMProj:   &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat", "vision"))
+	})
+
+	It("leaves a text-only entry at chat", func() {
+		e := RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat"))
+	})
+})
+
+var _ = Describe("localPath", func() {
+	It("keeps two repos with the same basename but different owners apart", func() {
+		// LiquidAI and unsloth both publish LFM2.5-8B-A1B-GGUF. A path built from
+		// the bare repo name gives both the same local file, so installing the
+		// second overwrites or skips the first and one of them then serves bytes
+		// that do not match its recorded sha256.
+		liquid := RenderChild(ChildInput{
+			Name:     "lfm2.5-8b-a1b-i-quality",
+			Repo:     "LiquidAI/LFM2.5-8B-A1B-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "33ab3b8c"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+		unsloth := RenderChild(ChildInput{
+			Name:     "lfm2.5-8b-a1b-q8-0",
+			Repo:     "unsloth/LFM2.5-8B-A1B-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "ec11666b"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(liquid.Files[0].Filename).ToNot(Equal(unsloth.Files[0].Filename))
+		Expect(unsloth.Files[0].Filename).To(Equal(
+			"llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf"))
+	})
+
+	It("namespaces the mmproj by owner too", func() {
+		e := RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			MMProj:   &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["mmproj"]).To(Equal(
+			"llama-cpp/mmproj/mudler/Example-APEX-GGUF/mmproj-F16.gguf"))
+	})
+})
+
+var _ = Describe("MTP builds", func() {
+	renderTier := func(repo string) GalleryEntry {
+		return RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     repo,
+			Template: "virtual.yaml",
+			SpecType: SpecTypeForRepo(repo),
+			Weights:  []GGUFFile{{Name: "Example-I-Quality.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+	}
+
+	It("turns MTP on for a build off an APEX-MTP repo", func() {
+		// These weights retain the model's own MTP heads, so shipping them with
+		// speculation off is a strictly larger download at the same speed,
+		// ranked identically to the plain rung at the same tier.
+		e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF")
+
+		Expect(e.Overrides["options"]).To(ContainElements(
+			"spec_type:draft-mtp", "spec_n_max:6", "spec_p_min:0.75"))
+		Expect(e.Tags).To(ContainElement("mtp"))
+	})
+
+	It("needs no drafter file, because the heads travel with the weights", func() {
+		e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF")
+
+		Expect(e.Overrides).ToNot(HaveKey("draft_model"))
+		Expect(e.Files).To(HaveLen(1))
+	})
+
+	It("leaves a build off a plain APEX repo alone", func() {
+		e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-GGUF")
+
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true"))
+	})
+
+	It("leaves an unsloth counterpart rung alone", func() {
+		// The counterpart quantizes the plain weights; nothing there carries heads.
+		e := renderTier("unsloth/Qwen3.6-35B-A3B-GGUF")
+
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true"))
+	})
+})

.github/ci/apexentries/unsloth.go

@@ -0,0 +1,71 @@
+package main
+
+import (
+	"regexp"
+	"sort"
+	"strings"
+)
+
+// WantedQuants is the fixed unsloth subset this generator emits. It is a
+// deliberate subset: unsloth publishes north of 20 quants per repo, and the
+// selector needs useful fitness points rather than every rung.
+var WantedQuants = []string{"UD-Q4_K_M", "UD-Q5_K_M", "UD-Q6_K", "Q8_0"}
+
+var shardRE = regexp.MustCompile(`-(\d{5})-of-(\d{5})\.gguf$`)
+
+// QuantBuild is one unsloth quantization, which may be a single file or an
+// ordered set of shards.
+type QuantBuild struct {
+	Quant   string
+	Files   []GGUFFile
+	Sharded bool
+}
+
+// CounterpartCandidates returns the unsloth repo base names worth probing, most
+// likely first. Both derivations are needed: the repo name finds
+// unsloth/gemma-4-26B-A4B-it-GGUF, while the file stem is what matches for
+// repos whose stem is the canonical model name.
+func CounterpartCandidates(repoName, fileStem string) []string {
+	clean := func(s string) string {
+		s = strings.TrimSuffix(s, "-GGUF")
+		s = regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "")
+		s = strings.TrimSuffix(s, "-APEX")
+		return regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "")
+	}
+
+	out := []string{clean(repoName)}
+	if stem := clean(fileStem); stem != out[0] {
+		out = append(out, stem)
+	}
+	return out
+}
+
+// DiscoverUnslothQuants returns the wanted quants a repo publishes, handling
+// both the flat single-file layout and the sharded layout where a quant lives
+// in its own subdirectory.
+func DiscoverUnslothQuants(files []GGUFFile) []QuantBuild {
+	var out []QuantBuild
+
+	for _, q := range WantedQuants {
+		var flat []GGUFFile
+		var shards []GGUFFile
+
+		for _, f := range files {
+			switch {
+			case !strings.Contains(f.Name, "/") && strings.HasSuffix(f.Name, "-"+q+".gguf"):
+				flat = append(flat, f)
+			case strings.HasPrefix(f.Name, q+"/") && shardRE.MatchString(f.Name):
+				shards = append(shards, f)
+			}
+		}
+
+		switch {
+		case len(flat) > 0:
+			out = append(out, QuantBuild{Quant: q, Files: flat})
+		case len(shards) > 0:
+			sort.Slice(shards, func(i, j int) bool { return shards[i].Name < shards[j].Name })
+			out = append(out, QuantBuild{Quant: q, Files: shards, Sharded: true})
+		}
+	}
+	return out
+}

.github/ci/apexentries/unsloth_test.go

@@ -0,0 +1,75 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("CounterpartCandidates", func() {
+	It("offers both the repo-derived and stem-derived names", func() {
+		// mudler/gemma-4-26B-A4B-it-APEX-GGUF ships gemma-4-26B-A4B-APEX-*.gguf,
+		// and only the repo-derived name finds unsloth/gemma-4-26B-A4B-it-GGUF.
+		got := CounterpartCandidates("gemma-4-26B-A4B-it-APEX-GGUF", "gemma-4-26B-A4B-APEX")
+
+		Expect(got).To(Equal([]string{"gemma-4-26B-A4B-it", "gemma-4-26B-A4B"}))
+	})
+
+	It("strips the MTP marker", func() {
+		got := CounterpartCandidates("Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF", "Qwopus3.6-35B-A3B-v1-APEX-MTP")
+
+		Expect(got[0]).To(Equal("Qwopus3.6-35B-A3B-v1"))
+	})
+
+	It("strips the TQ marker", func() {
+		// This is the branch that folds mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF into
+		// the qwen3.5-35b-a3b hub. Without it the probe is
+		// unsloth/Qwen3.5-35B-A3B-TQ-GGUF, which does not exist, so the family
+		// silently loses every unsloth rung.
+		got := CounterpartCandidates("Qwen3.5-35B-A3B-APEX-TQ-GGUF", "Qwen3.5-35B-A3B-APEX-TQ")
+
+		Expect(got).To(Equal([]string{"Qwen3.5-35B-A3B"}))
+	})
+
+	It("does not repeat a candidate when both derivations agree", func() {
+		got := CounterpartCandidates("Qwen3.6-35B-A3B-APEX-GGUF", "Qwen3.6-35B-A3B-APEX")
+
+		Expect(got).To(Equal([]string{"Qwen3.6-35B-A3B"}))
+	})
+})
+
+var _ = Describe("DiscoverUnslothQuants", func() {
+	It("finds flat single-file quants", func() {
+		files := []GGUFFile{
+			{Name: "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", SHA256: "a"},
+			{Name: "Qwen3.6-35B-A3B-UD-IQ1_M.gguf", SHA256: "b"},
+		}
+
+		got := DiscoverUnslothQuants(files)
+
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Quant).To(Equal("UD-Q4_K_M"))
+		Expect(got[0].Sharded).To(BeFalse())
+		Expect(got[0].Files).To(HaveLen(1))
+	})
+
+	It("collects a sharded quant from its subdirectory in shard order", func() {
+		files := []GGUFFile{
+			{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"},
+			{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"},
+		}
+
+		got := DiscoverUnslothQuants(files)
+
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Quant).To(Equal("UD-Q4_K_M"))
+		Expect(got[0].Sharded).To(BeTrue())
+		Expect(got[0].Files).To(HaveLen(2))
+		Expect(got[0].Files[0].Name).To(HaveSuffix("00001-of-00002.gguf"))
+	})
+
+	It("ignores quants outside the wanted subset", func() {
+		files := []GGUFFile{{Name: "Model-UD-IQ2_XXS.gguf", SHA256: "a"}}
+
+		Expect(DiscoverUnslothQuants(files)).To(BeEmpty())
+	})
+})

.github/ci/apexentries/verify.go

@@ -0,0 +1,312 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+)
+
+type verifyEntry struct {
+	Name      string       `yaml:"name"`
+	Tags      []string     `yaml:"tags"`
+	Variants  []VariantRef `yaml:"variants"`
+	Overrides struct {
+		// Backend scopes the checks that only hold for one engine. An entry that
+		// declares none takes its configuration from the referenced url: template,
+		// which this verifier never reads, so it cannot be judged either way.
+		Backend string   `yaml:"backend"`
+		Options []string `yaml:"options"`
+		// MMProj and DraftModel name the files that are not weights. They are
+		// the only signal for it: a drafter lands in the same models/ prefix as
+		// the weights, so the path alone cannot tell them apart.
+		MMProj     string `yaml:"mmproj"`
+		DraftModel string `yaml:"draft_model"`
+	} `yaml:"overrides"`
+	Files []struct {
+		Filename string `yaml:"filename"`
+		SHA256   string `yaml:"sha256"`
+		URI      string `yaml:"uri"`
+	} `yaml:"files"`
+}
+
+// Verify checks the invariants the variants schema and the tagging rule
+// require. It returns every problem rather than the first, so one run tells the
+// author everything that needs fixing.
+func Verify(path string) []string {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return []string{fmt.Sprintf("reading %s: %v", path, err)}
+	}
+
+	var entries []verifyEntry
+	if err := yaml.Unmarshal(raw, &entries); err != nil {
+		return []string{fmt.Sprintf("parsing %s: %v", path, err)}
+	}
+
+	var problems []string
+
+	byName := map[string]verifyEntry{}
+	for _, e := range entries {
+		if _, seen := byName[e.Name]; seen {
+			problems = append(problems, fmt.Sprintf("duplicate entry name: %s", e.Name))
+			continue
+		}
+		byName[e.Name] = e
+	}
+
+	for _, e := range entries {
+		for _, v := range e.Variants {
+			target, ok := byName[v.Model]
+			if !ok {
+				problems = append(problems, fmt.Sprintf("%s: variant %q does not exist", e.Name, v.Model))
+				continue
+			}
+			if len(target.Variants) > 0 {
+				problems = append(problems, fmt.Sprintf("%s: variant %q declares variants of its own", e.Name, v.Model))
+			}
+		}
+
+		for _, f := range e.Files {
+			if requiresSHA256(f.Filename) && f.SHA256 == "" {
+				problems = append(problems, fmt.Sprintf("%s: file %s has no sha256", e.Name, f.Filename))
+			}
+		}
+
+		problems = append(problems, checkWeightCount(e)...)
+		problems = append(problems, checkFeatureTag(e, "dflash")...)
+		problems = append(problems, checkFeatureTag(e, "mtp")...)
+	}
+
+	problems = append(problems, checkPathCollisions(entries)...)
+
+	return problems
+}
+
+// checkPathCollisions catches two different upstream files claiming one local
+// path. The install layer keys on the local filename, so whichever entry is
+// installed second either overwrites weights the first entry recorded a
+// different sha256 for or is skipped as already present. Either way some entry
+// afterwards serves bytes that do not match its own checksum, and nothing at
+// install time says so.
+//
+// This is an index-wide invariant rather than a per-entry one: neither entry is
+// wrong on its own and the collision exists only in their pairing. The usual
+// source is a path scheme built from the repo's BARE name, because two owners
+// publishing the same model name is routine for quantizers.
+//
+// Sharing a path is fine when the uri is the same, which is how several entries
+// legitimately reuse one projector. Files with no uri are skipped: there is
+// nothing to compare.
+func checkPathCollisions(entries []verifyEntry) []string {
+	type source struct{ uri, entry string }
+
+	first := map[string]source{}
+	reported := map[string]bool{}
+
+	var problems []string
+	for _, e := range entries {
+		for _, f := range e.Files {
+			if f.Filename == "" || f.URI == "" {
+				continue
+			}
+			prev, seen := first[f.Filename]
+			if !seen {
+				first[f.Filename] = source{uri: f.URI, entry: e.Name}
+				continue
+			}
+			if prev.uri == f.URI || reported[f.Filename] {
+				continue
+			}
+			// Reported once per path however many entries pile onto it, so one
+			// heavily reused filename cannot bury the rest of the report.
+			reported[f.Filename] = true
+			problems = append(problems, fmt.Sprintf(
+				"local path %s is claimed by two different uris: %s (%s) and %s (%s)",
+				f.Filename, prev.uri, prev.entry, f.URI, e.Name))
+		}
+	}
+	return problems
+}
+
+// auxiliaryExtensions are the metadata formats an entry ships beside its
+// weights, where an unverified download is a nuisance rather than a hole.
+//
+// The exclusion is stated as a list of metadata formats on purpose. Requiring
+// the checksum only on a blessed list of weight formats would silently exempt
+// every format nobody has shipped yet, and it already exempted safetensors
+// weights, which are downloaded and loaded exactly like GGUF ones.
+var auxiliaryExtensions = []string{".json", ".txt", ".md"}
+
+// requiresSHA256 reports whether an unverified download of this file would be
+// a supply-chain hole rather than a cosmetic gap.
+func requiresSHA256(filename string) bool {
+	for _, ext := range auxiliaryExtensions {
+		if strings.HasSuffix(filename, ext) {
+			return false
+		}
+	}
+	return true
+}
+
+// checkWeightCount catches an entry carrying two whole models. The flat-match
+// branch in DiscoverUnslothQuants appends every match, so a quant label that is
+// a suffix of another one (Q8_0 of UD-Q8_0) collects both files into one build
+// while the rendered model: points at only the first. The result downloads
+// twice the bytes and serves whichever file sorted first, silently.
+//
+// Shards are exempt because a sharded build is legitimately many files.
+//
+// The collision is a property of llama-cpp quant discovery, so the check is
+// scoped to that backend. Multi-component TTS, ASR and diffusion engines ship an
+// encoder, a decoder and a vocoder as one model, and there the second GGUF is
+// the design rather than a bug.
+func checkWeightCount(e verifyEntry) []string {
+	if e.Overrides.Backend != "llama-cpp" {
+		return nil
+	}
+
+	var weights []string
+	for _, f := range e.Files {
+		switch {
+		case !strings.HasSuffix(f.Filename, ".gguf"):
+		case shardRE.MatchString(f.Filename):
+		case f.Filename == e.Overrides.MMProj:
+		case f.Filename == e.Overrides.DraftModel:
+		default:
+			weights = append(weights, f.Filename)
+		}
+	}
+
+	if len(weights) > 1 {
+		return []string{fmt.Sprintf("%s: more than one weight file: %s", e.Name, strings.Join(weights, ", "))}
+	}
+	return nil
+}
+
+// checkFeatureTag enforces the rule in both directions. A tag without the
+// configuration promotes a build that is no faster; configuration without the
+// tag leaves a genuinely faster build ranked as plain.
+//
+// It only speaks about backends whose declaration it can actually read, because
+// a rule applied where the evidence is invisible reports noise rather than bugs.
+func checkFeatureTag(e verifyEntry, feature string) []string {
+	decl, configured, judgeable := featureDeclaration(e, feature)
+	if !judgeable {
+		return nil
+	}
+
+	tagged := false
+	for _, t := range e.Tags {
+		if t == feature {
+			tagged = true
+			break
+		}
+	}
+
+	switch {
+	case tagged && !configured:
+		return []string{fmt.Sprintf("%s: tagged %s but sets no %s", e.Name, feature, decl)}
+	case configured && !tagged:
+		return []string{fmt.Sprintf("%s: sets %s but is not tagged %s", e.Name, decl, feature)}
+	}
+	return nil
+}
+
+// featureDeclaration implements the per-backend table in
+// .agents/adding-gallery-models.md. It returns the declaration the backend uses
+// to configure the feature, whether the entry carries it, and whether this
+// verifier is in a position to answer at all.
+func featureDeclaration(e verifyEntry, feature string) (decl string, configured, judgeable bool) {
+	switch e.Overrides.Backend {
+	case "llama-cpp":
+		decl = "spec_type:draft-" + feature
+		for _, o := range e.Overrides.Options {
+			if strings.TrimSpace(o) == decl {
+				return decl, true, true
+			}
+		}
+		return decl, false, true
+
+	case "ds4":
+		// ds4 carries the MTP heads in the weights and turns them on with
+		// mtp_path / mtp_draft. It has no dflash counterpart, so dflash is not a
+		// question that can be asked of a ds4 entry.
+		if feature != "mtp" {
+			return "", false, false
+		}
+		decl = "mtp_path:"
+		for _, o := range e.Overrides.Options {
+			o = strings.TrimSpace(o)
+			if strings.HasPrefix(o, "mtp_path:") || strings.HasPrefix(o, "mtp_draft:") {
+				return decl, true, true
+			}
+		}
+		return decl, false, true
+
+	default:
+		// sglang configures the feature with speculative_algorithm: in the
+		// referenced gallery/*.yaml, and an entry that declares no backend takes
+		// its whole configuration from its url: template. Verify reads one index
+		// file and follows neither, so it must not judge these in either
+		// direction.
+		return "", false, false
+	}
+}
+
+// UnaccountedQuants reports a wanted quant the repo demonstrably publishes but
+// that discovery produced no build for. The layout that triggers it today is
+// root-level shards, which match neither branch of DiscoverUnslothQuants; no
+// counterpart ships that way yet, but a batch generator must not drop a build
+// with nothing said about it.
+func UnaccountedQuants(files []GGUFFile, builds []QuantBuild) []string {
+	built := map[string]bool{}
+	for _, b := range builds {
+		built[b.Quant] = true
+	}
+
+	var problems []string
+	for _, q := range WantedQuants {
+		if built[q] {
+			continue
+		}
+		for _, f := range files {
+			if filePublishesQuant(f.Name, q) {
+				problems = append(problems, fmt.Sprintf("quant %s is published upstream (%s) but produced no build", q, f.Name))
+				break
+			}
+		}
+	}
+	return problems
+}
+
+// filePublishesQuant reports whether an upstream file is a publication of
+// quant q. It anchors on the quant label the way DiscoverUnslothQuants does,
+// as the trailing token of the base name or as the sharding subdirectory, so
+// the diagnostic and the discovery it audits cannot disagree about what a file
+// is.
+//
+// An unanchored match would reproduce the very collision this diagnostic warns
+// about: Q8_0 is a substring of UD-Q8_0, so a repo publishing only UD-Q8_0
+// would be reported as publishing an unbuilt Q8_0, which it does not, and
+// UD-Q8_0 is not a wanted quant at all.
+func filePublishesQuant(name, q string) bool {
+	if strings.HasPrefix(name, q+"/") {
+		return true
+	}
+
+	base := name[strings.LastIndex(name, "/")+1:]
+	// Shard numbering sits between the quant label and the extension, so it has
+	// to come off before the label can be read as the trailing token. Root-level
+	// shards are the layout that matches neither branch of
+	// DiscoverUnslothQuants, and so the layout this diagnostic mainly catches.
+	base = shardRE.ReplaceAllString(base, ".gguf")
+
+	if !strings.HasSuffix(base, "-"+q+".gguf") {
+		return false
+	}
+	// UD- is unsloth's dynamic-quant modifier, and UD-<q> is a distinct quant
+	// label rather than a publication of <q>.
+	return !strings.HasSuffix(base, "-UD-"+q+".gguf")
+}

.github/ci/apexentries/verify_test.go

@@ -0,0 +1,480 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Verify", func() {
+	write := func(body string) string {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "index.yaml")
+		Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed())
+		return p
+	}
+
+	It("passes a sound index", func() {
+		Expect(Verify(write(`
+- name: parent
+  variants:
+    - model: child
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: child
+  files:
+    - filename: b.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(BeEmpty())
+	})
+
+	It("reports a variant pointing at a missing entry", func() {
+		Expect(Verify(write(`
+- name: parent
+  variants:
+    - model: ghost
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("ghost")))
+	})
+
+	It("reports a variant that itself declares variants", func() {
+		Expect(Verify(write(`
+- name: parent
+  variants:
+    - model: child
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: child
+  variants:
+    - model: grandchild
+  files:
+    - filename: b.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+- name: grandchild
+  files:
+    - filename: c.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+`))).To(ContainElement(ContainSubstring("declares variants of its own")))
+	})
+
+	It("reports duplicate entry names", func() {
+		Expect(Verify(write(`
+- name: dup
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: dup
+  files:
+    - filename: b.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(ContainElement(ContainSubstring("duplicate entry name")))
+	})
+
+	It("reports a file with no sha256", func() {
+		Expect(Verify(write(`
+- name: one
+  files:
+    - filename: a.gguf
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("no sha256")))
+	})
+
+	It("reports an entry tagged dflash without a matching spec_type", func() {
+		Expect(Verify(write(`
+- name: liar
+  tags:
+    - dflash
+  overrides:
+    backend: llama-cpp
+    options:
+      - use_jinja:true
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("tagged dflash")))
+	})
+
+	It("reports an entry configuring spec_type without the tag", func() {
+		Expect(Verify(write(`
+- name: shy
+  overrides:
+    backend: llama-cpp
+    options:
+      - spec_type:draft-mtp
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("not tagged mtp")))
+	})
+
+	// ds4 carries the MTP heads in the weights and names them with mtp_path, so
+	// the rule holds there in a different vocabulary rather than not at all.
+	It("reports a ds4 entry configuring mtp_path without the tag", func() {
+		Expect(Verify(write(`
+- name: ds4-shy
+  overrides:
+    backend: ds4
+    options:
+      - mtp_path:model-mtp.gguf
+      - mtp_draft:2
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("not tagged mtp")))
+	})
+
+	It("reports a ds4 entry tagged mtp that configures no mtp_path", func() {
+		Expect(Verify(write(`
+- name: ds4-liar
+  tags:
+    - mtp
+  overrides:
+    backend: ds4
+    options:
+      - context_size:4096
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("tagged mtp")))
+	})
+
+	It("accepts a ds4 entry that both configures mtp_path and carries the tag", func() {
+		Expect(Verify(write(`
+- name: ds4-honest
+  tags:
+    - mtp
+  overrides:
+    backend: ds4
+    options:
+      - mtp_path:model-mtp.gguf
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(BeEmpty())
+	})
+
+	// sglang declares speculative_algorithm in the referenced gallery/*.yaml,
+	// which Verify never reads, so it may not judge such an entry either way.
+	It("says nothing about an sglang entry tagged mtp", func() {
+		Expect(Verify(write(`
+- name: sglang-mtp
+  tags:
+    - mtp
+  overrides:
+    backend: sglang
+  files: []
+`))).To(BeEmpty())
+	})
+
+	It("says nothing about the tag on an entry with no declared backend", func() {
+		Expect(Verify(write(`
+- name: templated
+  tags:
+    - mtp
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(BeEmpty())
+	})
+
+	// The flat-match branch in unsloth.go appends every match, so a repo
+	// publishing both a plain and a UD Q8_0 renders one entry holding two full
+	// models while model: points at only the first.
+	It("reports an entry holding more than one non-shard weight file", func() {
+		Expect(Verify(write(`
+- name: greedy
+  overrides:
+    backend: llama-cpp
+    options:
+      - use_jinja:true
+    parameters:
+      model: llama-cpp/models/repo/Model-Q8_0.gguf
+  files:
+    - filename: llama-cpp/models/repo/Model-Q8_0.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: llama-cpp/models/repo/Model-UD-Q8_0.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(ContainElement(ContainSubstring("more than one weight file")))
+	})
+
+	It("accepts many shards alongside an mmproj and a drafter", func() {
+		Expect(Verify(write(`
+- name: sharded
+  tags:
+    - mtp
+  overrides:
+    backend: llama-cpp
+    options:
+      - spec_type:draft-mtp
+    mmproj: llama-cpp/mmproj/repo/mm.gguf
+    draft_model: llama-cpp/models/repo/Model-draft.gguf
+  files:
+    - filename: llama-cpp/models/repo/Model-00001-of-00002.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: llama-cpp/models/repo/Model-00002-of-00002.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+    - filename: llama-cpp/mmproj/repo/mm.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+    - filename: llama-cpp/models/repo/Model-draft.gguf
+      sha256: dd
+      uri: https://example.com/d.gguf
+`))).To(BeEmpty())
+	})
+
+	// Multi-component TTS and ASR engines legitimately ship an encoder, a
+	// tokenizer, a vocoder and so on as one model, so the collision the weight
+	// count catches does not exist for them.
+	It("accepts a multi-component non-llama-cpp entry declaring five weights", func() {
+		Expect(Verify(write(`
+- name: multi
+  overrides:
+    backend: qwen3-tts-cpp
+  files:
+    - filename: talker.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: tokenizer.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+    - filename: vocoder.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+    - filename: encoder.gguf
+      sha256: dd
+      uri: https://example.com/d.gguf
+    - filename: vae.gguf
+      sha256: ee
+      uri: https://example.com/e.gguf
+`))).To(BeEmpty())
+	})
+
+	It("says nothing about the weight count of an entry with no declared backend", func() {
+		Expect(Verify(write(`
+- name: templated-weights
+  files:
+    - filename: model-Q4_K_M.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: model-mmproj-f16.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(BeEmpty())
+	})
+
+	It("says nothing about an auxiliary metadata file carrying no sha256", func() {
+		Expect(Verify(write(`
+- name: aux
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: params.json
+      sha256: ""
+      uri: https://example.com/params.json
+`))).To(BeEmpty())
+	})
+
+	// safetensors weights are downloaded and loaded exactly like GGUF weights,
+	// so an unverified one is the same supply-chain hole.
+	It("reports a safetensors weight carrying no sha256", func() {
+		Expect(Verify(write(`
+- name: vae
+  files:
+    - filename: wan_2.1_vae.safetensors
+      sha256: ""
+      uri: https://example.com/vae.safetensors
+`))).To(ContainElement(ContainSubstring("no sha256")))
+	})
+
+	It("says nothing about a txt or md file carrying no sha256", func() {
+		Expect(Verify(write(`
+- name: docs
+  files:
+    - filename: notes.txt
+      sha256: ""
+      uri: https://example.com/notes.txt
+    - filename: README.md
+      sha256: ""
+      uri: https://example.com/README.md
+`))).To(BeEmpty())
+	})
+})
+
+var _ = Describe("UnaccountedQuants", func() {
+	// A quant published only as root-level shards matches neither branch in
+	// DiscoverUnslothQuants, so without this diagnostic the build would vanish
+	// from a batch run with nothing said about it.
+	It("reports a wanted quant upstream publishes but discovery dropped", func() {
+		files := []GGUFFile{
+			{Name: "Model-UD-Q4_K_M-00001-of-00003.gguf", SHA256: "aa"},
+			{Name: "Model-UD-Q4_K_M-00002-of-00003.gguf", SHA256: "bb"},
+			{Name: "Model-UD-Q4_K_M-00003-of-00003.gguf", SHA256: "cc"},
+		}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).
+			To(ContainElement(ContainSubstring("UD-Q4_K_M")))
+	})
+
+	It("says nothing when every published wanted quant produced a build", func() {
+		files := []GGUFFile{
+			{Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"},
+			{Name: "UD-Q6_K/Model-UD-Q6_K-00001-of-00002.gguf", SHA256: "bb"},
+			{Name: "UD-Q6_K/Model-UD-Q6_K-00002-of-00002.gguf", SHA256: "cc"},
+		}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
+	})
+
+	It("says nothing about a wanted quant the repo does not publish at all", func() {
+		files := []GGUFFile{{Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"}}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
+	})
+
+	// UD-Q8_0 is its own quant label and is not a wanted one. Reading it as a
+	// publication of Q8_0 is the substring collision this diagnostic exists to
+	// warn about, and subdirectory-sharded UD quants are the normal unsloth
+	// layout for large repos, so the false positive would fire on every batch.
+	It("does not read a subdirectory-sharded UD-Q8_0 as a published Q8_0", func() {
+		files := []GGUFFile{
+			{Name: "UD-Q8_0/Model-UD-Q8_0-00001-of-00002.gguf", SHA256: "aa"},
+			{Name: "UD-Q8_0/Model-UD-Q8_0-00002-of-00002.gguf", SHA256: "bb"},
+		}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
+	})
+
+	// A quant in its own subdirectory but not shard-numbered matches neither
+	// branch of DiscoverUnslothQuants, so it is genuinely published and
+	// genuinely undiscovered.
+	It("reports a wanted quant published in its own subdirectory without shard numbering", func() {
+		files := []GGUFFile{{Name: "Q8_0/Model-Q8_0.gguf", SHA256: "aa"}}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).
+			To(ContainElement(ContainSubstring("quant Q8_0 is published upstream")))
+	})
+
+	// builds is empty on purpose: it isolates the file-to-quant match from
+	// whatever DiscoverUnslothQuants would have made of the same file.
+	It("matches the flat single-file layout", func() {
+		files := []GGUFFile{{Name: "Model-Q8_0.gguf", SHA256: "aa"}}
+
+		Expect(UnaccountedQuants(files, nil)).
+			To(ConsistOf(ContainSubstring("quant Q8_0 is published upstream")))
+	})
+})
+
+var _ = Describe("Verify local path collisions", func() {
+	write := func(body string) string {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "index.yaml")
+		Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed())
+		return p
+	}
+
+	It("reports one local path claimed by two different uris", func() {
+		// The shape that shipped: LiquidAI and unsloth both publish
+		// LFM2.5-8B-A1B-GGUF, so a path built from the bare repo name gives both
+		// entries the same local file under two different checksums.
+		Expect(Verify(write(`
+- name: lfm2.5-8b-a1b
+  files:
+    - filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf
+      sha256: 33ab3b8c
+      uri: https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf
+- name: lfm2.5-8b-a1b-q8-0
+  files:
+    - filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf
+      sha256: ec11666b
+      uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf
+`))).To(ContainElement(SatisfyAll(
+			ContainSubstring("claimed by two different uris"),
+			ContainSubstring("lfm2.5-8b-a1b-q8-0"),
+		)))
+	})
+
+	It("accepts two entries reusing one file from the same uri", func() {
+		// Sibling builds of one repo legitimately share a projector.
+		Expect(Verify(write(`
+- name: a
+  files:
+    - filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf
+      sha256: cc
+      uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf
+- name: b
+  files:
+    - filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf
+      sha256: cc
+      uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf
+`))).To(BeEmpty())
+	})
+
+	It("reports a collision once however many entries pile onto the path", func() {
+		problems := Verify(write(`
+- name: a
+  files:
+    - filename: shared.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: b
+  files:
+    - filename: shared.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+- name: c
+  files:
+    - filename: shared.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+`))
+
+		var collisions int
+		for _, p := range problems {
+			if strings.Contains(p, "claimed by two different uris") {
+				collisions++
+			}
+		}
+		Expect(collisions).To(Equal(1))
+	})
+
+	It("says nothing about files that carry no uri", func() {
+		// A hand-written entry may record only a checksum. There is no upstream
+		// to compare, so the check cannot conclude anything either way.
+		Expect(Verify(write(`
+- name: a
+  files:
+    - filename: shared.gguf
+      sha256: aa
+- name: b
+  files:
+    - filename: shared.gguf
+      sha256: bb
+`))).To(BeEmpty())
+	})
+})

.github/ci/galleryedit/edit.go

@@ -0,0 +1,152 @@
+// Package galleryedit splices variant references into the LocalAI gallery index
+// as TEXT.
+//
+// Re-serialising the index through a YAML marshaller would reflow 40,000 lines,
+// drop the anchors and merge keys the gallery relies on, and produce a diff no
+// reviewer could read, which makes a pull request worthless even when the
+// content inside it is right. Every generator that adds variants to an entry the
+// gallery already ships therefore edits lines, and they share this package so
+// that two of them cannot drift apart on where a variants block belongs.
+package galleryedit
+
+import (
+	"fmt"
+	"regexp"
+	"sort"
+	"strings"
+)
+
+var (
+	entryStart   = regexp.MustCompile(`^-(?: |$)`)
+	inlineName   = regexp.MustCompile(`^- (?:&\S+ )?name:`)
+	keyName      = regexp.MustCompile(`^  name:`)
+	keyVariants  = regexp.MustCompile(`^  variants:\s*(.*)$`)
+	variantItem  = regexp.MustCompile(`^    - `)
+	unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`)
+)
+
+// Entry is the positional view of one gallery entry: what it is called and
+// which lines it occupies. Nothing about what the entry MEANS belongs here, so
+// each caller keeps its own semantic decode and only hands over the coordinates.
+type Entry struct {
+	Name string
+	// StartLine and EndLine bound the entry, zero based and half open.
+	StartLine int
+	EndLine   int
+}
+
+// Insert is one entry's pending variants addition. The caller owns the contents
+// of Variants: this package neither orders nor deduplicates them, because the
+// right order and the right dedup rule differ between generators.
+type Insert struct {
+	Entry    Entry
+	Variants []string
+}
+
+// Scan splits index text into lines and reports the line each top level list
+// item begins on.
+func Scan(text string) (lines []string, starts []int) {
+	lines = strings.Split(text, "\n")
+	for i, line := range lines {
+		if entryStart.MatchString(line) {
+			starts = append(starts, i)
+		}
+	}
+	return lines, starts
+}
+
+// Apply splices every insert into the index lines and returns the new text.
+func Apply(lines []string, inserts []Insert) ([]string, error) {
+	type edit struct {
+		at     int
+		remove int
+		insert []string
+	}
+	var edits []edit
+
+	for _, in := range inserts {
+		if len(in.Variants) == 0 {
+			continue
+		}
+
+		items := make([]string, 0, len(in.Variants))
+		for _, v := range in.Variants {
+			items = append(items, "    - model: "+QuoteName(v))
+		}
+
+		at, remove, err := insertionPoint(lines, in.Entry)
+		if err != nil {
+			return nil, err
+		}
+		block := items
+		if remove > 0 || !hasVariantsKey(lines, in.Entry) {
+			block = append([]string{"  variants:"}, items...)
+		}
+		edits = append(edits, edit{at: at, remove: remove, insert: block})
+	}
+
+	// Applying from the bottom up keeps every line number computed against the
+	// original text valid while earlier edits are still pending.
+	sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at })
+
+	out := append([]string(nil), lines...)
+	for _, e := range edits {
+		tail := append([]string(nil), out[e.at+e.remove:]...)
+		out = append(out[:e.at], append(append([]string(nil), e.insert...), tail...)...)
+	}
+	return out, nil
+}
+
+func hasVariantsKey(lines []string, e Entry) bool {
+	for i := e.StartLine; i < e.EndLine; i++ {
+		if keyVariants.MatchString(lines[i]) {
+			return true
+		}
+	}
+	return false
+}
+
+// insertionPoint reports where new variant items belong, and how many existing
+// lines the insertion replaces.
+//
+// An entry with no variants key gets one right after its name, which is where
+// the hand-written families put it. An entry with an empty "variants: []" has
+// that line replaced by a block. An entry with a block gets its items appended.
+func insertionPoint(lines []string, e Entry) (at int, remove int, err error) {
+	for i := e.StartLine; i < e.EndLine; i++ {
+		m := keyVariants.FindStringSubmatch(lines[i])
+		if m == nil {
+			continue
+		}
+		if strings.TrimSpace(m[1]) == "[]" {
+			return i, 1, nil
+		}
+		if strings.TrimSpace(m[1]) != "" {
+			return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1]))
+		}
+		last := i
+		for j := i + 1; j < e.EndLine && variantItem.MatchString(lines[j]); j++ {
+			last = j
+		}
+		return last + 1, 0, nil
+	}
+
+	if inlineName.MatchString(lines[e.StartLine]) {
+		return e.StartLine + 1, 0, nil
+	}
+	for i := e.StartLine; i < e.EndLine; i++ {
+		if keyName.MatchString(lines[i]) {
+			return i + 1, 0, nil
+		}
+	}
+	return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name)
+}
+
+// QuoteName quotes a variant reference when the name would otherwise change
+// meaning as bare YAML. Config-suffixed names carry a ":" and always need it.
+func QuoteName(name string) string {
+	if unsafeInName.MatchString(name) {
+		return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"`
+	}
+	return name
+}

.github/ci/variantproposals/edit.go

@@ -2,119 +2,41 @@ package main
 
 import (
 	"fmt"
-	"regexp"
-	"sort"
 	"strings"
-)
 
-var (
-	inlineName   = regexp.MustCompile(`^- (?:&\S+ )?name:`)
-	keyName      = regexp.MustCompile(`^  name:`)
-	keyVariants  = regexp.MustCompile(`^  variants:\s*(.*)$`)
-	variantItem  = regexp.MustCompile(`^    - `)
-	unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`)
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
 )
 
 // ApplyFamilies writes the proposed variant lists into the index text.
 //
-// The edit is textual on purpose. Re-serialising the index through a YAML
-// marshaller would reflow 40,000 lines, drop the anchors and merge keys the
-// gallery relies on, and produce a diff no reviewer could read, which would
-// make the pull request worthless even when the proposals inside it are right.
+// The line editing itself lives in galleryedit, shared with the apexentries
+// generator. Both jobs add variants to entries the gallery already ships, and a
+// second answer to "where does a variants block go" would drift from this one;
+// see that package for why the edit is textual rather than a YAML round trip.
 func ApplyFamilies(ix *Index, families []Family) ([]string, error) {
 	byName, _ := ix.ByName()
 
-	type edit struct {
-		at      int
-		remove  int
-		insert  []string
-		ordinal int
-	}
-	var edits []edit
-
+	var inserts []galleryedit.Insert
 	for _, f := range families {
 		entry, ok := byName[strings.ToLower(f.Parent)]
 		if !ok {
 			return nil, fmt.Errorf("parent %q is not in the index", f.Parent)
 		}
-		items := make([]string, 0, len(f.Proposals))
+
+		variants := make([]string, 0, len(f.Proposals))
 		for _, p := range f.Proposals {
-			items = append(items, "    - model: "+quoteName(p.Variant))
+			variants = append(variants, p.Variant)
 		}
 
-		at, remove, err := insertionPoint(ix, entry)
-		if err != nil {
-			return nil, err
-		}
-		insert := items
-		if remove > 0 || !hasVariantsKey(ix, entry) {
-			insert = append([]string{"  variants:"}, items...)
-		}
-		edits = append(edits, edit{at: at, remove: remove, insert: insert, ordinal: entry.Index})
+		inserts = append(inserts, galleryedit.Insert{
+			Entry: galleryedit.Entry{
+				Name:      entry.Name,
+				StartLine: entry.StartLine,
+				EndLine:   entry.EndLine,
+			},
+			Variants: variants,
+		})
 	}
 
-	// Applying from the bottom up keeps every line number computed against the
-	// original text valid while earlier edits are still pending.
-	sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at })
-
-	lines := append([]string(nil), ix.Lines...)
-	for _, e := range edits {
-		tail := append([]string(nil), lines[e.at+e.remove:]...)
-		lines = append(lines[:e.at], append(append([]string(nil), e.insert...), tail...)...)
-	}
-	return lines, nil
-}
-
-func hasVariantsKey(ix *Index, e *GalleryEntry) bool {
-	for i := e.StartLine; i < e.EndLine; i++ {
-		if keyVariants.MatchString(ix.Lines[i]) {
-			return true
-		}
-	}
-	return false
-}
-
-// insertionPoint reports where new variant items belong, and how many existing
-// lines the insertion replaces.
-//
-// An entry with no variants key gets one right after its name, which is where
-// the hand-written families put it. An entry with an empty "variants: []" has
-// that line replaced by a block. An entry with a block gets its items appended.
-func insertionPoint(ix *Index, e *GalleryEntry) (at int, remove int, err error) {
-	for i := e.StartLine; i < e.EndLine; i++ {
-		m := keyVariants.FindStringSubmatch(ix.Lines[i])
-		if m == nil {
-			continue
-		}
-		if strings.TrimSpace(m[1]) == "[]" {
-			return i, 1, nil
-		}
-		if strings.TrimSpace(m[1]) != "" {
-			return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1]))
-		}
-		last := i
-		for j := i + 1; j < e.EndLine && variantItem.MatchString(ix.Lines[j]); j++ {
-			last = j
-		}
-		return last + 1, 0, nil
-	}
-
-	if inlineName.MatchString(ix.Lines[e.StartLine]) {
-		return e.StartLine + 1, 0, nil
-	}
-	for i := e.StartLine; i < e.EndLine; i++ {
-		if keyName.MatchString(ix.Lines[i]) {
-			return i + 1, 0, nil
-		}
-	}
-	return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name)
-}
-
-// quoteName quotes a variant reference when the name would otherwise change
-// meaning as bare YAML. Config-suffixed names carry a ":" and always need it.
-func quoteName(name string) string {
-	if unsafeInName.MatchString(name) {
-		return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"`
-	}
-	return name
+	return galleryedit.Apply(ix.Lines, inserts)
 }

.github/ci/variantproposals/index.go

@@ -8,6 +8,8 @@ import (
 	"strings"
 
 	"gopkg.in/yaml.v3"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
 )
 
 // File is the subset of a gallery file entry the proposer reads.
@@ -57,7 +59,6 @@ type Index struct {
 }
 
 var (
-	entryStart  = regexp.MustCompile(`^-(?: |$)`)
 	anchorStart = regexp.MustCompile(`^- &(\S+)`)
 	mergeStart  = regexp.MustCompile(`^- !!merge <<: \*(\S+)`)
 )
@@ -82,13 +83,7 @@ func ParseIndex(text string) (*Index, error) {
 		return nil, fmt.Errorf("decoding gallery index: %w", err)
 	}
 
-	lines := strings.Split(text, "\n")
-	var starts []int
-	for i, line := range lines {
-		if entryStart.MatchString(line) {
-			starts = append(starts, i)
-		}
-	}
+	lines, starts := galleryedit.Scan(text)
 	if len(starts) != len(entries) {
 		return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number", len(entries), len(starts))
 	}

.github/gh_curl.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+# Shared curl wrapper for the nightly dependency-bump scripts.
+#
+# The bump workflow fans out to ~25 parallel matrix jobs, each querying
+# api.github.com. Anonymous API calls are capped at 60/hour per source IP and
+# GitHub-hosted runners egress through shared NAT addresses, so a random handful
+# of jobs were getting rate-limited (HTTP 403 -> curl exit 22, empty response)
+# every single night. Authenticating with GITHUB_TOKEN lifts the ceiling to
+# 1000/hour; the retries absorb whatever transient blips remain.
+
+# Wraps curl with GitHub auth (when a token is present) plus retry/timeout
+# hardening. Callers pass their own headers and the URL.
+gh_curl() {
+    # The bump scripts run under `set -x`; without this the Authorization header
+    # would be echoed into the job log on every call.
+    local had_xtrace=0
+    case "$-" in
+        *x*) had_xtrace=1; set +x ;;
+    esac
+
+    local args=(
+        --silent --show-error --location --fail
+        # --retry-all-errors so 403 rate-limit responses are retried too; plain
+        # --retry only covers 408/429/5xx. curl honours Retry-After when sent.
+        --retry 5 --retry-delay 3 --retry-all-errors
+        --connect-timeout 15 --max-time 60
+    )
+    if [ -n "${GITHUB_TOKEN:-}" ]; then
+        args+=(--header "Authorization: Bearer ${GITHUB_TOKEN}")
+    fi
+
+    curl "${args[@]}" "$@"
+    local rc=$?
+
+    if [ "$had_xtrace" -eq 1 ]; then
+        set -x
+    fi
+    return $rc
+}

.github/workflows/build-test.yaml

@@ -6,6 +6,14 @@ on:
       - master
   pull_request:
 
+# Supersede an in-flight run when a PR gets a new push. Keyed on the PR number
+# so every push to the same PR shares a group; on a master push the key falls
+# back to github.sha (unique per commit) and cancel-in-progress is false, so
+# master runs never cancel each other -- each commit is built on its own.
+concurrency:
+  group: ci-build-test-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 jobs:
   build-test:
     runs-on: ubuntu-latest

.github/workflows/bump_deps.yaml

@@ -50,6 +50,10 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
+          - repository: "mudler/vllm.cpp"
+            variable: "VLLM_CPP_VERSION"
+            branch: "main"
+            file: "backend/go/vllm-cpp/Makefile"
           - repository: "localai-org/moss-transcribe.cpp"
             variable: "MOSS_VERSION"
             branch: "master"
@@ -110,11 +114,21 @@ jobs:
             variable: "VIBEVOICE_CPP_VERSION"
             branch: "master"
             file: "backend/go/vibevoice-cpp/Makefile"
+          - repository: "mudler/magpie-tts.cpp"
+            variable: "MAGPIETTS_CPP_VERSION"
+            branch: "main"
+            file: "backend/go/magpie-tts-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7
       - name: Bump dependencies 🔧
         id: bump
+        env:
+          # This job fans out to ~25 parallel matrix entries, all querying
+          # api.github.com from runner IPs that share the 60/hour anonymous
+          # rate limit. Authenticating raises it to 1000/hour, which is what
+          # kept a random handful of these red every night.
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_deps.sh ${{ matrix.repository }} ${{ matrix.branch }} ${{ matrix.variable }} ${{ matrix.file }}
           {
@@ -151,6 +165,8 @@ jobs:
       - uses: actions/checkout@v7
       - name: Bump vLLM cu130 wheel pin 🔧
         id: bump
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_vllm_wheel.sh vllm-project/vllm backend/python/vllm/requirements-cublas13-after.txt VLLM_VERSION
           {
@@ -187,6 +203,8 @@ jobs:
       - uses: actions/checkout@v7
       - name: Bump vllm-metal pin 🔧
         id: bump
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_vllm_metal.sh vllm-project/vllm-metal backend/python/vllm/install.sh VLLM_METAL_VERSION
           {

.github/workflows/bump_docs.yaml

@@ -15,6 +15,10 @@ jobs:
     steps:
       - uses: actions/checkout@v7
       - name: Bump dependencies 🔧
+        env:
+          # Authenticated API calls get 1000 req/hour instead of the 60/hour
+          # anonymous cap that is shared across every job on the runner's IP.
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_docs.sh ${{ matrix.repository }}
       - name: Create Pull Request

.github/workflows/ci-tools-tests.yaml

@@ -0,0 +1,39 @@
+---
+# The packages under .github/ci/ are invisible to `go list ./...`, so neither
+# `make lint` nor the repository test run ever touches them. Their specs are
+# dead weight until a workflow names each package explicitly.
+name: 'CI tool tests'
+on:
+  pull_request:
+    paths:
+      - '.github/ci/**'
+      - '.github/workflows/ci-tools-tests.yaml'
+  push:
+    branches:
+      - master
+    paths:
+      - '.github/ci/**'
+jobs:
+  ci-tools:
+    name: 'Test the .github/ci generators'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      # The discovery heuristics are the risky part of these tools. A regression
+      # produces confident, wrong gallery entries, which is worse than no tool.
+      - name: 'Test the APEX entry generator'
+        run: go test ./.github/ci/apexentries/
+
+      - name: 'Test the variant proposer'
+        run: go test ./.github/ci/variantproposals/
+
+      # Shared by both generators above. Its behaviour is exercised through their
+      # specs; this step exists so a break in the shared package fails under its
+      # own name rather than as a puzzling failure in whichever caller ran first.
+      - name: 'Test the shared gallery editor'
+        run: go test ./.github/ci/galleryedit/

.github/workflows/image-pr.yml

@@ -52,7 +52,7 @@
               tag-latest: 'false'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:22.04"
+              base-image: "ubuntu:24.04"
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'hipblas'

.github/workflows/image.yml

@@ -113,7 +113,7 @@
               tag-latest: 'auto'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:22.04"
+              base-image: "ubuntu:24.04"
               skip-drivers: 'false'
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'

.github/workflows/lint.yml

@@ -61,7 +61,7 @@ jobs:
       # The backend matrix path filter fails silently: a miss emits an empty
       # matrix, every job goes green, and the change reaches no image (#10946).
       # Its tests need only node, so they ride along with this job.
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v7
         with:
           node-version: '20'
       - name: run CI script tests

.github/workflows/secscan.yaml

@@ -7,6 +7,19 @@ on:
   schedule:
   - cron: '0 0 * * 0'
 
+# `push:` is deliberately unfiltered, so this fires on every push to every
+# branch and there is no pull_request event to key on -- the usual
+# `github.event.pull_request.number || github.sha` idiom used elsewhere would
+# key on the unique-per-commit sha and dedup nothing. Group on the ref instead
+# so successive pushes to the same feature branch supersede one another.
+#
+# Cancelling is safe here: the only output is a SARIF upload, and code scanning
+# tracks the latest result per ref, so a superseded scan has nothing to lose.
+# master is excluded anyway -- every commit on master gets its own scan.
+concurrency:
+  group: ci-secscan-${{ github.ref }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
+
 jobs:
   tests:
     runs-on: ubuntu-latest

.github/workflows/test-extra.yml

@@ -37,6 +37,7 @@ jobs:
       sglang: ${{ steps.detect.outputs.sglang }}
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
       qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
+      magpie-tts-cpp: ${{ steps.detect.outputs.magpie-tts-cpp }}
       rfdetr-cpp: ${{ steps.detect.outputs.rfdetr-cpp }}
       locate-anything-cpp: ${{ steps.detect.outputs.locate-anything-cpp }}
       vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
@@ -866,6 +867,38 @@ jobs:
       - name: Test qwen3-tts-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/qwen3-tts-cpp test
+  tests-magpie-tts-cpp:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.magpie-tts-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
+      - name: Setup Go
+        uses: actions/setup-go@v5
+      - name: Display Go version
+        run: go version
+      - name: Proto Dependencies
+        run: |
+          # Install protoc
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Build magpie-tts-cpp
+        run: |
+          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp
+      - name: Test magpie-tts-cpp
+        run: |
+          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp test
   # Per-backend smoke for rfdetr-cpp: builds the .so + Go binary and runs
   # `make -C backend/go/rfdetr-cpp test`. test.sh fetches the small (~20 MB)
   # rfdetr-nano-q8_0 GGUF from the published mudler/rfdetr-cpp-nano HF repo

.github/workflows/yaml-check.yml

@@ -1,6 +1,11 @@
 name: 'Yamllint GitHub Actions'
 on:
   - pull_request
+
+concurrency:
+  group: ci-yamllint-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 jobs:
   yamllint:
     name: 'Yamllint'

.gitignore

@@ -111,3 +111,10 @@ core/http/react-ui/test-results/
 # the realtime-conformance gate; only the .fizz sources are authoritative.
 formal-verification/*.json
 formal-verification/out/
+
+# `go build ./.github/ci/apexentries` drops a binary of the package name into
+# whatever directory it runs in, one `git add -A` away from being committed.
+# Both paths are anchored: an unanchored `apexentries` would also match the
+# package directory itself and untrack the source.
+/apexentries
+/.github/ci/apexentries/apexentries

AGENTS.md

@@ -35,7 +35,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 
 ## Quick Reference
 
-- **Git hooks & coverage gates**: Run `make install-hooks` once per clone so the pre-commit lint + coverage gates run. **Never bypass them with `git commit --no-verify`, and never lower a coverage baseline or widen a gate's tolerance to turn a red gate green** — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
+- **Coverage gates**: Never lower a coverage baseline or widen a gate's tolerance to turn a red gate green — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
 - **Logging**: Use `github.com/mudler/xlog` (same API as slog)
 - **Go style**: Prefer `any` over `interface{}`
 - **Comments**: Explain *why*, not *what*

CONTRIBUTING.md

@@ -198,7 +198,6 @@ For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`C
 
 - Prefer modern Go idioms — for example, use `any` instead of `interface{}`.
 - Use [`golangci-lint`](https://golangci-lint.run) to catch common issues before submitting a PR.
-- Run `make install-hooks` once per clone to enable the pre-commit hook: Go changes run `make lint` + the coverage gate (`make test-coverage-check`); `core/http/react-ui/` changes run the Playwright e2e suite (`make test-ui`). Bypass a single commit with `git commit --no-verify`.
 - Use [`github.com/mudler/xlog`](https://github.com/mudler/xlog) for logging (same API as `slog`). Do not use `fmt.Println` or the standard `log` package for operational logging.
 - Use tab indentation for Go files (as defined in `.editorconfig`).
 
@@ -268,7 +267,7 @@ make test-e2e
 
 ### React UI tests and coverage
 
-The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI and pre-commit). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
+The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
 
 **If your change lowers UI coverage, raise it back by adding specs — do not widen the tolerance or hand-lower the baseline.** A *render-smoke* spec (navigate to a page, assert its header is visible) cheaply covers an entire lazy page. See `core/http/react-ui/e2e/page-render-smoke.spec.js` and the full policy in [.agents/building-and-testing.md](.agents/building-and-testing.md#react-ui-coverage).
 

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -103,7 +103,7 @@ COVERAGE_E2E_LABELS?=!real-models
 COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go
 
 
-.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check install-hooks build vendor lint lint-all
+.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
 
 all: help
 
@@ -269,8 +269,7 @@ LINT_EXCLUDE_DIRS_RE=/(backend/go/(piper|silero-vad|llm)|cmd/launcher)(/|$$)
 
 ## Set LINT_NEW_FROM to a git ref to override .golangci.yml's
 ## new-from-merge-base (origin/master). Useful from a fork clone where
-## origin/master is stale relative to the canonical repo — the pre-commit
-## hook passes the resolved upstream ref here so local lint matches CI.
+## origin/master is stale relative to the canonical repo.
 LINT_NEW_FROM?=
 lint:
 	@command -v golangci-lint >/dev/null 2>&1 || { \
@@ -289,17 +288,6 @@ lint-all:
 	}
 	golangci-lint run --new=false --new-from-merge-base= --new-from-rev= $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
 
-########################################################
-## Git hooks
-########################################################
-## Points git at the versioned .githooks/ directory so the pre-commit hook
-## (lint + coverage gate) runs locally. Run once per clone. Undo with:
-## `git config --unset core.hooksPath`. Skip a single commit with
-## `git commit --no-verify`.
-install-hooks:
-	git config core.hooksPath .githooks
-	@echo 'Installed git hooks: core.hooksPath -> .githooks (pre-commit runs lint + test-coverage-check on Go changes)'
-
 ########################################################
 ## E2E AIO tests (uses standard image with pre-configured models)
 ########################################################
@@ -637,6 +625,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/go/locate-anything-cpp test
 	$(MAKE) -C backend/go/depth-anything-cpp test
 	$(MAKE) -C backend/go/supertonic test
+	$(MAKE) -C backend/go/vllm-cpp test
 
 ##
 ## End-to-end gRPC tests that exercise a built backend container image.
@@ -1269,6 +1258,8 @@ BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
 BACKEND_MOSS_TTS_CPP = moss-tts-cpp|golang|.|false|true
+BACKEND_MAGPIE_TTS_CPP = magpie-tts-cpp|golang|.|false|true
+BACKEND_VLLM_CPP = vllm-cpp|golang|.|false|true
 BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
 BACKEND_LOCALVQE = localvqe|golang|.|false|true
@@ -1396,6 +1387,8 @@ $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TTS_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_MAGPIE_TTS_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
@@ -1415,7 +1408,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1450,7 +1443,7 @@ test-ui-e2e: build-ui-test-server
 UI_TEST_WORKERS ?=
 PLAYWRIGHT_WORKERS_FLAG = $(if $(UI_TEST_WORKERS),--workers=$(UI_TEST_WORKERS),)
 
-## Fast Playwright e2e run used by the pre-commit hook on React UI changes.
+## Fast Playwright e2e run for local React UI validation.
 ## Force-rebuilds the (non-instrumented) dist so the suite tests the working
 ## tree — not a stale dist the `react-ui` skip-guard would leave — re-embeds
 ## it into ui-test-server, and runs the specs. Uses the nix-provided browser

README.md

@@ -231,9 +231,11 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 
 | Backend | What it does |
 |---------|-------------|
+| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
 | [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
 | [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
+| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
 | [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
 | [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
 | [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |

backend/Dockerfile.golang

@@ -221,6 +221,33 @@ RUN if [ "${BACKEND}" = "crispasr" ]; then \
     apt-get clean && rm -rf /var/lib/apt/lists/*; \
 fi
 
+# sherpa-onnx links onnxruntime's CUDA execution provider, and
+# libonnxruntime_providers_cuda.so has cuDNN as a hard DT_NEEDED. The
+# onnxruntime GPU tarball does not ship cuDNN itself, so without this the
+# builder has none (the arm64 + CUDA 13 branch above is the only other place
+# that installs it) and package-gpu-libs.sh correctly refuses to produce a
+# package that references cuDNN with no cuDNN available to it.
+#
+# Installed per-backend rather than for every cublas build: the auto-detection
+# in package-gpu-libs.sh bundles only what a package actually references, so
+# the ggml backends would not grow either way, but they would all pay ~1.1 GB
+# of builder layer and registry cache for a library they never call.
+#
+# Runtime package only, no -dev: sherpa-onnx consumes onnxruntime's prebuilt
+# CUDA provider and never compiles against cuDNN headers. libcudnn9-cuda-N
+# carries the dispatcher plus all seven dlopen()ed sublibraries, which is what
+# complete_cudnn_family needs to assemble a whole bundle.
+RUN <<EOT bash
+    if [ "${BACKEND}" = "sherpa-onnx" ] && [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libcudnn9-cuda-${CUDA_MAJOR_VERSION} && \
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
 COPY . /LocalAI
 
 RUN git config --global --add safe.directory /LocalAI

backend/cpp/bonsai/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-BONSAI_VERSION?=9fcaed763ccda38ea81068ad9d7f991aaddca451
+BONSAI_VERSION?=7529fdaaf99ffdc5ca71ace9c7409a56b27ad92f
 LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/ds4/CMakeLists.txt

@@ -76,12 +76,13 @@ elseif(DS4_GPU STREQUAL "cpu")
     set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
 endif()
 
-# ds4.c now references ds4_distributed.c (distributed inference) and ds4_ssd.c
-# (SSD expert-cache), each split into its own translation unit upstream. Both
-# are GPU-agnostic objects shared by every GPU mode, so link them in regardless
-# of DS4_GPU.
+# Upstream splits distributed inference, tensor-parallel transport, the SSD
+# expert cache, and layer placement into GPU-agnostic translation units. Link
+# them regardless of DS4_GPU.
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_distributed.o")
+list(APPEND DS4_OBJS "${DS4_DIR}/ds4_tp.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_ssd.o")
+list(APPEND DS4_OBJS "${DS4_DIR}/ds4_layer_pack.o")
 
 add_executable(${TARGET}
     grpc-server.cpp

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=80ebbc396aee40eedc1d829222f3362d10fa4c6c
+# Upstream pin lives below as DS4_VERSION?=0a7ad776b9068348e6cb09df8cafa9cadd285298
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=80ebbc396aee40eedc1d829222f3362d10fa4c6c
+DS4_VERSION?=0a7ad776b9068348e6cb09df8cafa9cadd285298
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -18,20 +18,19 @@ UNAME_S := $(shell uname -s)
 
 CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
 
-# ds4_distributed.o and ds4_ssd.o are GPU-agnostic translation units that
-# ds4.c/ds4_cpu.o now reference (upstream split distributed inference and the
-# SSD expert-cache into their own .c files). Both objects are shared by every
-# GPU mode, so they are appended unconditionally below.
+# Upstream splits distributed inference, tensor-parallel transport, the SSD
+# expert cache, and layer placement into GPU-agnostic translation units. They
+# are shared by every GPU mode, so append them unconditionally below.
 ifeq ($(BUILD_TYPE),cublas)
     CMAKE_ARGS += -DDS4_GPU=cuda
-    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_ssd.o
+    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else ifeq ($(UNAME_S),Darwin)
     CMAKE_ARGS += -DDS4_GPU=metal
-    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_ssd.o
+    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
     # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
     CMAKE_ARGS += -DDS4_GPU=cpu
-    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_ssd.o
+    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 ifneq ($(NATIVE),true)
@@ -56,11 +55,11 @@ ds4:
 # the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
 ds4/ds4.o: ds4
 ifeq ($(BUILD_TYPE),cublas)
-	+$(MAKE) -C ds4 ds4.o ds4_cuda.o ds4_distributed.o ds4_ssd.o
+	+$(MAKE) -C ds4 ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else ifeq ($(UNAME_S),Darwin)
-	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_ssd.o
+	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
-	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_ssd.o
+	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 grpc-server: ds4/ds4.o

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=9d07d8681ece159a89fb4e16a1f9c9f3a5fac20f
+IK_LLAMA_VERSION?=0a4e10c7fb65d2dd5a4afb78339c7d373a8cdfaa
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=571d0d540df04f25298d0e159e520d9fc62ed121
+LLAMA_VERSION?=0d47ea7427463093e69128bf2c2f9cd06b3ee5b3
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/grpc-server.cpp

@@ -52,6 +52,7 @@
 #include "common.h"
 #include "arg.h"
 #include "chat-auto-parser.h"
+#include "llama_compat.h" // fork-skew switches, generated by prepare.sh
 #include "message_content.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
@@ -613,6 +614,13 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
     // starts with '-'. Applied once after the loop via common_params_parse.
     std::vector<std::string> extra_argv;
 
+    // O_DIRECT intent from the `direct_io` option. Upstream folded
+    // use_mmap/use_mlock/use_direct_io into a single common_params::load_mode
+    // enum (ggml-org/llama.cpp#20834), so the three independent LocalAI settings
+    // can only be reduced to one value once all of them have been read, held
+    // here until the mmap/mlock fields arrive further down.
+    bool want_direct_io = false;
+
     auto add_device_options = [&](const std::string & devices) {
         const std::regex regex{ R"([,]+)" };
         std::sregex_token_iterator it{ devices.begin(), devices.end(), regex, -1 };
@@ -868,9 +876,9 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         // --- O_DIRECT model loading (upstream --direct-io) ---
         } else if (!strcmp(optname, "direct_io") || !strcmp(optname, "use_direct_io")) {
             if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
-                params.use_direct_io = true;
+                want_direct_io = true;
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
-                params.use_direct_io = false;
+                want_direct_io = false;
             }
 
         // --- embedding normalization (upstream --embd-normalize) ---
@@ -1278,8 +1286,28 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
      lora_info.ptr = nullptr;
      params.lora_adapters.push_back(std::move(lora_info));
     }
-    params.use_mlock = request->mlock();
-    params.use_mmap = request->mmap();
+    // LocalAI keeps mmap, mlock and direct-I/O as three independent settings,
+    // while upstream now carries a single load mode. Fold them with the
+    // precedence the separate booleans used to give: direct I/O bypasses the
+    // page cache entirely, mlock implies mmap, and everything off is a plain
+    // buffered read. Forks that branched before ggml-org/llama.cpp#20834 still
+    // expose the booleans; prepare.sh probes the checkout and sets
+    // LOCALAI_LEGACY_LOAD_MODE in the generated llama_compat.h accordingly.
+#if LOCALAI_LEGACY_LOAD_MODE
+    params.use_mlock     = request->mlock();
+    params.use_mmap      = request->mmap();
+    params.use_direct_io = want_direct_io;
+#else
+    if (want_direct_io) {
+        params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO;
+    } else if (request->mlock()) {
+        params.load_mode = LLAMA_LOAD_MODE_MLOCK;
+    } else if (request->mmap()) {
+        params.load_mode = LLAMA_LOAD_MODE_MMAP;
+    } else {
+        params.load_mode = LLAMA_LOAD_MODE_NONE;
+    }
+#endif
 
     if (request->flashattention() == "on" || request->flashattention() == "enabled") {
         params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-chat-parser.patch

@@ -0,0 +1,225 @@
+# MiniMax-M3 chat-template parser, vendored from upstream llama.cpp PR #24523.
+#
+# Upstream has since merged the *model* half of #24523 (LLM_ARCH_MINIMAX_M3,
+# src/models/minimax-m3.cpp, the gguf-py constants and conversion/minimax.py), so
+# only the chat half is carried here: M3's namespace token "]<]minimax[>[" collides
+# with the autoparser's markup delimiters, so common/chat.cpp needs a dedicated
+# template detection + PEG parser that upstream does not have yet.
+#
+# Rebased against LLAMA_VERSION 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3, which also
+# renamed common_chat_params::thinking_end_tag to thinking_end_tags (a vector).
+# LLAMA_VERSION is auto-bumped nightly; if a bump rejects this patch, re-vendor from
+# #24523 — or, once the chat half merges upstream, delete this file.
+# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
+diff --git a/common/chat.cpp b/common/chat.cpp
+index 7a6e7238c..2dd015a2e 100644
+--- a/common/chat.cpp
++++ b/common/chat.cpp
+@@ -2121,6 +2121,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
+     return data;
+ }
+ 
++static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
++                                                             const autoparser::generation_params & inputs) {
++    common_chat_params data;
++
++    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
++    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
++    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
++    data.supports_thinking  = true;
++    data.thinking_start_tag = "<mm:think>";
++    data.thinking_end_tags  = {"</mm:think>"};
++
++    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
++    // params use the parameter name as the tag (<file_path>...</file_path>).
++    const std::string NS           = "]<]minimax[>[";
++    const std::string THINK_START  = "<mm:think>";
++    const std::string THINK_END    = "</mm:think>";
++    const std::string FC_START     = NS + "<tool_call>";
++    const std::string FC_END       = NS + "</tool_call>";
++    const std::string INVOKE_END   = NS + "</invoke>";
++
++    data.preserved_tokens   = {
++        NS,
++        "<tool_call>",
++        "</tool_call>",
++        THINK_START,
++        THINK_END,
++    };
++
++    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
++    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
++    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
++    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
++
++    const std::string GEN_PROMPT = data.generation_prompt;
++
++    if (inputs.has_continuation()) {
++        const auto & msg = inputs.continue_msg;
++
++        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
++        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
++            data.generation_prompt += THINK_END + msg.render_content();
++        }
++
++        data.prompt += data.generation_prompt;
++    }
++
++    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
++        auto generation_prompt = p.literal(GEN_PROMPT);
++        auto end = p.end();
++
++        auto reasoning = p.eps();
++        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
++        if (extract_reasoning && inputs.enable_thinking) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
++        } else if (extract_reasoning) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
++        }
++
++        if (has_response_format) {
++            auto response_format = p.rule("response-format",
++                p.literal("```json") + p.space() +
++                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
++                p.space() + p.literal("```"));
++            return generation_prompt + reasoning + response_format + end;
++        }
++
++        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
++            return generation_prompt + reasoning + p.content(p.rest()) + end;
++        }
++
++        auto tool_choice = p.choice();
++        foreach_function(inputs.tools, [&](const json & tool) {
++            const auto & function = tool.at("function");
++            std::string  name     = function.at("name");
++            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
++            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
++
++            std::set<std::string> required;
++            if (params.contains("required")) {
++                params.at("required").get_to(required);
++            }
++
++            auto schema_info = common_schema_info();
++            schema_info.resolve_refs(params);
++
++            std::vector<common_peg_parser> required_parsers;
++            std::vector<common_peg_parser> optional_parsers;
++            for (const auto & [param_name, param_schema] : props.items()) {
++                bool is_required = required.find(param_name) != required.end();
++                bool is_string   = schema_info.resolves_to_string(param_schema);
++
++                const std::string p_close = NS + "</" + param_name + ">";
++
++                auto arg = p.tool_arg(
++                    p.tool_arg_open(
++                        p.literal(NS + "<") +
++                        p.tool_arg_name(p.literal(param_name)) +
++                        p.literal(">")) +
++                    (is_string
++                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
++                                p.tool_arg_close(p.literal(p_close)), p_close)
++                         : p.tool_arg_json_value(p.schema(p.json(),
++                                                          "tool-" + name + "-arg-" + param_name + "-schema",
++                                                          param_schema, false)) +
++                           p.tool_arg_close(p.literal(p_close))));
++
++                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
++                if (is_required) {
++                    required_parsers.push_back(named_arg);
++                } else {
++                    optional_parsers.push_back(named_arg);
++                }
++            }
++
++            common_peg_parser args_seq = p.eps();
++            for (size_t i = 0; i < required_parsers.size(); i++) {
++                if (i > 0) {
++                    args_seq = args_seq + p.space();
++                }
++                args_seq = args_seq + required_parsers[i];
++            }
++
++            if (!optional_parsers.empty()) {
++                common_peg_parser any_opt = p.choice();
++                for (const auto & opt : optional_parsers) {
++                    any_opt |= opt;
++                }
++                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
++            }
++
++            common_peg_parser invoke_body = args_seq;
++            auto func_parser = p.tool(
++                p.tool_open(p.literal(NS + "<invoke name=\"") +
++                            p.tool_name(p.literal(name)) + p.literal("\">")) +
++                p.space() + invoke_body + p.space() +
++                p.tool_close(p.literal(INVOKE_END)));
++
++            tool_choice |= p.rule("tool-" + name, func_parser);
++        });
++
++        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
++
++        common_peg_parser tool_calls = p.eps();
++        if (inputs.parallel_tool_calls) {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice +
++                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
++        } else {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
++        }
++
++        if (!require_tools) {
++            tool_calls = p.optional(tool_calls);
++        }
++
++        auto content_before_tools = p.content(p.until(FC_START));
++        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
++    });
++
++    data.parser = parser.save();
++
++    if (include_grammar) {
++        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
++        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
++            foreach_function(inputs.tools, [&](const json & tool) {
++                const auto & function = tool.at("function");
++                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
++                builder.resolve_refs(schema);
++            });
++            if (has_response_format) {
++                auto schema = inputs.json_schema;
++                builder.resolve_refs(schema);
++            }
++            parser.build_grammar(builder, data.grammar_lazy);
++        });
++
++        data.grammar_triggers = {
++            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
++        };
++    }
++
++    return data;
++}
++
+ // Cohere2 MoE (a.k.a. "North Code") parser.
+ //
+ // The assistant turn is fully marker-wrapped:
+@@ -2707,6 +2892,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
+         return common_chat_params_init_gigachat_v3(tmpl, params);
+     }
+ 
++    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
++    // markup delimiters, so detect the template and use a dedicated parser.
++    if (src.find("]<]minimax[>[") != std::string::npos &&
++        src.find("<tool_call>") != std::string::npos &&
++        src.find("<invoke name=") != std::string::npos) {
++        LOG_DBG("Using specialized template: MiniMax-M3\n");
++        return common_chat_params_init_minimax_m3(tmpl, params);
++    }
++
+     // DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
+     // The template source contains the token as a variable assignment, not as a literal in markup.
+     // V3.2 names the tool call block "function_calls", V4 names it "tool_calls".

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch

@@ -1,814 +0,0 @@
-# Vendored from upstream llama.cpp PR #24523 (Preliminary MiniMax-M3 support).
-# Rebased against LLAMA_VERSION 00fa7cb284cbf133fc426733bd64238a3588a33e (also applies cleanly
-# to the later pin 505b1ed15ca80e2a19f12ff4ac365e40fb374053). LLAMA_VERSION is auto-bumped
-# nightly; if a bump rejects this patch, re-vendor from #24523 — or, once #24523 merges
-# upstream, delete this file and bump LLAMA_VERSION normally.
-# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
-diff --git a/common/chat.cpp b/common/chat.cpp
-index 22d2ee4..440be9a 100644
---- a/common/chat.cpp
-+++ b/common/chat.cpp
-@@ -2035,6 +2035,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
-     return data;
- }
- 
-+static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
-+                                                             const autoparser::generation_params & inputs) {
-+    common_chat_params data;
-+
-+    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
-+    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
-+    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
-+    data.supports_thinking  = true;
-+    data.thinking_start_tag = "<mm:think>";
-+    data.thinking_end_tag   = "</mm:think>";
-+
-+    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
-+    // params use the parameter name as the tag (<file_path>...</file_path>).
-+    const std::string NS           = "]<]minimax[>[";
-+    const std::string THINK_START  = "<mm:think>";
-+    const std::string THINK_END    = "</mm:think>";
-+    const std::string FC_START     = NS + "<tool_call>";
-+    const std::string FC_END       = NS + "</tool_call>";
-+    const std::string INVOKE_END   = NS + "</invoke>";
-+
-+    data.preserved_tokens   = {
-+        NS,
-+        "<tool_call>",
-+        "</tool_call>",
-+        THINK_START,
-+        THINK_END,
-+    };
-+
-+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
-+    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
-+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
-+
-+    const std::string GEN_PROMPT = data.generation_prompt;
-+
-+    if (inputs.has_continuation()) {
-+        const auto & msg = inputs.continue_msg;
-+
-+        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
-+        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
-+            data.generation_prompt += THINK_END + msg.render_content();
-+        }
-+
-+        data.prompt += data.generation_prompt;
-+    }
-+
-+    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
-+        auto generation_prompt = p.literal(GEN_PROMPT);
-+        auto end = p.end();
-+
-+        auto reasoning = p.eps();
-+        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
-+        if (extract_reasoning && inputs.enable_thinking) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
-+        } else if (extract_reasoning) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
-+        }
-+
-+        if (has_response_format) {
-+            auto response_format = p.rule("response-format",
-+                p.literal("```json") + p.space() +
-+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
-+                p.space() + p.literal("```"));
-+            return generation_prompt + reasoning + response_format + end;
-+        }
-+
-+        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
-+            return generation_prompt + reasoning + p.content(p.rest()) + end;
-+        }
-+
-+        auto tool_choice = p.choice();
-+        foreach_function(inputs.tools, [&](const json & tool) {
-+            const auto & function = tool.at("function");
-+            std::string  name     = function.at("name");
-+            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
-+            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
-+
-+            std::set<std::string> required;
-+            if (params.contains("required")) {
-+                params.at("required").get_to(required);
-+            }
-+
-+            auto schema_info = common_schema_info();
-+            schema_info.resolve_refs(params);
-+
-+            std::vector<common_peg_parser> required_parsers;
-+            std::vector<common_peg_parser> optional_parsers;
-+            for (const auto & [param_name, param_schema] : props.items()) {
-+                bool is_required = required.find(param_name) != required.end();
-+                bool is_string   = schema_info.resolves_to_string(param_schema);
-+
-+                const std::string p_close = NS + "</" + param_name + ">";
-+
-+                auto arg = p.tool_arg(
-+                    p.tool_arg_open(
-+                        p.literal(NS + "<") +
-+                        p.tool_arg_name(p.literal(param_name)) +
-+                        p.literal(">")) +
-+                    (is_string
-+                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
-+                                p.tool_arg_close(p.literal(p_close)), p_close)
-+                         : p.tool_arg_json_value(p.schema(p.json(),
-+                                                          "tool-" + name + "-arg-" + param_name + "-schema",
-+                                                          param_schema, false)) +
-+                           p.tool_arg_close(p.literal(p_close))));
-+
-+                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
-+                if (is_required) {
-+                    required_parsers.push_back(named_arg);
-+                } else {
-+                    optional_parsers.push_back(named_arg);
-+                }
-+            }
-+
-+            common_peg_parser args_seq = p.eps();
-+            for (size_t i = 0; i < required_parsers.size(); i++) {
-+                if (i > 0) {
-+                    args_seq = args_seq + p.space();
-+                }
-+                args_seq = args_seq + required_parsers[i];
-+            }
-+
-+            if (!optional_parsers.empty()) {
-+                common_peg_parser any_opt = p.choice();
-+                for (const auto & opt : optional_parsers) {
-+                    any_opt |= opt;
-+                }
-+                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
-+            }
-+
-+            common_peg_parser invoke_body = args_seq;
-+            auto func_parser = p.tool(
-+                p.tool_open(p.literal(NS + "<invoke name=\"") +
-+                            p.tool_name(p.literal(name)) + p.literal("\">")) +
-+                p.space() + invoke_body + p.space() +
-+                p.tool_close(p.literal(INVOKE_END)));
-+
-+            tool_choice |= p.rule("tool-" + name, func_parser);
-+        });
-+
-+        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
-+
-+        common_peg_parser tool_calls = p.eps();
-+        if (inputs.parallel_tool_calls) {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice +
-+                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
-+        } else {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
-+        }
-+
-+        if (!require_tools) {
-+            tool_calls = p.optional(tool_calls);
-+        }
-+
-+        auto content_before_tools = p.content(p.until(FC_START));
-+        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
-+    });
-+
-+    data.parser = parser.save();
-+
-+    if (include_grammar) {
-+        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
-+        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
-+            foreach_function(inputs.tools, [&](const json & tool) {
-+                const auto & function = tool.at("function");
-+                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
-+                builder.resolve_refs(schema);
-+            });
-+            if (has_response_format) {
-+                auto schema = inputs.json_schema;
-+                builder.resolve_refs(schema);
-+            }
-+            parser.build_grammar(builder, data.grammar_lazy);
-+        });
-+
-+        data.grammar_triggers = {
-+            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
-+        };
-+    }
-+
-+    return data;
-+}
-+
- // Cohere2 MoE (a.k.a. "North Code") parser.
- //
- // The assistant turn is fully marker-wrapped:
-@@ -2612,6 +2797,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
-         return common_chat_params_init_gigachat_v3(tmpl, params);
-     }
- 
-+    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
-+    // markup delimiters, so detect the template and use a dedicated parser.
-+    if (src.find("]<]minimax[>[") != std::string::npos &&
-+        src.find("<tool_call>") != std::string::npos &&
-+        src.find("<invoke name=") != std::string::npos) {
-+        LOG_DBG("Using specialized template: MiniMax-M3\n");
-+        return common_chat_params_init_minimax_m3(tmpl, params);
-+    }
-+
-     // DeepSeek V3.2 format detection: template defines dsml_token and uses it for tool calls.
-     // The template source contains the token as a variable assignment, not as a literal in markup.
-     if (src.find("dsml_token") != std::string::npos &&
-diff --git a/conversion/__init__.py b/conversion/__init__.py
-index 02ea638..71de528 100644
---- a/conversion/__init__.py
-+++ b/conversion/__init__.py
-@@ -155,6 +155,8 @@ TEXT_MODEL_MAP: dict[str, str] = {
-     "MiniCPMForCausalLM": "minicpm",
-     "MiniCPMV4_6ForConditionalGeneration": "minicpm",
-     "MiniMaxM2ForCausalLM": "minimax",
-+    "MiniMaxM3SparseForCausalLM": "minimax",
-+    "MiniMaxM3SparseForConditionalGeneration": "minimax",
-     "Ministral3ForCausalLM": "mistral3",
-     "Mistral3ForConditionalGeneration": "mistral3",
-     "MistralForCausalLM": "llama",
-diff --git a/conversion/base.py b/conversion/base.py
-index 0421aa4..224481a 100644
---- a/conversion/base.py
-+++ b/conversion/base.py
-@@ -1154,7 +1154,8 @@ class TextModel(ModelBase):
-                 or "projector." in name or "pre_mm_projector_norm" in name \
-                 or "image_newline" in name or "view_seperator" in name \
-                 or "patch_embed" in name or "patch_embedding" in name \
--                or "patch_merger." in name or "model.connector." in name:
-+                or "patch_merger." in name or "patch_merge_mlp" in name \
-+                or "model.connector." in name:
-             return None
- 
-         return super().filter_tensors(item)
-@@ -1201,7 +1202,7 @@ class TextModel(ModelBase):
-             self.gguf_writer.add_embedding_length(n_embd)
-             logger.info(f"gguf: embedding length = {n_embd}")
- 
--        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-+        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-             self.gguf_writer.add_feed_forward_length(n_ff)
-             logger.info(f"gguf: feed forward length = {n_ff}")
- 
-diff --git a/conversion/minimax.py b/conversion/minimax.py
-index 4857775..4f637f5 100644
---- a/conversion/minimax.py
-+++ b/conversion/minimax.py
-@@ -52,3 +52,67 @@ class MiniMaxM2Model(TextModel):
-             return
- 
-         yield from super().modify_tensors(data_torch, name, bid)
-+
-+
-+@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
-+class MiniMaxM3Model(TextModel):
-+    # Text-only MiniMax-M3: MiniMax-M2 GQA + DeepSeek-V3 shared/leading-dense experts (swigluoai).
-+    model_arch = gguf.MODEL_ARCH.MINIMAXM3
-+    _experts_cache: dict[int, dict[str, Tensor]] = {}
-+
-+    def set_gguf_parameters(self):
-+        # feed_forward_length comes from dense_intermediate_size (base); experts use intermediate_size.
-+        super().set_gguf_parameters()
-+
-+        self.gguf_writer.add_expert_feed_forward_length(self.find_hparam(["intermediate_size"]))
-+        self.gguf_writer.add_rope_dimension_count(self.find_hparam(["rotary_dim"]))
-+        self.gguf_writer.add_expert_shared_count(self.find_hparam(["n_shared_experts"]))
-+        self.gguf_writer.add_expert_weights_scale(self.find_hparam(["routed_scaling_factor"]))
-+        self.gguf_writer.add_expert_weights_norm(True)
-+
-+        # leading dense layers: moe_layer_freq (ints) or mlp_layer_types (Transformers 5.12, strings)
-+        moe_layer_freq = self.find_hparam(["moe_layer_freq", "mlp_layer_types"])
-+        n_dense = 0
-+        for v in moe_layer_freq:
-+            if v == 0 or v == "dense":
-+                n_dense += 1
-+            else:
-+                break
-+        self.gguf_writer.add_leading_dense_block_count(n_dense)
-+
-+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
-+        # index_* (sparse-attn indexer) tensors are preserved but unused; the loader skips them
-+        if name.startswith("language_model."):
-+            name = name[len("language_model."):]
-+
-+        # Gemma-style (1+w) RMSNorm: bake +1 in so llama.cpp can use plain RMSNorm
-+        if name.endswith("norm.weight"):
-+            data_torch = data_torch + 1.0
-+
-+        # merge routed experts (w1/w2/w3); shared_experts.* passes through to *_shexp
-+        if "block_sparse_moe.experts." in name:
-+            n_experts = self.find_hparam(["num_local_experts", "num_experts"])
-+            assert bid is not None
-+
-+            expert_cache = self._experts_cache.setdefault(bid, {})
-+            expert_cache[name] = data_torch
-+            expert_weights = ["w1", "w2", "w3"]
-+
-+            if len(expert_cache) < n_experts * len(expert_weights):
-+                return
-+
-+            for w_name in expert_weights:
-+                datas: list[Tensor] = []
-+                for xid in range(n_experts):
-+                    ename = f"model.layers.{bid}.block_sparse_moe.experts.{xid}.{w_name}.weight"
-+                    datas.append(expert_cache[ename])
-+                    del expert_cache[ename]
-+
-+                data_torch = torch.stack(datas, dim=0)
-+                merged_name = f"model.layers.{bid}.block_sparse_moe.experts.{w_name}.weight"
-+                yield from super().modify_tensors(data_torch, merged_name, bid)
-+
-+            del self._experts_cache[bid]
-+            return
-+
-+        yield from super().modify_tensors(data_torch, name, bid)
-diff --git a/gguf-py/gguf/constants.py b/gguf-py/gguf/constants.py
-index 869e436..760e3dd 100644
---- a/gguf-py/gguf/constants.py
-+++ b/gguf-py/gguf/constants.py
-@@ -525,6 +525,7 @@ class MODEL_ARCH(IntEnum):
-     APERTUS          = auto()
-     COGVLM           = auto()
-     MINIMAXM2        = auto()
-+    MINIMAXM3        = auto()
-     RND1             = auto()
-     PANGU_EMBED      = auto()
-     MISTRAL3         = auto()
-@@ -613,6 +614,10 @@ class MODEL_TENSOR(IntEnum):
-     MOE_LATENT_UP        = auto() # nemotron 3 super
-     ATTN_Q_NORM          = auto()
-     ATTN_K_NORM          = auto()
-+    ATTN_INDEX_Q         = auto() # minimax-m3 sparse-attn indexer (unused)
-+    ATTN_INDEX_K         = auto()
-+    ATTN_INDEX_Q_NORM    = auto()
-+    ATTN_INDEX_K_NORM    = auto()
-     LAYER_OUT_NORM       = auto()
-     LAYER_OUT_SCALE      = auto()
-     PER_LAYER_TOKEN_EMBD = auto() # gemma3n
-@@ -1105,6 +1110,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
-     MODEL_ARCH.GROVEMOE:         "grovemoe",
-     MODEL_ARCH.APERTUS:          "apertus",
-     MODEL_ARCH.MINIMAXM2:        "minimax-m2",
-+    MODEL_ARCH.MINIMAXM3:        "minimax-m3",
-     MODEL_ARCH.COGVLM:           "cogvlm",
-     MODEL_ARCH.RND1:             "rnd1",
-     MODEL_ARCH.PANGU_EMBED:      "pangu-embedded",
-@@ -1163,6 +1169,10 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
-     MODEL_TENSOR.ATTN_GATE:                 "blk.{bid}.attn_gate",
-     MODEL_TENSOR.ATTN_Q_NORM:               "blk.{bid}.attn_q_norm",
-     MODEL_TENSOR.ATTN_K_NORM:               "blk.{bid}.attn_k_norm",
-+    MODEL_TENSOR.ATTN_INDEX_Q:              "blk.{bid}.attn_index_q",
-+    MODEL_TENSOR.ATTN_INDEX_K:              "blk.{bid}.attn_index_k",
-+    MODEL_TENSOR.ATTN_INDEX_Q_NORM:         "blk.{bid}.attn_index_q_norm",
-+    MODEL_TENSOR.ATTN_INDEX_K_NORM:         "blk.{bid}.attn_index_k_norm",
-     MODEL_TENSOR.ATTN_OUT_NORM:             "blk.{bid}.attn_output_norm",
-     MODEL_TENSOR.ATTN_POST_NORM:            "blk.{bid}.post_attention_norm",
-     MODEL_TENSOR.FFN_GATE_INP:              "blk.{bid}.ffn_gate_inp",
-@@ -4102,6 +4112,30 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.FFN_UP_EXP,
-         MODEL_TENSOR.FFN_EXP_PROBS_B,
-     ],
-+    MODEL_ARCH.MINIMAXM3: [
-+        MODEL_TENSOR.TOKEN_EMBD,
-+        MODEL_TENSOR.OUTPUT_NORM,
-+        MODEL_TENSOR.OUTPUT,
-+        MODEL_TENSOR.ATTN_NORM,
-+        MODEL_TENSOR.ATTN_Q,
-+        MODEL_TENSOR.ATTN_Q_NORM,
-+        MODEL_TENSOR.ATTN_K,
-+        MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_V,
-+        MODEL_TENSOR.ATTN_OUT,
-+        MODEL_TENSOR.FFN_NORM,
-+        MODEL_TENSOR.FFN_GATE_INP,
-+        MODEL_TENSOR.FFN_EXP_PROBS_B,
-+        MODEL_TENSOR.FFN_GATE_EXP,
-+        MODEL_TENSOR.FFN_DOWN_EXP,
-+        MODEL_TENSOR.FFN_UP_EXP,
-+        MODEL_TENSOR.FFN_GATE_SHEXP,
-+        MODEL_TENSOR.FFN_DOWN_SHEXP,
-+        MODEL_TENSOR.FFN_UP_SHEXP,
-+        MODEL_TENSOR.FFN_GATE,
-+        MODEL_TENSOR.FFN_DOWN,
-+        MODEL_TENSOR.FFN_UP,
-+    ],
-     MODEL_ARCH.COGVLM: [
-         MODEL_TENSOR.TOKEN_EMBD,
-         MODEL_TENSOR.OUTPUT_NORM,
-@@ -4128,6 +4162,10 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.ATTN_Q_NORM,
-         MODEL_TENSOR.ATTN_K,
-         MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_Q,
-+        MODEL_TENSOR.ATTN_INDEX_K,
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM,
-         MODEL_TENSOR.ATTN_V,
-         MODEL_TENSOR.ATTN_OUT,
-         MODEL_TENSOR.FFN_NORM,
-diff --git a/gguf-py/gguf/tensor_mapping.py b/gguf-py/gguf/tensor_mapping.py
-index 9efb36f..a62040b 100644
---- a/gguf-py/gguf/tensor_mapping.py
-+++ b/gguf-py/gguf/tensor_mapping.py
-@@ -717,6 +717,22 @@ class TensorNameMap:
-             "model.layers.{bid}.attention.key_layernorm",                     # apertus
-         ),
- 
-+        MODEL_TENSOR.ATTN_INDEX_Q: (
-+            "model.layers.{bid}.self_attn.index_q_proj",                      # minimax-m3 (sparse-attn indexer)
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K: (
-+            "model.layers.{bid}.self_attn.index_k_proj",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM: (
-+            "model.layers.{bid}.self_attn.index_q_norm",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM: (
-+            "model.layers.{bid}.self_attn.index_k_norm",                      # minimax-m3
-+        ),
-+
-         MODEL_TENSOR.ROPE_FREQS: (
-             "encoder.layers.{bid}.self_attention.rotary_emb.inv_freq",  # persimmon
-         ),
-diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
-index b890e66..cb8bfc8 100644
---- a/src/llama-arch.cpp
-+++ b/src/llama-arch.cpp
-@@ -125,6 +125,7 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
-     { LLM_ARCH_GROVEMOE,         "grovemoe"         },
-     { LLM_ARCH_APERTUS,          "apertus"          },
-     { LLM_ARCH_MINIMAX_M2,       "minimax-m2"       },
-+    { LLM_ARCH_MINIMAX_M3,       "minimax-m3"       },
-     { LLM_ARCH_COGVLM,           "cogvlm"           },
-     { LLM_ARCH_RND1,             "rnd1"             },
-     { LLM_ARCH_PANGU_EMBED,      "pangu-embedded"   },
-@@ -395,6 +396,10 @@ static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
-     { LLM_TENSOR_ATTN_POST_NORM,                         "blk.%d.post_attention_norm" },
-     { LLM_TENSOR_ATTN_Q_NORM,                            "blk.%d.attn_q_norm" },
-     { LLM_TENSOR_ATTN_K_NORM,                            "blk.%d.attn_k_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_Q,                           "blk.%d.attn_index_q" },
-+    { LLM_TENSOR_ATTN_INDEX_K,                           "blk.%d.attn_index_k" },
-+    { LLM_TENSOR_ATTN_INDEX_Q_NORM,                      "blk.%d.attn_index_q_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_K_NORM,                      "blk.%d.attn_index_k_norm" },
-     { LLM_TENSOR_ATTN_GATE,                              "blk.%d.attn_gate" },
-     { LLM_TENSOR_FFN_POST_NORM,                          "blk.%d.post_ffw_norm" },
-     { LLM_TENSOR_FFN_POST_NORM_1,                        "blk.%d.post_ffw_norm_1" },
-@@ -761,6 +766,11 @@ static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
-     {LLM_TENSOR_FFN_NORM_EXPS,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_K_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-+    // minimax-m3 sparse-attn indexer: unused (GGML_OP_NONE) so the loader skips it
-+    {LLM_TENSOR_ATTN_INDEX_Q,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_Q_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-     {LLM_TENSOR_LAYER_OUT_NORM,             {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_LAYER_OUT_SCALE,            {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_A_NORM,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-@@ -998,6 +1008,7 @@ bool llm_arch_supports_sm_tensor(const llm_arch & arch) {
-         case LLM_ARCH_LFM2:
-         case LLM_ARCH_LFM2MOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_MISTRAL4:
-         case LLM_ARCH_KIMI_LINEAR:
-             return false;
-diff --git a/src/llama-arch.h b/src/llama-arch.h
-index a4f5091..2d50ead 100644
---- a/src/llama-arch.h
-+++ b/src/llama-arch.h
-@@ -144,6 +144,7 @@ enum llm_arch {
-     LLM_ARCH_TALKIE,
-     LLM_ARCH_MELLUM,
-     LLM_ARCH_EAGLE3,
-+    LLM_ARCH_MINIMAX_M3,
-     LLM_ARCH_DFLASH,
-     LLM_ARCH_UNKNOWN,
- };
-@@ -429,6 +430,10 @@ enum llm_tensor {
-     LLM_TENSOR_FFN_LATENT_UP,
-     LLM_TENSOR_ATTN_Q_NORM,
-     LLM_TENSOR_ATTN_K_NORM,
-+    LLM_TENSOR_ATTN_INDEX_Q,      // minimax-m3 sparse-attn indexer (unused)
-+    LLM_TENSOR_ATTN_INDEX_K,
-+    LLM_TENSOR_ATTN_INDEX_Q_NORM,
-+    LLM_TENSOR_ATTN_INDEX_K_NORM,
-     LLM_TENSOR_LAYER_OUT_NORM,
-     LLM_TENSOR_LAYER_OUT_SCALE,
-     LLM_TENSOR_POST_ATTN_NORM,
-diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
-index c8ecb0a..4c2c286 100644
---- a/src/llama-graph.cpp
-+++ b/src/llama-graph.cpp
-@@ -1719,6 +1719,16 @@ ggml_tensor * llm_graph_context::build_ffn(
-                 cur = ggml_reglu(ctx0, cur);
-                 cb(cur, "ffn_reglu", il);
-             } break;
-+        case LLM_FFN_SWIGLU_OAI:
-+            {
-+                // clamped SwiGLU: parallel gate path (cur=gate, tmp=up)
-+                GGML_ASSERT(gate && type_gate == LLM_FFN_PAR);
-+                constexpr float alpha = 1.702f;
-+                constexpr float limit = 7.0f;
-+                cur = ggml_swiglu_oai(ctx0, cur, tmp, alpha, limit);
-+                cb(cur, "ffn_swiglu_oai", il);
-+                type_gate = LLM_FFN_SEQ; // gate*up already fused; skip the par multiply
-+            } break;
-         default:
-             GGML_ABORT("fatal error");
-     }
-diff --git a/src/llama-graph.h b/src/llama-graph.h
-index c84cb6a..806ce7b 100644
---- a/src/llama-graph.h
-+++ b/src/llama-graph.h
-@@ -54,6 +54,7 @@ enum llm_ffn_op_type : int {
-     LLM_FFN_SWIGLU,
-     LLM_FFN_GEGLU,
-     LLM_FFN_REGLU,
-+    LLM_FFN_SWIGLU_OAI,
-     LLM_FFN_SWIGLU_OAI_MOE,
- };
- 
-diff --git a/src/llama-model.cpp b/src/llama-model.cpp
-index d874813..7bb71c0 100644
---- a/src/llama-model.cpp
-+++ b/src/llama-model.cpp
-@@ -280,6 +280,8 @@ static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params
-             return new llama_model_apertus(params);
-         case LLM_ARCH_MINIMAX_M2:
-             return new llama_model_minimax_m2(params);
-+        case LLM_ARCH_MINIMAX_M3:
-+            return new llama_model_minimax_m3(params);
-         case LLM_ARCH_COGVLM:
-             return new llama_model_cogvlm(params);
-         case LLM_ARCH_PANGU_EMBED:
-@@ -807,6 +809,7 @@ const char * llm_type_name(llm_type type) {
-         case LLM_TYPE_310B_A15B:     return "310B.A15B";
-         case LLM_TYPE_355B_A32B:     return "355B.A32B";
-         case LLM_TYPE_397B_A17B:     return "397B.A17B";
-+        case LLM_TYPE_428B_A23B:     return "428B.A23B";
-         case LLM_TYPE_685B_A37B:     return "685B.A37B";
-         case LLM_TYPE_744B_A40B:     return "744B.A40B";
-         case LLM_TYPE_E2B:           return "E2B";
-@@ -2532,6 +2535,7 @@ llama_rope_type llama_model_rope_type(const llama_model * model) {
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_APERTUS:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_COGVLM:
-         case LLM_ARCH_PANGU_EMBED:
-         case LLM_ARCH_AFMOE:
-diff --git a/src/llama-model.h b/src/llama-model.h
-index 45b054c..540e0d2 100644
---- a/src/llama-model.h
-+++ b/src/llama-model.h
-@@ -139,6 +139,7 @@ enum llm_type {
-     LLM_TYPE_310B_A15B, // /MiMo-V2-Flash
-     LLM_TYPE_355B_A32B, // GLM-4.5
-     LLM_TYPE_397B_A17B, // Qwen3.5
-+    LLM_TYPE_428B_A23B, // MiniMax M3
-     LLM_TYPE_685B_A37B, // DeepSeek V3.2
-     LLM_TYPE_744B_A40B, // GLM-5
-     LLM_TYPE_E2B,
-diff --git a/src/models/minimax-m3.cpp b/src/models/minimax-m3.cpp
-new file mode 100644
-index 0000000..137852a
---- /dev/null
-+++ b/src/models/minimax-m3.cpp
-@@ -0,0 +1,197 @@
-+#include "models.h"
-+
-+// MiniMax-M3, text-only: MiniMax-M2 GQA (per-head QK-norm, partial rotary) + DeepSeek-V3
-+// leading-dense/routed/shared experts (swigluoai). Sparse attn -> dense; vision + MTP dropped.
-+
-+void llama_model_minimax_m3::load_arch_hparams(llama_model_loader & ml) {
-+    ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
-+    ml.get_key(LLM_KV_LEADING_DENSE_BLOCK_COUNT,   hparams.n_layer_dense_lead, false);
-+    ml.get_key(LLM_KV_EXPERT_FEED_FORWARD_LENGTH,  hparams.n_ff_exp);
-+    ml.get_key(LLM_KV_EXPERT_SHARED_COUNT,         hparams.n_expert_shared);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE,        hparams.expert_weights_scale, false);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_NORM,         hparams.expert_weights_norm, false);
-+    ml.get_key(LLM_KV_EXPERT_GATING_FUNC,          hparams.expert_gating_func);
-+
-+    switch (hparams.n_layer()) {
-+        case 60: type = LLM_TYPE_428B_A23B; break;
-+        default: type = LLM_TYPE_UNKNOWN;
-+    }
-+}
-+
-+void llama_model_minimax_m3::load_arch_tensors(llama_model_loader &) {
-+    LLAMA_LOAD_LOCALS;
-+    const int64_t n_expert_shared = hparams.n_expert_shared;
-+    const int64_t n_ff_exp        = hparams.n_ff_exp;
-+
-+    tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
-+
-+    output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
-+    output      = create_tensor(tn(LLM_TENSOR_OUTPUT,      "weight"), {n_embd, n_vocab}, 0);
-+
-+    for (int i = 0; i < n_layer; ++i) {
-+        auto & layer = layers[i];
-+
-+        create_tensor_qkv(layer, i, n_embd, n_embd_head_k * n_head, n_embd_gqa, n_embd_gqa, 0);
-+        layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), { n_embd_head_k * n_head, n_embd }, 0);
-+
-+        layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
-+        // per-head QK-norm (one head_dim vector)
-+        layer.attn_q_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_NORM, "weight", i), {n_embd_head_k}, 0);
-+        layer.attn_k_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_NORM, "weight", i), {n_embd_head_k}, 0);
-+
-+        // sparse-attn indexer (unused): GGML_OP_NONE -> loader skips; NOT_REQUIRED -> older GGUFs still load;
-+        // SKIP_IF_VIRTUAL -> no-file loader (test-llama-archs) skips them too
-+        const int64_t n_index_head = 4;    // sparse_num_index_heads
-+        const int64_t d_index      = 128;  // sparse_index_dim
-+        const int idx_flags = TENSOR_NOT_REQUIRED | TENSOR_SKIP_IF_VIRTUAL;
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q,      "weight", i), {n_embd, n_index_head * d_index}, idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K,      "weight", i), {n_embd, d_index},                idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q_NORM, "weight", i), {d_index},                        idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K_NORM, "weight", i), {d_index},                        idx_flags);
-+
-+        layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
-+
-+        if (i < (int) hparams.n_layer_dense_lead) {
-+            layer.ffn_gate = create_tensor(tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd,   n_ff}, 0);
-+            layer.ffn_down = create_tensor(tn(LLM_TENSOR_FFN_DOWN, "weight", i), {  n_ff, n_embd}, 0);
-+            layer.ffn_up   = create_tensor(tn(LLM_TENSOR_FFN_UP,   "weight", i), {n_embd,   n_ff}, 0);
-+        } else {
-+            layer.ffn_gate_inp    = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP,    "weight", i), {n_embd, n_expert}, 0);
-+            layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias",   i), {n_expert}, 0);
-+            layer.ffn_gate_exps   = create_tensor(tn(LLM_TENSOR_FFN_GATE_EXPS,   "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+            layer.ffn_down_exps   = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS,   "weight", i), {n_ff_exp, n_embd, n_expert}, 0);
-+            layer.ffn_up_exps     = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,     "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+
-+            layer.ffn_gate_shexp = create_tensor(tn(LLM_TENSOR_FFN_GATE_SHEXP, "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+            layer.ffn_down_shexp = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "weight", i), {        n_ff_exp * n_expert_shared, n_embd}, 0);
-+            layer.ffn_up_shexp   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+        }
-+    }
-+}
-+
-+std::unique_ptr<llm_graph_context> llama_model_minimax_m3::build_arch_graph(const llm_graph_params & params) const {
-+    return std::make_unique<graph>(*this, params);
-+}
-+
-+llama_model_minimax_m3::graph::graph(const llama_model & model, const llm_graph_params & params) : llm_graph_context(params) {
-+    const int64_t n_embd_head = hparams.n_embd_head_v();
-+
-+    GGML_ASSERT(n_embd_head == hparams.n_embd_head_k());
-+    // partial rotary: head_dim != n_rot, so don't assert n_embd_head == n_rot
-+
-+    ggml_tensor * cur;
-+    ggml_tensor * inpL;
-+
-+    inpL = build_inp_embd(model.tok_embd);
-+
-+    ggml_tensor * inp_pos = build_inp_pos();
-+    auto inp_attn = build_attn_inp_kv();
-+    ggml_tensor * inp_out_ids = build_inp_out_ids();
-+
-+    for (int il = 0; il < n_layer; ++il) {
-+        ggml_tensor * inpSA = inpL;
-+
-+        // self-attention
-+        {
-+            cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);
-+            cb(cur, "attn_norm", il);
-+
-+            auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur,
-+                    n_embd_head, n_head, n_head_kv, il);
-+
-+            // per-head QK RMSNorm (weights include Gemma +1)
-+            Qcur = build_norm(Qcur, model.layers[il].attn_q_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Qcur, "Qcur_normed", il);
-+            Kcur = build_norm(Kcur, model.layers[il].attn_k_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Kcur, "Kcur_normed", il);
-+
-+            Qcur = ggml_rope_ext(
-+                ctx0, Qcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+            Kcur = ggml_rope_ext(
-+                ctx0, Kcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+
-+            cb(Qcur, "Qcur", il);
-+            cb(Kcur, "Kcur", il);
-+            cb(Vcur, "Vcur", il);
-+
-+            cur = build_attn(inp_attn,
-+                    model.layers[il].wo, NULL, model.layers[il].wo_s,
-+                    Qcur, Kcur, Vcur, nullptr, nullptr, nullptr, 1.0f/sqrtf(float(n_embd_head)), il);
-+        }
-+
-+        if (il == n_layer - 1 && inp_out_ids) {
-+            cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);
-+            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids);
-+        }
-+
-+        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);
-+        cb(ffn_inp, "ffn_inp", il);
-+
-+        cur = build_norm(ffn_inp, model.layers[il].ffn_norm, NULL, LLM_NORM_RMS, il);
-+        cb(cur, "ffn_norm", il);
-+
-+        if ((uint32_t) il < hparams.n_layer_dense_lead) {
-+            // leading dense
-+            cur = build_ffn(cur,
-+                    model.layers[il].ffn_up,   NULL, NULL,
-+                    model.layers[il].ffn_gate, NULL, NULL,
-+                    model.layers[il].ffn_down, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(cur, "ffn_out", il);
-+        } else {
-+            // routed experts
-+            ggml_tensor * moe_out = build_moe_ffn(cur,
-+                    model.layers[il].ffn_gate_inp,
-+                    model.layers[il].ffn_up_exps,
-+                    model.layers[il].ffn_gate_exps,
-+                    model.layers[il].ffn_down_exps,
-+                    model.layers[il].ffn_exp_probs_b,
-+                    n_expert, n_expert_used,
-+                    LLM_FFN_SWIGLU_OAI_MOE, hparams.expert_weights_norm,
-+                    hparams.expert_weights_scale,
-+                    (llama_expert_gating_func_type) hparams.expert_gating_func,
-+                    il);
-+            cb(moe_out, "ffn_moe_out", il);
-+
-+            // shared expert
-+            ggml_tensor * ffn_shexp = build_ffn(cur,
-+                    model.layers[il].ffn_up_shexp,   NULL, NULL,
-+                    model.layers[il].ffn_gate_shexp, NULL, NULL,
-+                    model.layers[il].ffn_down_shexp, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(ffn_shexp, "ffn_shexp", il);
-+
-+            cur = ggml_add(ctx0, moe_out, ffn_shexp);
-+            cb(cur, "ffn_out", il);
-+        }
-+
-+        cur = ggml_add(ctx0, cur, ffn_inp);
-+
-+        cur = build_cvec(cur, il);
-+        cb(cur, "l_out", il);
-+
-+        // input for next layer
-+        inpL = cur;
-+    }
-+
-+    cur = inpL;
-+
-+    cur = build_norm(cur, model.output_norm, NULL, LLM_NORM_RMS, -1);
-+    cb(cur, "result_norm", -1);
-+    res->t_embd = cur;
-+
-+    // lm_head
-+    cur = build_lora_mm(model.output, cur, model.output_s);
-+    cb(cur, "result_output", -1);
-+    res->t_logits = cur;
-+
-+    ggml_build_forward_expand(gf, cur);
-+}
-diff --git a/src/models/models.h b/src/models/models.h
-index 7a52e7b..5e2a826 100644
---- a/src/models/models.h
-+++ b/src/models/models.h
-@@ -1870,6 +1870,17 @@ struct llama_model_minimax_m2 : public llama_model_base {
-     std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
- };
- 
-+struct llama_model_minimax_m3 : public llama_model_base {
-+    llama_model_minimax_m3(const struct llama_model_params & params) : llama_model_base(params) {}
-+    void load_arch_hparams(llama_model_loader & ml) override;
-+    void load_arch_tensors(llama_model_loader & ml) override;
-+
-+    struct graph : public llm_graph_context {
-+        graph(const llama_model & model, const llm_graph_params & params);
-+    };
-+
-+    std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
-+};
- 
- struct llama_model_cogvlm : public llama_model_base {
-     llama_model_cogvlm(const struct llama_model_params & params) : llama_model_base(params) {}
-diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp
-index f39abe7..2085f43 100644
---- a/tests/test-llama-archs.cpp
-+++ b/tests/test-llama-archs.cpp
-@@ -352,6 +352,7 @@ static bool moe_mandatory(const llm_arch arch) {
-         case LLM_ARCH_LLADA_MOE:
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_RND1:
-         case LLM_ARCH_PADDLEOCR:
-         case LLM_ARCH_MIMO2:

backend/cpp/llama-cpp/prepare.sh

@@ -31,6 +31,26 @@ cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 
+## Fork-skew probe. Upstream folded common_params::use_mmap / use_mlock /
+## use_direct_io into a single `load_mode` enum (ggml-org/llama.cpp#20834).
+## turboquant and bonsai compile this very same grpc-server.cpp against forks
+## that branched before that change, so the field set is decided from the
+## checkout in front of us rather than from a per-fork build flag: the flavor
+## targets disagree on whether they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and
+## probing heals itself the moment a fork rebases past the refactor.
+if grep -q "LLAMA_LOAD_MODE_MMAP" llama.cpp/include/llama.h; then
+    echo "==> llama.cpp carries the load-mode enum, using common_params::load_mode"
+    LEGACY_LOAD_MODE=0
+else
+    echo "==> llama.cpp predates the load-mode enum, using the legacy mmap/mlock/direct-io booleans"
+    LEGACY_LOAD_MODE=1
+fi
+cat > llama.cpp/tools/grpc-server/llama_compat.h <<EOF
+// Generated by backend/cpp/llama-cpp/prepare.sh. Do not edit.
+#pragma once
+#define LOCALAI_LEGACY_LOAD_MODE ${LEGACY_LOAD_MODE}
+EOF
+
 set +e
 if grep -q "grpc-server" llama.cpp/tools/CMakeLists.txt; then
     echo "grpc-server already added"

backend/cpp/turboquant/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-TURBOQUANT_VERSION?=7d9715f1f071fa07c7b2ad3dbfd320b314139e65
+TURBOQUANT_VERSION?=c26cbdffcf6fc9b7430cd6b117757e9a3f70b7ea
 LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
 
 CMAKE_ARGS?=

backend/cpp/turboquant/patches/0001-hip-guard-copy2d-peer-fastpath.patch

@@ -1,50 +1,18 @@
 hip: port the turboquant CUDA additions that ggml's HIP shim doesn't cover
 
-The turboquant fork adds/modifies a few ggml-cuda.cu spots with CUDA APIs
-that ggml's HIP (and MUSA) compatibility layer does not provide, breaking
-the -gpu-rocm-hipblas-turboquant build:
+The turboquant fork creates backend events with plain cudaEventCreate,
+which ggml's HIP shim does not alias (it only aliases
+cudaEventCreateWithFlags). Use cudaEventCreateWithFlags(...,
+cudaEventDisableTiming), exactly as the rest of this file does.
 
-  1. ggml_cuda_copy2d_across_devices() (host-staged cross-device copy for
-     split mul_mat output) uses the CUDA 3D-peer copy APIs
-     cudaMemcpy3DPeerParms / make_cudaPitchedPtr / make_cudaExtent /
-     cudaMemcpy3DPeerAsync. HIP genuinely does not support these (see the
-     fork's own comment "HIP does not support cudaMemcpy3DPeerAsync"), so
-     guard the peer fast path with #if !defined(GGML_USE_HIP) &&
-     !defined(GGML_USE_MUSA) -- matching how the fork already guards the
-     same API for the sibling 2D copy -- and fall through to the existing
-     cudaMemcpyAsync staging fallback below (functionally identical,
-     slightly slower on multi-GPU ROCm).
-
-  2. ggml_backend_cuda_device_event_new() creates its event with plain
-     cudaEventCreate, which ggml's HIP shim does not alias (it only aliases
-     cudaEventCreateWithFlags). Use cudaEventCreateWithFlags(..., 
-     cudaEventDisableTiming) -- exactly what the rest of this file already
-     does (cf. lines ~1034, ~3461) and HIP-safe.
-
-CUDA builds are unaffected. Drop the relevant hunk once the fork HIP-ports
-these; apply-patches.sh fails fast if an anchor goes stale.
+CUDA builds are unaffected. Drop this patch once the fork HIP-ports the
+event creation; apply-patches.sh fails fast if the anchor goes stale.
 
 diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
-index 0427e6b..6352e6a 100644
+index 7d35c1a..2908acb 100644
 --- a/ggml/src/ggml-cuda/ggml-cuda.cu
 +++ b/ggml/src/ggml-cuda/ggml-cuda.cu
-@@ -1933,6 +1933,7 @@ static cudaError_t ggml_cuda_copy2d_across_devices(
-     size_t width, size_t height, cudaStream_t dst_stream, cudaStream_t src_stream) {
- 
-     const auto & info = ggml_cuda_info();
-+#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)  // 3D-peer copy types unmapped by ggml's HIP/MUSA shim; use staging fallback below
-     if (info.peer_access[src_device][dst_device]) {
-         cudaMemcpy3DPeerParms p = {};
-         p.dstDevice = dst_device;
-@@ -1942,6 +1943,7 @@ static cudaError_t ggml_cuda_copy2d_across_devices(
-         p.extent = make_cudaExtent(width, height, 1);
-         return cudaMemcpy3DPeerAsync(&p, dst_stream);
-     }
-+#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
- 
-     // Fallback: stage all rows through a single contiguous pinned buffer
-     int prev_device = ggml_cuda_get_device();
-@@ -5714,7 +5716,7 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_
+@@ -5795,7 +5795,7 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_
      ggml_cuda_set_device(dev_ctx->device);
  
      cudaEvent_t event;

backend/go/ced/Makefile

@@ -1,6 +1,6 @@
 # ced sound-classification backend Makefile.
 #
-# Upstream pin lives below as CED_VERSION?=<sha> so .github/bump_deps.sh can find
+# Upstream pin lives below as CED_VERSION?=db5aae02973a745722d6fbd2157cab1999106777
 # and update it (matches the parakeet-cpp / whisper.cpp convention).
 #
 # Local dev shortcut: symlink an out-of-tree ced.cpp shared build + header and
@@ -9,7 +9,7 @@
 #   ln -sf /path/to/ced.cpp/include/ced_capi.h .
 #   go build -o ced-grpc .
 
-CED_VERSION?=c04ac14b7992d00584d9e812c9bb6268598a6ce7
+CED_VERSION?=db5aae02973a745722d6fbd2157cab1999106777
 CED_REPO?=https://github.com/localai-org/ced.cpp
 
 GOCMD?=go

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=5fca47ecf05cd68bb0075f8a00fe04da06f208d0
+CRISPASR_VERSION?=306faee45fab641d54f9f941f075de1e9c0d3278
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/locate-anything-cpp/Makefile

@@ -10,7 +10,7 @@ JOBS?=$(shell nproc --ignore=1)
 # this on `master` always picks up the latest C-API surface (incl. the
 # per-detection accessor functions used by golocateanythingcpp.go).
 LOCATEANYTHING_REPO?=https://github.com/mudler/locate-anything.cpp.git
-LOCATEANYTHING_VERSION?=ade2634f7f79b56121125e5885628744795a478f
+LOCATEANYTHING_VERSION?=77376ab332de918220f7a7e391542eefb5407c9f
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/magpie-tts-cpp/.gitignore

@@ -0,0 +1,6 @@
+magpie-tts-cpp
+*.so
+*.dylib
+sources/
+package/
+magpie-tts-models/

backend/go/magpie-tts-cpp/CMakeLists.txt

@@ -0,0 +1,35 @@
+cmake_minimum_required(VERSION 3.16)
+project(gomagpiettscpp LANGUAGES C CXX)
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
+set(MAGPIE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/magpie-tts.cpp)
+
+# Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
+if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
+    set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
+endif()
+
+# The magpie-tts C-API is exported directly by the upstream shared library
+# (magpie_tts_capi_* in libmagpie-tts.so, ggml statically linked inside), so
+# unlike qwen3-tts-cpp / moss-tts-cpp no local C shim is needed -- this wrapper
+# only configures the upstream project as a purego-loadable shared library.
+set(MAGPIE_SHARED       ON  CACHE BOOL "" FORCE)
+set(MAGPIE_BUILD_CLI    OFF CACHE BOOL "" FORCE)
+set(MAGPIE_BUILD_TESTS  OFF CACHE BOOL "" FORCE)
+
+# Upstream FORCE-overwrites GGML_CUDA / GGML_METAL / GGML_VULKAN / GGML_HIP from
+# its own MAGPIE_GGML_* toggles, which would silently discard the -DGGML_*=ON
+# flags the LocalAI Makefile passes for GPU BUILD_TYPEs. Translate them into the
+# MAGPIE_GGML_* vocabulary before add_subdirectory. (GGML_SYCL / GGML_BLAS and
+# the CPU ISA flags are not touched by upstream and reach ggml unchanged.)
+foreach(_be CUDA METAL VULKAN HIP)
+    if(GGML_${_be})
+        set(MAGPIE_GGML_${_be} ON CACHE BOOL "" FORCE)
+    endif()
+endforeach()
+
+add_subdirectory(${MAGPIE_DIR} magpie EXCLUDE_FROM_ALL)
+
+# Place libmagpie-tts.so at the build root, where the Makefile picks it up.
+set_target_properties(magpie-tts PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/magpie-tts-cpp/Makefile

@@ -0,0 +1,140 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+JOBS?=$(shell nproc --ignore=1)
+
+# magpie-tts.cpp version
+MAGPIETTS_REPO?=https://github.com/mudler/magpie-tts.cpp
+MAGPIETTS_CPP_VERSION?=3008ff73fc2d2da9e4d743b09350aa7023e8980c
+SO_TARGET?=libgomagpiettscpp.so
+
+CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
+
+ifeq ($(NATIVE),false)
+	CMAKE_ARGS+=-DGGML_NATIVE=OFF
+endif
+
+ifeq ($(BUILD_TYPE),cublas)
+	CMAKE_ARGS+=-DGGML_CUDA=ON
+else ifeq ($(BUILD_TYPE),openblas)
+	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
+else ifeq ($(BUILD_TYPE),clblas)
+	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
+else ifeq ($(BUILD_TYPE),hipblas)
+	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
+	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
+	ROCM_HOME ?= /opt/rocm
+	ROCM_PATH ?= /opt/rocm
+	export CXX=$(ROCM_HOME)/llvm/bin/clang++
+	export CC=$(ROCM_HOME)/llvm/bin/clang
+	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DGGML_VULKAN=ON
+else ifeq ($(OS),Darwin)
+	ifneq ($(BUILD_TYPE),metal)
+		CMAKE_ARGS+=-DGGML_METAL=OFF
+	else
+		CMAKE_ARGS+=-DGGML_METAL=ON
+		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
+	endif
+endif
+
+ifeq ($(BUILD_TYPE),sycl_f16)
+	CMAKE_ARGS+=-DGGML_SYCL=ON \
+		-DCMAKE_C_COMPILER=icx \
+		-DCMAKE_CXX_COMPILER=icpx \
+		-DGGML_SYCL_F16=ON
+endif
+
+ifeq ($(BUILD_TYPE),sycl_f32)
+	CMAKE_ARGS+=-DGGML_SYCL=ON \
+		-DCMAKE_C_COMPILER=icx \
+		-DCMAKE_CXX_COMPILER=icpx
+endif
+
+sources/magpie-tts.cpp:
+	mkdir -p sources/magpie-tts.cpp
+	cd sources/magpie-tts.cpp && \
+	git init && \
+	git remote add origin $(MAGPIETTS_REPO) && \
+	git fetch origin && \
+	git checkout $(MAGPIETTS_CPP_VERSION) && \
+	git submodule update --init --recursive --depth 1 --single-branch
+
+# Detect OS
+UNAME_S := $(shell uname -s)
+
+# Only build CPU variants on Linux
+ifeq ($(UNAME_S),Linux)
+	VARIANT_TARGETS = libgomagpiettscpp-avx.so libgomagpiettscpp-avx2.so libgomagpiettscpp-avx512.so libgomagpiettscpp-fallback.so
+else
+	# On non-Linux (e.g., Darwin), build only fallback variant (as a dylib)
+	VARIANT_TARGETS = libgomagpiettscpp-fallback.dylib
+endif
+
+magpie-tts-cpp: main.go gomagpiettscpp.go $(VARIANT_TARGETS)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o magpie-tts-cpp ./
+
+package: magpie-tts-cpp
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf libgomagpiettscpp*.so libgomagpiettscpp*.dylib package sources/magpie-tts.cpp magpie-tts-cpp
+
+purge:
+	rm -rf build*
+
+# Variants must build sequentially
+.NOTPARALLEL:
+
+# Build all variants (Linux only)
+ifeq ($(UNAME_S),Linux)
+libgomagpiettscpp-avx.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx${RESET})
+	SO_TARGET=libgomagpiettscpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx.so
+
+libgomagpiettscpp-avx2.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx2${RESET})
+	SO_TARGET=libgomagpiettscpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx2.so
+
+libgomagpiettscpp-avx512.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx512${RESET})
+	SO_TARGET=libgomagpiettscpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx512.so
+endif
+
+# Build fallback variant (all platforms)
+libgomagpiettscpp-fallback.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:fallback${RESET})
+	SO_TARGET=libgomagpiettscpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-fallback.so
+
+# Build fallback variant as a dylib (Darwin)
+libgomagpiettscpp-fallback.dylib: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:fallback (dylib)${RESET})
+	SO_TARGET=libgomagpiettscpp-fallback.dylib CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-fallback.dylib
+
+libgomagpiettscpp-custom: CMakeLists.txt
+	mkdir -p build-$(SO_TARGET) && \
+	cd build-$(SO_TARGET) && \
+	cmake .. $(CMAKE_ARGS) && \
+	cmake --build . --config Release -j$(JOBS) --target magpie-tts && \
+	cd .. && \
+	(mv build-$(SO_TARGET)/libmagpie-tts.so ./$(SO_TARGET) 2>/dev/null || \
+	 mv build-$(SO_TARGET)/libmagpie-tts.dylib ./$(SO_TARGET) 2>/dev/null)
+
+test: magpie-tts-cpp
+	@echo "Running magpie-tts-cpp tests..."
+	bash test.sh
+	@echo "magpie-tts-cpp tests completed."
+
+all: magpie-tts-cpp package

backend/go/magpie-tts-cpp/README.md

@@ -0,0 +1,69 @@
+# Magpie TTS C++ backend
+
+This backend runs NVIDIA's **Magpie TTS Multilingual 357M** GGUF through
+[magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp), a from-scratch
+C++/ggml port (model + NanoCodec + tokenizer + G2P dictionaries in one
+self-contained GGUF, no Python at inference time). It generates **22.05 kHz
+mono** speech in 5 baked voices across 9+ languages.
+
+The library is loaded via purego (cgo-less `dlopen`) exactly like
+`qwen3-tts-cpp` / `moss-tts-cpp`; the flat C-API (`magpie_tts_capi_*`) is
+exported directly by the upstream shared library, so there is no local C shim.
+
+## Model configuration
+
+The model path points at the single GGUF:
+
+```yaml
+name: magpie-tts-cpp
+backend: magpie-tts-cpp
+parameters:
+  model: magpie-tts-multilingual-357m-q8_0.gguf
+known_usecases:
+  - tts
+options:
+  - "speaker:Aria"   # optional default voice (Aria, Jason, John, Leo, Sofia, or 0-4)
+  - "language:en"    # optional default language
+```
+
+GGUFs live at
+[mudler/magpie-tts.cpp-gguf](https://huggingface.co/mudler/magpie-tts.cpp-gguf)
+(q8_0 recommended: near-lossless, ~624 MB, fastest decode).
+
+## Voices and languages
+
+Magpie has 5 baked speakers - `Aria`, `Jason`, `John`, `Leo`, `Sofia` - and no
+voice cloning. The request `voice` accepts the names case-insensitively or the
+indices `0`-`4`; empty selects Aria. Languages: `en`, `es`, `de`, `fr`, `it`,
+`pt-BR`, `hi`, `vi`, `ko`, `ar-AE`, `ar-SA`, `ar-MSA` (case-insensitive;
+default `en`).
+
+## API example
+
+```bash
+curl http://localhost:8080/v1/audio/speech \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "magpie-tts-cpp",
+    "input": "Hello world, this is a test of the text to speech system.",
+    "voice": "sofia",
+    "language": "en"
+  }' \
+  --output speech.wav
+```
+
+## Native end-to-end test
+
+The labeled test loads a real GGUF, synthesizes WAVs (verifying rate, layout
+and non-silence), and exercises the streaming path:
+
+```bash
+make -C backend/go/magpie-tts-cpp magpie-tts-cpp
+
+MAGPIETTS_MODEL=/path/to/magpie-tts-multilingual-357m-q8_0.gguf \
+MAGPIETTS_LIBRARY=backend/go/magpie-tts-cpp/libgomagpiettscpp-fallback.so \
+  go test ./backend/go/magpie-tts-cpp -ginkgo.label-filter=e2e
+```
+
+`bash test.sh` does the same and auto-downloads the q8_0 GGUF when
+`MAGPIETTS_MODEL` is unset.

backend/go/magpie-tts-cpp/audio.go

@@ -0,0 +1,93 @@
+package main
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"os"
+
+	"github.com/go-audio/audio"
+	"github.com/go-audio/wav"
+)
+
+// magpieSampleRate is the fixed Magpie TTS Multilingual (NanoCodec) output
+// rate: 22.05 kHz.
+const magpieSampleRate = 22050
+
+// magpieChannels is the fixed output layout: mono.
+const magpieChannels = 1
+
+// wavHeaderMono returns a 44-byte WAV header for a streaming 16-bit mono PCM
+// stream at 22050 Hz, with placeholder (0xFFFFFFFF) sizes since the total
+// length is unknown up front. Emitted as the first chunk of TTSStream so the
+// HTTP layer receives a self-describing WAV.
+func wavHeaderMono() []byte {
+	const blockAlign = magpieChannels * 2 // 16-bit mono
+	var buf bytes.Buffer
+	w := func(v any) { _ = binary.Write(&buf, binary.LittleEndian, v) }
+	buf.WriteString("RIFF")
+	w(uint32(0xFFFFFFFF))
+	buf.WriteString("WAVE")
+	buf.WriteString("fmt ")
+	w(uint32(16))                            // Subchunk1Size
+	w(uint16(1))                             // PCM
+	w(uint16(magpieChannels))                // mono
+	w(uint32(magpieSampleRate))              // sample rate
+	w(uint32(magpieSampleRate * blockAlign)) // byte rate = SR * blockAlign
+	w(uint16(blockAlign))                    // block align
+	w(uint16(16))                            // bits per sample
+	buf.WriteString("data")
+	w(uint32(0xFFFFFFFF))
+	return buf.Bytes()
+}
+
+// floatToPCM16LE clamps each sample to [-1,1] and encodes it as little-endian
+// signed 16-bit PCM.
+func floatToPCM16LE(samples []float32) []byte {
+	out := make([]byte, len(samples)*2)
+	for i, s := range samples {
+		if s > 1 {
+			s = 1
+		} else if s < -1 {
+			s = -1
+		}
+		v := int16(s * 32767)
+		out[i*2] = byte(v)        // #nosec G115 -- intentional little-endian split of a clamped int16
+		out[i*2+1] = byte(v >> 8) // #nosec G115 -- high byte of the same clamped int16
+	}
+	return out
+}
+
+// writeWAVMono writes float samples as a finalized 16-bit mono WAV at
+// 22050 Hz.
+func writeWAVMono(dst string, samples []float32) error {
+	f, err := os.Create(dst) // #nosec G304 -- dst is the server-chosen output path from the TTS request, not user-traversable
+	if err != nil {
+		return fmt.Errorf("magpie-tts: create %q: %w", dst, err)
+	}
+	enc := wav.NewEncoder(f, magpieSampleRate, 16, magpieChannels, 1)
+	ints := make([]int, len(samples))
+	for i, s := range samples {
+		if s > 1 {
+			s = 1
+		} else if s < -1 {
+			s = -1
+		}
+		ints[i] = int(s * 32767)
+	}
+	b := &audio.IntBuffer{
+		Format:         &audio.Format{NumChannels: magpieChannels, SampleRate: magpieSampleRate},
+		Data:           ints,
+		SourceBitDepth: 16,
+	}
+	if err := enc.Write(b); err != nil {
+		_ = enc.Close()
+		_ = f.Close()
+		return fmt.Errorf("magpie-tts: encode WAV: %w", err)
+	}
+	if err := enc.Close(); err != nil {
+		_ = f.Close()
+		return fmt.Errorf("magpie-tts: finalize WAV: %w", err)
+	}
+	return f.Close()
+}

backend/go/magpie-tts-cpp/e2e_test.go

@@ -0,0 +1,127 @@
+package main
+
+import (
+	"encoding/binary"
+	"math"
+	"os"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func ttsReq(text, voice, lang, dst string) *pb.TTSRequest {
+	r := &pb.TTSRequest{Text: text, Voice: voice, Dst: dst}
+	if lang != "" {
+		r.Language = &lang
+	}
+	return r
+}
+
+// wavRMS parses a 16-bit PCM WAV file and returns (sampleRate, channels, RMS
+// of the normalized samples).
+func wavRMS(path string) (int, int, float64) {
+	data, err := os.ReadFile(path) // #nosec G304 -- test-owned temp file
+	Expect(err).ToNot(HaveOccurred())
+	Expect(len(data)).To(BeNumerically(">", 44))
+	Expect(string(data[0:4])).To(Equal("RIFF"))
+	Expect(string(data[8:12])).To(Equal("WAVE"))
+	channels := int(binary.LittleEndian.Uint16(data[22:24]))
+	rate := int(binary.LittleEndian.Uint32(data[24:28]))
+	// Find the data chunk (go-audio writes fmt first, data after).
+	off := 12
+	for off+8 <= len(data) {
+		id := string(data[off : off+4])
+		sz := int(binary.LittleEndian.Uint32(data[off+4 : off+8]))
+		if id == "data" {
+			pcm := data[off+8:]
+			if sz < len(pcm) {
+				pcm = pcm[:sz]
+			}
+			var sum float64
+			n := len(pcm) / 2
+			for i := 0; i < n; i++ {
+				s := float64(int16(binary.LittleEndian.Uint16(pcm[i*2:]))) / 32768.0
+				sum += s * s
+			}
+			Expect(n).To(BeNumerically(">", 0))
+			return rate, channels, math.Sqrt(sum / float64(n))
+		}
+		off += 8 + sz
+	}
+	Fail("no data chunk found in " + path)
+	return 0, 0, 0
+}
+
+var _ = Describe("magpie-tts-cpp e2e", Label("e2e"), func() {
+	var (
+		loaded  bool
+		backend *MagpieTtsCpp
+	)
+
+	BeforeEach(func() {
+		modelPath := os.Getenv("MAGPIETTS_MODEL")
+		if modelPath == "" {
+			Skip("MAGPIETTS_MODEL not set; skipping e2e")
+		}
+		if !loaded {
+			lib := os.Getenv("MAGPIETTS_LIBRARY")
+			if lib == "" {
+				lib = "./libgomagpiettscpp-fallback.so"
+			}
+			h, err := purego.Dlopen(lib, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+			Expect(err).ToNot(HaveOccurred())
+			purego.RegisterLibFunc(&CppAbiVersion, h, "magpie_tts_capi_abi_version")
+			purego.RegisterLibFunc(&CppLoad, h, "magpie_tts_capi_load")
+			purego.RegisterLibFunc(&CppFree, h, "magpie_tts_capi_free")
+			purego.RegisterLibFunc(&CppSynthesize, h, "magpie_tts_capi_synthesize")
+			purego.RegisterLibFunc(&CppFreeAudio, h, "magpie_tts_capi_free_audio")
+			purego.RegisterLibFunc(&CppLastError, h, "magpie_tts_capi_last_error")
+
+			backend = &MagpieTtsCpp{}
+			Expect(backend.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
+			loaded = true
+		}
+	})
+
+	It("synthesizes a non-silent 22.05 kHz mono WAV via TTS", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		Expect(backend.TTS(ttsReq("Hello world, this is a test.", "Aria", "en", dst))).To(Succeed())
+		rate, channels, rms := wavRMS(dst)
+		Expect(rate).To(Equal(22050))
+		Expect(channels).To(Equal(1))
+		Expect(rms).To(BeNumerically(">", 0.01), "audio should not be silent")
+	})
+
+	It("accepts a case-insensitive voice and a speaker index", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		Expect(backend.TTS(ttsReq("Short test.", "sofia", "en", dst))).To(Succeed())
+		Expect(backend.TTS(ttsReq("Short test.", "1", "en", dst))).To(Succeed())
+	})
+
+	It("rejects an unknown voice before reaching the engine", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		err := backend.TTS(ttsReq("Short test.", "not-a-speaker", "en", dst))
+		Expect(err).To(MatchError(ContainSubstring("unknown voice")))
+	})
+
+	It("streams a self-describing WAV via TTSStream", func() {
+		results := make(chan []byte, 4096)
+		done := make(chan error, 1)
+		go func() { done <- backend.TTSStream(ttsReq("Hello there, streaming test.", "", "", ""), results) }()
+
+		var chunks int
+		var first []byte
+		for c := range results {
+			if chunks == 0 {
+				first = c
+			}
+			chunks++
+		}
+		Expect(<-done).ToNot(HaveOccurred())
+		Expect(chunks).To(BeNumerically(">=", 2))
+		Expect(string(first[0:4])).To(Equal("RIFF"))
+		Expect(string(first[8:12])).To(Equal("WAVE"))
+	})
+})

backend/go/magpie-tts-cpp/gomagpiettscpp.go

@@ -0,0 +1,147 @@
+package main
+
+import (
+	"fmt"
+	"path/filepath"
+	"unsafe"
+
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// capiABIVersion is the magpie_tts_capi.h surface this backend binds. Bumped
+// upstream on any breaking signature/semantics change; refuse to run on a
+// mismatch instead of crashing inside a miscompiled call.
+const capiABIVersion = 1
+
+var (
+	// magpie_tts_capi_abi_version() int
+	CppAbiVersion func() int
+	// magpie_tts_capi_load(gguf_path) -> ctx (NULL on failure)
+	CppLoad func(path string) uintptr
+	// magpie_tts_capi_free(ctx)
+	CppFree func(ctx uintptr)
+	// magpie_tts_capi_synthesize(ctx, text, language, speaker, out_n) -> float*
+	// 22050 Hz mono f32 PCM in [-1,1]; NULL on failure (see last_error).
+	CppSynthesize func(ctx uintptr, text, language, speaker string, outN unsafe.Pointer) uintptr
+	// magpie_tts_capi_free_audio(ptr)
+	CppFreeAudio func(ptr uintptr)
+	// magpie_tts_capi_last_error(ctx) -> const char* (ctx-owned, "" if none)
+	CppLastError func(ctx uintptr) string
+)
+
+// MagpieTtsCpp serves the Magpie TTS Multilingual 357M GGUF through the
+// magpie-tts.cpp C-API. The context is stateful (per-call error buffer, reused
+// graph allocator) and NOT safe for concurrent synthesize calls, so
+// base.SingleThread serializes everything behind the server-level lock.
+type MagpieTtsCpp struct {
+	base.SingleThread
+	ctx  uintptr
+	opts loadOptions
+}
+
+func (m *MagpieTtsCpp) Load(opts *pb.ModelOptions) error {
+	model := opts.ModelFile
+	if model == "" {
+		model = opts.ModelPath
+	}
+	if !filepath.IsAbs(model) && opts.ModelPath != "" {
+		model = filepath.Join(opts.ModelPath, model)
+	}
+
+	m.opts = parseOptions(opts.Options)
+
+	if abi := CppAbiVersion(); abi != capiABIVersion {
+		return fmt.Errorf("magpie-tts: C-API ABI mismatch: library reports v%d, backend built for v%d", abi, capiABIVersion)
+	}
+
+	xlog.Info("[magpie-tts-cpp] Load", "model", model)
+
+	ctx := CppLoad(model)
+	if ctx == 0 {
+		// Load failures have no context to query last_error on; the C side
+		// logs the reason to stderr.
+		return fmt.Errorf("magpie-tts: failed to load model %q", model)
+	}
+	m.ctx = ctx
+	return nil
+}
+
+// lastError surfaces the context's last error, falling back to a generic
+// message when the C side left it empty.
+func (m *MagpieTtsCpp) lastError() string {
+	if m.ctx == 0 {
+		return "no model loaded"
+	}
+	if e := CppLastError(m.ctx); e != "" {
+		return e
+	}
+	return "unknown error"
+}
+
+// synthesize runs one C-API synthesis and copies the PCM out of C memory.
+func (m *MagpieTtsCpp) synthesize(req *pb.TTSRequest) ([]float32, error) {
+	if m.ctx == 0 {
+		return nil, fmt.Errorf("magpie-tts: no model loaded")
+	}
+	if req.Text == "" {
+		return nil, fmt.Errorf("magpie-tts: TTS requires text")
+	}
+	speaker, err := resolveSpeaker(req.Voice, m.opts.speaker)
+	if err != nil {
+		return nil, err
+	}
+	lang := resolveLanguage(req.Language, m.opts.language)
+
+	var n int32
+	ptr := CppSynthesize(m.ctx, req.Text, lang, speaker, unsafe.Pointer(&n)) // #nosec G103 -- out-param for the purego-bound C-API
+	if ptr == 0 {
+		return nil, fmt.Errorf("magpie-tts: synthesis failed: %s", m.lastError())
+	}
+	// Register the free as soon as we own a non-null buffer, so the n<=0 guard
+	// below cannot leak it (defensive: the C contract returns NULL on failure).
+	defer CppFreeAudio(ptr)
+	if n <= 0 {
+		return nil, fmt.Errorf("magpie-tts: synthesis produced no samples")
+	}
+	//nolint:govet // C-allocated PCM, copied out before free
+	src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) // #nosec G103 -- C-allocated PCM, copied out before free
+	out := make([]float32, int(n))
+	copy(out, src)
+	return out, nil
+}
+
+func (m *MagpieTtsCpp) TTS(req *pb.TTSRequest) error {
+	if req.Dst == "" {
+		return fmt.Errorf("magpie-tts: TTS requires a destination path")
+	}
+	samples, err := m.synthesize(req)
+	if err != nil {
+		return err
+	}
+	return writeWAVMono(req.Dst, samples)
+}
+
+// TTSStream synthesizes one-shot (the magpie C-API has no streaming call) and
+// then emits a self-describing mono WAV: a header chunk followed by the PCM in
+// fixed-size slices, so the HTTP layer still receives a streamed WAV (the gRPC
+// TTSStream path never sets Message, so the backend owns the header - see
+// core/backend/tts.go:ModelTTSStream).
+func (m *MagpieTtsCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
+	defer close(results)
+	samples, err := m.synthesize(req)
+	if err != nil {
+		return err
+	}
+	results <- wavHeaderMono()
+	const sampleChunk = 4096 // mono samples per emitted chunk
+	for off := 0; off < len(samples); off += sampleChunk {
+		end := off + sampleChunk
+		if end > len(samples) {
+			end = len(samples)
+		}
+		results <- floatToPCM16LE(samples[off:end])
+	}
+	return nil
+}

backend/go/magpie-tts-cpp/magpiettscpp_test.go

@@ -0,0 +1,115 @@
+package main
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestMagpieTtsCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "magpie-tts-cpp suite")
+}
+
+var _ = Describe("resolveSpeaker", func() {
+	It("canonicalizes case-insensitive names", func() {
+		for in, want := range map[string]string{
+			"aria": "Aria", "JASON": "Jason", "john": "John",
+			"Leo": "Leo", "sofia": "Sofia",
+		} {
+			got, err := resolveSpeaker(in, "")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(got).To(Equal(want))
+		}
+	})
+	It("accepts indices 0-4", func() {
+		for in, want := range map[string]string{
+			"0": "Aria", "1": "Jason", "2": "John", "3": "Leo", "4": "Sofia",
+		} {
+			got, err := resolveSpeaker(in, "")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(got).To(Equal(want))
+		}
+	})
+	It("selects the engine default on empty", func() {
+		got, err := resolveSpeaker("", "")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(BeEmpty())
+	})
+	It("falls back to the model-level default speaker", func() {
+		got, err := resolveSpeaker("", "sofia")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("Sofia"))
+	})
+	It("prefers the request voice over the fallback", func() {
+		got, err := resolveSpeaker("leo", "sofia")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("Leo"))
+	})
+	It("rejects out-of-range indices", func() {
+		_, err := resolveSpeaker("5", "")
+		Expect(err).To(HaveOccurred())
+		_, err = resolveSpeaker("-1", "")
+		Expect(err).To(HaveOccurred())
+	})
+	It("rejects unknown names with the valid choices", func() {
+		_, err := resolveSpeaker("serena", "")
+		Expect(err).To(MatchError(ContainSubstring("Aria, Jason, John, Leo, Sofia")))
+	})
+})
+
+var _ = Describe("resolveLanguage", func() {
+	strp := func(s string) *string { return &s }
+
+	It("defaults to empty (engine picks en)", func() {
+		Expect(resolveLanguage(nil, "")).To(BeEmpty())
+	})
+	It("canonicalizes case for known codes", func() {
+		Expect(resolveLanguage(strp("EN"), "")).To(Equal("en"))
+		Expect(resolveLanguage(strp("pt-br"), "")).To(Equal("pt-BR"))
+		Expect(resolveLanguage(strp("AR-MSA"), "")).To(Equal("ar-MSA"))
+	})
+	It("falls back to the model-level default language", func() {
+		Expect(resolveLanguage(nil, "de")).To(Equal("de"))
+		Expect(resolveLanguage(strp(""), "PT-BR")).To(Equal("pt-BR"))
+	})
+	It("prefers the request language over the fallback", func() {
+		Expect(resolveLanguage(strp("it"), "de")).To(Equal("it"))
+	})
+	It("passes unknown codes through verbatim", func() {
+		Expect(resolveLanguage(strp("zh"), "")).To(Equal("zh"))
+	})
+})
+
+var _ = Describe("parseOptions", func() {
+	It("reads speaker and language defaults", func() {
+		o := parseOptions([]string{"speaker:Jason", "language:de"})
+		Expect(o.speaker).To(Equal("Jason"))
+		Expect(o.language).To(Equal("de"))
+	})
+	It("accepts the voice/lang aliases and ignores unknown keys", func() {
+		o := parseOptions([]string{"voice: sofia ", "lang: pt-BR", "bogus:1", "novalue"})
+		Expect(o.speaker).To(Equal("sofia"))
+		Expect(o.language).To(Equal("pt-BR"))
+	})
+})
+
+var _ = Describe("audio encoding", func() {
+	It("emits a well-formed streaming mono WAV header", func() {
+		h := wavHeaderMono()
+		Expect(h).To(HaveLen(44))
+		Expect(string(h[0:4])).To(Equal("RIFF"))
+		Expect(string(h[8:12])).To(Equal("WAVE"))
+		// channels (offset 22) == 1, sample rate (offset 24) == 22050
+		Expect(int(h[22]) | int(h[23])<<8).To(Equal(1))
+		Expect(int(h[24]) | int(h[25])<<8 | int(h[26])<<16 | int(h[27])<<24).To(Equal(22050))
+	})
+	It("clamps float PCM to int16", func() {
+		b := floatToPCM16LE([]float32{2, -2, 0})
+		Expect(b).To(HaveLen(6))
+		Expect(int16(uint16(b[0]) | uint16(b[1])<<8)).To(Equal(int16(32767)))
+		Expect(int16(uint16(b[2]) | uint16(b[3])<<8)).To(Equal(int16(-32767)))
+		Expect(int16(uint16(b[4]) | uint16(b[5])<<8)).To(Equal(int16(0)))
+	})
+})

backend/go/magpie-tts-cpp/main.go

@@ -0,0 +1,54 @@
+package main
+
+// Note: this is started internally by LocalAI and a server is allocated for each model
+import (
+	"flag"
+	"os"
+	"runtime"
+
+	"github.com/ebitengine/purego"
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+type LibFuncs struct {
+	FuncPtr any
+	Name    string
+}
+
+func main() {
+	libName := os.Getenv("MAGPIETTS_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "./libgomagpiettscpp-fallback.dylib"
+		} else {
+			libName = "./libgomagpiettscpp-fallback.so"
+		}
+	}
+
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		panic(err)
+	}
+
+	libFuncs := []LibFuncs{
+		{&CppAbiVersion, "magpie_tts_capi_abi_version"},
+		{&CppLoad, "magpie_tts_capi_load"},
+		{&CppFree, "magpie_tts_capi_free"},
+		{&CppSynthesize, "magpie_tts_capi_synthesize"},
+		{&CppFreeAudio, "magpie_tts_capi_free_audio"},
+		{&CppLastError, "magpie_tts_capi_last_error"},
+	}
+	for _, lf := range libFuncs {
+		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &MagpieTtsCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/magpie-tts-cpp/options.go

@@ -0,0 +1,108 @@
+package main
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+)
+
+// loadOptions holds the parsed model-level options. Magpie is a single
+// self-contained GGUF (model + codec + tokenizer + G2P dictionaries), so the
+// options only cover synthesis defaults.
+type loadOptions struct {
+	// speaker is the default baked speaker when a request has no voice.
+	speaker string
+	// language is the default language when a request has none ("" = engine
+	// default, which is "en").
+	language string
+}
+
+func splitOption(o string) (key, value string, ok bool) {
+	i := strings.Index(o, ":")
+	if i < 0 {
+		return "", "", false
+	}
+	return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true
+}
+
+// parseOptions reads the backend "key:value" option slice. Unknown keys are
+// ignored.
+func parseOptions(opts []string) loadOptions {
+	var o loadOptions
+	for _, oo := range opts {
+		key, value, ok := splitOption(oo)
+		if !ok {
+			continue
+		}
+		switch key {
+		case "speaker", "voice":
+			o.speaker = value
+		case "language", "lang":
+			o.language = value
+		}
+	}
+	return o
+}
+
+// magpieSpeakers are the baked speakers of Magpie TTS Multilingual 357M, in
+// index order (the engine matches names exactly, so the Go side canonicalizes
+// case-insensitive names and 0-4 indices to these strings).
+var magpieSpeakers = []string{"Aria", "Jason", "John", "Leo", "Sofia"}
+
+// resolveSpeaker maps the request voice (falling back to the model-level
+// default) onto a canonical baked speaker name. Accepted forms:
+// case-insensitive names (aria, JASON, ...) and indices 0-4. Empty selects the
+// engine default (speaker 0, Aria). Anything else is rejected with the valid
+// choices, instead of surfacing the engine's late error.
+func resolveSpeaker(voice, fallback string) (string, error) {
+	v := strings.TrimSpace(voice)
+	if v == "" {
+		v = strings.TrimSpace(fallback)
+	}
+	if v == "" {
+		return "", nil
+	}
+	if idx, err := strconv.Atoi(v); err == nil {
+		if idx < 0 || idx >= len(magpieSpeakers) {
+			return "", fmt.Errorf("magpie-tts: speaker index %d out of range 0-%d", idx, len(magpieSpeakers)-1)
+		}
+		return magpieSpeakers[idx], nil
+	}
+	for _, s := range magpieSpeakers {
+		if strings.EqualFold(s, v) {
+			return s, nil
+		}
+	}
+	return "", fmt.Errorf("magpie-tts: unknown voice %q (valid: %s, or 0-%d)",
+		voice, strings.Join(magpieSpeakers, ", "), len(magpieSpeakers)-1)
+}
+
+// magpieLanguages are the canonical language codes the tokenizer's language
+// map knows (exact-match on the C side), keyed by their lowercase form so
+// requests can be case-insensitive.
+var magpieLanguages = map[string]string{
+	"en": "en", "es": "es", "de": "de", "fr": "fr", "it": "it",
+	"pt-br": "pt-BR", "hi": "hi", "vi": "vi", "ko": "ko",
+	"ar-ae": "ar-AE", "ar-sa": "ar-SA", "ar-msa": "ar-MSA",
+}
+
+// resolveLanguage picks the request language, else the model-level default,
+// else "" (the engine defaults to "en"), canonicalizing case for the known
+// codes. Unknown codes pass through verbatim so the engine reports them with
+// its own exact-vocabulary error.
+func resolveLanguage(reqLang *string, fallback string) string {
+	l := ""
+	if reqLang != nil {
+		l = strings.TrimSpace(*reqLang)
+	}
+	if l == "" {
+		l = strings.TrimSpace(fallback)
+	}
+	if l == "" {
+		return ""
+	}
+	if canon, ok := magpieLanguages[strings.ToLower(l)]; ok {
+		return canon
+	}
+	return l
+}

backend/go/magpie-tts-cpp/package.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/magpie-tts-cpp $CURDIR/package/
+cp -fv $CURDIR/libgomagpiettscpp-*.so $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/libgomagpiettscpp-*.dylib $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/magpie-tts-cpp/run.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+echo "CPU info:"
+if [ "$(uname)" != "Darwin" ]; then
+	grep -e "model\sname" /proc/cpuinfo | head -1
+	grep -e "flags" /proc/cpuinfo | head -1
+fi
+
+if [ "$(uname)" = "Darwin" ]; then
+	# macOS: single dylib variant (Metal or Accelerate)
+	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.dylib"
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.so"
+
+	if grep -q -e "\savx\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX    found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx.so"
+		fi
+	fi
+
+	if grep -q -e "\savx2\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX2   found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx2.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx2.so"
+		fi
+	fi
+
+	# Check avx 512
+	if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX512F found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx512.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx512.so"
+		fi
+	fi
+
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+fi
+
+export MAGPIETTS_LIBRARY=$LIBRARY
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LIBRARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/magpie-tts-cpp "$@"
+fi
+
+echo "Using library: $LIBRARY"
+exec "$CURDIR"/magpie-tts-cpp "$@"

backend/go/magpie-tts-cpp/test.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+echo "Running magpie-tts-cpp backend tests..."
+
+# Auto-download the q8_0 GGUF only when MAGPIETTS_MODEL is not set.
+if [ -z "$MAGPIETTS_MODEL" ]; then
+    MODEL_DIR="./magpie-tts-models"
+    mkdir -p "$MODEL_DIR"
+    REPO_ID="mudler/magpie-tts.cpp-gguf"
+    BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
+    FILE="magpie-tts-multilingual-357m-q8_0.gguf"
+    dest="${MODEL_DIR}/${FILE}"
+    if [ -f "${dest}" ]; then
+        echo "  [skip] ${FILE}"
+    else
+        echo "  [download] ${FILE}..."
+        curl -L -o "${dest}" "${BASE_URL}/${FILE}" --progress-bar
+    fi
+    export MAGPIETTS_MODEL="${dest}"
+fi
+
+go test -v -timeout 1200s .
+
+echo "All magpie-tts-cpp tests passed."

backend/go/omnivoice-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # omnivoice.cpp version
 OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=f39cc4a3af988091f662313b336dddf8c83a3fb5
+OMNIVOICE_VERSION?=4f33af825d66e6ef1cb185e87b4589cacf747291
 SO_TARGET?=libgomnivoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/qwen3-tts-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # qwentts.cpp version
 QWEN3TTS_REPO?=https://github.com/ServeurpersoCom/qwentts.cpp
-QWEN3TTS_CPP_VERSION?=e93292bee1778854ab7dcb2d325ffe531fef910f
+QWEN3TTS_CPP_VERSION?=35ebe5376b82a0a59d008586d55bbe623d449011
 SO_TARGET?=libgoqwen3ttscpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=ea4e566ccffa10f853ecc3f29e74b1820bc91beb
+STABLEDIFFUSION_GGML_VERSION?=2d0385ba85af358f7115dda608a63eafd9de7ffd
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/vllm-cpp/.gitignore

@@ -0,0 +1,6 @@
+sources/
+build/
+package/
+vllm-cpp
+libvllm.so
+libvllm.dylib

backend/go/vllm-cpp/Makefile

@@ -0,0 +1,102 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+# nproc doesn't exist on the macOS runners: an empty JOBS turns `-j$(JOBS)`
+# into bare `-j` (unlimited clang jobs), which swap-thrashes the 3-core Mac
+# until the 6h GHA timeout. Fall back to sysctl there, then to a constant.
+JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+# vllm.cpp version
+VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
+VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845
+
+# The backend consumes only the stable C ABI (libvllm + include/vllm.h), so the
+# server, examples and tests of the engine are never built here.
+CMAKE_ARGS+=-DVLLM_CPP_SERVER=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_BUILD_EXAMPLES=OFF
+CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
+
+# vllm.cpp sets no global -march: SIMD tiers are per-file with runtime dispatch,
+# so ONE portable library serves every CPU of the target arch (unlike the
+# ggml-based backends and their avx/avx2/avx512 variant builds).
+UNAME_M := $(shell uname -m)
+
+ifeq ($(BUILD_TYPE),cublas)
+	# Blackwell-family targets only: other CUDA arches are build-supported
+	# upstream but have no runtime-proven fast path. amd64 gets the consumer
+	# (120a) + GB10 (121a) fat binary; arm64 CUDA (l4t-style images, DGX
+	# Spark) is GB10 only. Triton-AOT GDN cubins are vendored per-arch, no
+	# Python needed to consume them.
+	ifeq ($(UNAME_M),x86_64)
+		# NO -DVLLM_CPP_TRITON on fat builds: the vendored Triton-AOT cubin
+		# trees are per-arch and the engine refuses a multi-arch build unless
+		# pinned to one tree (unsound for the other arch). The non-AOT GDN
+		# path serves the fat binary; single-arch builds keep the cubins.
+		#
+		# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks
+		# compute_121a (GB10) and its ptxas rejects the sm_120a NVFP4 MMA
+		# kernels ("Vector type too large"), so no cuda-12 variant is shipped.
+		ifeq ($(CUDA_MAJOR_VERSION),12)
+      $(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
+		endif
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=120a;121a"
+	else
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
+	endif
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF
+else ifeq ($(BUILD_TYPE),metal)
+	CMAKE_ARGS+=-DVLLM_CPP_METAL=ON
+else
+	CMAKE_ARGS+=-DVLLM_CPP_CUDA=OFF
+endif
+
+UNAME_S := $(shell uname -s)
+ifeq ($(UNAME_S),Darwin)
+	LIB=libvllm.dylib
+else
+	LIB=libvllm.so
+endif
+
+sources/vllm.cpp:
+	mkdir -p sources/vllm.cpp
+	cd sources/vllm.cpp && \
+	git init && \
+	git remote add origin $(VLLM_CPP_REPO) && \
+	git fetch --depth 1 origin $(VLLM_CPP_VERSION) && \
+	git checkout FETCH_HEAD
+
+$(LIB): sources/vllm.cpp
+	mkdir -p build && \
+	cd build && \
+	cmake ../sources/vllm.cpp $(CMAKE_ARGS) && \
+	cmake --build . --config Release -j$(JOBS) --target vllm_shared
+	cp -fL build/$(LIB) ./$(LIB)
+
+vllm-cpp: main.go govllmcpp.go backend.go options.go $(LIB)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vllm-cpp ./
+
+package: vllm-cpp
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf libvllm.so libvllm.dylib package sources/vllm.cpp vllm-cpp
+
+purge:
+	rm -rf build
+
+.NOTPARALLEL:
+
+# The unit specs are pure Go (struct mirrors, option mapping, load
+# validation): no libvllm build is needed. The e2e specs skip unless
+# VLLM_CPP_MODEL points at a real model (then build the lib first).
+test:
+	@echo "Running vllm-cpp tests..."
+	bash test.sh
+	@echo "vllm-cpp tests completed."
+
+all: vllm-cpp package

backend/go/vllm-cpp/README.md

@@ -0,0 +1,68 @@
+# vllm-cpp backend
+
+LocalAI text-generation backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
+the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching,
+safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
+inference time.
+
+The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
+ABI v2) through purego:
+
+- `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
+  directory (`config.json` + safetensors). `context_size` maps to
+  `max_model_len`; `options: ["block_size:<n>", "num_blocks:<n>",
+  "max_num_seqs:<n>"]` size the KV cache and scheduler admission.
+- `Predict` -> `vllm_complete` (blocking).
+- `PredictStream` -> `vllm_complete_stream`; concurrent gRPC requests batch
+  continuously in the engine's shared AsyncLLM scheduler.
+- Chat / tool calling rides the SAME code path as the llama.cpp autoparser:
+  with `use_tokenizer_template: true` the backend implements
+  `PredictRich`/`PredictStreamRich` over the ABI v3 chat entry points
+  (`vllm_chat` / `vllm_chat_stream`). The ENGINE applies the model's chat
+  template (GGUF `tokenizer.chat_template` or `tokenizer_config.json`),
+  decides when a tool call engages (`tool_choice: auto` lowers to a LAZY
+  structural-tag decode constraint; `required`/named force one), parses tool
+  calls with its streaming Hermes-style parser, and the backend maps each
+  `chat.completion.chunk` onto `ChatDelta`/`ToolCallDelta` protos.
+- Without structured messages the plain path applies:
+  `PredictOptions.Grammar` -> the ABI's `structured_grammar` (GBNF) for
+  LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
+  choice constraints are also exposed by the ABI.
+
+## Hardware coverage
+
+The CUDA builds require the CUDA 13 toolchain and target Blackwell only:
+`sm_120a` + `sm_121a` on x86_64, `sm_121a` (GB10 / DGX Spark) on arm64. CUDA
+12.x nvcc cannot compile the Blackwell fp4 kernels, so no CUDA 12 variant is
+shipped and `backend/index.yaml` maps the `nvidia-cuda-12` /
+`nvidia-l4t-cuda-12` capabilities at the CPU build. Practically:
+
+| Host | Installed build |
+|---|---|
+| x86_64 + CUDA 13 | `cuda13-vllm-cpp` |
+| DGX Spark / GB10 (JetPack 7, CUDA 13) | `nvidia-l4t-arm64-vllm-cpp` |
+| Jetson AGX Orin (sm_87, JetPack 6, CUDA 12) | `cpu-vllm-cpp` |
+| Apple Silicon | `metal-vllm-cpp` |
+| Anything else | `vulkan-vllm-cpp` or `cpu-vllm-cpp` |
+
+The capability a host reports comes from `/run/localai/capability` inside the
+LocalAI container, which the image bakes in at build time (see `Dockerfile`).
+A DGX Spark running the CUDA 12 `-nvidia-l4t-arm64` image therefore reports
+`nvidia-l4t-cuda-12` and gets the CPU build; use the `-nvidia-l4t-arm64-cuda-13`
+image, or set `LOCALAI_FORCE_META_BACKEND_CAPABILITY=nvidia-l4t-cuda-13`, to
+get the GPU one.
+
+Model config example:
+
+```yaml
+name: qwen3-vllm
+backend: vllm-cpp
+context_size: 8192
+parameters:
+  model: Qwen3-4B   # model dir (safetensors) or .gguf file
+options:
+- max_num_seqs:16
+```
+
+Testing: `make test` runs the unit specs; export `VLLM_CPP_MODEL=<model>` (and
+optionally `VLLM_CPP_LIBRARY=<libvllm path>`) to enable the e2e specs.

backend/go/vllm-cpp/backend.go

@@ -0,0 +1,245 @@
+package main
+
+// LocalAI gRPC backend over the vllm.cpp C ABI.
+//
+// Predict maps to the blocking vllm_complete; PredictStream maps to
+// vllm_complete_stream, whose per-delta C callback bridges into the gRPC
+// stream channel. Concurrent calls are intentional: every completion entry
+// point submits into the engine's shared AsyncLLM scheduler, so parallel
+// LocalAI requests batch continuously inside the engine (the reason this
+// backend embeds base.Base and not base.SingleThread).
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+type VllmCpp struct {
+	base.Base
+
+	engine uintptr
+	opts   loadOptions
+}
+
+// Stream registry: the per-request bridge between the C token callback and
+// the gRPC stream channel, keyed by an integer handle round-tripped through
+// the C user_data pointer (never a Go pointer across the ABI). The host gRPC
+// server drains the channel even after a client disconnect, so sends here
+// cannot wedge the engine's delivery loop.
+var (
+	streamsMu   sync.Mutex
+	streams     = map[uintptr]chan string{}
+	streamNext  uintptr
+	tokenCbOnce sync.Once
+	tokenCbPtr  uintptr
+)
+
+// tokenCallback is the single C-shared callback for every stream; it
+// dispatches on the user_data handle. Returning 0 aborts the in-flight
+// request (vllm_token_callback contract).
+func tokenCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	streamsMu.Lock()
+	results := streams[userData]
+	streamsMu.Unlock()
+	if results == nil {
+		return 0 // unknown request: stop generation.
+	}
+	if text := goString(delta); text != "" {
+		results <- text
+	}
+	return 1
+}
+
+func registerStream(results chan string) uintptr {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	streamNext++
+	streams[streamNext] = results
+	return streamNext
+}
+
+func unregisterStream(h uintptr) {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	delete(streams, h)
+}
+
+// validModelPath enforces the greedy-probe rule: when a model config has no
+// explicit backend, the loader probes every backend with the model name, so
+// Load must refuse anything vllm.cpp cannot serve (a GGUF file, or a HF-style
+// directory with config.json + safetensors).
+func validModelPath(model string) error {
+	info, err := os.Stat(model)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: model path %q not found: %w", model, err)
+	}
+	if info.IsDir() {
+		if _, err := os.Stat(filepath.Join(model, "config.json")); err != nil {
+			return fmt.Errorf("vllm-cpp: model dir %q has no config.json", model)
+		}
+		return nil
+	}
+	if strings.EqualFold(filepath.Ext(model), ".gguf") {
+		return nil
+	}
+	return fmt.Errorf("vllm-cpp: model %q is neither a .gguf file nor a config.json model dir", model)
+}
+
+func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
+	model := opts.ModelFile
+	if model == "" {
+		model = opts.ModelPath
+	}
+	if !filepath.IsAbs(model) && opts.ModelPath != "" {
+		model = filepath.Join(opts.ModelPath, model)
+	}
+	if err := validModelPath(model); err != nil {
+		return err
+	}
+
+	v.opts = parseOptions(opts)
+
+	mp := defaultModelParams()
+	if v.opts.blockSize > 0 {
+		mp.BlockSize = v.opts.blockSize
+	}
+	if v.opts.numBlocks > 0 {
+		mp.NumBlocks = v.opts.numBlocks
+	}
+	if opts.ContextSize > 0 {
+		mp.MaxModelLen = opts.ContextSize
+	}
+	if v.opts.maxNumSeqs > 0 {
+		mp.MaxNumSeqs = v.opts.maxNumSeqs
+	}
+
+	modelC := cString(model)
+	mp.ModelPath = uintptr(unsafe.Pointer(&modelC[0])) // #nosec G103 -- borrowed by C for the load call only
+	var toolParserC, reasoningParserC []byte
+	if v.opts.toolParser != "" {
+		toolParserC = cString(v.opts.toolParser)
+		mp.ToolParser = uintptr(unsafe.Pointer(&toolParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+	if v.opts.reasoningParser != "" {
+		reasoningParserC = cString(v.opts.reasoningParser)
+		mp.ReasoningParser = uintptr(unsafe.Pointer(&reasoningParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+
+	xlog.Info("[vllm-cpp] Load", "model", model, "engine", vllmVersion(),
+		"blockSize", mp.BlockSize, "numBlocks", mp.NumBlocks,
+		"maxModelLen", mp.MaxModelLen, "maxNumSeqs", mp.MaxNumSeqs)
+
+	var engine uintptr
+	rc := vllmEngineLoad(unsafe.Pointer(&mp), unsafe.Pointer(&engine)) // #nosec G103 -- POD out-params
+	runtime.KeepAlive(modelC)
+	runtime.KeepAlive(toolParserC)
+	runtime.KeepAlive(reasoningParserC)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: engine load failed: %s", vllmLastError())
+	}
+	v.engine = engine
+	return nil
+}
+
+func (v *VllmCpp) Free() error {
+	if v.engine != 0 {
+		vllmEngineFree(v.engine)
+		v.engine = 0
+	}
+	return nil
+}
+
+// samplingFromPredict lowers PredictOptions into the C sampling POD plus the
+// backing buffers that must stay alive for the duration of the C call.
+func samplingFromPredict(opts *pb.PredictOptions) (sp cSamplingParams, keep []any) {
+	sp = defaultSamplingParams()
+	sp.Temperature = opts.Temperature
+	if opts.TopP > 0 {
+		sp.TopP = opts.TopP
+	}
+	if opts.TopK > 0 {
+		sp.TopK = opts.TopK
+	}
+	if opts.MinP > 0 {
+		sp.MinP = opts.MinP
+	}
+	if opts.Tokens > 0 {
+		sp.MaxTokens = opts.Tokens
+	} else {
+		sp.MaxTokens = 0 // unbounded; the engine caps at max_model_len.
+	}
+	if opts.Seed > 0 {
+		sp.Seed = uint64(opts.Seed)
+		sp.HasSeed = 1
+	}
+	sp.PresencePenalty = opts.PresencePenalty
+	sp.FrequencyPenalty = opts.FrequencyPenalty
+	if opts.Penalty > 0 {
+		sp.RepetitionPenalty = opts.Penalty
+	}
+	if opts.IgnoreEOS {
+		sp.IgnoreEOS = 1
+	}
+	if len(opts.StopPrompts) > 0 {
+		ptrs, backing := cStringArray(opts.StopPrompts)
+		sp.Stop = uintptr(unsafe.Pointer(&ptrs[0])) // #nosec G103 -- borrowed by C for the call only
+		sp.NStop = int32(len(ptrs))
+		keep = append(keep, ptrs, backing)
+	}
+	if opts.Grammar != "" {
+		g := cString(opts.Grammar)
+		sp.StructuredGrammar = uintptr(unsafe.Pointer(&g[0])) // #nosec G103 -- borrowed by C for the call only
+		keep = append(keep, g)
+	}
+	return sp, keep
+}
+
+func (v *VllmCpp) Predict(opts *pb.PredictOptions) (string, error) {
+	if v.engine == 0 {
+		return "", fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	sp, keep := samplingFromPredict(opts)
+	var out cCompletion
+	rc := vllmComplete(v.engine, opts.Prompt, unsafe.Pointer(&sp), unsafe.Pointer(&out)) // #nosec G103 -- POD in/out params
+	runtime.KeepAlive(keep)
+	if rc != vllmOK {
+		return "", fmt.Errorf("vllm-cpp: completion failed: %s", vllmLastError())
+	}
+	text := goString(out.Text)
+	vllmCompletionFree(unsafe.Pointer(&out)) // #nosec G103 -- frees out.Text
+	return text, nil
+}
+
+func (v *VllmCpp) PredictStream(opts *pb.PredictOptions, results chan string) error {
+	if v.engine == 0 {
+		close(results)
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	tokenCbOnce.Do(func() {
+		tokenCbPtr = purego.NewCallback(tokenCallback)
+	})
+
+	sp, keep := samplingFromPredict(opts)
+	handle := registerStream(results)
+
+	go func() {
+		defer close(results)
+		defer unregisterStream(handle)
+		rc := vllmCompleteStream(v.engine, opts.Prompt, unsafe.Pointer(&sp), tokenCbPtr, handle) // #nosec G103 -- POD in-params
+		runtime.KeepAlive(keep)
+		if rc != vllmOK {
+			xlog.Error("[vllm-cpp] stream failed", "error", vllmLastError())
+		}
+	}()
+	return nil
+}

backend/go/vllm-cpp/chat.go

@@ -0,0 +1,289 @@
+package main
+
+// The rich chat path (AIModelRich): rides the ENGINE's serving pipeline via
+// the ABI v3 chat entry points, exactly like the llama-cpp autoparser flow.
+// The engine applies the model's chat template, decides when a tool call
+// engages (tool_choice auto lowers to a LAZY structural-tag decode
+// constraint), parses tool calls with its streaming-stateful Hermes-style
+// parser, and hands back chat.completion.chunk JSON that this file maps 1:1
+// onto pb.Reply ChatDelta / ToolCallDelta.
+
+import (
+	"encoding/json"
+	"fmt"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// useChatPath reports whether the request should go through the engine-side
+// chat pipeline: the model config asked for backend-side templating and the
+// host handed us structured messages.
+func useChatPath(opts *pb.PredictOptions) bool {
+	return opts.UseTokenizerTemplate && len(opts.Messages) > 0
+}
+
+// chatRequestJSON lowers PredictOptions into one OpenAI chat-completions
+// request object for the ABI (the engine ignores `model`/`stream`).
+func chatRequestJSON(opts *pb.PredictOptions, stream bool) (string, error) {
+	messages := make([]map[string]any, 0, len(opts.Messages))
+	for _, m := range opts.Messages {
+		msg := map[string]any{"role": m.Role, "content": m.Content}
+		if m.ToolCalls != "" {
+			var toolCalls any
+			if err := json.Unmarshal([]byte(m.ToolCalls), &toolCalls); err == nil {
+				msg["tool_calls"] = toolCalls
+			}
+		}
+		// Multi-turn tool identity + prior reasoning: a role="tool" reply
+		// carries the id (and optionally the name) of the assistant call it
+		// answers, and assistant history may carry its reasoning span. The
+		// engine's template context needs all three or a second turn after
+		// tool execution is malformed.
+		if m.ToolCallId != "" {
+			msg["tool_call_id"] = m.ToolCallId
+		}
+		if m.Name != "" {
+			msg["name"] = m.Name
+		}
+		if m.ReasoningContent != "" {
+			msg["reasoning"] = m.ReasoningContent
+		}
+		messages = append(messages, msg)
+	}
+	req := map[string]any{"messages": messages}
+
+	if opts.Tools != "" {
+		var tools any
+		if err := json.Unmarshal([]byte(opts.Tools), &tools); err != nil {
+			return "", fmt.Errorf("vllm-cpp: tools is not valid JSON: %w", err)
+		}
+		req["tools"] = tools
+	}
+	if opts.ToolChoice != "" {
+		var choice any
+		// ToolChoice arrives either as a bare string ("auto"/"required"/"none")
+		// or as the OpenAI named-function JSON object.
+		if err := json.Unmarshal([]byte(opts.ToolChoice), &choice); err == nil {
+			req["tool_choice"] = choice
+		} else {
+			req["tool_choice"] = opts.ToolChoice
+		}
+	}
+
+	req["temperature"] = opts.Temperature
+	if opts.TopP > 0 {
+		req["top_p"] = opts.TopP
+	}
+	if opts.TopK > 0 {
+		req["top_k"] = opts.TopK
+	}
+	if opts.Tokens > 0 {
+		req["max_tokens"] = opts.Tokens
+	}
+	if opts.Seed > 0 {
+		req["seed"] = opts.Seed
+	}
+	if len(opts.StopPrompts) > 0 {
+		req["stop"] = opts.StopPrompts
+	}
+	if opts.PresencePenalty != 0 {
+		req["presence_penalty"] = opts.PresencePenalty
+	}
+	if opts.FrequencyPenalty != 0 {
+		req["frequency_penalty"] = opts.FrequencyPenalty
+	}
+	if stream {
+		// The engine's request parser validates stream_options against the
+		// stream flag at parse time (before the ABI entry point forces it),
+		// so state the intent explicitly.
+		req["stream"] = true
+		req["stream_options"] = map[string]any{"include_usage": true}
+	}
+
+	b, err := json.Marshal(req)
+	if err != nil {
+		return "", err
+	}
+	return string(b), nil
+}
+
+// chatChunk is the subset of an OpenAI chat.completion(.chunk) object the
+// backend consumes.
+type chatChunk struct {
+	Object  string `json:"object"`
+	Choices []struct {
+		Delta        *chatDelta `json:"delta"`   // streaming chunks
+		Message      *chatDelta `json:"message"` // non-stream response
+		FinishReason string     `json:"finish_reason"`
+	} `json:"choices"`
+	Usage *struct {
+		PromptTokens     int32 `json:"prompt_tokens"`
+		CompletionTokens int32 `json:"completion_tokens"`
+	} `json:"usage"`
+}
+
+type chatDelta struct {
+	Content          string `json:"content"`
+	ReasoningContent string `json:"reasoning"`
+	ToolCalls        []struct {
+		Index    int32  `json:"index"`
+		ID       string `json:"id"`
+		Function struct {
+			Name      string `json:"name"`
+			Arguments string `json:"arguments"`
+		} `json:"function"`
+	} `json:"tool_calls"`
+}
+
+// toReply maps one parsed chunk onto a pb.Reply carrying the content bytes
+// plus the structured ChatDelta (the host prefers ChatDeltas when present).
+func (c *chatChunk) toReply() *pb.Reply {
+	reply := &pb.Reply{}
+	if c.Usage != nil {
+		reply.PromptTokens = c.Usage.PromptTokens
+		reply.Tokens = c.Usage.CompletionTokens
+	}
+	if len(c.Choices) == 0 {
+		return reply
+	}
+	d := c.Choices[0].Delta
+	if d == nil {
+		d = c.Choices[0].Message
+	}
+	if d == nil {
+		return reply
+	}
+	delta := &pb.ChatDelta{
+		Content:          d.Content,
+		ReasoningContent: d.ReasoningContent,
+	}
+	for _, tc := range d.ToolCalls {
+		delta.ToolCalls = append(delta.ToolCalls, &pb.ToolCallDelta{
+			Index:     tc.Index,
+			Id:        tc.ID,
+			Name:      tc.Function.Name,
+			Arguments: tc.Function.Arguments,
+		})
+	}
+	reply.Message = []byte(d.Content)
+	if delta.Content != "" || delta.ReasoningContent != "" ||
+		len(delta.ToolCalls) > 0 {
+		reply.ChatDeltas = []*pb.ChatDelta{delta}
+	}
+	return reply
+}
+
+// Chat-stream registry: chunk JSON arrives on the engine's delivery thread
+// through one shared C callback; the integer handle in user_data selects the
+// destination channel (never a Go pointer across the ABI).
+var (
+	chatStreamsMu  sync.Mutex
+	chatStreams    = map[uintptr]chan<- *pb.Reply{}
+	chatStreamNext uintptr
+	chatCbOnce     sync.Once
+	chatCbPtr      uintptr
+)
+
+func chatCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	chatStreamsMu.Lock()
+	results := chatStreams[userData]
+	chatStreamsMu.Unlock()
+	if results == nil {
+		return 0
+	}
+	_ = finished // the terminal call carries an empty delta; nothing to emit.
+	payload := goString(delta)
+	if payload == "" {
+		return 1
+	}
+	var chunk chatChunk
+	if err := json.Unmarshal([]byte(payload), &chunk); err != nil {
+		xlog.Error("[vllm-cpp] unparseable chat chunk", "error", err)
+		return 1
+	}
+	results <- chunk.toReply()
+	return 1
+}
+
+func registerChatStream(results chan<- *pb.Reply) uintptr {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	chatStreamNext++
+	chatStreams[chatStreamNext] = results
+	return chatStreamNext
+}
+
+func unregisterChatStream(h uintptr) {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	delete(chatStreams, h)
+}
+
+// PredictRich implements the non-streaming rich path. Without structured
+// messages it falls back to the plain Predict flow (LocalAI-side templating,
+// optional grammar constraint).
+func (v *VllmCpp) PredictRich(opts *pb.PredictOptions) (*pb.Reply, error) {
+	if !useChatPath(opts) {
+		text, err := v.Predict(opts)
+		if err != nil {
+			return nil, err
+		}
+		return &pb.Reply{Message: []byte(text)}, nil
+	}
+	if v.engine == 0 {
+		return nil, fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, false)
+	if err != nil {
+		return nil, err
+	}
+	var out uintptr
+	rc := vllmChat(v.engine, request, unsafe.Pointer(&out)) // #nosec G103 -- char** out-param
+	if rc != vllmOK {
+		return nil, fmt.Errorf("vllm-cpp: chat failed: %s", vllmLastError())
+	}
+	payload := goString(out)
+	vllmStringFree(out)
+	var response chatChunk
+	if err := json.Unmarshal([]byte(payload), &response); err != nil {
+		return nil, fmt.Errorf("vllm-cpp: unparseable chat response: %w", err)
+	}
+	return response.toReply(), nil
+}
+
+// PredictStreamRich implements the streaming rich path. Contract: send into
+// the channel and return when finished; the host closes the channel.
+func (v *VllmCpp) PredictStreamRich(opts *pb.PredictOptions, results chan<- *pb.Reply) error {
+	if !useChatPath(opts) {
+		// Legacy bridge: run the plain stream and wrap deltas.
+		plain := make(chan string)
+		if err := v.PredictStream(opts, plain); err != nil {
+			return err
+		}
+		for delta := range plain {
+			results <- &pb.Reply{Message: []byte(delta)}
+		}
+		return nil
+	}
+	if v.engine == 0 {
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, true)
+	if err != nil {
+		return err
+	}
+	chatCbOnce.Do(func() {
+		chatCbPtr = purego.NewCallback(chatCallback)
+	})
+	handle := registerChatStream(results)
+	defer unregisterChatStream(handle)
+	rc := vllmChatStream(v.engine, request, chatCbPtr, handle)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: chat stream failed: %s", vllmLastError())
+	}
+	return nil
+}

backend/go/vllm-cpp/chat_test.go

@@ -0,0 +1,158 @@
+package main
+
+import (
+	"encoding/json"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("useChatPath", func() {
+	It("requires tokenizer templating AND structured messages", func() {
+		Expect(useChatPath(&pb.PredictOptions{})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{UseTokenizerTemplate: true})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages:             []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeTrue())
+	})
+})
+
+var _ = Describe("chatRequestJSON", func() {
+	It("lowers messages, tools, tool_choice and sampling onto one request", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "system", Content: "be brief"},
+				{Role: "user", Content: "weather in Rome?"},
+			},
+			Tools:       `[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object"}}}]`,
+			ToolChoice:  "required",
+			Temperature: 0.2,
+			TopP:        0.9,
+			Tokens:      64,
+			StopPrompts: []string{"<|im_end|>"},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req["messages"]).To(HaveLen(2))
+		Expect(req["tools"]).To(HaveLen(1))
+		Expect(req["tool_choice"]).To(Equal("required"))
+		Expect(req["max_tokens"]).To(BeNumerically("==", 64))
+		Expect(req["top_p"]).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(req["stop"]).To(ConsistOf("<|im_end|>"))
+		Expect(req).NotTo(HaveKey("stream_options"))
+	})
+
+	It("parses a named-function tool_choice object and asks for stream usage", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			Messages:   []*pb.Message{{Role: "user", Content: "hi"}},
+			ToolChoice: `{"type":"function","function":{"name":"get_weather"}}`,
+		}, true)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		choice, ok := req["tool_choice"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		Expect(choice["type"]).To(Equal("function"))
+		Expect(req["stream_options"]).To(HaveKeyWithValue("include_usage", true))
+	})
+
+	It("rejects malformed tools JSON", func() {
+		_, err := chatRequestJSON(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+			Tools:    "{not json",
+		}, false)
+		Expect(err).To(HaveOccurred())
+	})
+})
+
+var _ = Describe("chatChunk.toReply", func() {
+	It("maps a streaming tool-call delta onto ChatDelta/ToolCallDelta", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{
+			"tool_calls":[{"index":0,"id":"call_1","function":{"name":"get_weather","arguments":"{\"city\":"}}]
+		},"finish_reason":null}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].ToolCalls).To(HaveLen(1))
+		tc := reply.ChatDeltas[0].ToolCalls[0]
+		Expect(tc.Name).To(Equal("get_weather"))
+		Expect(tc.Id).To(Equal("call_1"))
+		Expect(tc.Arguments).To(Equal(`{"city":`))
+	})
+
+	It("maps a non-stream response message and usage", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion","choices":[{"message":{
+			"role":"assistant","content":"Sunny."},"finish_reason":"stop"}],
+			"usage":{"prompt_tokens":12,"completion_tokens":3}}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(string(reply.Message)).To(Equal("Sunny."))
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].Content).To(Equal("Sunny."))
+		Expect(reply.PromptTokens).To(BeNumerically("==", 12))
+		Expect(reply.Tokens).To(BeNumerically("==", 3))
+	})
+
+	It("emits no ChatDelta for an empty role-only chunk", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":""}}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+		Expect(chunk.toReply().ChatDeltas).To(BeEmpty())
+	})
+})
+
+var _ = Describe("chatRequestJSON multi-turn tool round trip", func() {
+	It("forwards tool_call_id, name, reasoning and assistant tool_calls", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "What is the weather in Rome?"},
+				{
+					Role:             "assistant",
+					ReasoningContent: "need the weather tool",
+					ToolCalls:        `[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Rome\"}"}}]`,
+				},
+				{Role: "tool", ToolCallId: "call_1", Name: "get_weather", Content: `{"temp": 21}`},
+			},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req struct {
+			Messages []map[string]any `json:"messages"`
+		}
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req.Messages).To(HaveLen(3))
+
+		assistant := req.Messages[1]
+		Expect(assistant["reasoning"]).To(Equal("need the weather tool"))
+		calls, ok := assistant["tool_calls"].([]any)
+		Expect(ok).To(BeTrue())
+		Expect(calls).To(HaveLen(1))
+		call := calls[0].(map[string]any)
+		Expect(call["id"]).To(Equal("call_1"))
+
+		tool := req.Messages[2]
+		Expect(tool["role"]).To(Equal("tool"))
+		Expect(tool["tool_call_id"]).To(Equal("call_1"))
+		Expect(tool["name"]).To(Equal("get_weather"))
+		Expect(tool["content"]).To(Equal(`{"temp": 21}`))
+
+		user := req.Messages[0]
+		Expect(user).NotTo(HaveKey("tool_call_id"))
+		Expect(user).NotTo(HaveKey("name"))
+		Expect(user).NotTo(HaveKey("reasoning"))
+	})
+})

backend/go/vllm-cpp/e2e_test.go

@@ -0,0 +1,281 @@
+package main
+
+// E2E over a real model + the built libvllm. Gated on VLLM_CPP_MODEL (a .gguf
+// file or a safetensors model dir): without it the suite skips, so CI runs
+// only the unit specs. test.sh auto-downloads a small GGUF when the gate is
+// unset and the download is allowed.
+
+import (
+	"encoding/json"
+	"os"
+	"runtime"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("vllm-cpp e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	It("refuses a foreign model artefact", func() {
+		other := &VllmCpp{}
+		Expect(other.Load(&pb.ModelOptions{ModelFile: "/nonexistent/foreign.bin"})).NotTo(Succeed())
+	})
+
+	It("completes a prompt (greedy)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "The capital of France is",
+			Tokens:      16,
+			Temperature: 0,
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(BeEmpty())
+	})
+
+	It("is deterministic under greedy decoding", func() {
+		opts := &pb.PredictOptions{Prompt: "1 2 3 4", Tokens: 8, Temperature: 0}
+		a, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		b, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(a).To(Equal(b))
+	})
+
+	It("streams deltas that concatenate to the blocking result", func() {
+		opts := &pb.PredictOptions{Prompt: "Count: one two", Tokens: 12, Temperature: 0}
+		blocking, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan string)
+		Expect(backend.PredictStream(opts, results)).To(Succeed())
+		var sb strings.Builder
+		for delta := range results {
+			sb.WriteString(delta)
+		}
+		Expect(sb.String()).To(Equal(blocking))
+	})
+
+	It("honors stop words", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "a b c d e f",
+			Tokens:      64,
+			Temperature: 0,
+			StopPrompts: []string{"g"},
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(ContainSubstring("g h"))
+	})
+
+	It("constrains generation with a GBNF grammar (tool-call path)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "Answer strictly yes or no: is water wet?",
+			Tokens:      4,
+			Temperature: 0,
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).To(Or(HavePrefix("yes"), HavePrefix("no")))
+	})
+
+	It("serves concurrent streams", func() {
+		const n = 4
+		type result struct {
+			text string
+			err  error
+		}
+		done := make(chan result, n)
+		for i := 0; i < n; i++ {
+			go func() {
+				results := make(chan string)
+				err := backend.PredictStream(&pb.PredictOptions{
+					Prompt: "Hello", Tokens: 8, Temperature: 0,
+				}, results)
+				var sb strings.Builder
+				for delta := range results {
+					sb.WriteString(delta)
+				}
+				done <- result{sb.String(), err}
+			}()
+		}
+		for i := 0; i < n; i++ {
+			r := <-done
+			Expect(r.err).NotTo(HaveOccurred())
+			Expect(r.text).NotTo(BeEmpty())
+		}
+	})
+})
+
+var _ = Describe("vllm-cpp chat e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	weatherTools := `[{"type":"function","function":{"name":"get_weather",` +
+		`"description":"Get the current weather for a city.",` +
+		`"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]`
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping chat e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	chatOpts := func() *pb.PredictOptions {
+		return &pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "Reply with one short sentence: what is the capital of France?"},
+			},
+			Tokens:      512,
+			Temperature: 0,
+		}
+	}
+
+	It("answers a plain chat turn through the engine-side template", func() {
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(BeEmpty())
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+	})
+
+	It("splits reasoning from content engine-side (auto-detected from the template)", func() {
+		// The Qwen3.5 chat template carries <think>, so the engine auto-selects
+		// the think_auto reasoning parser: a markerless answer stays pure
+		// content; if the model DOES think, the block arrives as
+		// ReasoningContent and never leaks into Message.
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(ContainSubstring("<think>"))
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+		for _, d := range reply.ChatDeltas {
+			Expect(d.ReasoningContent).NotTo(ContainSubstring("Paris"),
+				"the user-visible answer must not be swallowed into reasoning")
+		}
+	})
+
+	It("streams chat deltas that concatenate to the blocking answer", func() {
+		blocking, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan *pb.Reply, 64)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(chatOpts(), results)
+			close(results)
+		}()
+		var sb strings.Builder
+		for r := range results {
+			sb.WriteString(string(r.Message))
+		}
+		Expect(<-done).To(Succeed())
+		Expect(sb.String()).To(Equal(string(blocking.Message)))
+	})
+
+	It("emits a parsed tool call when tool_choice requires it", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "What is the weather in Rome right now?"},
+		}
+		opts.Tools = weatherTools
+		opts.ToolChoice = "required"
+		opts.Tokens = 256
+
+		reply, err := backend.PredictRich(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(reply.ChatDeltas).NotTo(BeEmpty())
+		var name, args string
+		for _, d := range reply.ChatDeltas {
+			for _, tc := range d.ToolCalls {
+				if tc.Name != "" {
+					name = tc.Name
+				}
+				args += tc.Arguments
+			}
+		}
+		Expect(name).To(Equal("get_weather"))
+		var parsed map[string]any
+		Expect(json.Unmarshal([]byte(args), &parsed)).To(Succeed(),
+			"tool arguments must be valid JSON: %q", args)
+		Expect(parsed).To(HaveKey("city"))
+	})
+
+	It("lets the engine decide on auto tool choice and streams tool deltas", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "Use the get_weather tool to check the weather in Rome."},
+		}
+		opts.Tools = weatherTools
+		opts.Tokens = 512
+
+		results := make(chan *pb.Reply, 128)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(opts, results)
+			close(results)
+		}()
+		sawToolDelta := false
+		for r := range results {
+			for _, d := range r.ChatDeltas {
+				if len(d.ToolCalls) > 0 {
+					sawToolDelta = true
+				}
+			}
+		}
+		Expect(<-done).To(Succeed())
+		// tool_choice auto is a LAZY constraint: the model may or may not call.
+		// With an explicit instruction the gate model reliably does; treat a
+		// no-call run as a soft signal rather than a hard failure only if the
+		// engine produced SOME output.
+		Expect(sawToolDelta).To(BeTrue(), "expected the engine to engage the tool")
+	})
+})

backend/go/vllm-cpp/govllmcpp.go

@@ -0,0 +1,182 @@
+package main
+
+// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v2).
+//
+// The structs below are hand-mirrored PODs of the C declarations, with
+// explicit padding so the Go layout matches the C layout on linux/darwin
+// amd64+arm64. Struct-by-value entry points (the *_default helpers) are NOT
+// bound - purego's struct-return support is platform-dependent - so the
+// defaults are replicated here and guarded by the vllm_abi_version check at
+// startup: a library whose ABI differs from what these mirrors were written
+// against is refused before any request runs.
+
+import (
+	"fmt"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+)
+
+// abiVersion is the VLLM_ABI_VERSION this file mirrors (vllm.h).
+const abiVersion = 5
+
+// vllm_status (vllm.h).
+const (
+	vllmOK = 0
+)
+
+// cModelParams mirrors vllm_model_params.
+type cModelParams struct {
+	ModelPath           uintptr // const char*
+	TokenizerConfigPath uintptr // const char*
+	BlockSize           int32
+	NumBlocks           int32
+	MaxModelLen         int32
+	MaxNumSeqs          int32
+	ToolParser          uintptr // const char*; NULL = auto-detect (ABI v4)
+	ReasoningParser     uintptr // const char*; NULL = auto-detect (ABI v5)
+}
+
+// cSamplingParams mirrors vllm_sampling_params (ABI v2, structured fields
+// included). Padding matches the C compiler's: the uint64 seed is 8-aligned,
+// and each pointer following an int32 is 8-aligned.
+type cSamplingParams struct {
+	Temperature          float32
+	TopP                 float32
+	TopK                 int32
+	MinP                 float32
+	MaxTokens            int32
+	_                    [4]byte
+	Seed                 uint64
+	HasSeed              int32
+	PresencePenalty      float32
+	FrequencyPenalty     float32
+	RepetitionPenalty    float32
+	MinTokens            int32
+	IgnoreEOS            int32
+	Stop                 uintptr // const char* const*
+	NStop                int32
+	_                    [4]byte
+	StructuredJSON       uintptr // const char*
+	StructuredRegex      uintptr // const char*
+	StructuredChoice     uintptr // const char* const*
+	NStructuredChoice    int32
+	_                    [4]byte
+	StructuredGrammar    uintptr // const char*
+	StructuredJSONObject int32
+	_                    [4]byte
+}
+
+// cCompletion mirrors vllm_completion.
+type cCompletion struct {
+	Text             uintptr // char*, caller-owned
+	FinishReason     uintptr // const char*, library-owned
+	PromptTokens     int32
+	CompletionTokens int32
+}
+
+// defaultSamplingParams mirrors vllm_sampling_params_default().
+func defaultSamplingParams() cSamplingParams {
+	return cSamplingParams{
+		Temperature:       1.0,
+		TopP:              1.0,
+		MaxTokens:         16,
+		RepetitionPenalty: 1.0,
+	}
+}
+
+// defaultModelParams mirrors vllm_model_params_default().
+func defaultModelParams() cModelParams {
+	return cModelParams{
+		BlockSize:  32,
+		NumBlocks:  256,
+		MaxNumSeqs: 8,
+	}
+}
+
+var (
+	vllmEngineLoad     func(params, out unsafe.Pointer) int32
+	vllmEngineFree     func(engine uintptr)
+	vllmComplete       func(engine uintptr, prompt string, params, out unsafe.Pointer) int32
+	vllmCompleteStream func(engine uintptr, prompt string, params unsafe.Pointer, cb uintptr, userData uintptr) int32
+	vllmChat           func(engine uintptr, requestJSON string, out unsafe.Pointer) int32
+	vllmChatStream     func(engine uintptr, requestJSON string, cb uintptr, userData uintptr) int32
+	vllmStringFree     func(s uintptr)
+	vllmCompletionFree func(out unsafe.Pointer)
+	vllmLastError      func() string
+	vllmVersion        func() string
+	vllmABIVersion     func() int32
+)
+
+type libFunc struct {
+	ptr  any
+	name string
+}
+
+// registerLib dlopens libvllm and binds the C ABI, refusing an ABI-version
+// mismatch (the struct mirrors above would be undefined behavior against a
+// different layout).
+func registerLib(libName string) error {
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: dlopen %s: %w", libName, err)
+	}
+	for _, lf := range []libFunc{
+		{&vllmEngineLoad, "vllm_engine_load"},
+		{&vllmEngineFree, "vllm_engine_free"},
+		{&vllmComplete, "vllm_complete"},
+		{&vllmCompleteStream, "vllm_complete_stream"},
+		{&vllmChat, "vllm_chat"},
+		{&vllmChatStream, "vllm_chat_stream"},
+		{&vllmStringFree, "vllm_string_free"},
+		{&vllmCompletionFree, "vllm_completion_free"},
+		{&vllmLastError, "vllm_last_error"},
+		{&vllmVersion, "vllm_version"},
+		{&vllmABIVersion, "vllm_abi_version"},
+	} {
+		purego.RegisterLibFunc(lf.ptr, lib, lf.name)
+	}
+	if v := vllmABIVersion(); v != abiVersion {
+		return fmt.Errorf("vllm-cpp: ABI mismatch: library reports v%d, backend built against v%d", v, abiVersion)
+	}
+	return nil
+}
+
+// cString returns a NUL-terminated byte slice for s. The backing array may be
+// passed to C for the duration of a call (the ABI borrows and copies); keep it
+// alive across the call with runtime.KeepAlive.
+func cString(s string) []byte {
+	b := make([]byte, len(s)+1)
+	copy(b, s)
+	return b
+}
+
+// cStringArray builds a NULL-free array of C-string pointers plus the backing
+// buffers that must stay alive for the duration of the C call.
+func cStringArray(ss []string) (ptrs []uintptr, backing [][]byte) {
+	backing = make([][]byte, 0, len(ss))
+	ptrs = make([]uintptr, 0, len(ss))
+	for _, s := range ss {
+		b := cString(s)
+		backing = append(backing, b)
+		ptrs = append(ptrs, uintptr(unsafe.Pointer(&b[0]))) // #nosec G103 -- borrowed by C for the call only
+	}
+	return ptrs, backing
+}
+
+// goString copies a NUL-terminated C string.
+func goString(p uintptr) string {
+	if p == 0 {
+		return ""
+	}
+	//nolint:govet // C-owned pointer handed over by purego, valid for this call
+	base := unsafe.Pointer(p) // #nosec G103 -- C-owned, copied out immediately
+	n := 0
+	for *(*byte)(unsafe.Add(base, n)) != 0 {
+		n++
+	}
+	if n == 0 {
+		return ""
+	}
+	return string(unsafe.Slice((*byte)(base), n))
+}

backend/go/vllm-cpp/main.go

@@ -0,0 +1,35 @@
+package main
+
+// Note: this is started internally by LocalAI and a server is allocated for each model
+import (
+	"flag"
+	"os"
+	"runtime"
+
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+func main() {
+	libName := os.Getenv("VLLM_CPP_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "./libvllm.dylib"
+		} else {
+			libName = "./libvllm.so"
+		}
+	}
+
+	if err := registerLib(libName); err != nil {
+		panic(err)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &VllmCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/vllm-cpp/options.go

@@ -0,0 +1,54 @@
+package main
+
+// Engine-sizing knobs carried through the model config's free-form
+// `options:` list ("key:value" entries), mirroring how the other in-house
+// backends pass engine-specific settings that have no proto field.
+
+import (
+	"strconv"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+type loadOptions struct {
+	blockSize  int32 // KV block size (tokens/block); engine default 32.
+	numBlocks  int32 // KV blocks to allocate; engine default 256.
+	maxNumSeqs int32 // max concurrent sequences; engine default 8.
+	// Engine-side parser selection (ABI v4/v5). Empty = the engine
+	// auto-detects from the chat template; "none" disables the reasoning
+	// split; unknown names fail the first chat call.
+	toolParser      string
+	reasoningParser string
+}
+
+func parseOptions(opts *pb.ModelOptions) loadOptions {
+	lo := loadOptions{}
+	for _, o := range opts.GetOptions() {
+		k, v, found := strings.Cut(o, ":")
+		if !found {
+			continue
+		}
+		switch strings.TrimSpace(k) {
+		case "block_size":
+			lo.blockSize = parseInt32(v, lo.blockSize)
+		case "num_blocks":
+			lo.numBlocks = parseInt32(v, lo.numBlocks)
+		case "max_num_seqs":
+			lo.maxNumSeqs = parseInt32(v, lo.maxNumSeqs)
+		case "tool_parser":
+			lo.toolParser = strings.TrimSpace(v)
+		case "reasoning_parser":
+			lo.reasoningParser = strings.TrimSpace(v)
+		}
+	}
+	return lo
+}
+
+func parseInt32(s string, fallback int32) int32 {
+	n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 32)
+	if err != nil || n <= 0 {
+		return fallback
+	}
+	return int32(n)
+}

backend/go/vllm-cpp/package.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/vllm-cpp $CURDIR/package/
+cp -fLv $CURDIR/libvllm.so $CURDIR/package/ 2>/dev/null || true
+cp -fLv $CURDIR/libvllm.dylib $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/vllm-cpp/run.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+# vllm.cpp ships ONE portable library per platform (SIMD tiers are per-file
+# with runtime dispatch), so there is no per-CPU variant probing here.
+if [ "$(uname)" = "Darwin" ]; then
+	LIBRARY="$CURDIR/libvllm.dylib"
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	LIBRARY="$CURDIR/libvllm.so"
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+fi
+
+export VLLM_CPP_LIBRARY=$LIBRARY
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LIBRARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/vllm-cpp "$@"
+fi
+
+echo "Using library: $LIBRARY"
+exec "$CURDIR"/vllm-cpp "$@"

backend/go/vllm-cpp/test.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+echo "Running vllm-cpp backend tests..."
+
+# Unit specs always run (struct-mirror layout, option/sampling mapping, load
+# validation). The e2e specs need a real model: set VLLM_CPP_MODEL to a .gguf
+# file or a safetensors model dir to enable them (see e2e_test.go).
+go test -v -timeout 1200s .
+
+echo "All vllm-cpp tests passed."

backend/go/vllm-cpp/vllmcpp_test.go

@@ -0,0 +1,162 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestVllmCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "vllm-cpp suite")
+}
+
+// The Go POD mirrors must match the C struct layout of vllm.h (ABI v2)
+// byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
+// amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
+var _ = Describe("C ABI struct mirrors", func() {
+	It("cModelParams matches vllm_model_params", func() {
+		var p cModelParams
+		Expect(unsafe.Offsetof(p.ModelPath)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TokenizerConfigPath)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.BlockSize)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.NumBlocks)).To(Equal(uintptr(20)))
+		Expect(unsafe.Offsetof(p.MaxModelLen)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.MaxNumSeqs)).To(Equal(uintptr(28)))
+		Expect(unsafe.Offsetof(p.ToolParser)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.ReasoningParser)).To(Equal(uintptr(40)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(48)))
+	})
+
+	It("cSamplingParams matches vllm_sampling_params (ABI v2)", func() {
+		var p cSamplingParams
+		Expect(unsafe.Offsetof(p.Temperature)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TopP)).To(Equal(uintptr(4)))
+		Expect(unsafe.Offsetof(p.TopK)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.MinP)).To(Equal(uintptr(12)))
+		Expect(unsafe.Offsetof(p.MaxTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.Seed)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.HasSeed)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.PresencePenalty)).To(Equal(uintptr(36)))
+		Expect(unsafe.Offsetof(p.FrequencyPenalty)).To(Equal(uintptr(40)))
+		Expect(unsafe.Offsetof(p.RepetitionPenalty)).To(Equal(uintptr(44)))
+		Expect(unsafe.Offsetof(p.MinTokens)).To(Equal(uintptr(48)))
+		Expect(unsafe.Offsetof(p.IgnoreEOS)).To(Equal(uintptr(52)))
+		Expect(unsafe.Offsetof(p.Stop)).To(Equal(uintptr(56)))
+		Expect(unsafe.Offsetof(p.NStop)).To(Equal(uintptr(64)))
+		Expect(unsafe.Offsetof(p.StructuredJSON)).To(Equal(uintptr(72)))
+		Expect(unsafe.Offsetof(p.StructuredRegex)).To(Equal(uintptr(80)))
+		Expect(unsafe.Offsetof(p.StructuredChoice)).To(Equal(uintptr(88)))
+		Expect(unsafe.Offsetof(p.NStructuredChoice)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.StructuredGrammar)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.StructuredJSONObject)).To(Equal(uintptr(112)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
+	})
+
+	It("cCompletion matches vllm_completion", func() {
+		var c cCompletion
+		Expect(unsafe.Offsetof(c.Text)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(c.FinishReason)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(c.PromptTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(c.CompletionTokens)).To(Equal(uintptr(20)))
+		Expect(unsafe.Sizeof(c)).To(Equal(uintptr(24)))
+	})
+})
+
+var _ = Describe("parseOptions", func() {
+	It("extracts the engine sizing knobs", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:64", "num_blocks:512", "max_num_seqs:32", "unknown:ignored",
+		}})
+		Expect(lo.blockSize).To(Equal(int32(64)))
+		Expect(lo.numBlocks).To(Equal(int32(512)))
+		Expect(lo.maxNumSeqs).To(Equal(int32(32)))
+	})
+	It("ignores malformed and non-positive values", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:abc", "num_blocks:-1", "max_num_seqs", "block_size:0",
+		}})
+		Expect(lo).To(Equal(loadOptions{}))
+	})
+})
+
+var _ = Describe("samplingFromPredict", func() {
+	It("maps the sampling fields onto the C POD", func() {
+		sp, _ := samplingFromPredict(&pb.PredictOptions{
+			Temperature:      0.7,
+			TopP:             0.9,
+			TopK:             40,
+			MinP:             0.05,
+			Tokens:           128,
+			Seed:             42,
+			Penalty:          1.1,
+			PresencePenalty:  0.5,
+			FrequencyPenalty: 0.25,
+			IgnoreEOS:        true,
+		})
+		Expect(sp.Temperature).To(BeNumerically("~", 0.7, 1e-6))
+		Expect(sp.TopP).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(sp.TopK).To(Equal(int32(40)))
+		Expect(sp.MinP).To(BeNumerically("~", 0.05, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(128)))
+		Expect(sp.HasSeed).To(Equal(int32(1)))
+		Expect(sp.Seed).To(Equal(uint64(42)))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.1, 1e-6))
+		Expect(sp.PresencePenalty).To(BeNumerically("~", 0.5, 1e-6))
+		Expect(sp.FrequencyPenalty).To(BeNumerically("~", 0.25, 1e-6))
+		Expect(sp.IgnoreEOS).To(Equal(int32(1)))
+	})
+
+	It("keeps the engine defaults for unset fields and stays unseeded", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{})
+		Expect(sp.TopP).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(0))) // unbounded, engine-capped.
+		Expect(sp.HasSeed).To(Equal(int32(0)))
+		Expect(sp.Stop).To(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).To(Equal(uintptr(0)))
+		Expect(keep).To(BeEmpty())
+	})
+
+	It("wires stop prompts and the grammar constraint", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{
+			StopPrompts: []string{"</s>", "\n\n"},
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(sp.NStop).To(Equal(int32(2)))
+		Expect(sp.Stop).NotTo(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).NotTo(Equal(uintptr(0)))
+		Expect(keep).NotTo(BeEmpty())
+	})
+})
+
+var _ = Describe("validModelPath", func() {
+	It("accepts a .gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "model.gguf")
+		Expect(os.WriteFile(p, []byte("GGUF"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).To(Succeed())
+	})
+	It("accepts a directory with config.json", func() {
+		dir := GinkgoT().TempDir()
+		Expect(os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o600)).To(Succeed())
+		Expect(validModelPath(dir)).To(Succeed())
+	})
+	It("refuses a directory without config.json (greedy-probe rule)", func() {
+		Expect(validModelPath(GinkgoT().TempDir())).NotTo(Succeed())
+	})
+	It("refuses a non-gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "weights.bin")
+		Expect(os.WriteFile(p, []byte("x"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).NotTo(Succeed())
+	})
+	It("refuses a missing path", func() {
+		Expect(validModelPath("/nonexistent/model.gguf")).NotTo(Succeed())
+	})
+})

backend/index.yaml

@@ -156,6 +156,56 @@
     nvidia-cuda-12: "cuda12-whisper"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-whisper"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-whisper"
+- &vllm-cpp
+  name: "vllm-cpp"
+  alias: "vllm-cpp"
+  license: apache-2.0
+  description: |
+    vllm.cpp is a from-scratch C++20 port of vLLM created and maintained by the LocalAI team.
+    It mirrors vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching,
+    scheduler, sampler) on a portable tensor runtime with no Python, PyTorch or ggml at
+    inference time. It loads Hugging Face safetensors and GGUF checkpoints, supports
+    structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine,
+    and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan.
+    The CUDA builds require the CUDA 13 toolchain and target Blackwell only: sm_120a
+    plus sm_121a on x86_64, and sm_121a (GB10 / DGX Spark) on arm64. Older NVIDIA
+    hardware and CUDA 12 hosts - including Jetson AGX Orin (sm_87, JetPack 6) - run
+    the CPU build instead.
+  urls:
+    - https://github.com/mudler/vllm.cpp
+  tags:
+    - text-to-text
+    - LLM
+    - CPU
+    - GPU
+    - CUDA
+    - metal
+  capabilities:
+    default: "cpu-vllm-cpp"
+    nvidia: "cuda13-vllm-cpp"
+    metal: "metal-vllm-cpp"
+    vulkan: "vulkan-vllm-cpp"
+    nvidia-cuda-13: "cuda13-vllm-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp"
+    # No CUDA 12 variant exists: 12.x nvcc cannot compile the Blackwell fp4
+    # kernels, so those hosts run the CPU build. Mapped explicitly rather than
+    # left to the "default" catch-all so the fallback is visible here instead
+    # of looking like an oversight.
+    nvidia-cuda-12: "cpu-vllm-cpp"
+    nvidia-l4t-cuda-12: "cpu-vllm-cpp"
+- !!merge <<: *vllm-cpp
+  name: "vllm-cpp-development"
+  capabilities:
+    default: "cpu-vllm-cpp-development"
+    nvidia: "cuda13-vllm-cpp-development"
+    metal: "metal-vllm-cpp-development"
+    vulkan: "vulkan-vllm-cpp-development"
+    nvidia-cuda-13: "cuda13-vllm-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp-development"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp-development"
+    nvidia-cuda-12: "cpu-vllm-cpp-development"
+    nvidia-l4t-cuda-12: "cpu-vllm-cpp-development"
 - &crispasr
   name: "crispasr"
   alias: "crispasr"
@@ -1171,6 +1221,48 @@
     nvidia-l4t: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-moss-tts-cpp-development"
+- &magpiettscpp
+  name: "magpie-tts-cpp"
+  description: |
+    Magpie TTS C++ backend using GGML (magpie-tts.cpp). Native C++
+    text-to-speech for NVIDIA's Magpie TTS Multilingual 357M model (encoder +
+    autoregressive decoder over NanoCodec tokens), running from a single
+    self-contained GGUF (model, codec, tokenizer, G2P dictionaries) with no
+    Python at inference time. 22.05kHz mono output, 5 baked voices (Aria,
+    Jason, John, Leo, Sofia), 9+ languages.
+  urls:
+    - https://github.com/mudler/magpie-tts.cpp
+    - https://huggingface.co/mudler/magpie-tts.cpp-gguf
+  tags:
+    - text-to-speech
+    - tts
+  alias: "magpie-tts-cpp"
+  capabilities:
+    default: "cpu-magpie-tts-cpp"
+    nvidia: "cuda12-magpie-tts-cpp"
+    nvidia-cuda-13: "cuda13-magpie-tts-cpp"
+    nvidia-cuda-12: "cuda12-magpie-tts-cpp"
+    intel: "intel-sycl-f16-magpie-tts-cpp"
+    metal: "metal-magpie-tts-cpp"
+    amd: "rocm-magpie-tts-cpp"
+    vulkan: "vulkan-magpie-tts-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
+- !!merge <<: *magpiettscpp
+  name: "magpie-tts-cpp-development"
+  capabilities:
+    default: "cpu-magpie-tts-cpp-development"
+    nvidia: "cuda12-magpie-tts-cpp-development"
+    nvidia-cuda-13: "cuda13-magpie-tts-cpp-development"
+    nvidia-cuda-12: "cuda12-magpie-tts-cpp-development"
+    intel: "intel-sycl-f16-magpie-tts-cpp-development"
+    metal: "metal-magpie-tts-cpp-development"
+    amd: "rocm-magpie-tts-cpp-development"
+    vulkan: "vulkan-magpie-tts-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
 - &omnivoicecpp
   name: "omnivoice-cpp"
   description: |
@@ -4754,6 +4846,107 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp
+## magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "nvidia-l4t-arm64-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cpu-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "metal-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "metal-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cpu-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda12-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "rocm-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f32-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f16-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "vulkan-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "vulkan-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda12-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "rocm-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f32-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f16-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp
 ## omnivoice-cpp
 - !!merge <<: *omnivoicecpp
   name: "omnivoice-cpp-development"
@@ -6430,3 +6623,53 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-supertonic"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-supertonic
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-vllm-cpp

backend/python/ace-step/backend.py

@@ -401,16 +401,16 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 "sample_query": request.text or "",
                 "sample_mode": True,
                 "thinking": True,
-                "vocal_language": request.language or request.GetLanguage() or "en",
+                "vocal_language": request.language or "en",
                 "instrumental": request.instrumental if request.HasField("instrumental") else False,
             }
         else:
-            caption = request.caption or request.GetCaption() or request.text
+            caption = request.caption or request.text
             payload = {
                 "prompt": caption,
                 "lyrics": request.lyrics or request.lyrics or "",
                 "thinking": request.think if request.HasField("think") else False,
-                "vocal_language": request.language or request.GetLanguage() or "en",
+                "vocal_language": request.language or "en",
             }
             if request.HasField("bpm"):
                 payload["bpm"] = request.bpm

backend/python/common/libbackend.sh

@@ -398,40 +398,80 @@ function ensureVenv() {
 function runProtogen() {
     ensureVenv
 
-    # Match grpcio-tools to the grpcio already installed by the backend's
-    # requirements. grpcio and grpcio-tools are released in lockstep, and the
-    # protoc that grpcio-tools bundles stamps a Protobuf "gencode" version into
-    # backend_pb2.py. Left unpinned, `uv pip install grpcio-tools` pulls the
-    # newest release, whose newer gencode (e.g. 7.35.0) trips Protobuf's
-    # runtime >= gencode guarantee at import time when a backend caps the
-    # protobuf runtime lower (vLLM pins it to 6.33.6), crashing the backend with
-    # "grpc service not ready" before it ever loads a model. Pinning
-    # grpcio-tools to the installed grpcio version keeps the gencode in step with
-    # the runtime. Backends whose protobuf runtime trails grpcio can set
-    # GRPCIO_TOOLS_VERSION to the newest generator their runtime accepts. Falls
-    # back to unpinned when grpcio isn't installed yet.
-    # See mudler/LocalAI#10718.
-    local grpcio_tools_spec="grpcio-tools"
-    local grpcio_version
-    if [ -n "${GRPCIO_TOOLS_VERSION:-}" ]; then
-        grpcio_tools_spec="grpcio-tools==${GRPCIO_TOOLS_VERSION}"
-    else
-        grpcio_version="$(python -c 'import importlib.metadata as m; print(m.version("grpcio"))' 2>/dev/null || true)"
+    # The protoc that grpcio-tools bundles stamps a Protobuf "gencode" version
+    # into backend_pb2.py, and Protobuf refuses to import a stub whose gencode is
+    # newer than the installed runtime. Left unpinned, grpcio-tools resolves to
+    # the newest release (1.82.1, gencode 7.35.0) which a backend holding the
+    # runtime lower (vLLM resolves protobuf to 6.33.6) then cannot import,
+    # crashing with "grpc service not ready" before it ever loads a model.
+    #
+    # Resolve the generator in a THROWAWAY environment so the backend's own venv
+    # keeps exactly the dependency set its requirements files declared. The
+    # generator is a build tool and has no business editing the runtime deps: the
+    # spec below is chosen to fit the venv as it stands, never to change it.
+    # Falls back to unpinned when neither version can be detected.
+    # See mudler/LocalAI#10718, #10940.
+    local protobuf_version grpcio_version protogen_env protogen_python
+    protobuf_version="$(python -c 'import importlib.metadata as m; print(m.version("protobuf"))' 2>/dev/null || true)"
+    grpcio_version="$(python -c 'import importlib.metadata as m; print(m.version("grpcio"))' 2>/dev/null || true)"
+
+    # The stubs impose TWO independent constraints on the generator, and both
+    # have to hold or the backend dies on import:
+    #
+    #   backend_pb2.py       needs  protobuf runtime >= gencode
+    #   backend_pb2_grpc.py  needs  installed grpcio >= grpcio-tools
+    #
+    # Bound grpcio-tools from both sides and let the resolver find the newest
+    # version that satisfies them. The protobuf ceiling makes it back off to an
+    # older grpcio-tools when the runtime is behind, which is what bounds the
+    # gencode; the grpcio ceiling keeps the _grpc stub loadable. Pinning only one
+    # side is what made the earlier attempts fail, in both directions.
+    local -a protogen_spec=("grpcio-tools")
+    if [ -n "${grpcio_version}" ]; then
+        protogen_spec=("grpcio-tools<=${grpcio_version}")
     fi
-    if [ "${grpcio_tools_spec}" = "grpcio-tools" ] && [ -n "${grpcio_version}" ]; then
-        grpcio_tools_spec="grpcio-tools==${grpcio_version}"
+    if [ -n "${protobuf_version}" ]; then
+        protogen_spec+=("protobuf<=${protobuf_version}")
     fi
 
+    protogen_env="$(mktemp -d)"
+
     if [ "x${USE_PIP}" == "xtrue" ]; then
-        pip install "${grpcio_tools_spec}"
+        python -m venv "${protogen_env}"
+        "${protogen_env}/bin/pip" install "${protogen_spec[@]}"
     else
-        uv pip install "${grpcio_tools_spec}"
+        uv venv "${protogen_env}"
+        VIRTUAL_ENV="${protogen_env}" uv pip install "${protogen_spec[@]}"
     fi
+    protogen_python="${protogen_env}/bin/python"
+
     pushd "${EDIR}" >/dev/null
+        # Drop the cached bytecode along with the sources. CPython validates a
+        # .pyc against the source's mtime *and size*, both of which can be
+        # unchanged across a regeneration (the gencode triple is the same width
+        # whether it reads 7.35.0 or 6.33.5), so a stale backend_pb2.pyc can be
+        # reused in place of the stub we just generated.
         rm -f backend_pb2.py backend_pb2.pyi backend_pb2_grpc.py
-        # use the venv python (ensures correct interpreter & sys.path)
-        python -m grpc_tools.protoc -I../../ -I./ --python_out=. --grpc_python_out=. backend.proto
+        rm -rf __pycache__
+        # Generate with the throwaway toolchain; the output is plain Python and
+        # carries no dependency on the interpreter that produced it.
+        "${protogen_python}" -m grpc_tools.protoc -I../../ -I./ --python_out=. --grpc_python_out=. backend.proto
+
+        # Verify with the BACKEND's python, which is the interpreter that has to
+        # import these at model load. Fail the build rather than ship a backend
+        # that cannot import its own stubs: otherwise the gencode/runtime
+        # mismatch only surfaces in a released image, as an opaque
+        # "grpc service not ready".
+        # Check BOTH stubs: backend_pb2 catches a gencode ahead of the protobuf
+        # runtime, backend_pb2_grpc catches generated code ahead of the installed
+        # grpcio. Checking only the former lets the latter reach CI, or users.
+        if ! python -c 'import backend_pb2, backend_pb2_grpc' >/dev/null; then
+            echo "runProtogen: generated stubs are not importable by the backend venv (protobuf ${protobuf_version:-unknown}, grpcio ${grpcio_version:-unknown})" >&2
+            exit 1
+        fi
     popd >/dev/null
+
+    rm -rf "${protogen_env}"
 }
 
 

backend/python/common/model_identity.py

@@ -20,6 +20,8 @@ Enforcement is deliberately narrow: it compares two strings and never inspects
 the model itself.
 """
 
+import asyncio
+import inspect
 import threading
 
 import grpc
@@ -181,6 +183,40 @@ class ModelIdentityInterceptor(grpc.ServerInterceptor):
         return _rebuild(handler, guard)
 
 
+_STREAM_DONE = object()
+
+
+def _next_or_done(iterator):
+    """next(iterator), returning the _STREAM_DONE sentinel at exhaustion.
+
+    StopIteration must not propagate out of a function run via run_in_executor:
+    it cannot travel through a Future and would surface as an opaque error.
+    """
+    try:
+        return next(iterator)
+    except StopIteration:
+        return _STREAM_DONE
+
+
+async def _call_behavior(behavior, request, context):
+    """Invoke a unary servicer behavior without blocking the event loop.
+
+    Native async behavior is awaited directly. A sync behavior -- many backends
+    define `def LoadModel` / `def Embedding`, not `async def` -- is dispatched to
+    a worker thread so a slow load/inference cannot freeze all aio RPC handling,
+    mirroring grpc.aio's own sync-handler adaptation. A callable wrapper that
+    returns an awaitable is supported too: the (cheap) call runs in the thread,
+    then the awaitable is awaited back on the loop.
+    """
+    if inspect.iscoroutinefunction(behavior):
+        return await behavior(request, context)
+    loop = asyncio.get_running_loop()
+    result = await loop.run_in_executor(None, behavior, request, context)
+    if inspect.isawaitable(result):
+        result = await result
+    return result
+
+
 class AsyncModelIdentityInterceptor(grpc.aio.ServerInterceptor):
     """Async counterpart for backends running grpc.aio servers."""
 
@@ -200,7 +236,11 @@ class AsyncModelIdentityInterceptor(grpc.aio.ServerInterceptor):
             original = handler.unary_unary
 
             async def record(request, context):
-                result = await original(request, context)
+                # A backend's LoadModel may be a plain sync method (many define
+                # `def LoadModel`, not `async def`). Dispatch it so it neither
+                # crashes with "object <T> can't be used in 'await'" nor runs its
+                # (potentially slow) body on the event loop thread.
+                result = await _call_behavior(original, request, context)
                 if getattr(result, "success", True):
                     self.state.record(getattr(request, "Model", ""))
                 return result
@@ -214,8 +254,21 @@ class AsyncModelIdentityInterceptor(grpc.aio.ServerInterceptor):
                 message = self.state.mismatch(getattr(request, "ModelIdentity", ""))
                 if message is not None:
                     await context.abort(grpc.StatusCode.NOT_FOUND, message)
-                async for response in original_stream(request, context):
-                    yield response
+                # A sync backend yields a plain generator, an async one an async
+                # generator. Async: iterate directly. Sync: pull each item via a
+                # worker thread so a slow producer doesn't block the event loop
+                # (and so StopIteration can't escape through a Future).
+                stream = original_stream(request, context)
+                if hasattr(stream, "__aiter__"):
+                    async for response in stream:
+                        yield response
+                else:
+                    loop = asyncio.get_running_loop()
+                    while True:
+                        item = await loop.run_in_executor(None, _next_or_done, stream)
+                        if item is _STREAM_DONE:
+                            break
+                        yield item
 
             return _rebuild(handler, guard_stream)
 
@@ -225,6 +278,6 @@ class AsyncModelIdentityInterceptor(grpc.aio.ServerInterceptor):
             message = self.state.mismatch(getattr(request, "ModelIdentity", ""))
             if message is not None:
                 await context.abort(grpc.StatusCode.NOT_FOUND, message)
-            return await original_unary(request, context)
+            return await _call_behavior(original_unary, request, context)
 
         return _rebuild(handler, guard)

backend/python/common/model_identity_test.py

@@ -9,7 +9,9 @@ failure modes are silent: enforcement that is wired up but never installed, and
 enforcement that rejects requests it should serve.
 """
 
+import asyncio
 import os
+import threading
 import unittest
 
 import grpc
@@ -64,6 +66,13 @@ def _handler(behavior, response_streaming=False):
     return grpc.unary_unary_rpc_method_handler(behavior)
 
 
+def _const_continuation(handler):
+    async def continuation(_):
+        return handler
+
+    return continuation
+
+
 class TestInterceptorInstalled(unittest.TestCase):
     """The wiring, which is where this can silently do nothing.
 
@@ -304,5 +313,176 @@ class TestModalityMethods(unittest.TestCase):
             self.assertNotIn(method, model_identity._GUARDED_METHODS)
 
 
+class TestAsyncInterceptorBehavior(unittest.TestCase):
+    """The grpc.aio counterpart, which had no behavioral coverage.
+
+    AsyncModelIdentityInterceptor wraps a backend's own servicer behavior. That
+    behavior may be sync or async: several backends define `def LoadModel` and
+    `def Embedding` (not `async def`), and grpc.aio's dispatch adapts both. The
+    interceptor invokes the behavior itself, so if it awaits unconditionally it
+    breaks every sync method it guards with "object <T> can't be used in
+    'await'". These tests exercise both shapes; the sync ones are the regression.
+    """
+
+    def setUp(self):
+        self.interceptor = model_identity.AsyncModelIdentityInterceptor()
+
+    def _wrap(self, method, handler):
+        async def continuation(_):
+            return handler
+
+        return asyncio.run(
+            self.interceptor.intercept_service(continuation, _FakeCallDetails(method))
+        )
+
+    def _load(self, behavior, model):
+        wrapped = self._wrap("/backend.Backend/LoadModel", _handler(behavior))
+        return asyncio.run(wrapped.unary_unary(_Request(Model=model), _FakeContext()))
+
+    def _call_unary(self, behavior, identity):
+        wrapped = self._wrap("/backend.Backend/Predict", _handler(behavior))
+        return asyncio.run(
+            wrapped.unary_unary(_Request(ModelIdentity=identity), _FakeContext())
+        )
+
+    def _drain_stream(self, behavior, identity):
+        wrapped = self._wrap(
+            "/backend.Backend/PredictStream", _handler(behavior, response_streaming=True)
+        )
+
+        async def drain():
+            out = []
+            async for item in wrapped.unary_stream(
+                _Request(ModelIdentity=identity), _FakeContext()
+            ):
+                out.append(item)
+            return out
+
+        return asyncio.run(drain())
+
+    # --- LoadModel: sync behavior is the regression, async must still work ---
+
+    def test_load_records_with_sync_behavior(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+        self.assertEqual(self.interceptor.state.loaded, "a.gguf")
+
+    def test_load_records_with_async_behavior(self):
+        async def behavior(request, context):
+            return _Result()
+
+        self._load(behavior, "a.gguf")
+        self.assertEqual(self.interceptor.state.loaded, "a.gguf")
+
+    def test_failed_sync_load_records_nothing(self):
+        self._load(lambda request, context: _Result(success=False), "a.gguf")
+        self.assertEqual(self.interceptor.state.loaded, "")
+
+    # --- guarded unary: sync and async behaviors both served / rejected ---
+
+    def test_guard_serves_sync_behavior(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+        result = self._call_unary(lambda request, context: "served", "a.gguf")
+        self.assertEqual(result, "served")
+
+    def test_guard_serves_async_behavior(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+
+        async def behavior(request, context):
+            return "served"
+
+        self.assertEqual(self._call_unary(behavior, "a.gguf"), "served")
+
+    def test_guard_rejects_mismatch(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+        with self.assertRaises(_Aborted):
+            self._call_unary(lambda request, context: "served", "b.gguf")
+
+    # --- guarded stream: sync generator and async generator both work ---
+
+    def test_guard_stream_serves_sync_generator(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+
+        def behavior(request, context):
+            yield "a"
+            yield "b"
+
+        self.assertEqual(self._drain_stream(behavior, "a.gguf"), ["a", "b"])
+
+    def test_guard_stream_serves_async_generator(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+
+        async def behavior(request, context):
+            yield "a"
+            yield "b"
+
+        self.assertEqual(self._drain_stream(behavior, "a.gguf"), ["a", "b"])
+
+    # --- sync behavior must not run on the event-loop thread ---
+    #
+    # Awaiting a sync method's return fixed the TypeError, but calling the
+    # (possibly slow) sync behavior on the event loop still froze all aio RPC
+    # handling. These record the thread each behavior runs on and assert it is a
+    # worker thread, not the loop thread.
+
+    def _run_capturing_loop_thread(self, method, handler, request):
+        captured = {}
+
+        async def run():
+            captured["loop"] = threading.get_ident()
+            wrapped = await self.interceptor.intercept_service(
+                _const_continuation(handler), _FakeCallDetails(method)
+            )
+            behavior = wrapped.unary_stream if handler.response_streaming else wrapped.unary_unary
+            if handler.response_streaming:
+                async for _ in behavior(request, _FakeContext()):
+                    pass
+            else:
+                await behavior(request, _FakeContext())
+
+        asyncio.run(run())
+        return captured["loop"]
+
+    def test_sync_load_runs_off_the_event_loop(self):
+        ran = {}
+
+        def behavior(request, context):
+            ran["thread"] = threading.get_ident()
+            return _Result()
+
+        loop_thread = self._run_capturing_loop_thread(
+            "/backend.Backend/LoadModel", _handler(behavior), _Request(Model="a.gguf")
+        )
+        self.assertIn("thread", ran)
+        self.assertNotEqual(ran["thread"], loop_thread)
+
+    def test_sync_guarded_unary_runs_off_the_event_loop(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+        ran = {}
+
+        def behavior(request, context):
+            ran["thread"] = threading.get_ident()
+            return "served"
+
+        loop_thread = self._run_capturing_loop_thread(
+            "/backend.Backend/Predict", _handler(behavior), _Request(ModelIdentity="a.gguf")
+        )
+        self.assertNotEqual(ran["thread"], loop_thread)
+
+    def test_sync_stream_next_runs_off_the_event_loop(self):
+        self._load(lambda request, context: _Result(), "a.gguf")
+        ran = {}
+
+        def behavior(request, context):
+            ran["thread"] = threading.get_ident()
+            yield "a"
+
+        loop_thread = self._run_capturing_loop_thread(
+            "/backend.Backend/PredictStream",
+            _handler(behavior, response_streaming=True),
+            _Request(ModelIdentity="a.gguf"),
+        )
+        self.assertNotEqual(ran["thread"], loop_thread)
+
+
 if __name__ == "__main__":
     unittest.main()

backend/python/coqui/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.82.1
+grpcio==1.83.0
 protobuf
 certifi
 packaging==26.2

backend/python/fish-speech/install.sh

@@ -60,3 +60,9 @@ if [ "x${USE_PIP}" == "xtrue" ]; then
 else
     uv pip install "protobuf>=5.29.0"
 fi
+
+# Regenerate the stubs against the protobuf runtime settled on just above. The
+# pin exists because transitive deps drag protobuf down; generating before that
+# pin is applied is what stamps a gencode the runtime then rejects.
+# See mudler/LocalAI#10718.
+runProtogen

backend/python/llama-cpp-quantization/install.sh

@@ -68,3 +68,11 @@ if [ ! -x "${QUANTIZE_BIN}" ] && ! command -v llama-quantize &>/dev/null; then
         echo "Warning: cmake not found — llama-quantize will not be available. Install cmake or provide llama-quantize on PATH."
     fi
 fi
+
+# The stubs generated at the end of installRequirements were built against the
+# protobuf runtime as it stood before the installs above, which resolve their own
+# dependencies and can move that runtime. Regenerate now that the dependency set
+# is final, so the gencode stamped into backend_pb2.py cannot be newer than the
+# runtime that ships. Same failure mode as mudler/LocalAI#10718; runProtogen
+# clears the previous stubs first, so this is idempotent.
+runProtogen

backend/python/transformers/requirements-cpu.txt

@@ -4,7 +4,7 @@ numba==0.60.0
 accelerate
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-cublas12.txt

@@ -4,7 +4,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-cublas13.txt

@@ -4,7 +4,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-hipblas.txt

@@ -5,7 +5,7 @@ transformers>=5.14.1
 llvmlite==0.43.0
 numba==0.60.0
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-intel.txt

@@ -5,7 +5,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0