Merge branch 'master' into fix/7461-mlx-file-uri

* feat(llama-cpp): route Score through the slot loop

Score previously bypassed the slot loop with a direct llama_decode: a
conflict guard aborted the whole process if scoring raced generation, the
config validator had to reject score alongside chat/completion/embeddings,
and every candidate re-decoded the full shared prompt.

Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks
are scheduled like any other slot work: generation and scoring serialize
naturally, the shared prompt is decoded once per call, and the slot's
prompt cache carries the conversation prefix across calls. Context
checkpoints at the score boundary and at the cache-divergence point keep
SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole
prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s
to ~0.5s on a desktop CPU.

The conflict guard and the validation split are removed; declaring score
with generation usecases on one config is now supported and shares the
slot cache.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier wire types and pipeline config

Wire types and YAML config for realtime classifier mode: sessions carry a
localai_classifier extension (options with canned replies/tool calls,
softmax threshold, normalization, history trimming, fallback modes, and a
deterministic wake-word address gate), mirrored by pipeline.classifier in
the model YAML and surfaced in the config-meta registry. The
localai.classifier.result server event reports the full score distribution
per turn.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier response flow

Classifier-mode responses: instead of autoregressive generation, each user
turn is prefill-scored against the option list (router.ScoreClassifier
prompt/candidate shapes over the Score primitive) and the winning option's
canned reply and tool call are emitted through the existing response
machinery. Below-threshold turns take the configured fallback (none /
canned reply / generate); empty transcripts and unaddressed turns (wake
word not mentioned) skip scoring entirely. The scoring probe defaults to
the latest user message only — small scorers echo canned replies from
prior turns back as the top option otherwise.

Built for hardware that can afford prompt processing but not decode: with
slot-based Score the option list stays KV-cached across turns, so a turn
costs roughly one forward pass over the new words.

session_update_error events now carry the validation cause instead of a
generic message.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): bound the VAD tick's scan window and buffer retention

The VAD tick loop re-scanned the entire input buffer every 300ms and only
trimmed it on zero-segment ticks or commits. Audio that keeps producing
segments without a committing pause (steady noise a mic pipeline lets
through, music, continuous speech) grew the buffer toward the 100MB cap
with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero
per buffered second: past ~90s retained, ticks run back to back and pin
~4 cores until the stream stops.

Silero's recurrent state only carries a few hundred ms of context, so
rescanning old audio buys nothing. Clip the slice handed to the VAD to the
largest silence the commit test can need to measure (server_vad silence
window or the semantic eagerness fallback) plus a warm-up margin, and
rebase the returned segment times so every downstream consumer keeps
whole-buffer coordinates. An open turn whose clipped window is all silence
now commits (the silence outran the window) instead of being discarded as
no-speech. Independently, retain at most 90s of raw buffer, rebasing the
live-feed and EOU cursors on trim — this also bounds the previously
unbounded VAD-error path. Turn boundaries are otherwise unchanged: no
forced commits, no new coordinator states.

pipeline.turn_detection.vad_window_sec can widen the scan window; values
below the automatic floor are ignored. The tick body is extracted into
vadTick so specs can drive turn detection synchronously (same shape as
classifySoundWindow); the babble reproduction that pinned 4 cores now
plateaus under 10% of one core.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(backend): let per-model threads override the global default

ModelOptions overrode a set per-model threads value with the app-level
--threads whenever the latter was non-zero — and WithThreads defaults it
to the physical core count, so it always was. The YAML threads: knob has
been dead config: a tiny VAD model could never opt down from the global
pool size.

SetDefaults already fills an unset per-model value from the app config,
which is the intended precedence; resolve threads through a helper that
honors it (explicit threads: 0 still means unset).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(gallery): single-thread the silero VAD

Silero is a ~2MB recurrent model with no exploitable graph parallelism:
measured per-call latency is identical at 1 and 10 ORT threads, while
every extra pool thread just spin-waits between the realtime loop's
frequent tiny inferences.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(realtime): classifier mode, VAD scan window, threads precedence

Document the realtime classifier mode (options, threshold guidance,
wake-word address gate, empty-transcript handling), the VAD scan window
and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the
per-model threads precedence, and the M3 classifier note in the realtime
state-machine design doc.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): score all candidates in one batched decode

One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot
decodes the shared prefix (prompt + longest common candidate token
prefix) once, then forks one sequence per candidate off it
(metadata-only for the unified KV cache, copy-on-write for recurrent
state) and decodes every candidate's unique tail in one llama_decode.
Previously each candidate was its own task that restored the boundary
checkpoint and re-decoded its full tail sequentially, paying
per-candidate task and decode overhead.

The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and
recurrent-state cells) beyond the parallel slots via the new
common_params::n_seq_score_forks. Forking requires the unified KV cache
(already this backend's default) since per-sequence streams would shrink
n_ctx_seq; an explicit kv_unified:false disables forking and Score calls
that need it fail cleanly. Candidates beyond the fork/output budget
decode in successive chunks.

Wire contract and scores are unchanged: per-token logprobs are stitched
from the shared region and the forked tails. Verified bitwise
deterministic call-to-call and independent of candidate order (no
cross-fork leakage via equal-length candidate swap); ranking matches the
per-candidate implementation on the drone battery (winner softmax
0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and
empty candidates all pass.

Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm
realtime classifier turns 196-303ms. The 9-candidate drone turn decodes
~17 unique tail tokens in one batch instead of nine sequential ~220ms
checkpoint-restore tasks.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): gate scoring capacity by model usecase

Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity.

Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step

The builder-prebuilt path installs gcc-14 with apt directly and ignored
the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already
honors, so an ubuntu mirror outage broke every arm64 backend build. Pass
the args into the stage and run apt-mirror.sh (already in the build
context via COPY . /LocalAI) before the apt step.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier argument slots via constrained completion

Hybrid classify-then-complete: a classifier option's canned tool call can
declare typed argument slots (number | enum | string, with defaults and
prompt hints) referenced as "{{name}}" in the arguments template. When
the option wins, the slots are filled by a short grammar-constrained
completion that continues the exact scoring prompt — rendered by the same
cached ScoreClassifier, so the llama.cpp prompt cache is already warm —
with the chosen route JSON re-opened at the first slot field. A GBNF
grammar pins the field skeleton and frees only the values; temperature 0,
a couple dozen tokens at most (~300ms on a desktop CPU for two slots).

Slot declarations and hints ride the option descriptions in the shared
system prompt, informing scoring and the fill alike at no per-turn token
cost. The localai.classifier.result event carries the final arguments and
a fill_latency_ms. On inference failure the slots' defaults apply; a slot
without a default fails the response (or falls through with
fallback.mode: generate). Slot filling requires completion alongside
score in the scoring model's known_usecases.

Verified end-to-end on the Pi drone demo: "fly forward three meters" in
distance mode classifies forward and infers {"distance": 3, "units":
"meters"} in ~310ms, and the drone flies exactly 3 units.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): splice filled slot values into classifier replies

A classifier option's spoken reply can now reference its tool's argument
slots ("Going forward {{distance}} {{units}}."): the values inferred by
the slot-fill completion — or the recovery defaults — are spliced into
the reply as plain text before it is emitted, so what the assistant says
confirms what it actually inferred. Placeholders without a value stay
literal, and options without slots are untouched.

FillToolArguments now returns the raw slot values alongside the spliced
arguments JSON to make the reply templating possible.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): harden classifier slot completion

Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): prewarm the classifier scoring prompt on registration

Swapping a session's classifier option list (a voice-switched command
mode, for instance) made the next turns pay a full re-prefill of the new
option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and
worse: on hybrid-memory models like LFM2.5, whose state cannot be
partially rewound (llama.cpp can only restore checkpoints), *every*
probe change re-prefilled from scratch whenever the last checkpoint
missed the probe boundary, so even same-list turns intermittently cost
full prefills.

Registering an option list (pipeline seed or session.update) now fires a
best-effort background prewarm: two throwaway scores with distinct
probes. The first prefills the new option-list prompt; the second,
diverging exactly where per-turn probe text starts, plants the backend's
rewind point (KV checkpoint) at the stable-prefix boundary that every
real turn reuses. The prewarm hides behind the canned mode-switch reply
— by the time it finishes speaking, the cache is warm. Idempotent per
option set, detached from the registering request's lifetime.

Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after
a mode switch 2374ms -> 340ms; intermittent same-list full prefills
(1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently,
options: [parallel:2] on the scoring model additionally keeps one slot
per list via prefix-similarity routing (+26MB RSS, unified KV).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix

Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new
small models are headed) cannot rewind their state, so any prompt-cache
reuse that needs a rewind falls back to a full re-prefill. For classifier
scoring that meant every probe change re-processed the whole option-list
prompt: the server's checkpoints were placed reactively (at wherever the
previous task happened to diverge), so a checkpoint past the next
divergence was erased rather than restored — measured as intermittent
2-10s turns on prompts with a 95%+ common prefix.

The classifier now computes the probe-invariant prompt prefix once (the
byte-wise common prefix of two synthetic probe renders) and declares its
length with every Score request; the server maps it to a token boundary
and forces a KV checkpoint exactly there on each score prefill. That
checkpoint sits at or before every future divergence under the same
option list, so it always survives and always restores — repeat scoring
costs probe+candidates regardless of how the probe changes.

Also:
- prewarm reruns on every option-list registration instead of memoizing
  per list: with boundary checkpoints a redundant rewarm costs two
  probe-sized decodes, while skipping one after a slot eviction (three
  lists sharing fewer slots evict in LRU cascades) silently moves a full
  re-prefill onto the user's next turn
- new llama.cpp backend option rs_seq:N exposes bounded recurrent-state
  rollback outside speculative decoding; measured impractical for
  deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap
  insurance for small-state models
- docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot
  similarity threshold funnels distinct lists onto one slot)

Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady
state: every turn 285-421ms including mode switches, vs 2.4s post-switch
and intermittent 1.3-2.9s re-prefills before.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): align classifier cache guidance

Document the single-score prewarm behavior and clean the vendored score patch formatting.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(llama-cpp): guard score task for fork backends

TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields.

Assisted-by: Codex:gpt-5 [gh]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(dev): generate gRPC code before commit lint

The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>

.agents/building-and-testing.md

@@ -28,7 +28,6 @@ The core Go suites (`./pkg`, `./core`, plus the in-process integration suite `./
   - **Build tags (`COVERAGE_TAGS`, passed via `GINKGO_TAGS`):** defaults to `debug auth`. The `auth` tag is required to compile the real (sqlite-backed) auth implementation and its ~150 `//go:build auth` tests — without it those files aren't built, the tests don't run, and the gate scores auth against a stub (~3.7% instead of ~38%). If you add new tag-gated tests, extend `COVERAGE_TAGS` or they won't count (and likely won't run in CI at all).
 - `make test-coverage-check` — runs `test-coverage`, then `scripts/coverage-check.sh` fails the build if total coverage is **below** the committed baseline in `coverage-baseline.txt`. The Linux job in `.github/workflows/test.yml` runs this instead of `make test`.
 - `make test-coverage-baseline` — regenerates and overwrites `coverage-baseline.txt` from the current run.
-- `make install-hooks` — sets `core.hooksPath` to the versioned `.githooks/`, whose `pre-commit` runs checks scoped to what's staged: Go changes → `make lint` + `make test-coverage-check`; `core/http/react-ui/` changes → `make test-ui-coverage-check` (Playwright e2e + UI coverage gate). A commit touching neither is skipped; bypass with `git commit --no-verify`. The hook resolves golangci-lint's new-from base to `upstream/master` → `origin/master` → `master`, so it works from a fork clone where `origin/master` is stale (passed to `make lint` via `LINT_NEW_FROM`).
 
 ### React UI coverage
 
@@ -38,12 +37,11 @@ The React UI (`core/http/react-ui/`) has **no component/unit tests** — its onl
 - **Browser:** the flake dev shell ships `chromium` and exports `PLAYWRIGHT_CHROMIUM_PATH`; `playwright.config.js` uses it via `launchOptions.executablePath`, and the Makefile skips `playwright install` when it's set. This avoids Playwright's downloaded browser, which can't resolve system libs (`libglib-2.0`, …) on NixOS. In CI (no `PLAYWRIGHT_CHROMIUM_PATH`) the Makefile falls back to `playwright install --with-deps chromium`.
 - The app is a React SPA, so coverage accumulates across in-app navigation within a test; a full `page.goto`/reload resets it.
 - `.nycrc.json` uses `all: true`, so **every `src/**` file is in the report**, including 0%-coverage ones — that's how you spot features with no test at all (sort the HTML report or `coverage-summary.json` by line% ascending). 
-- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`) and pre-commit on `core/http/react-ui/` changes.
+- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`).
 - **Why it has a tolerance (unlike the strict Go gate):** UI e2e coverage is *non-deterministic*. Specs that assert on state and end while async/lazy render work is still in flight collect those lines only when the render beats the coverage teardown — so the total drifts with machine speed/load (a fast local box reads higher than a slow CI runner), diffusely across many specs. The tolerance absorbs that drift, so set the baseline *below* the slow-CI floor, never to a fast-local `make test-ui-coverage-baseline` number, or CI flaps.
 - **Raising coverage is cheap:** a *render-smoke* spec (navigate to a route, assert its header renders) mounts a lazy page and runs its full render + initial effects, capturing most of its lines in a few lines of test — see `e2e/page-render-smoke.spec.js`. Auth is disabled in the test server (`isAdmin=true`), so `RequireAdmin`/`RequireFeature` routes render without a mock. The most *deterministic* win is removing a race: make a spec `await` a rendered element before ending (see `e2e/agents.spec.js` → AgentCreate) so its lines count every run.
 
 Rules (both gates):
-- **Install the hooks:** `make install-hooks` once per clone so lint + coverage run pre-commit. Don't lean on CI for what the hook catches.
-- **Don't work around the gate:** never `git commit --no-verify`, and never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
+- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
 - If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
 - The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.

.docker/llama-cpp-compile.sh

@@ -28,6 +28,10 @@ if [ -z "${BUILD_TYPE:-}" ]; then
   # variants with it (the host never *selects* SME unless it has it, but every variant must
   # still compile).
   if [ "${TARGETARCH}" = "arm64" ]; then
+    # The prebuilt base inherits default ports.ubuntu.com sources; honor the
+    # APT_*_MIRROR build args here like the from-source path does, so this
+    # apt step survives a mirror outage.
+    sh /LocalAI/.docker/apt-mirror.sh || true
     apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
     export CC=gcc-14 CXX=g++-14
   fi

.githooks/pre-commit

@@ -1,72 +0,0 @@
-#!/usr/bin/env sh
-#
-# LocalAI pre-commit hook. Install it (once per clone) with:
-#
-#     make install-hooks
-#
-# Runs only the checks relevant to what's staged:
-#   - Go files          -> make lint + make test-coverage-check
-#   - core/http/react-ui -> make test-ui-coverage-check (Playwright e2e + gate)
-#   - realtime state machines / specs -> make test-realtime-conformance
-#       (respcoord/**, turncoord/**, or formal-verification/** -- a pure .fizz
-#        spec edit must still re-verify the design, detected separately from Go)
-# A commit touching none of these is skipped entirely (other docs/YAML can't
-# change lint findings, Go coverage, the UI, or the realtime conformance gate).
-#
-# To bypass for a single commit (e.g. a WIP checkpoint): git commit --no-verify
-set -eu
-
-repo_root="$(git rev-parse --show-toplevel)"
-cd "$repo_root"
-
-staged="$(git diff --cached --name-only --diff-filter=ACMRD)"
-
-go_changed=0
-ui_changed=0
-rt_changed=0
-if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
-if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
-if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
-
-if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
-	echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
-	exit 0
-fi
-
-if [ "$go_changed" -eq 1 ]; then
-	# Resolve the ref golangci-lint's new-from-merge-base should compare
-	# against. .golangci.yml pins origin/master, which is correct in CI
-	# (origin == the canonical repo) but wrong from a fork clone, where
-	# origin/master lags behind and lint would report the whole upstream
-	# backlog. Prefer upstream/master, then origin/master, then master.
-	lint_base=""
-	for ref in upstream/master origin/master master; do
-		if git rev-parse --verify --quiet "${ref}^{commit}" >/dev/null 2>&1; then
-			lint_base="$ref"
-			break
-		fi
-	done
-
-	echo "pre-commit ▶ golangci-lint (make lint${lint_base:+, new-from $lint_base})"
-	make lint LINT_NEW_FROM="$lint_base"
-
-	echo "pre-commit ▶ coverage gate (make test-coverage-check) — builds and runs the"
-	echo "             pkg/core suites plus tests/e2e; can take a few minutes."
-	make test-coverage-check
-fi
-
-if [ "$ui_changed" -eq 1 ]; then
-	echo "pre-commit ▶ React UI e2e + coverage gate (make test-ui-coverage-check) —"
-	echo "             rebuilds the UI + ui-test-server, runs the Playwright specs, and"
-	echo "             fails if line coverage regressed; can take a couple of minutes."
-	make test-ui-coverage-check
-fi
-
-if [ "$rt_changed" -eq 1 ]; then
-	echo "pre-commit ▶ realtime state-machine conformance (make test-realtime-conformance) —"
-	echo "             Go transition/rapid tests under -race + FizzBee model check of the"
-	echo "             authoritative specs. Fail-closed: needs FizzBee (make install-fizzbee)."
-	make test-realtime-conformance
-fi
-
-echo "pre-commit ✓ all relevant checks passed"

.github/backend-matrix.yml

@@ -66,6 +66,34 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-kokoro'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'true'
+    backend: "kokoro"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-kokoro'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'true'
+    backend: "kokoro"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -871,6 +899,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1935,6 +1976,32 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2000,6 +2067,32 @@ include:
     backend: "moss-tts-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp'
+    base-image: "ubuntu:24.04"
+    ubuntu-version: '2404'
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2963,6 +3056,19 @@ include:
     dockerfile: "./backend/Dockerfile.privacy-filter"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # Vulkan: base-grpc-vulkan-amd64 carries the SDK. arm64 vulkan is a one-line
   # add once amd64 is proven in CI.
   - build-type: 'vulkan'
@@ -4583,6 +4689,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4597,7 +4717,50 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # vllm-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # omnivoice-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-magpie-tts-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4652,6 +4815,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f32'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f32-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4691,6 +4867,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f16'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f16-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4732,6 +4921,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4774,6 +4977,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4814,6 +5031,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2204'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-arm64-magpie-tts-cpp'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2204'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -4853,6 +5083,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-magpie-tts-cpp'
+    base-image: "rocm/dev-ubuntu-24.04:6.4.4"
+    runs-on: 'ubuntu-latest'
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -5774,6 +6017,14 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-moss-tts-cpp"
     build-type: "metal"
     lang: "go"
+  - backend: "magpie-tts-cpp"
+    tag-suffix: "-metal-darwin-arm64-magpie-tts-cpp"
+    build-type: "metal"
+    lang: "go"
+  - backend: "vllm-cpp"
+    tag-suffix: "-metal-darwin-arm64-vllm-cpp"
+    build-type: "metal"
+    lang: "go"
   - backend: "omnivoice-cpp"
     tag-suffix: "-metal-darwin-arm64-omnivoice-cpp"
     build-type: "metal"

.github/bump_deps.sh

@@ -1,5 +1,8 @@
 #!/bin/bash
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1
 BRANCH=$2
 VAR=$3
@@ -9,10 +12,11 @@ if [ -z "$FILE" ]; then
     FILE="Makefile"
 fi
 
-# -L so a renamed/transferred upstream repo (GitHub answers 301) still
-# resolves instead of handing us the redirect body, and -f so an HTTP error
-# aborts the run rather than letting an error page reach sed below.
-LAST_COMMIT=$(curl -sfL -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")
+# gh_curl follows redirects so a renamed/transferred upstream repo (GitHub
+# answers 301) still resolves, and fails on HTTP errors rather than letting an
+# error page reach sed below. `|| true` keeps a failed lookup from aborting the
+# script at exit 22 with no context — the SHA guard below reports it instead.
+LAST_COMMIT=$(gh_curl -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH" || true)
 
 # Guard the sed input: anything that is not a bare 40-hex SHA (an API error
 # body, an empty response) would otherwise be spliced into the Makefile pin —

.github/bump_docs.sh

@@ -1,7 +1,18 @@
 #!/bin/bash
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1
 
-LATEST_TAG=$(curl -s "https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
+LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
+    "https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
+
+# jq prints the string "null" for a missing key, so a throttled or otherwise
+# unexpected API response would otherwise be published as the docs version.
+if [ -z "$LATEST_TAG" ] || [ "$LATEST_TAG" = "null" ]; then
+    echo "Refusing to bump docs version: could not resolve the latest release tag for $REPO." >&2
+    exit 1
+fi
 
 cat <<< $(jq ".version = \"$LATEST_TAG\"" docs/data/version.json) > docs/data/version.json

.github/bump_vllm_metal.sh

@@ -11,6 +11,9 @@
 # darwin build can only use the exact vLLM version vllm-metal supports, so it may
 # lag the Linux pin (requirements-cublas13-after.txt) until vllm-metal catches up.
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1   # vllm-project/vllm-metal
 FILE=$2   # backend/python/vllm/install.sh
 VAR=$3    # VLLM_METAL_VERSION (used for the workflow's output file names)
@@ -22,12 +25,12 @@ fi
 
 # vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
 # /releases/latest returns the newest one (with its cp312 wheel asset).
-LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
+LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
     "https://api.github.com/repos/$REPO/releases/latest" \
     | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
 
 # The coupled vLLM source version lives in vllm-metal's installer at that tag.
-NEW_VLLM_VERSION=$(curl -fsSL \
+NEW_VLLM_VERSION=$(gh_curl \
     "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
     | grep -oE 'vllm_v="[0-9]+\.[0-9]+\.[0-9]+"' | head -1 | cut -d'"' -f2)
 

.github/bump_vllm_wheel.sh

@@ -9,6 +9,9 @@
 # vars in Makefiles; this script handles the two-value rewrite specific to the
 # vLLM requirements file.
 set -xe
+
+source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
+
 REPO=$1   # vllm-project/vllm
 FILE=$2   # backend/python/vllm/requirements-cublas13-after.txt
 VAR=$3    # VLLM_VERSION (used for output file names so the workflow can read them)
@@ -19,7 +22,7 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
 fi
 
 # /releases/latest returns the most recent non-prerelease tag.
-LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
+LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
     "https://api.github.com/repos/$REPO/releases/latest" \
     | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
 

.github/ci/apexentries/README.md

@@ -0,0 +1,194 @@
+# apexentries
+
+Generates gallery entries for the `mudler/*-APEX-GGUF` HuggingFace repositories.
+
+Each APEX repo becomes one **family**: one entry per quality rung the repo
+publishes and one per quantization rung its unsloth counterpart publishes, all
+gathered under the **base model's** entry. LocalAI's variant selector then picks
+the build that fits the hardware in front of it.
+
+## The hub is the base model entry, never a generated `*-apex` parent
+
+Somebody looking for `qwen3.6-35b-a3b` must find every build of those weights
+under that one name: the APEX imatrix rungs, the unsloth quant rungs and any
+speculative build. A separate `qwen3.6-35b-a3b-apex` hub competing with the base
+entry would split the family in two and leave whichever half the user did not
+search for effectively invisible.
+
+So the generator resolves the hub by stripping the `-APEX`, `-MTP` and `-TQ`
+markers and looking the result up in the index, trying both the repo-derived and
+the stem-derived candidate the same way `CounterpartCandidates` does. Then:
+
+- **The hub exists** (14 of the 45 repos, resolving to 10 distinct entries).
+  Nothing new is emitted for the family root. A `variants:` block is spliced into
+  the entry that is already there, textually, leaving its description, icon,
+  tags, overrides and files untouched. The line editing is shared with the
+  `variantproposals` job via `.github/ci/galleryedit`.
+- **The hub is absent** (the other 31). A new hub is emitted, named for the base
+  model and never for the APEX repo. It carries one of the discovered builds as
+  its own payload so it is a complete installable entry rather than a bare index,
+  and that payload is what gives it an `overrides.backend`. Without a declared
+  backend the verifier would skip it, so a hub carrying feature tags would escape
+  the tagging check in silence.
+
+Several APEX repos routinely resolve to one base model, so both paths accumulate
+by hub name rather than assuming one family per hub.
+
+Two references are always filtered out of a hub's list: anything the entry
+already declares, and the hub's own name. The self reference is not merely
+redundant. An unsloth rung whose weights the gallery already ships under the base
+name resolves, through the merge, straight back to the hub, and the verifier
+reads a self reference as a variant that declares variants of its own.
+
+The four hand-written `*-apex` entries (`qwen3.6-35b-a3b-apex`,
+`gemma-4-26b-a4b-it-apex`, `qwen3.5-35b-a3b-apex`,
+`nemotron-3-nano-omni-30b-a3b-reasoning-apex`) are **ordinary builds**, not hubs.
+They are referenced from their hub's variants list like any other rung, and are
+never deleted or renamed.
+
+## Flags
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `-index <path>` | `gallery/index.yaml` | Gallery index to dedup against. Read only, unless `-apply` is passed. |
+| `-only <a,b,c>` | (all) | Comma-separated full repo names (`mudler/Foo-APEX-GGUF`) to restrict generation to. A name that matches nothing is reported as a warning, since it is a typo rather than an empty result. |
+| `-out <path>` | (none) | Write the entries to add to this file. Nothing is written to the gallery. |
+| `-apply` | `false` | Splice the variants into `-index` and append the new entries to it. |
+| `-verify <path>` | (none) | Verify a gallery index and exit. Ignores every other flag. |
+
+Either `-out` or `-apply` is required, otherwise the run has nothing to do.
+
+`-apply` splices variant lines into existing entries and **appends** new ones. It
+never re-serialises the index: it is roughly 40,000 lines, and a YAML round trip
+would reflow the whole file, drop the anchors and merge keys the gallery relies
+on, and produce a diff nobody can review. On the three-family sample the splice
+is 24 added lines across 3 hunks with zero deletions.
+
+## Discovery is by filename suffix, never by repo name
+
+Builds come from the files a repo actually publishes. A filename is never
+constructed from a repo name, because the two disagree:
+`mudler/gemma-4-26B-A4B-it-APEX-GGUF` ships `gemma-4-26B-A4B-APEX-*.gguf`, and
+five other repos likewise drop a suffix (`-it`, `-2603`) or a vendor prefix
+(`NVIDIA-`) that the repo name carries. Composing a URL from the repo name would
+produce a 404 for every one of them, and the 404 would only surface after the
+entry shipped.
+
+The quality ladder is matched on the trailing tier marker, `-(I-)?(Quality|
+Balanced|Compact|Mini|Nano).gguf`. The `I-` prefix marks the imatrix ladder. The
+imatrix ladder is emitted when it is non-empty and the plain ladder is used only
+as a fallback, because two of the 45 repos publish no imatrix tiers at all and
+must still contribute. Eleven repos carry a fifth `I-Nano` rung, so nothing
+assumes a fixed number of rungs.
+
+Every run prints, per repo, the counts that discovery accounted for. If the
+number of classified files is short of the number of `.gguf` files the repo
+publishes, the shortfall is printed as `UNCLASSIFIED`. That check is a set
+difference on counts rather than a second pass over filenames: a second matcher
+would duplicate the tier regex and the two copies would drift. The failure it
+catches is quiet. A publishing-script typo that breaks every imatrix filename in
+a repo does not produce a short ladder; it makes the imatrix ladder empty, and
+the fallback then downgrades the whole family to the plain ladder with nothing
+said. A downstream HTTP check cannot catch it either, because it validates the
+URLs that were emitted, and an undiscovered tier emits none.
+
+The same reasoning applies to `UNACCOUNTED QUANT`, printed when the unsloth
+counterpart demonstrably publishes a wanted quant that produced no build. It is
+reported at discovery time because a dropped quant leaves no trace at all in the
+finished gallery file.
+
+## sha256 always comes from the API
+
+Every file stanza takes its `sha256` from the HuggingFace models API
+(`lfs.sha256`). A GGUF the API describes without one is a fatal error for that
+family: the repo is reported by name and the run ends non-zero. It is never
+substituted from another field, because that is exactly how a Xet hash ends up
+masquerading as a content hash.
+
+## The dflash / mtp tagging rule
+
+An entry is tagged `dflash` or `mtp` **if and only if** it configures the
+matching `spec_type:draft-<feature>`. Variant ranking reads tags and nothing
+else, so a tag that does not match the configuration either promotes a build
+that is no faster or hides one that genuinely is.
+
+A repo name is not configuration. `mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF` ships
+weights that carry MTP heads; an entry that does not enable them is not an MTP
+entry and is not tagged as one.
+
+A generated hub inherits the tags of the build it carries as its payload, rather
+than rebuilding them from the base set, so a hub whose payload configures a
+`spec_type` stays tagged consistently with the overrides copied alongside it.
+
+## Reuse reporting: two categories, not one
+
+Generated entries are deduped against the gallery and against the batch itself.
+The run prints the result under two separate headings, because the two cases are
+not equivalent:
+
+- **URI MATCHES** mean the gallery, or an earlier entry in this batch, already
+  ships exactly these weights. Pointing the hub at the existing entry is correct
+  and needs no thought.
+- **NAME COLLISIONS** mean an entry already owns the name but holds different
+  weights. Referencing it would point the hub at a build other than the one
+  generated. Every one of these must be inspected by hand.
+
+The run then prints `HUBS SPLICED`, listing every reference that will be added to
+an entry the gallery already ships along with the line it will be added at, and
+`HUBS CREATED` for the families that get a new hub. The splices are the part a
+review has to read closely, because they modify entries somebody else wrote.
+
+Hubs are deliberately kept out of the merge. A new hub carries the family's top
+rung as its own payload, so URI dedup would fold the hub into that rung and the
+family would lose the very entry point this command exists to create.
+
+## Workflow: sample first, then the full set
+
+Never run the full generation straight into the gallery. Generate a small,
+deliberately awkward sample, have it reviewed, then run the rest.
+
+```bash
+# 1. Sample three families that between them cover the awkward shapes:
+#    a standard four-rung repo, one with the extra I-Nano rung AND a file stem
+#    that differs from its repo name, and one whose unsloth counterpart shards
+#    its quants across subdirectories.
+go run ./.github/ci/apexentries \
+  -index gallery/index.yaml \
+  -only mudler/Qwen3.6-35B-A3B-APEX-GGUF,mudler/gemma-4-26B-A4B-it-APEX-GGUF,mudler/Step-3.7-Flash-APEX-GGUF \
+  -out /tmp/sample.yaml
+
+# 2. Verify the sample against the gallery it would join, splices included. Apply
+#    to a COPY, never to the real index, and check that the diff is only the
+#    intended variant lines. Compare the verifier output to the gallery's own
+#    baseline: what matters is that the sample adds no new problem, not that the
+#    total is zero.
+cp gallery/index.yaml /tmp/index-copy.yaml
+go run ./.github/ci/apexentries -index /tmp/index-copy.yaml -only <same list> -apply
+diff -u gallery/index.yaml /tmp/index-copy.yaml   # expect zero deletions
+
+go run ./.github/ci/apexentries -verify gallery/index.yaml > /tmp/baseline.log 2>&1
+go run ./.github/ci/apexentries -verify /tmp/index-copy.yaml > /tmp/spliced.log 2>&1
+diff /tmp/baseline.log /tmp/spliced.log
+
+# 3. Have a human review /tmp/sample.yaml and every reported name collision.
+
+# 4. Only then, the full set.
+go run ./.github/ci/apexentries -index gallery/index.yaml -apply
+```
+
+## Tests
+
+```bash
+go test ./.github/ci/apexentries/
+```
+
+The shared line editor has its own package:
+
+```bash
+go test ./.github/ci/galleryedit/
+```
+
+`.github/ci/` is invisible to `go list ./...`, so these specs are not covered by
+`make lint` or the repository test run. `.github/workflows/ci-tools-tests.yaml`
+names the package explicitly; keep that workflow in step with any package added
+under `.github/ci/`.

.github/ci/apexentries/discover.go

@@ -0,0 +1,70 @@
+package main
+
+import (
+	"regexp"
+	"strings"
+)
+
+// tierRE matches the tier marker APEX repos put at the end of a weight
+// filename. Discovery is by suffix because the stem is not predictable from
+// the repo name: six of the 45 repos drop a suffix ("-it", "-2603") or a
+// vendor prefix ("NVIDIA-") that the repo name carries.
+var tierRE = regexp.MustCompile(`-(I-)?(Quality|Balanced|Compact|Mini|Nano)\.gguf$`)
+
+// fullPrecisionRE matches the unquantized source weights an APEX repo publishes
+// alongside its ladder, flat (-F16.gguf) or sharded across a numbered set
+// (-F16-00001-of-00010.gguf). bf16 is accepted because some repos publish that
+// instead, and the match is case-insensitive because the casing varies between
+// publishing scripts.
+//
+// These are deliberately not tiers: they are the weights the ladder is quantized
+// FROM, and generation is scoped to the ladder itself.
+var fullPrecisionRE = regexp.MustCompile(`(?i)-b?f16(-\d{5}-of-\d{5})?\.gguf$`)
+
+// IsFullPrecision reports whether a weight filename is an unquantized source.
+func IsFullPrecision(name string) bool {
+	return fullPrecisionRE.MatchString(name)
+}
+
+// Tier is one discovered build of an APEX repo.
+type Tier struct {
+	Label string
+	File  GGUFFile
+}
+
+// DiscoverAPEXTiers splits a repo's weight files into the imatrix ladder and
+// the plain ladder. mmproj files are never tiers.
+func DiscoverAPEXTiers(files []GGUFFile) (imatrix, plain []Tier) {
+	for _, f := range files {
+		if strings.HasPrefix(f.Name, "mmproj") {
+			continue
+		}
+		m := tierRE.FindStringSubmatch(f.Name)
+		if m == nil {
+			continue
+		}
+		if m[1] != "" {
+			imatrix = append(imatrix, Tier{Label: "I-" + m[2], File: f})
+			continue
+		}
+		plain = append(plain, Tier{Label: m[2], File: f})
+	}
+	return imatrix, plain
+}
+
+// DiscoverMMProj returns the repo's projector file, if it publishes one. The
+// name varies across repos (mmproj.gguf, mmproj-F16.gguf,
+// mmproj-step3.7-flash-f16.gguf), so match the prefix rather than a fixed name.
+func DiscoverMMProj(files []GGUFFile) (GGUFFile, bool) {
+	for _, f := range files {
+		if strings.HasPrefix(f.Name, "mmproj") {
+			return f, true
+		}
+	}
+	return GGUFFile{}, false
+}
+
+// FileStem returns a tier's filename with its tier suffix removed.
+func FileStem(t Tier) string {
+	return tierRE.ReplaceAllString(t.File.Name, "")
+}

.github/ci/apexentries/discover_test.go

@@ -0,0 +1,68 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("DiscoverAPEXTiers", func() {
+	It("finds tiers regardless of how the stem relates to the repo name", func() {
+		// This repo is mudler/gemma-4-26B-A4B-it-APEX-GGUF but its files drop "-it".
+		files := []GGUFFile{
+			{Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf", SHA256: "a"},
+			{Name: "gemma-4-26B-A4B-APEX-I-Nano.gguf", SHA256: "b"},
+			{Name: "gemma-4-26B-A4B-APEX-Quality.gguf", SHA256: "c"},
+			{Name: "mmproj-F16.gguf", SHA256: "d"},
+		}
+
+		imatrix, plain := DiscoverAPEXTiers(files)
+
+		Expect(labels(imatrix)).To(ConsistOf("I-Quality", "I-Nano"))
+		Expect(labels(plain)).To(ConsistOf("Quality"))
+	})
+
+	It("excludes mmproj from the tier list", func() {
+		files := []GGUFFile{{Name: "mmproj.gguf", SHA256: "d"}}
+
+		imatrix, plain := DiscoverAPEXTiers(files)
+
+		Expect(imatrix).To(BeEmpty())
+		Expect(plain).To(BeEmpty())
+	})
+})
+
+var _ = Describe("DiscoverMMProj", func() {
+	It("finds an mmproj whatever its suffix", func() {
+		files := []GGUFFile{
+			{Name: "Model-APEX-I-Mini.gguf", SHA256: "a"},
+			{Name: "mmproj-step3.7-flash-f16.gguf", SHA256: "b"},
+		}
+
+		got, ok := DiscoverMMProj(files)
+
+		Expect(ok).To(BeTrue())
+		Expect(got.Name).To(Equal("mmproj-step3.7-flash-f16.gguf"))
+	})
+
+	It("reports absence when the repo ships none", func() {
+		_, ok := DiscoverMMProj([]GGUFFile{{Name: "Model-APEX-Quality.gguf", SHA256: "a"}})
+
+		Expect(ok).To(BeFalse())
+	})
+})
+
+var _ = Describe("FileStem", func() {
+	It("strips the tier suffix", func() {
+		t := Tier{Label: "I-Quality", File: GGUFFile{Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf"}}
+
+		Expect(FileStem(t)).To(Equal("gemma-4-26B-A4B-APEX"))
+	})
+})
+
+func labels(ts []Tier) []string {
+	out := make([]string, 0, len(ts))
+	for _, t := range ts {
+		out = append(out, t.Label)
+	}
+	return out
+}

.github/ci/apexentries/hf.go

@@ -0,0 +1,130 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+
+	"github.com/mudler/LocalAI/pkg/httpclient"
+)
+
+// ErrNoSHA256 marks a GGUF the HuggingFace API describes without an
+// lfs.sha256. Emitting an entry without a hash would ship an unverifiable
+// download, and guessing one from another field is how a Xet hash ends up
+// masquerading as a content hash, so this is fatal rather than skippable.
+var ErrNoSHA256 = errors.New("gguf file has no lfs.sha256")
+
+// GGUFFile is one .gguf sibling of a HuggingFace repo.
+type GGUFFile struct {
+	Name   string
+	Size   int64
+	SHA256 string
+}
+
+type apiSibling struct {
+	RFilename string `json:"rfilename"`
+	Size      int64  `json:"size"`
+	LFS       *struct {
+		SHA256 string `json:"sha256"`
+	} `json:"lfs"`
+}
+
+type apiModel struct {
+	Siblings []apiSibling `json:"siblings"`
+}
+
+// ParseRepoFiles returns every .gguf sibling described by a models API body.
+func ParseRepoFiles(body []byte) ([]GGUFFile, error) {
+	var m apiModel
+	if err := json.Unmarshal(body, &m); err != nil {
+		return nil, fmt.Errorf("decoding model response: %w", err)
+	}
+
+	var out []GGUFFile
+	for _, s := range m.Siblings {
+		if !strings.HasSuffix(s.RFilename, ".gguf") {
+			continue
+		}
+		if s.LFS == nil || s.LFS.SHA256 == "" {
+			return nil, fmt.Errorf("%s: %w", s.RFilename, ErrNoSHA256)
+		}
+		out = append(out, GGUFFile{Name: s.RFilename, Size: s.Size, SHA256: s.LFS.SHA256})
+	}
+	return out, nil
+}
+
+// FetchOptionalRepoFiles asks the models API for a repo the caller can do
+// without, and reports separately whether the repo was merely unreadable.
+//
+// HuggingFace answers 401 Unauthorized, not 404, for a repo that does not exist
+// when the request carries no credentials. Without a token there is therefore no
+// way to tell "this repo was never published" from "this repo is private", so an
+// optional probe has to treat 401 and 403 exactly like 404: whatever the reason,
+// there is nothing here for us to read, so there is no counterpart.
+//
+// The second return value exists because that collapse is lossy in one
+// direction: 401/403 can also mean a real, gated repo whose quants we would
+// genuinely want. The caller reports those repos so a silently dropped
+// counterpart is visible to a human rather than invisible.
+func FetchOptionalRepoFiles(client *http.Client, repo string) ([]GGUFFile, bool, error) {
+	files, status, err := fetchRepoFiles(client, repo)
+	if err != nil && (status == http.StatusUnauthorized || status == http.StatusForbidden) {
+		return nil, true, nil
+	}
+	return files, false, err
+}
+
+// FetchRepoFiles asks the models API for one repo. A 404 yields (nil, nil) so
+// that probing for an optional counterpart repo is not an error. Every other
+// non-200, 401 and 403 included, is an error: for a repo the run REQUIRES there
+// is no benign reading of "we cannot see it".
+func FetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, error) {
+	files, _, err := fetchRepoFiles(client, repo)
+	return files, err
+}
+
+// fetchRepoFiles does the request and returns the HTTP status alongside the
+// result, so the optional and required callers can apply different policies to
+// the same response without duplicating the request.
+func fetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, int, error) {
+	url := fmt.Sprintf("https://huggingface.co/api/models/%s?blobs=true", repo)
+	req, err := http.NewRequest(http.MethodGet, url, nil)
+	if err != nil {
+		return nil, 0, err
+	}
+	req.Header.Set("User-Agent", "localai-apexentries/1.0")
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, 0, err
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusNotFound {
+		return nil, resp.StatusCode, nil
+	}
+	if resp.StatusCode != http.StatusOK {
+		return nil, resp.StatusCode, fmt.Errorf("%s: unexpected status %d", repo, resp.StatusCode)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, resp.StatusCode, err
+	}
+	files, err := ParseRepoFiles(body)
+	return files, resp.StatusCode, err
+}
+
+// newHTTPClient builds the client used against the HuggingFace API. It goes
+// through pkg/httpclient rather than a bare &http.Client{} because the std
+// client follows redirects and forwards custom credential headers to the
+// redirect target on a cross-host hop (GHSA-3mj3-57v2-4636). This caller sends
+// only a User-Agent today, but it talks to an external API that could start
+// redirecting, and an HF_TOKEN header here later would then leak.
+func newHTTPClient() *http.Client {
+	return httpclient.NewWithTimeout(60 * time.Second)
+}

.github/ci/apexentries/hf_test.go

@@ -0,0 +1,142 @@
+package main
+
+import (
+	"bytes"
+	"io"
+	"net/http"
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestApexEntries(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "apexentries")
+}
+
+// stubTransport answers every request with one canned status and body, so the
+// status handling of the fetchers can be exercised without reaching the real
+// HuggingFace API.
+type stubTransport struct {
+	status int
+	body   string
+}
+
+func (t stubTransport) RoundTrip(req *http.Request) (*http.Response, error) {
+	return &http.Response{
+		StatusCode: t.status,
+		Body:       io.NopCloser(bytes.NewBufferString(t.body)),
+		Header:     make(http.Header),
+		Request:    req,
+	}, nil
+}
+
+func stubClient(status int, body string) *http.Client {
+	return &http.Client{Transport: stubTransport{status: status, body: body}}
+}
+
+const oneGGUFBody = `{"siblings":[{"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}}]}`
+
+var _ = Describe("FetchOptionalRepoFiles", func() {
+	// HuggingFace answers 401 rather than 404 for a repo that does not exist
+	// when the client carries no credentials, so an optional probe cannot tell
+	// "absent" from "unauthorized" and must treat both as "no counterpart".
+	It("treats a 401 as an absent repo and flags it as unavailable", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusUnauthorized, ""), "unsloth/Nope-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+		Expect(unavailable).To(BeTrue())
+	})
+
+	It("treats a 403 as an absent repo and flags it as unavailable", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusForbidden, ""), "unsloth/Gated-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+		Expect(unavailable).To(BeTrue())
+	})
+
+	// A clean 404 is an unambiguous absence, so it must NOT be reported as
+	// unavailable: the whole point of the flag is to separate the ambiguous
+	// case a human may need to look at from the settled one.
+	It("treats a 404 as an absent repo without flagging it as unavailable", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusNotFound, ""), "unsloth/Nope-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+		Expect(unavailable).To(BeFalse())
+	})
+
+	It("parses a 200 body as usual", func() {
+		files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusOK, oneGGUFBody), "unsloth/Real-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(unavailable).To(BeFalse())
+		Expect(files).To(HaveLen(1))
+		Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf"))
+		Expect(files[0].SHA256).To(Equal("aa"))
+	})
+
+	// Tolerating 401/403 must not widen into tolerating everything: a 500 is a
+	// broken API, not evidence about whether the repo exists.
+	It("still errors on a 500", func() {
+		_, _, err := FetchOptionalRepoFiles(stubClient(http.StatusInternalServerError, ""), "unsloth/Real-GGUF")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("unexpected status 500"))
+	})
+})
+
+var _ = Describe("FetchRepoFiles", func() {
+	// The APEX repo itself is not optional. A 401 there means the repo the run
+	// was asked to publish cannot be read, which is a real failure and must not
+	// be quietly downgraded to "no files".
+	It("errors on a 401 for a required repo", func() {
+		_, err := FetchRepoFiles(stubClient(http.StatusUnauthorized, ""), "mudler/Model-APEX-GGUF")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("unexpected status 401"))
+	})
+
+	It("errors on a 403 for a required repo", func() {
+		_, err := FetchRepoFiles(stubClient(http.StatusForbidden, ""), "mudler/Model-APEX-GGUF")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("unexpected status 403"))
+	})
+
+	It("still treats a 404 as an absent repo", func() {
+		files, err := FetchRepoFiles(stubClient(http.StatusNotFound, ""), "mudler/Model-APEX-GGUF")
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(BeEmpty())
+	})
+})
+
+var _ = Describe("ParseRepoFiles", func() {
+	It("returns gguf siblings with their lfs sha256", func() {
+		body := []byte(`{"siblings":[
+			{"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}},
+			{"rfilename":"README.md"},
+			{"rfilename":"mmproj.gguf","size":5,"lfs":{"sha256":"bb","size":5}}
+		]}`)
+
+		files, err := ParseRepoFiles(body)
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(files).To(HaveLen(2))
+		Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf"))
+		Expect(files[0].SHA256).To(Equal("aa"))
+		Expect(files[1].Name).To(Equal("mmproj.gguf"))
+	})
+
+	It("reports a gguf that carries no lfs sha256", func() {
+		body := []byte(`{"siblings":[{"rfilename":"mmproj.gguf","size":5}]}`)
+
+		_, err := ParseRepoFiles(body)
+
+		Expect(err).To(MatchError(ErrNoSHA256))
+	})
+})

.github/ci/apexentries/hub.go

@@ -0,0 +1,143 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
+)
+
+// IndexText is the gallery index seen as text: the entries it declares plus the
+// exact lines each one occupies, which is what splicing a variants block into an
+// entry the gallery already ships requires.
+//
+// It is a second, narrower read of the same file LoadExisting parses. The two
+// answer different questions: LoadExisting answers "do these weights already
+// exist anywhere", this one answers "where in the file does this entry live".
+type IndexText struct {
+	Lines   []string
+	Entries []*indexEntry
+
+	byName map[string]*indexEntry
+}
+
+// indexEntry is one entry of the index: its name, the variants it already
+// declares, and its coordinates in the file.
+type indexEntry struct {
+	Name     string       `yaml:"name"`
+	Variants []VariantRef `yaml:"variants"`
+
+	Pos galleryedit.Entry `yaml:"-"`
+}
+
+// LoadIndexText reads the gallery index for editing.
+func LoadIndexText(path string) (*IndexText, error) {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return nil, err
+	}
+	return ParseIndexText(string(raw))
+}
+
+// ParseIndexText pairs the decoded entries with the top level list items the
+// text actually contains.
+//
+// If the two views disagree on how many entries there are then every line number
+// a splice would compute is suspect, and the failure mode is writing a variants
+// block into the wrong model. The parse refuses instead.
+func ParseIndexText(text string) (*IndexText, error) {
+	var entries []*indexEntry
+	if err := yaml.Unmarshal([]byte(text), &entries); err != nil {
+		return nil, fmt.Errorf("decoding gallery index: %w", err)
+	}
+
+	lines, starts := galleryedit.Scan(text)
+	if len(starts) != len(entries) {
+		return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number",
+			len(entries), len(starts))
+	}
+
+	ix := &IndexText{Lines: lines, Entries: entries, byName: map[string]*indexEntry{}}
+	for i, e := range entries {
+		if e == nil {
+			return nil, fmt.Errorf("gallery index list item %d is empty; refusing to edit by line number", i)
+		}
+		end := len(lines)
+		if i+1 < len(starts) {
+			end = starts[i+1]
+		}
+		e.Pos = galleryedit.Entry{Name: e.Name, StartLine: starts[i], EndLine: end}
+
+		// First occurrence wins, matching the gallery's own resolution.
+		key := strings.ToLower(e.Name)
+		if _, seen := ix.byName[key]; !seen {
+			ix.byName[key] = e
+		}
+	}
+	return ix, nil
+}
+
+// Find looks an entry up by name, case insensitively.
+func (ix *IndexText) Find(name string) *indexEntry {
+	return ix.byName[strings.ToLower(name)]
+}
+
+// ResolveHub returns the gallery name of a family's hub and whether the gallery
+// already ships an entry under it.
+//
+// The hub is the BASE model entry, never a generated *-apex parent. Somebody
+// looking for qwen3.6-35b-a3b has to find every build of those weights under
+// that one name: the APEX imatrix rungs, the unsloth quant rungs and any
+// speculative build. A separate qwen3.6-35b-a3b-apex hub competing with the base
+// entry would split the family in two and leave whichever half the user did not
+// search for invisible.
+//
+// Both candidates are tried for the same reason CounterpartCandidates tries
+// both. The repo name and the published file stem disagree for several of these
+// repos, and either one may be what the base entry was named after.
+func ResolveHub(ix *IndexText, repoBase, stem string) (name string, exists bool) {
+	candidates := CounterpartCandidates(repoBase, stem)
+	for _, c := range candidates {
+		if n := slug(c); ix.Find(n) != nil {
+			return n, true
+		}
+	}
+	// Nothing matched, so the family needs a hub of its own under the repo
+	// derived name, which is the more reliable of the two.
+	return slug(candidates[0]), false
+}
+
+// HubLabel is the human-cased base model name, for prose rather than lookup.
+func HubLabel(repoBase, stem string) string {
+	return CounterpartCandidates(repoBase, stem)[0]
+}
+
+// filterVariants drops the references a hub must not carry: itself, and anything
+// it already lists.
+//
+// The self reference is not merely redundant. A hub that names itself makes the
+// verifier resolve the reference back to the hub, see that the hub declares
+// variants, and report a variant that declares variants of its own. It arises
+// for real rather than in theory: an unsloth rung whose weights the gallery
+// already ships under the base model name resolves, through Merge, straight back
+// to the hub that is about to reference it.
+func filterVariants(hub string, already []VariantRef, want []string) []string {
+	seen := map[string]bool{strings.ToLower(hub): true}
+	for _, v := range already {
+		seen[strings.ToLower(v.Model)] = true
+	}
+
+	var out []string
+	for _, w := range want {
+		key := strings.ToLower(w)
+		if seen[key] {
+			continue
+		}
+		seen[key] = true
+		out = append(out, w)
+	}
+	return out
+}

.github/ci/apexentries/main.go

@@ -0,0 +1,795 @@
+// Command apexentries generates gallery entries for the mudler APEX GGUF
+// repositories: one entry per imatrix tier and per unsloth quant rung, all
+// gathered under the BASE model's entry. Builds off a *-APEX-MTP-GGUF repo turn
+// speculative decoding on, because those weights retain the model's MTP heads
+// and are only worth their extra size with the heads in use.
+//
+// The base model entry is the hub. Somebody looking for qwen3.6-35b-a3b must
+// find every build of those weights under that one name, so when the gallery
+// already ships the base entry this command splices a variants block into it
+// rather than emitting a competing *-apex parent beside it. Only a family whose
+// base model the gallery does not ship at all gets a new hub entry, and that one
+// is still named for the base model.
+//
+// Builds are discovered by inspecting the filenames a repo actually publishes.
+// Repo names do not reliably predict them: mudler/gemma-4-26B-A4B-it-APEX-GGUF
+// ships gemma-4-26B-A4B-APEX-*.gguf, and six of the 45 repos drop a suffix or a
+// vendor prefix in the same way.
+package main
+
+import (
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path"
+	"sort"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
+)
+
+const (
+	// entryTemplate carries no backend and no parameters of its own, which is
+	// why RenderChild states everything inline.
+	entryTemplate = "virtual.yaml"
+	unslothOwner  = "unsloth"
+	authorListURL = "https://huggingface.co/api/models?author=mudler&limit=300"
+)
+
+// rungRank orders the quality ladder from best to smallest. The HuggingFace API
+// returns siblings alphabetically and DiscoverAPEXTiers preserves that order, so
+// an unsorted variants list reads I-Balanced, I-Compact, I-Mini, I-Nano,
+// I-Quality. Selection ignores authored order, so this is purely so the file a
+// human reviews scans in a meaningful sequence.
+var rungRank = map[string]int{
+	"I-Quality": 0, "I-Balanced": 1, "I-Compact": 2, "I-Mini": 3, "I-Nano": 4,
+	"Quality": 5, "Balanced": 6, "Compact": 7, "Mini": 8, "Nano": 9,
+}
+
+// baseTags are the tags every generated entry carries. dflash and mtp are never
+// among them: RenderChild adds those if and only if the entry configures the
+// matching spec_type.
+var baseTags = []string{"llm", "gguf", "cpu", "gpu"}
+
+// childBuild pairs a rendered entry with its position on the quality ladder, so
+// the parent's variants list can be sorted without re-parsing entry names.
+type childBuild struct {
+	entry GalleryEntry
+	rank  int
+}
+
+// family is one APEX repo's full generated output.
+type family struct {
+	repo      string
+	repoBase  string
+	stem      string
+	hasMMProj bool
+	children  []childBuild
+
+	// skippedRepos are counterpart candidates HuggingFace would not describe.
+	// Carried on the family rather than printed and forgotten so the run can
+	// summarize them next to everything else a reviewer has to eyeball.
+	skippedRepos []string
+	census       fileCensus
+	unaccounted  int
+}
+
+// fileCensus splits the files discovery emitted nothing for into the ones a
+// reviewer must chase and the ones that are deliberately out of scope.
+//
+// Full-precision sources are the second kind: they are the unquantized weights
+// the ladder is derived FROM, not a rung of it. Folding them into the
+// unclassified total would leave a permanent benign baseline, and a permanent
+// baseline is exactly what hides the one file that ever genuinely matters.
+type fileCensus struct {
+	unclassified  int
+	fullPrecision int
+}
+
+// add accumulates one repo's census into a running total.
+func (c *fileCensus) add(o fileCensus) {
+	c.unclassified += o.unclassified
+	c.fullPrecision += o.fullPrecision
+}
+
+// sortedChildren returns the family's builds in ladder order, best first.
+func (f *family) sortedChildren() []childBuild {
+	sorted := append([]childBuild{}, f.children...)
+	sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].rank < sorted[j].rank })
+	return sorted
+}
+
+func main() {
+	verify := flag.String("verify", "", "verify a gallery index and exit")
+	index := flag.String("index", "gallery/index.yaml", "gallery index to dedup against")
+	only := flag.String("only", "", "comma-separated repo names to restrict generation to")
+	out := flag.String("out", "", "write the entries to add to this file")
+	apply := flag.Bool("apply", false, "append the entries to add to -index")
+	flag.Parse()
+
+	if *verify != "" {
+		problems := Verify(*verify)
+		for _, p := range problems {
+			fmt.Fprintln(os.Stderr, p)
+		}
+		if len(problems) > 0 {
+			fmt.Fprintf(os.Stderr, "%d problem(s)\n", len(problems))
+			os.Exit(1)
+		}
+		fmt.Println("index is sound")
+		return
+	}
+
+	if err := generate(*index, *only, *out, *apply); err != nil {
+		fmt.Fprintln(os.Stderr, "error:", err)
+		os.Exit(1)
+	}
+}
+
+func generate(indexPath, only, outPath string, apply bool) error {
+	if outPath == "" && !apply {
+		return fmt.Errorf("nothing to do: pass -out <file> or -apply")
+	}
+
+	client := newHTTPClient()
+
+	repos, err := listAPEXRepos(client)
+	if err != nil {
+		return err
+	}
+	if only != "" {
+		repos = restrict(repos, only)
+	}
+	if len(repos) == 0 {
+		return fmt.Errorf("no APEX repos selected")
+	}
+	fmt.Printf("repos selected: %d\n", len(repos))
+
+	var families []family
+	var failed []string
+
+	for _, repo := range repos {
+		f, err := buildFamily(client, repo)
+		if err != nil {
+			// A missing sha256 is fatal for the family rather than skippable: an
+			// entry without one ships an unverifiable download. Report which repo
+			// and keep going, so one bad repo does not hide the state of the rest.
+			fmt.Fprintf(os.Stderr, "FAILED %s: %v\n", repo, err)
+			failed = append(failed, repo)
+			continue
+		}
+		families = append(families, *f)
+	}
+
+	existing, err := LoadExisting(indexPath)
+	if err != nil {
+		return err
+	}
+	ixText, err := LoadIndexText(indexPath)
+	if err != nil {
+		return err
+	}
+	fmt.Printf("existing index: %d names, %d weight URIs, %d lines\n",
+		len(existing.ByName), len(existing.ByURI), len(ixText.Lines))
+
+	// Only the builds go through Merge. A hub is deliberately kept out of it: a
+	// new hub carries the family's top rung as its own payload, so Merge's URI
+	// dedup would fold the hub into that rung and the family would lose the very
+	// entry point this command exists to create. Hub names are checked against
+	// the index directly, by ResolveHub.
+	var generated []GalleryEntry
+	for _, f := range families {
+		for _, c := range f.children {
+			generated = append(generated, c.entry)
+		}
+	}
+
+	add, reused := Merge(existing, generated)
+	reportReuse(existing, generated, reused)
+
+	// Variant references are resolved from `reused`, never used to decide what to
+	// emit: on a within-batch name collision Merge records reused[name] = name
+	// while the first entry of that name is still in `add`, so treating presence
+	// in `reused` as "dropped" would silently emit nothing for it.
+	added := map[string]bool{}
+	for _, e := range add {
+		added[e.Name] = true
+	}
+
+	inserts, newHubs, err := planHubs(families, ixText, reused, added)
+	if err != nil {
+		return err
+	}
+	reportHubs(ixText, inserts, newHubs)
+
+	skipped, census, fullPrecisionRepos, unaccounted := reportSkipped(families)
+
+	add = append(add, newHubs...)
+
+	fmt.Printf("\nentries generated: %d\nentries to add:    %d\nentries reused:    %d\nhubs spliced:      %d\nhubs created:      %d\nrepos skipped:     %d\nexcluded (full precision): %d files across %d repos\nunclassified:      %d\nunaccounted:       %d\n",
+		len(generated), len(add), len(reused), len(inserts), len(newHubs), len(skipped),
+		census.fullPrecision, fullPrecisionRepos, census.unclassified, unaccounted)
+
+	lines, err := galleryedit.Apply(ixText.Lines, inserts)
+	if err != nil {
+		return err
+	}
+
+	if err := writeEntries(add, lines, outPath, apply, indexPath); err != nil {
+		return err
+	}
+
+	if len(failed) > 0 {
+		return fmt.Errorf("%d repo(s) failed: %s", len(failed), strings.Join(failed, ", "))
+	}
+	return nil
+}
+
+// resolveVariant maps a generated child name onto whatever entry actually stands
+// for it after the merge. `added` is consulted first because a within-batch name
+// collision puts a name in BOTH add and reused, and the entry that was emitted
+// is the one the parent must reference.
+func resolveVariant(name string, reused map[string]string, added map[string]bool) string {
+	if added[name] {
+		return name
+	}
+	if target, ok := reused[name]; ok {
+		return target
+	}
+	return name
+}
+
+// SpecTypeForRepo reports the speculative decoding mechanism a repo's builds can
+// turn on with no extra download.
+//
+// The *-APEX-MTP-GGUF repos republish the base weights with the model's own MTP
+// heads retained, so those builds are only worth their extra size if the heads
+// are actually used. Every other APEX repo drops them, and switching MTP on
+// there would name a mechanism the weights cannot serve.
+//
+// The suffix is read off the repo the FILES come from, so nothing downstream has
+// to infer a capability from an entry name.
+func SpecTypeForRepo(repo string) string {
+	if strings.HasSuffix(path.Base(repo), "-APEX-MTP-GGUF") {
+		return "draft-mtp"
+	}
+	return ""
+}
+
+// buildFamily discovers everything one APEX repo and its unsloth counterpart
+// publish, and renders it.
+func buildFamily(client *http.Client, repo string) (*family, error) {
+	files, err := FetchRepoFiles(client, repo)
+	if err != nil {
+		return nil, err
+	}
+	if len(files) == 0 {
+		return nil, fmt.Errorf("no gguf files")
+	}
+
+	imatrix, plain := DiscoverAPEXTiers(files)
+	mmproj, hasMMProj := DiscoverMMProj(files)
+
+	census := reportUnclassified(repo, files, imatrix, plain)
+
+	// The imatrix ladder is preferred, but two of the 45 repos publish no
+	// imatrix tiers at all and must still contribute their plain ladder.
+	ladder := imatrix
+	ladderKind := "imatrix"
+	if len(ladder) == 0 {
+		ladder = plain
+		ladderKind = "plain"
+	}
+	if len(ladder) == 0 {
+		return nil, fmt.Errorf("no tiers discovered")
+	}
+
+	sortTiers(ladder)
+
+	var mm *GGUFFile
+	if hasMMProj {
+		mm = &mmproj
+	}
+
+	repoBase := strings.TrimSuffix(path.Base(repo), "-GGUF")
+	f := &family{repo: repo, repoBase: repoBase, hasMMProj: hasMMProj, census: census}
+
+	// Only the APEX ladder can carry MTP heads; the unsloth counterpart quantizes
+	// the plain weights and gets nothing from this.
+	specType := SpecTypeForRepo(repo)
+
+	for _, t := range ladder {
+		f.children = append(f.children, childBuild{
+			rank: rungRank[t.Label],
+			entry: RenderChild(ChildInput{
+				Name:     slug(repoBase) + "-" + slug(t.Label),
+				Repo:     repo,
+				Template: entryTemplate,
+				SpecType: specType,
+				Weights:  []GGUFFile{t.File},
+				MMProj:   mm,
+				BaseTags: baseTags,
+			}),
+		})
+	}
+
+	stem := FileStem(ladder[0])
+	f.stem = stem
+	fmt.Printf("%s: %d %s tier(s) [%s], stem %s, mmproj %v\n",
+		repo, len(ladder), ladderKind, tierLabels(ladder), stem, hasMMProj)
+
+	counterpart, cpFiles, skipped, err := resolveCounterpart(client, repoBase, stem)
+	f.skippedRepos = skipped
+	if err != nil {
+		return nil, err
+	}
+	if counterpart != "" {
+		builds := DiscoverUnslothQuants(cpFiles)
+
+		// Called here rather than inside Verify: a quant dropped at discovery
+		// leaves no trace at all in the finished gallery file, so the only place
+		// the shortfall is still visible is the moment of discovery.
+		unaccounted := UnaccountedQuants(cpFiles, builds)
+		f.unaccounted = len(unaccounted)
+		for _, p := range unaccounted {
+			fmt.Fprintf(os.Stderr, "UNACCOUNTED QUANT %s: %s\n", counterpart, p)
+		}
+
+		cpMMProj, hasCPMMProj := DiscoverMMProj(cpFiles)
+		var cpMM *GGUFFile
+		if hasCPMMProj {
+			cpMM = &cpMMProj
+		}
+		cpBase := strings.TrimSuffix(path.Base(counterpart), "-GGUF")
+
+		for i, b := range builds {
+			f.children = append(f.children, childBuild{
+				rank: 100 + i,
+				entry: RenderChild(ChildInput{
+					Name:     slug(cpBase) + "-" + slug(b.Quant),
+					Repo:     counterpart,
+					Template: entryTemplate,
+					Weights:  b.Files,
+					MMProj:   cpMM,
+					BaseTags: baseTags,
+				}),
+			})
+		}
+		fmt.Printf("%s: counterpart %s, %d quant build(s) %s\n", repo, counterpart, len(builds), quantLabels(builds))
+	} else {
+		fmt.Printf("%s: no unsloth counterpart\n", repo)
+	}
+
+	return f, nil
+}
+
+// planHubs decides, per family, whether the family's builds are spliced into a
+// base model entry the gallery already ships or gathered under a new hub.
+//
+// Splicing is strongly preferred and is the measured majority-adjacent case. The
+// existing entry keeps its description, icon, tags, overrides and files
+// untouched; only variant lines are added to it.
+func planHubs(families []family, ix *IndexText, reused map[string]string, added map[string]bool) ([]galleryedit.Insert, []GalleryEntry, error) {
+	// Several APEX repos can resolve to one base model, so both paths accumulate
+	// by hub name rather than assuming one family per hub.
+	wantByHub := map[string][]string{}
+	var spliceOrder []string
+
+	var newHubs []GalleryEntry
+	hubAt := map[string]int{}
+
+	for i := range families {
+		f := &families[i]
+
+		hubName, exists := ResolveHub(ix, f.repoBase, f.stem)
+		want := hubVariants(f, ix, reused, added)
+
+		if exists {
+			if _, seen := wantByHub[hubName]; !seen {
+				spliceOrder = append(spliceOrder, hubName)
+			}
+			wantByHub[hubName] = append(wantByHub[hubName], want...)
+			continue
+		}
+
+		if at, dup := hubAt[hubName]; dup {
+			for _, v := range filterVariants(hubName, newHubs[at].Variants, want) {
+				newHubs[at].Variants = append(newHubs[at].Variants, VariantRef{Model: v})
+			}
+			continue
+		}
+
+		builds := f.sortedChildren()
+		if len(builds) == 0 {
+			return nil, nil, fmt.Errorf("%s: no builds to hang a hub on", f.repo)
+		}
+		hubAt[hubName] = len(newHubs)
+		newHubs = append(newHubs, renderHub(hubName, f, builds[0], filterVariants(hubName, nil, want)))
+	}
+
+	var inserts []galleryedit.Insert
+	for _, name := range spliceOrder {
+		e := ix.Find(name)
+		items := filterVariants(name, e.Variants, wantByHub[name])
+		if len(items) == 0 {
+			continue
+		}
+		inserts = append(inserts, galleryedit.Insert{Entry: e.Pos, Variants: items})
+	}
+	return inserts, newHubs, nil
+}
+
+// hubVariants is a family's full build list, in ladder order, named as the hub
+// must reference them after the merge.
+func hubVariants(f *family, ix *IndexText, reused map[string]string, added map[string]bool) []string {
+	var out []string
+
+	// A hand-written *-apex entry is an ordinary build of these weights. It is
+	// never deleted, never renamed and never treated as a hub; it is simply
+	// referenced like any other rung.
+	if apex := slug(f.repoBase); ix.Find(apex) != nil {
+		out = append(out, apex)
+	}
+	for _, c := range f.sortedChildren() {
+		out = append(out, resolveVariant(c.entry.Name, reused, added))
+	}
+	return out
+}
+
+// renderHub builds the hub for a family whose base model the gallery does not
+// ship at all. It is named for the BASE model, never for the APEX repo.
+//
+// It carries one of the discovered builds as its own payload so it is a complete
+// installable entry rather than a bare index pointing at other entries. That
+// payload is what supplies overrides.backend, which matters beyond installation:
+// the verifier can only judge the tagging rule for a backend it can read, so a
+// hub carrying feature tags and no backend would escape the check in silence.
+//
+// The payload's own tags are kept rather than rebuilt from baseTags, so a hub
+// whose payload configures a spec_type stays tagged for it and consistent with
+// the overrides copied alongside.
+func renderHub(name string, f *family, payload childBuild, variants []string) GalleryEntry {
+	e := payload.entry
+	e.Name = name
+	e.Description = fmt.Sprintf(
+		"%s. Quality ladder and quantization rungs published by %s and its unsloth counterpart; LocalAI picks the build that fits the hardware.",
+		HubLabel(f.repoBase, f.stem), f.repo)
+
+	e.Tags = append([]string{}, payload.entry.Tags...)
+	if f.hasMMProj && !hasTag(e.Tags, "vision") {
+		e.Tags = append(e.Tags, "vision")
+	}
+
+	e.Variants = nil
+	for _, v := range variants {
+		e.Variants = append(e.Variants, VariantRef{Model: v})
+	}
+	return e
+}
+
+func hasTag(tags []string, want string) bool {
+	for _, t := range tags {
+		if t == want {
+			return true
+		}
+	}
+	return false
+}
+
+// resolveCounterpart probes the unsloth candidates in order and returns the
+// first that publishes files.
+//
+// CounterpartCandidates is handed a BARE repo name: its cleaner does not strip
+// an owner prefix, so passing "mudler/Foo-APEX-GGUF" would yield "mudler/Foo"
+// and compose into the nonsense probe "unsloth/mudler/Foo".
+//
+// It also returns the candidates HuggingFace refused to describe. Those are
+// indistinguishable from absent without credentials, so they are skipped, but
+// they are named rather than dropped: one of them could be a real gated repo
+// whose quants belong in the gallery.
+func resolveCounterpart(client *http.Client, repoBase, stem string) (string, []GGUFFile, []string, error) {
+	var unavailable []string
+	for _, cand := range CounterpartCandidates(repoBase, stem) {
+		repo := unslothOwner + "/" + cand + "-GGUF"
+		files, unreadable, err := FetchOptionalRepoFiles(client, repo)
+		if err != nil {
+			return "", nil, unavailable, fmt.Errorf("probing %s: %w", repo, err)
+		}
+		if unreadable {
+			unavailable = append(unavailable, repo)
+			continue
+		}
+		if len(files) > 0 {
+			return repo, files, unavailable, nil
+		}
+	}
+	return "", nil, unavailable, nil
+}
+
+// reportUnclassified prints the files discovery turned into nothing.
+//
+// It is a set difference on COUNTS, not a re-match of filenames: re-matching
+// would duplicate the tier regex from discover.go and the two copies would
+// drift. The likeliest trigger is a typo or case change from a publishing script
+// rather than a genuine sixth tier, and because generation falls back to the
+// plain ladder when the imatrix one is empty, a repo whose imatrix files all
+// fail to match silently downgrades the whole family instead of erroring. The
+// downstream HTTP check cannot catch that: it validates URLs that were emitted,
+// and an undiscovered tier emits none.
+// It returns the census so the run can total it.
+func reportUnclassified(repo string, files []GGUFFile, imatrix, plain []Tier) fileCensus {
+	mmprojCount, fullPrecision := 0, 0
+	for _, f := range files {
+		// The mmproj test comes first because projectors are themselves often
+		// published at f16 (mmproj-F16.gguf), and counting such a file in both
+		// buckets would understate the unclassified remainder.
+		if strings.HasPrefix(f.Name, "mmproj") {
+			mmprojCount++
+			continue
+		}
+		if IsFullPrecision(f.Name) {
+			fullPrecision++
+		}
+	}
+
+	classified := len(imatrix) + len(plain) + mmprojCount + fullPrecision
+	if classified >= len(files) {
+		return fileCensus{fullPrecision: fullPrecision}
+	}
+	fmt.Fprintf(os.Stderr, "UNCLASSIFIED %s: %d of %d .gguf files classified, %d unaccounted for\n",
+		repo, classified, len(files), len(files)-classified)
+	return fileCensus{unclassified: len(files) - classified, fullPrecision: fullPrecision}
+}
+
+// reportReuse splits Merge's single reused map into the two cases it conflates.
+//
+// A URI match means the gallery already ships exactly these weights, and
+// pointing the parent at the existing entry is correct. A NAME match with a
+// different URI means an unrelated entry happens to own the name, and
+// referencing it would point the parent at different weights than were
+// generated, substituting a build without saying so. Only the first is safe to
+// wave through.
+func reportReuse(existing *ExistingIndex, generated []GalleryEntry, reused map[string]string) {
+	byName := map[string]GalleryEntry{}
+	for _, e := range generated {
+		if _, seen := byName[e.Name]; !seen {
+			byName[e.Name] = e
+		}
+	}
+
+	var nameCollisions, uriMatches []string
+	for name, target := range reused {
+		gen := byName[name]
+		uri := ""
+		if len(gen.Files) > 0 {
+			uri = gen.Files[0].URI
+		}
+
+		switch {
+		case hasName(existing, name):
+			nameCollisions = append(nameCollisions,
+				fmt.Sprintf("  %s -> gallery entry of the same name (generated uri: %s)", name, orNone(uri)))
+		case target == name:
+			nameCollisions = append(nameCollisions,
+				fmt.Sprintf("  %s -> earlier entry of the same name in this batch (generated uri: %s)", name, orNone(uri)))
+		default:
+			uriMatches = append(uriMatches, fmt.Sprintf("  %s -> %s (same weights: %s)", name, target, orNone(uri)))
+		}
+	}
+	sort.Strings(nameCollisions)
+	sort.Strings(uriMatches)
+
+	fmt.Printf("\nNAME COLLISIONS (%d) - inspect each by hand, the target may hold different weights\n", len(nameCollisions))
+	for _, l := range nameCollisions {
+		fmt.Println(l)
+	}
+	fmt.Printf("\nURI MATCHES (%d) - the gallery or this batch already ships these exact weights\n", len(uriMatches))
+	for _, l := range uriMatches {
+		fmt.Println(l)
+	}
+}
+
+// reportHubs prints exactly what will be written where. The splices are the part
+// a human has to read: they modify entries the gallery already ships, so the
+// review needs the target, the line, and every added reference spelled out.
+func reportHubs(ix *IndexText, inserts []galleryedit.Insert, newHubs []GalleryEntry) {
+	fmt.Printf("\nHUBS SPLICED (%d) - variants added to the EXISTING base model entry, nothing else touched\n", len(inserts))
+	for _, in := range inserts {
+		e := ix.Find(in.Entry.Name)
+		fmt.Printf("  %s (line %d, %d variant(s) already declared):\n", in.Entry.Name, in.Entry.StartLine+1, len(e.Variants))
+		for _, v := range in.Variants {
+			fmt.Printf("    + - model: %s\n", galleryedit.QuoteName(v))
+		}
+	}
+
+	fmt.Printf("\nHUBS CREATED (%d) - the gallery ships no base model entry, so one is emitted for it\n", len(newHubs))
+	for _, h := range newHubs {
+		fmt.Printf("  %s:\n", h.Name)
+		for _, v := range h.Variants {
+			fmt.Printf("    - model: %s\n", v.Model)
+		}
+	}
+}
+
+// reportSkipped names the counterpart repos HuggingFace would not describe, and
+// totals the other two silent-shortfall counters alongside them.
+//
+// A skipped repo is not the same as a clean 404. HuggingFace answers 401 for a
+// nonexistent repo to an unauthenticated client, so the overwhelmingly likely
+// reading is "there is no such counterpart", which is the normal case for the
+// community merges. But a private or gated repo answers 401 too, and that one
+// WOULD have quants worth shipping. Printing the list is what keeps that
+// possibility auditable instead of silently discarded.
+func reportSkipped(families []family) ([]string, fileCensus, int, int) {
+	var skipped []string
+	var census fileCensus
+	fullPrecisionRepos, unaccounted := 0, 0
+	for _, f := range families {
+		skipped = append(skipped, f.skippedRepos...)
+		census.add(f.census)
+		if f.census.fullPrecision > 0 {
+			fullPrecisionRepos++
+		}
+		unaccounted += f.unaccounted
+	}
+	sort.Strings(skipped)
+
+	fmt.Printf("\nREPOS SKIPPED AS UNAVAILABLE (%d) - HuggingFace answered 401/403, which is indistinguishable from absent without a token; check none of these is a real gated repo\n", len(skipped))
+	for _, r := range skipped {
+		fmt.Printf("  %s\n", r)
+	}
+	return skipped, census, fullPrecisionRepos, unaccounted
+}
+
+func hasName(ix *ExistingIndex, name string) bool {
+	_, ok := ix.ByName[name]
+	return ok
+}
+
+func orNone(s string) string {
+	if s == "" {
+		return "(no files)"
+	}
+	return s
+}
+
+// writeEntries emits the additions.
+//
+// -apply does two things in one pass: it writes back the spliced lines, which
+// differ from the original only by the variant lines galleryedit inserted, and
+// then appends the new entries. New entries are APPENDED rather than merged into
+// the structure, for the same reason the splice is textual: a YAML round trip
+// over 40,000 lines would reflow the whole file into an unreviewable diff.
+func writeEntries(add []GalleryEntry, lines []string, outPath string, apply bool, indexPath string) error {
+	if apply {
+		if err := os.WriteFile(indexPath, []byte(strings.Join(lines, "\n")), 0o644); err != nil {
+			return err
+		}
+		fmt.Printf("spliced %s\n", indexPath)
+	}
+
+	if len(add) == 0 {
+		fmt.Println("nothing to append")
+		return nil
+	}
+
+	blob, err := yaml.Marshal(add)
+	if err != nil {
+		return err
+	}
+
+	if outPath != "" {
+		if err := os.WriteFile(outPath, blob, 0o644); err != nil {
+			return err
+		}
+		fmt.Printf("wrote %d entries to %s\n", len(add), outPath)
+	}
+
+	if apply {
+		f, err := os.OpenFile(indexPath, os.O_APPEND|os.O_WRONLY, 0o644)
+		if err != nil {
+			return err
+		}
+		defer f.Close()
+		if _, err := f.Write(blob); err != nil {
+			return err
+		}
+		fmt.Printf("appended %d entries to %s\n", len(add), indexPath)
+	}
+	return nil
+}
+
+// listAPEXRepos returns the mudler repos whose name marks them as APEX builds.
+func listAPEXRepos(client *http.Client) ([]string, error) {
+	req, err := http.NewRequest(http.MethodGet, authorListURL, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("User-Agent", "localai-apexentries/1.0")
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("listing models: unexpected status %d", resp.StatusCode)
+	}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+
+	var models []struct {
+		ID string `json:"id"`
+	}
+	if err := json.Unmarshal(body, &models); err != nil {
+		return nil, fmt.Errorf("decoding model list: %w", err)
+	}
+
+	var out []string
+	for _, m := range models {
+		if strings.Contains(m.ID, "APEX") {
+			out = append(out, m.ID)
+		}
+	}
+	sort.Strings(out)
+	return out, nil
+}
+
+func restrict(repos []string, only string) []string {
+	want := map[string]bool{}
+	for _, r := range strings.Split(only, ",") {
+		if r = strings.TrimSpace(r); r != "" {
+			want[r] = true
+		}
+	}
+
+	var out []string
+	for _, r := range repos {
+		if want[r] {
+			out = append(out, r)
+			delete(want, r)
+		}
+	}
+	// A name in -only that matched nothing is a typo, not an empty result.
+	for r := range want {
+		fmt.Fprintf(os.Stderr, "WARNING: -only names %s, which is not an APEX repo of this author\n", r)
+	}
+	return out
+}
+
+func sortTiers(tiers []Tier) {
+	sort.SliceStable(tiers, func(i, j int) bool { return rungRank[tiers[i].Label] < rungRank[tiers[j].Label] })
+}
+
+func tierLabels(tiers []Tier) string {
+	var out []string
+	for _, t := range tiers {
+		out = append(out, t.Label)
+	}
+	return strings.Join(out, ",")
+}
+
+func quantLabels(builds []QuantBuild) string {
+	var out []string
+	for _, b := range builds {
+		l := b.Quant
+		if b.Sharded {
+			l += fmt.Sprintf("(%d shards)", len(b.Files))
+		}
+		out = append(out, l)
+	}
+	return strings.Join(out, ",")
+}
+
+// slug turns a repo, tier or quant label into a gallery entry name component.
+func slug(s string) string {
+	return strings.ReplaceAll(strings.ToLower(s), "_", "-")
+}

.github/ci/apexentries/main_test.go

@@ -0,0 +1,344 @@
+package main
+
+import (
+	"strings"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
+)
+
+func mustIndex(text string) *IndexText {
+	ix, err := ParseIndexText(text)
+	ExpectWithOffset(1, err).ToNot(HaveOccurred())
+	return ix
+}
+
+// buildOf renders a realistic child so the specs exercise the payload a hub
+// actually inherits rather than a bare name.
+func buildOf(name, repo, file string, rank int) childBuild {
+	return childBuild{
+		rank: rank,
+		entry: RenderChild(ChildInput{
+			Name:     name,
+			Repo:     repo,
+			Template: entryTemplate,
+			Weights:  []GGUFFile{{Name: file, SHA256: "aa"}},
+			BaseTags: baseTags,
+		}),
+	}
+}
+
+var _ = Describe("ResolveHub", func() {
+	It("picks the base model name over the APEX name, even when both are in the gallery", func() {
+		// The hub is the entry a user searches for. If the *-apex entry were
+		// chosen the family would be gathered under a name nobody looks up, and
+		// the base entry would go on advertising only its own build.
+		ix := mustIndex("- name: qwen3.6-35b-a3b\n  url: u\n- name: qwen3.6-35b-a3b-apex\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX", "Qwen3.6-35B-A3B-APEX")
+
+		Expect(name).To(Equal("qwen3.6-35b-a3b"))
+		Expect(exists).To(BeTrue())
+	})
+
+	It("falls back to the stem-derived candidate when the repo-derived one is absent", func() {
+		// gemma's repo says "-it" and its published files do not, so only one of
+		// the two candidates can match whatever the base entry was named after.
+		ix := mustIndex("- name: gemma-4-26b-a4b\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "gemma-4-26B-A4B-it-APEX", "gemma-4-26B-A4B-APEX")
+
+		Expect(name).To(Equal("gemma-4-26b-a4b"))
+		Expect(exists).To(BeTrue())
+	})
+
+	It("reports the base name as absent rather than settling for the APEX entry", func() {
+		ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "Qwen3.5-35B-A3B-APEX", "Qwen3.5-35B-A3B-APEX")
+
+		Expect(name).To(Equal("qwen3.5-35b-a3b"))
+		Expect(exists).To(BeFalse())
+	})
+
+	It("strips the MTP and TQ markers as well as APEX", func() {
+		ix := mustIndex("- name: qwen3.6-35b-a3b\n  url: u\n")
+
+		name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX-MTP", "Qwen3.6-35B-A3B-APEX-MTP")
+
+		Expect(name).To(Equal("qwen3.6-35b-a3b"))
+		Expect(exists).To(BeTrue())
+	})
+})
+
+var _ = Describe("planHubs", func() {
+	noReuse := map[string]string{}
+	allAdded := func(names ...string) map[string]bool {
+		out := map[string]bool{}
+		for _, n := range names {
+			out[n] = true
+		}
+		return out
+	}
+
+	It("splices into the existing base entry instead of emitting an *-apex parent", func() {
+		ix := mustIndex("- name: step-3.7-flash\n  url: u\n- name: other\n  url: u\n")
+		fams := []family{{
+			repo:     "mudler/Step-3.7-Flash-APEX-GGUF",
+			repoBase: "Step-3.7-Flash-APEX",
+			stem:     "Step-3.7-Flash-APEX",
+			children: []childBuild{buildOf("step-3.7-flash-apex-i-quality", "mudler/Step-3.7-Flash-APEX-GGUF", "a.gguf", 0)},
+		}}
+
+		inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("step-3.7-flash-apex-i-quality"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(newHubs).To(BeEmpty())
+		Expect(inserts).To(HaveLen(1))
+		Expect(inserts[0].Entry.Name).To(Equal("step-3.7-flash"))
+		Expect(inserts[0].Variants).To(Equal([]string{"step-3.7-flash-apex-i-quality"}))
+	})
+
+	It("merges into an entry that already declares variants, without repeating one", func() {
+		// The gallery's qwen3.6-35b-a3b already lists its APEX build. Re-adding it
+		// would put a duplicate key's worth of noise in the diff and a duplicate
+		// reference in the entry.
+		ix := mustIndex("- name: qwen3.6-35b-a3b\n  variants:\n    - model: qwen3.6-35b-a3b-apex\n  url: u\n" +
+			"- name: qwen3.6-35b-a3b-apex\n  url: u\n")
+		fams := []family{{
+			repo:     "mudler/Qwen3.6-35B-A3B-APEX-GGUF",
+			repoBase: "Qwen3.6-35B-A3B-APEX",
+			stem:     "Qwen3.6-35B-A3B-APEX",
+			children: []childBuild{buildOf("qwen3.6-35b-a3b-apex-i-quality", "mudler/Qwen3.6-35B-A3B-APEX-GGUF", "a.gguf", 0)},
+		}}
+
+		inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("qwen3.6-35b-a3b-apex-i-quality"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(newHubs).To(BeEmpty())
+		Expect(inserts[0].Variants).To(Equal([]string{"qwen3.6-35b-a3b-apex-i-quality"}))
+
+		out, err := galleryedit.Apply(ix.Lines, inserts)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(strings.Count(strings.Join(out, "\n"), "variants:")).To(Equal(1))
+		Expect(out).To(HaveLen(len(ix.Lines) + 1))
+	})
+
+	It("never lets the hub reference itself", func() {
+		// An unsloth rung whose weights the gallery already ships under the base
+		// name resolves, through Merge, straight back to the hub. The verifier
+		// reads a self reference as a variant that declares variants of its own.
+		ix := mustIndex("- name: step-3.7-flash\n  url: u\n")
+		fams := []family{{
+			repo:     "mudler/Step-3.7-Flash-APEX-GGUF",
+			repoBase: "Step-3.7-Flash-APEX",
+			stem:     "Step-3.7-Flash-APEX",
+			children: []childBuild{buildOf("step-3.7-flash-ud-q4-k-m", "unsloth/Step-3.7-Flash-GGUF", "a.gguf", 100)},
+		}}
+
+		inserts, _, err := planHubs(fams, ix, map[string]string{"step-3.7-flash-ud-q4-k-m": "step-3.7-flash"}, map[string]bool{})
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(inserts).To(BeEmpty())
+	})
+
+	It("emits a hub named for the base model when the gallery has none", func() {
+		ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n  url: u\n")
+		fams := []family{{
+			repo:      "mudler/Qwen3.5-35B-A3B-APEX-GGUF",
+			repoBase:  "Qwen3.5-35B-A3B-APEX",
+			stem:      "Qwen3.5-35B-A3B-APEX",
+			hasMMProj: true,
+			children: []childBuild{
+				buildOf("qwen3.5-35b-a3b-apex-i-quality", "mudler/Qwen3.5-35B-A3B-APEX-GGUF", "a.gguf", 0),
+				buildOf("qwen3.5-35b-a3b-ud-q6-k", "unsloth/Qwen3.5-35B-A3B-GGUF", "b.gguf", 102),
+			},
+		}}
+
+		inserts, newHubs, err := planHubs(fams, ix, noReuse,
+			allAdded("qwen3.5-35b-a3b-apex-i-quality", "qwen3.5-35b-a3b-ud-q6-k"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(inserts).To(BeEmpty())
+		Expect(newHubs).To(HaveLen(1))
+
+		hub := newHubs[0]
+		Expect(hub.Name).To(Equal("qwen3.5-35b-a3b"))
+		Expect(hub.Name).ToNot(HaveSuffix("-apex"))
+
+		// A hand-written *-apex entry is an ordinary build, referenced like any
+		// other rung and never deleted or renamed.
+		Expect(hub.Variants).To(Equal([]VariantRef{
+			{Model: "qwen3.5-35b-a3b-apex"},
+			{Model: "qwen3.5-35b-a3b-apex-i-quality"},
+			{Model: "qwen3.5-35b-a3b-ud-q6-k"},
+		}))
+
+		// The verifier skips entries with no declared backend, so a hub without
+		// one would escape the tagging check in silence.
+		Expect(hub.Overrides).To(HaveKeyWithValue("backend", "llama-cpp"))
+		Expect(hub.Files).ToNot(BeEmpty())
+		Expect(hub.Tags).To(ContainElement("vision"))
+	})
+
+	It("gathers two APEX repos that share one base model under a single hub", func() {
+		ix := mustIndex("- name: unrelated\n  url: u\n")
+		fams := []family{
+			{
+				repo:     "mudler/Solo-APEX-GGUF",
+				repoBase: "Solo-APEX",
+				stem:     "Solo-APEX",
+				children: []childBuild{buildOf("solo-apex-i-quality", "mudler/Solo-APEX-GGUF", "a.gguf", 0)},
+			},
+			{
+				repo:     "mudler/Solo-APEX-MTP-GGUF",
+				repoBase: "Solo-APEX-MTP",
+				stem:     "Solo-APEX-MTP",
+				children: []childBuild{buildOf("solo-apex-mtp-i-quality", "mudler/Solo-APEX-MTP-GGUF", "b.gguf", 0)},
+			},
+		}
+
+		_, newHubs, err := planHubs(fams, ix, noReuse, allAdded("solo-apex-i-quality", "solo-apex-mtp-i-quality"))
+
+		Expect(err).ToNot(HaveOccurred())
+		Expect(newHubs).To(HaveLen(1))
+		Expect(newHubs[0].Name).To(Equal("solo"))
+		Expect(newHubs[0].Variants).To(Equal([]VariantRef{
+			{Model: "solo-apex-i-quality"},
+			{Model: "solo-apex-mtp-i-quality"},
+		}))
+	})
+})
+
+var _ = Describe("hubVariants", func() {
+	It("orders builds by quality rung rather than discovery order", func() {
+		// DiscoverAPEXTiers preserves input order and the HF API returns siblings
+		// alphabetically, so an unsorted list reads I-Balanced, I-Compact, I-Mini,
+		// I-Nano, I-Quality. Selection ignores authored order; this is for the
+		// human reading the file.
+		f := family{repoBase: "X-APEX", stem: "X-APEX", children: []childBuild{
+			{rank: rungRank["I-Nano"], entry: GalleryEntry{Name: "x-i-nano"}},
+			{rank: 100, entry: GalleryEntry{Name: "x-ud-q4-k-m"}},
+			{rank: rungRank["I-Quality"], entry: GalleryEntry{Name: "x-i-quality"}},
+			{rank: rungRank["I-Compact"], entry: GalleryEntry{Name: "x-i-compact"}},
+		}}
+
+		got := hubVariants(&f, mustIndex("- name: x\n  url: u\n"), map[string]string{}, map[string]bool{})
+
+		Expect(got).To(Equal([]string{"x-i-quality", "x-i-compact", "x-i-nano", "x-ud-q4-k-m"}))
+	})
+})
+
+var _ = Describe("ParseIndexText", func() {
+	It("refuses to edit by line number when the two views of the file disagree", func() {
+		_, err := ParseIndexText("- name: one\n  url: u\n-\n")
+		Expect(err).To(MatchError(ContainSubstring("empty")))
+	})
+
+	It("records the line range of each entry", func() {
+		ix := mustIndex("- name: first\n  url: u\n- name: second\n  url: u\n")
+
+		Expect(ix.Find("FIRST").Pos.StartLine).To(Equal(0))
+		Expect(ix.Find("first").Pos.EndLine).To(Equal(2))
+		Expect(ix.Find("second").Pos.StartLine).To(Equal(2))
+	})
+})
+
+var _ = Describe("resolveVariant", func() {
+	It("keeps an entry that was emitted even when it is also in reused", func() {
+		// A within-batch name collision records reused[name] = name while the
+		// FIRST entry of that name is still in add. Treating presence in reused as
+		// "dropped" would emit nothing for it.
+		added := map[string]bool{"dup": true}
+		reused := map[string]string{"dup": "dup"}
+
+		Expect(resolveVariant("dup", reused, added)).To(Equal("dup"))
+	})
+
+	It("redirects a reused name at the entry that stands in for it", func() {
+		added := map[string]bool{}
+		reused := map[string]string{"generated": "already-in-gallery"}
+
+		Expect(resolveVariant("generated", reused, added)).To(Equal("already-in-gallery"))
+	})
+})
+
+var _ = Describe("slug", func() {
+	It("lowercases and turns quant underscores into hyphens", func() {
+		Expect(slug("UD-Q4_K_M")).To(Equal("ud-q4-k-m"))
+		Expect(slug("gemma-4-26B-A4B-it-APEX")).To(Equal("gemma-4-26b-a4b-it-apex"))
+		Expect(slug("I-Nano")).To(Equal("i-nano"))
+	})
+})
+
+var _ = Describe("sortTiers", func() {
+	It("puts the imatrix ladder in descending quality order", func() {
+		tiers := []Tier{
+			{Label: "I-Balanced"}, {Label: "I-Compact"}, {Label: "I-Mini"},
+			{Label: "I-Nano"}, {Label: "I-Quality"},
+		}
+		sortTiers(tiers)
+		Expect(tierLabels(tiers)).To(Equal("I-Quality,I-Balanced,I-Compact,I-Mini,I-Nano"))
+	})
+})
+
+var _ = Describe("restrict", func() {
+	It("keeps only the named repos", func() {
+		got := restrict([]string{"mudler/A-APEX-GGUF", "mudler/B-APEX-GGUF"}, "mudler/B-APEX-GGUF")
+		Expect(got).To(Equal([]string{"mudler/B-APEX-GGUF"}))
+	})
+
+	It("returns nothing when the filter matches nothing", func() {
+		Expect(restrict([]string{"mudler/A-APEX-GGUF"}, "mudler/typo")).To(BeEmpty())
+	})
+})
+
+var _ = Describe("reportUnclassified", func() {
+	// One real imatrix rung is always present so the specs measure how the
+	// remaining files are bucketed, not an empty-repo edge case.
+	tier := Tier{Label: "I-Quality", File: GGUFFile{Name: "Model-APEX-I-Quality.gguf"}}
+
+	censusOf := func(names ...string) fileCensus {
+		files := []GGUFFile{tier.File}
+		for _, n := range names {
+			files = append(files, GGUFFile{Name: n})
+		}
+		return reportUnclassified("mudler/Model-APEX-GGUF", files, []Tier{tier}, nil)
+	}
+
+	It("counts a flat full-precision source as excluded, not unclassified", func() {
+		got := censusOf("Carnice-MoE-35B-A3B-F16.gguf")
+		Expect(got.fullPrecision).To(Equal(1))
+		Expect(got.unclassified).To(Equal(0))
+	})
+
+	It("counts every shard of a sharded full-precision source as excluded", func() {
+		got := censusOf(
+			"MiniMax-M2.7-APEX-F16-00001-of-00003.gguf",
+			"MiniMax-M2.7-APEX-F16-00002-of-00003.gguf",
+			"MiniMax-M2.7-APEX-F16-00003-of-00003.gguf",
+		)
+		Expect(got.fullPrecision).To(Equal(3))
+		Expect(got.unclassified).To(Equal(0))
+	})
+
+	It("treats bf16 the same as f16, in either case", func() {
+		got := censusOf("Model-APEX-BF16.gguf", "Model-APEX-bf16-00001-of-00002.gguf", "Model-APEX-f16.gguf")
+		Expect(got.fullPrecision).To(Equal(3))
+		Expect(got.unclassified).To(Equal(0))
+	})
+
+	It("still reports a genuinely unknown filename as unclassified", func() {
+		got := censusOf("Model-APEX-Turbo.gguf")
+		Expect(got.unclassified).To(Equal(1))
+		Expect(got.fullPrecision).To(Equal(0))
+	})
+
+	It("separates the two kinds when a repo publishes both", func() {
+		got := censusOf("Model-APEX-F16.gguf", "Model-APEX-Turbo.gguf")
+		Expect(got.fullPrecision).To(Equal(1))
+		Expect(got.unclassified).To(Equal(1))
+	})
+})

.github/ci/apexentries/merge.go

@@ -0,0 +1,143 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+)
+
+const (
+	hfShorthandPrefix = "huggingface://"
+	hfResolvePrefix   = "https://huggingface.co/"
+	hfResolveInfix    = "/resolve/main/"
+)
+
+// canonicalURI reduces the two interchangeable spellings of a HuggingFace file
+// to one key, so a generated resolve/main URI dedups against the shorthand the
+// gallery uses for the majority of its entries.
+//
+// The repo is exactly the first two path segments; everything after is the file
+// path, which may itself contain slashes because sharded quants live in a
+// subdirectory. Anything that is not recognisably one of the two forms is
+// returned unchanged rather than guessed at, so mirrors and other hosts still
+// dedup on their literal string.
+func canonicalURI(uri string) string {
+	switch {
+	case strings.HasPrefix(uri, hfShorthandPrefix):
+		rest := strings.TrimPrefix(uri, hfShorthandPrefix)
+		owner, after, ok := strings.Cut(rest, "/")
+		if !ok {
+			return uri
+		}
+		name, file, ok := strings.Cut(after, "/")
+		if !ok || owner == "" || name == "" || file == "" {
+			return uri
+		}
+		return hfShorthandPrefix + owner + "/" + name + "/" + file
+
+	case strings.HasPrefix(uri, hfResolvePrefix):
+		rest := strings.TrimPrefix(uri, hfResolvePrefix)
+		repo, file, ok := strings.Cut(rest, hfResolveInfix)
+		if !ok || file == "" {
+			return uri
+		}
+		// A repo is owner/name and nothing more; a longer prefix means this is
+		// some other huggingface.co URL that must not be rewritten.
+		owner, name, ok := strings.Cut(repo, "/")
+		if !ok || owner == "" || name == "" || strings.Contains(name, "/") {
+			return uri
+		}
+		return hfShorthandPrefix + repo + "/" + file
+
+	default:
+		return uri
+	}
+}
+
+// ExistingIndex is the lookup built from the current gallery: entry names, and
+// which entry claims each weight URI.
+type ExistingIndex struct {
+	ByName map[string]int
+	ByURI  map[string]string
+}
+
+// LoadExisting reads the gallery index for dedup purposes only. It is
+// deliberately not used to rewrite the file: the index is 40,000 lines, and a
+// YAML round trip would reflow the whole thing into an unreviewable diff.
+func LoadExisting(path string) (*ExistingIndex, error) {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return nil, err
+	}
+
+	var entries []struct {
+		Name  string `yaml:"name"`
+		Files []struct {
+			URI string `yaml:"uri"`
+		} `yaml:"files"`
+	}
+	if err := yaml.Unmarshal(raw, &entries); err != nil {
+		return nil, fmt.Errorf("parsing %s: %w", path, err)
+	}
+
+	ix := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+	for i, e := range entries {
+		ix.ByName[e.Name] = i
+		for _, f := range e.Files {
+			if f.URI != "" {
+				ix.ByURI[canonicalURI(f.URI)] = e.Name
+			}
+		}
+	}
+	return ix, nil
+}
+
+// Merge splits generated entries into those to add and those already covered.
+// reused maps a generated name to the existing entry that stands in for it, so
+// a parent can reference what is already there instead of duplicating weights.
+// Several APEX repos share one base model, so the same counterpart rungs are
+// generated more than once in a batch. The batch has to dedup against itself as
+// well as against the gallery, tracked locally because the caller may reuse the
+// ExistingIndex it passed in.
+func Merge(existing *ExistingIndex, generated []GalleryEntry) (add []GalleryEntry, reused map[string]string) {
+	reused = map[string]string{}
+	batchNames := map[string]string{}
+	batchURIs := map[string]string{}
+
+	// Canonicalized into a local copy rather than in place: an ExistingIndex may
+	// be hand-built or reused by the caller, so Merge must not rewrite it.
+	existingURIs := make(map[string]string, len(existing.ByURI))
+	for uri, owner := range existing.ByURI {
+		existingURIs[canonicalURI(uri)] = owner
+	}
+
+	for _, e := range generated {
+		// Name is checked before URI: a name collision must block the add
+		// whatever the weights say, since duplicate names corrupt the index.
+		if _, clash := existing.ByName[e.Name]; clash {
+			reused[e.Name] = e.Name
+			continue
+		}
+		if claimant, clash := batchNames[e.Name]; clash {
+			reused[e.Name] = claimant
+			continue
+		}
+		if len(e.Files) > 0 {
+			uri := canonicalURI(e.Files[0].URI)
+			if owner, ok := existingURIs[uri]; ok {
+				reused[e.Name] = owner
+				continue
+			}
+			if claimant, ok := batchURIs[uri]; ok {
+				reused[e.Name] = claimant
+				continue
+			}
+			batchURIs[uri] = e.Name
+		}
+		batchNames[e.Name] = e.Name
+		add = append(add, e)
+	}
+	return add, reused
+}

.github/ci/apexentries/merge_test.go

@@ -0,0 +1,183 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Merge", func() {
+	It("drops a generated entry whose weight URI already exists and reports the existing name", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"qwen3.6-35b-a3b-apex": 0},
+			ByURI: map[string]string{
+				"https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf": "qwen3.6-35b-a3b-apex",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "x-apex-i-quality",
+			Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("x-apex-i-quality", "qwen3.6-35b-a3b-apex"))
+	})
+
+	It("keeps a generated entry whose weights are new", func() {
+		existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+		gen := []GalleryEntry{{
+			Name:  "x-apex-i-mini",
+			Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Mini.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(reused).To(BeEmpty())
+	})
+
+	It("refuses to add an entry whose name collides with an existing one", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"x-apex-i-mini": 0},
+			ByURI:  map[string]string{},
+		}
+		gen := []GalleryEntry{{
+			Name:  "x-apex-i-mini",
+			Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/other.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("x-apex-i-mini", "x-apex-i-mini"))
+	})
+
+	// The gallery records most of its URIs in huggingface:// shorthand while
+	// render.go only ever emits the resolve/main form, so without
+	// canonicalization the majority of the file is invisible to the dedup.
+	It("matches a generated https URI against the shorthand form recorded in the gallery", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"foo-gguf-q8-0": 0},
+			ByURI: map[string]string{
+				"huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf": "foo-gguf-q8-0",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "foo-apex-q8-0",
+			Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0"))
+	})
+
+	It("matches a generated shorthand URI against the https form recorded in the gallery", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"foo-gguf-q8-0": 0},
+			ByURI: map[string]string{
+				"https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf": "foo-gguf-q8-0",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "foo-apex-q8-0",
+			Files: []EntryFile{{URI: "huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0"))
+	})
+
+	// Sharded quants live under a subdirectory, so the file path carries slashes
+	// of its own and only the first two segments are the repo.
+	It("matches across both forms when the file path has a subdirectory", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"model-ud-q4-k-m": 0},
+			ByURI: map[string]string{
+				"huggingface://unsloth/Model-GGUF/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf": "model-ud-q4-k-m",
+			},
+		}
+		gen := []GalleryEntry{{
+			Name:  "model-apex-ud-q4-k-m",
+			Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Model-GGUF/resolve/main/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf"}},
+		}}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(BeEmpty())
+		Expect(reused).To(HaveKeyWithValue("model-apex-ud-q4-k-m", "model-ud-q4-k-m"))
+	})
+
+	// Several APEX repos share one base model, so the same unsloth rungs are
+	// generated more than once in a single batch.
+	It("adds only the first of two generated entries sharing a name", func() {
+		existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+		gen := []GalleryEntry{
+			{
+				Name:  "shared-rung-q8-0",
+				Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}},
+			},
+			{
+				Name:  "shared-rung-q8-0",
+				Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Other-GGUF/resolve/main/Other-Q8_0.gguf"}},
+			},
+		}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(add[0].Files[0].URI).To(Equal("https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"))
+		Expect(reused).To(HaveKeyWithValue("shared-rung-q8-0", "shared-rung-q8-0"))
+	})
+
+	It("adds only the first of two generated entries sharing a primary URI", func() {
+		existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
+		gen := []GalleryEntry{
+			{
+				Name:  "shared-rung-from-apex",
+				Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}},
+			},
+			{
+				Name:  "shared-rung-from-apex-mtp",
+				Files: []EntryFile{{URI: "huggingface://unsloth/Shared-GGUF/Shared-Q8_0.gguf"}},
+			},
+		}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(add[0].Name).To(Equal("shared-rung-from-apex"))
+		Expect(reused).To(HaveKeyWithValue("shared-rung-from-apex-mtp", "shared-rung-from-apex"))
+	})
+
+	// Anything that is not a HuggingFace URI must survive untouched, so an
+	// unrecognised scheme still dedups against the very same string.
+	It("leaves a URI in neither recognised form alone and still dedups it exactly", func() {
+		existing := &ExistingIndex{
+			ByName: map[string]int{"mirrored-model": 0},
+			ByURI: map[string]string{
+				"https://mirror.example.com/weights/Model-Q8_0.gguf": "mirrored-model",
+			},
+		}
+		gen := []GalleryEntry{
+			{
+				Name:  "mirrored-apex",
+				Files: []EntryFile{{URI: "https://mirror.example.com/weights/Model-Q8_0.gguf"}},
+			},
+			{
+				Name:  "elsewhere-apex",
+				Files: []EntryFile{{URI: "https://mirror.example.com/weights/Other-Q8_0.gguf"}},
+			},
+		}
+
+		add, reused := Merge(existing, gen)
+
+		Expect(add).To(HaveLen(1))
+		Expect(add[0].Name).To(Equal("elsewhere-apex"))
+		Expect(reused).To(HaveKeyWithValue("mirrored-apex", "mirrored-model"))
+	})
+})

.github/ci/apexentries/render.go

@@ -0,0 +1,175 @@
+package main
+
+import (
+	"fmt"
+	"path"
+	"strings"
+)
+
+// EntryFile is one downloadable file of a gallery entry.
+type EntryFile struct {
+	Filename string `yaml:"filename"`
+	SHA256   string `yaml:"sha256"`
+	URI      string `yaml:"uri"`
+}
+
+// GalleryEntry is the subset of a gallery entry this generator writes.
+//
+// Named GalleryEntry rather than Entry because the test files dot-import
+// Ginkgo, whose table DSL exports an Entry that a package-level Entry would
+// collide with. The yaml tags are what the gallery index sees, so the Go
+// identifier is free to differ.
+type GalleryEntry struct {
+	Name        string         `yaml:"name"`
+	URL         string         `yaml:"url"`
+	Description string         `yaml:"description,omitempty"`
+	Tags        []string       `yaml:"tags,omitempty"`
+	Overrides   map[string]any `yaml:"overrides,omitempty"`
+	Files       []EntryFile    `yaml:"files,omitempty"`
+	Variants    []VariantRef   `yaml:"variants,omitempty"`
+}
+
+// VariantRef mirrors the gallery's variant reference: a name and nothing else.
+type VariantRef struct {
+	Model string `yaml:"model"`
+}
+
+// ChildInput is everything needed to render one non-parent entry.
+type ChildInput struct {
+	Name string
+	Repo string
+	// DraftRepo is the repo publishing the drafter, when it is not the repo
+	// publishing the weights. Speculative pairings routinely cross repos, so
+	// the drafter cannot be assumed to sit next to the weights. Empty means
+	// same-repo, which is how the *-APEX-MTP-GGUF repos ship.
+	DraftRepo string
+	Template  string
+	Weights   []GGUFFile
+	MMProj    *GGUFFile
+	SpecType  string
+	DraftFile *GGUFFile
+	BaseTags  []string
+}
+
+// specTuning is the acceptance-window tuning each spec type ships with, copied
+// from the hand-written entries that already run these two mechanisms rather
+// than invented here. The two differ because the drafters differ: self-drafted
+// MTP heads produce a short, high-confidence proposal (15+ hand-written entries
+// use 6 with a 0.75 floor), while a separate DFlash drafter is cheap enough to
+// run far ahead unconditionally (the five hand-written dflash entries use 15 and
+// set no floor).
+var specTuning = map[string][]string{
+	"draft-mtp":    {"spec_n_max:6", "spec_p_min:0.75"},
+	"draft-dflash": {"spec_n_max:15"},
+}
+
+func hfURI(repo, file string) string {
+	return fmt.Sprintf("https://huggingface.co/%s/resolve/main/%s", repo, file)
+}
+
+// localPath is where a downloaded file lands.
+//
+// The hand-written entries namespace by the repo's BARE name
+// (llama-cpp/models/<repo>/<file>), which is not unique. LiquidAI/LFM2.5-8B-A1B-GGUF
+// and unsloth/LFM2.5-8B-A1B-GGUF share a basename, so both claim
+// llama-cpp/models/LFM2.5-8B-A1B-GGUF/, and installing the second after the first
+// either overwrites weights whose recorded sha256 belongs to the other file or is
+// skipped as already present. Two owners publishing the same model name is the
+// normal case for quantizers, not an edge case, so the owner has to be in the path.
+//
+// The owner becomes its own path segment rather than being folded into the
+// directory name: owner/repo is unique on HuggingFace and "/" cannot occur inside
+// either half, so this is the only form that is collision-proof by construction.
+// It still reads as the hand-written convention with the owner restored, and the
+// extra depth is already present in the index for sharded builds.
+func localPath(kind, repo, file string) string {
+	// path.Dir yields "." for a repo named without an owner, which path.Join
+	// drops, so such a caller keeps the historical two-segment layout.
+	return path.Join("llama-cpp", kind, path.Dir(repo), path.Base(repo), file)
+}
+
+// RenderChild builds one child entry.
+//
+// The dflash/mtp tag is added if and only if this entry sets a spec_type,
+// because variant ranking reads tags and nothing else, and a tag that does not
+// match what the entry configures either promotes a build that is no faster or
+// hides one that is.
+func RenderChild(in ChildInput) GalleryEntry {
+	e := GalleryEntry{
+		Name:      in.Name,
+		URL:       fmt.Sprintf("github:mudler/LocalAI/gallery/%s@master", in.Template),
+		Tags:      append([]string{}, in.BaseTags...),
+		Overrides: map[string]any{},
+	}
+
+	// gallery/virtual.yaml carries no backend, so nothing else would name an
+	// engine for these entries. Matching the hand-written entries on
+	// known_usecases too: LocalAI would fall back to the backend defaults, but
+	// generated entries should not read differently from their neighbours.
+	e.Overrides["backend"] = "llama-cpp"
+	e.Overrides["known_usecases"] = []string{"chat"}
+
+	options := []string{"use_jinja:true"}
+
+	for _, w := range in.Weights {
+		e.Files = append(e.Files, EntryFile{
+			Filename: localPath("models", in.Repo, w.Name),
+			SHA256:   w.SHA256,
+			URI:      hfURI(in.Repo, w.Name),
+		})
+	}
+	e.Overrides["parameters"] = map[string]any{
+		"model": localPath("models", in.Repo, in.Weights[0].Name),
+	}
+
+	if in.MMProj != nil {
+		// An explicit known_usecases SUPPRESSES the backend-default fallback in
+		// core/gallery/models_types.go, so a multimodal entry left at chat-only
+		// never matches FilterGalleryModelsByUsecase(FLAG_VISION) or
+		// FilterGalleryModelsByMultimodal and vanishes from the UI's vision and
+		// multimodal filters. 19 of the 45 APEX repos ship an mmproj.
+		e.Overrides["known_usecases"] = []string{"chat", "vision"}
+		e.Overrides["mmproj"] = localPath("mmproj", in.Repo, in.MMProj.Name)
+		e.Files = append(e.Files, EntryFile{
+			Filename: localPath("mmproj", in.Repo, in.MMProj.Name),
+			SHA256:   in.MMProj.SHA256,
+			URI:      hfURI(in.Repo, in.MMProj.Name),
+		})
+	}
+
+	// A spec type is configured independently of a drafter FILE. Weights that
+	// carry their own MTP heads need no second download, and requiring one left
+	// the *-APEX-MTP-GGUF builds shipping the larger heads-bearing weights with
+	// the heads switched off: a strictly bigger download at the same speed,
+	// ranked identically to the plain rung at the same tier.
+	if in.SpecType != "" {
+		options = append(options, "spec_type:"+in.SpecType)
+		options = append(options, specTuning[in.SpecType]...)
+		// The tag is derived from the spec type this entry sets and from nothing
+		// else. Variant ranking reads tags only, so a tag taken from a repo or
+		// entry NAME would promote a build that is no faster whenever the name
+		// and the configuration disagree.
+		e.Tags = append(e.Tags, strings.TrimPrefix(in.SpecType, "draft-"))
+	}
+
+	if in.SpecType != "" && in.DraftFile != nil {
+		// Fall back to the weights repo so pairings that publish the drafter
+		// alongside the weights keep working without restating the repo.
+		draftRepo := in.DraftRepo
+		if draftRepo == "" {
+			draftRepo = in.Repo
+		}
+		draftPath := localPath("models", draftRepo, in.DraftFile.Name)
+
+		e.Overrides["draft_model"] = draftPath
+		e.Overrides["flash_attention"] = "on"
+		e.Files = append(e.Files, EntryFile{
+			Filename: draftPath,
+			SHA256:   in.DraftFile.SHA256,
+			URI:      hfURI(draftRepo, in.DraftFile.Name),
+		})
+	}
+
+	e.Overrides["options"] = options
+	return e
+}

.github/ci/apexentries/render_test.go

@@ -0,0 +1,249 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("RenderChild", func() {
+	It("tags an entry that configures draft-dflash", func() {
+		e := RenderChild(ChildInput{
+			Name:      "qwen3.5-9b-dflash",
+			Repo:      "mudler/Example-APEX-GGUF",
+			Template:  "virtual.yaml",
+			Weights:   []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			SpecType:  "draft-dflash",
+			DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"},
+			BaseTags:  []string{"llm", "gguf"},
+		})
+
+		Expect(e.Tags).To(ContainElement("dflash"))
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Overrides["options"]).To(ContainElement("spec_type:draft-dflash"))
+		Expect(e.Overrides["draft_model"]).ToNot(BeNil())
+	})
+
+	It("does not tag an MTP-named repo that configures no speculation", func() {
+		// mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF ships MTP-bearing weights. Weights
+		// that carry the heads are not an entry that enables them, and tagging it
+		// would win the feature axis without being any faster.
+		e := RenderChild(ChildInput{
+			Name:     "qwen3.6-35b-a3b-apex-mtp-i-quality",
+			Repo:     "mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Qwen3.6-35B-A3B-APEX-MTP-I-Quality.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Tags).ToNot(ContainElement("dflash"))
+		Expect(e.Overrides).ToNot(HaveKey("draft_model"))
+	})
+
+	It("lists every shard of a sharded build and points the model at the first", func() {
+		e := RenderChild(ChildInput{
+			Name:     "step-3.7-flash-ud-q4-k-m",
+			Repo:     "unsloth/Step-3.7-Flash-GGUF",
+			Template: "virtual.yaml",
+			Weights: []GGUFFile{
+				{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"},
+				{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"},
+			},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Files).To(HaveLen(2))
+		params, ok := e.Overrides["parameters"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		Expect(params["model"]).To(HaveSuffix("00001-of-00002.gguf"))
+		Expect(e.Files[0].URI).To(Equal(
+			"https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf"))
+	})
+
+	It("wires mmproj when the repo publishes one", func() {
+		e := RenderChild(ChildInput{
+			Name:     "example-i-mini",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}},
+			MMProj:   &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["mmproj"]).ToNot(BeNil())
+		Expect(e.Files).To(HaveLen(2))
+	})
+
+	It("names the engine and the usecases the hand-written entries name", func() {
+		// gallery/virtual.yaml supplies no backend, so an entry that omits one
+		// names no engine at all and cannot load.
+		e := RenderChild(ChildInput{
+			Name:     "example-i-mini",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["backend"]).To(Equal("llama-cpp"))
+		Expect(e.Overrides["known_usecases"]).To(ContainElement("chat"))
+	})
+
+	It("draws the drafter from DraftRepo when the pairing spans two repos", func() {
+		// unsloth/Qwen3-4B-GGUF pairs with a drafter published separately by
+		// AtomicChat, so a drafter URI built from the weights repo 404s.
+		e := RenderChild(ChildInput{
+			Name:      "qwen3-4b-dflash",
+			Repo:      "unsloth/Qwen3-4B-GGUF",
+			DraftRepo: "AtomicChat/Qwen3-4B-DFlash-GGUF",
+			Template:  "virtual.yaml",
+			Weights:   []GGUFFile{{Name: "Qwen3-4B-Q4_K_M.gguf", SHA256: "a"}},
+			SpecType:  "draft-dflash",
+			DraftFile: &GGUFFile{Name: "Qwen3-4B-DFlash.Q8_0.gguf", SHA256: "b"},
+			BaseTags:  []string{"llm", "gguf"},
+		})
+
+		Expect(e.Files[0].URI).To(Equal(
+			"https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf"))
+		Expect(e.Files[1].URI).To(Equal(
+			"https://huggingface.co/AtomicChat/Qwen3-4B-DFlash-GGUF/resolve/main/Qwen3-4B-DFlash.Q8_0.gguf"))
+		Expect(e.Files[1].Filename).To(Equal(
+			"llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf"))
+		Expect(e.Overrides["draft_model"]).To(Equal(
+			"llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf"))
+	})
+
+	It("falls back to the weights repo for the drafter when DraftRepo is empty", func() {
+		// The *-APEX-MTP-GGUF repos ship the drafter alongside the weights.
+		e := RenderChild(ChildInput{
+			Name:      "example-apex-dflash",
+			Repo:      "mudler/Example-APEX-GGUF",
+			Template:  "virtual.yaml",
+			Weights:   []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			SpecType:  "draft-dflash",
+			DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"},
+			BaseTags:  []string{"llm", "gguf"},
+		})
+
+		Expect(e.Files[1].URI).To(Equal(
+			"https://huggingface.co/mudler/Example-APEX-GGUF/resolve/main/Example-DFlash.Q8_0.gguf"))
+		Expect(e.Files[1].Filename).To(Equal(
+			"llama-cpp/models/mudler/Example-APEX-GGUF/Example-DFlash.Q8_0.gguf"))
+	})
+})
+
+var _ = Describe("RenderChild known_usecases", func() {
+	It("declares vision alongside chat when the entry carries an mmproj", func() {
+		// An explicit known_usecases suppresses the backend-default fallback, so a
+		// chat-only multimodal entry disappears from the UI's vision filter.
+		e := RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			MMProj:   &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat", "vision"))
+	})
+
+	It("leaves a text-only entry at chat", func() {
+		e := RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat"))
+	})
+})
+
+var _ = Describe("localPath", func() {
+	It("keeps two repos with the same basename but different owners apart", func() {
+		// LiquidAI and unsloth both publish LFM2.5-8B-A1B-GGUF. A path built from
+		// the bare repo name gives both the same local file, so installing the
+		// second overwrites or skips the first and one of them then serves bytes
+		// that do not match its recorded sha256.
+		liquid := RenderChild(ChildInput{
+			Name:     "lfm2.5-8b-a1b-i-quality",
+			Repo:     "LiquidAI/LFM2.5-8B-A1B-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "33ab3b8c"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+		unsloth := RenderChild(ChildInput{
+			Name:     "lfm2.5-8b-a1b-q8-0",
+			Repo:     "unsloth/LFM2.5-8B-A1B-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "ec11666b"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(liquid.Files[0].Filename).ToNot(Equal(unsloth.Files[0].Filename))
+		Expect(unsloth.Files[0].Filename).To(Equal(
+			"llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf"))
+	})
+
+	It("namespaces the mmproj by owner too", func() {
+		e := RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     "mudler/Example-APEX-GGUF",
+			Template: "virtual.yaml",
+			Weights:  []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
+			MMProj:   &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
+			BaseTags: []string{"llm", "gguf"},
+		})
+
+		Expect(e.Overrides["mmproj"]).To(Equal(
+			"llama-cpp/mmproj/mudler/Example-APEX-GGUF/mmproj-F16.gguf"))
+	})
+})
+
+var _ = Describe("MTP builds", func() {
+	renderTier := func(repo string) GalleryEntry {
+		return RenderChild(ChildInput{
+			Name:     "example-i-quality",
+			Repo:     repo,
+			Template: "virtual.yaml",
+			SpecType: SpecTypeForRepo(repo),
+			Weights:  []GGUFFile{{Name: "Example-I-Quality.gguf", SHA256: "a"}},
+			BaseTags: []string{"llm", "gguf"},
+		})
+	}
+
+	It("turns MTP on for a build off an APEX-MTP repo", func() {
+		// These weights retain the model's own MTP heads, so shipping them with
+		// speculation off is a strictly larger download at the same speed,
+		// ranked identically to the plain rung at the same tier.
+		e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF")
+
+		Expect(e.Overrides["options"]).To(ContainElements(
+			"spec_type:draft-mtp", "spec_n_max:6", "spec_p_min:0.75"))
+		Expect(e.Tags).To(ContainElement("mtp"))
+	})
+
+	It("needs no drafter file, because the heads travel with the weights", func() {
+		e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF")
+
+		Expect(e.Overrides).ToNot(HaveKey("draft_model"))
+		Expect(e.Files).To(HaveLen(1))
+	})
+
+	It("leaves a build off a plain APEX repo alone", func() {
+		e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-GGUF")
+
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true"))
+	})
+
+	It("leaves an unsloth counterpart rung alone", func() {
+		// The counterpart quantizes the plain weights; nothing there carries heads.
+		e := renderTier("unsloth/Qwen3.6-35B-A3B-GGUF")
+
+		Expect(e.Tags).ToNot(ContainElement("mtp"))
+		Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true"))
+	})
+})

.github/ci/apexentries/unsloth.go

@@ -0,0 +1,71 @@
+package main
+
+import (
+	"regexp"
+	"sort"
+	"strings"
+)
+
+// WantedQuants is the fixed unsloth subset this generator emits. It is a
+// deliberate subset: unsloth publishes north of 20 quants per repo, and the
+// selector needs useful fitness points rather than every rung.
+var WantedQuants = []string{"UD-Q4_K_M", "UD-Q5_K_M", "UD-Q6_K", "Q8_0"}
+
+var shardRE = regexp.MustCompile(`-(\d{5})-of-(\d{5})\.gguf$`)
+
+// QuantBuild is one unsloth quantization, which may be a single file or an
+// ordered set of shards.
+type QuantBuild struct {
+	Quant   string
+	Files   []GGUFFile
+	Sharded bool
+}
+
+// CounterpartCandidates returns the unsloth repo base names worth probing, most
+// likely first. Both derivations are needed: the repo name finds
+// unsloth/gemma-4-26B-A4B-it-GGUF, while the file stem is what matches for
+// repos whose stem is the canonical model name.
+func CounterpartCandidates(repoName, fileStem string) []string {
+	clean := func(s string) string {
+		s = strings.TrimSuffix(s, "-GGUF")
+		s = regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "")
+		s = strings.TrimSuffix(s, "-APEX")
+		return regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "")
+	}
+
+	out := []string{clean(repoName)}
+	if stem := clean(fileStem); stem != out[0] {
+		out = append(out, stem)
+	}
+	return out
+}
+
+// DiscoverUnslothQuants returns the wanted quants a repo publishes, handling
+// both the flat single-file layout and the sharded layout where a quant lives
+// in its own subdirectory.
+func DiscoverUnslothQuants(files []GGUFFile) []QuantBuild {
+	var out []QuantBuild
+
+	for _, q := range WantedQuants {
+		var flat []GGUFFile
+		var shards []GGUFFile
+
+		for _, f := range files {
+			switch {
+			case !strings.Contains(f.Name, "/") && strings.HasSuffix(f.Name, "-"+q+".gguf"):
+				flat = append(flat, f)
+			case strings.HasPrefix(f.Name, q+"/") && shardRE.MatchString(f.Name):
+				shards = append(shards, f)
+			}
+		}
+
+		switch {
+		case len(flat) > 0:
+			out = append(out, QuantBuild{Quant: q, Files: flat})
+		case len(shards) > 0:
+			sort.Slice(shards, func(i, j int) bool { return shards[i].Name < shards[j].Name })
+			out = append(out, QuantBuild{Quant: q, Files: shards, Sharded: true})
+		}
+	}
+	return out
+}

.github/ci/apexentries/unsloth_test.go

@@ -0,0 +1,75 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("CounterpartCandidates", func() {
+	It("offers both the repo-derived and stem-derived names", func() {
+		// mudler/gemma-4-26B-A4B-it-APEX-GGUF ships gemma-4-26B-A4B-APEX-*.gguf,
+		// and only the repo-derived name finds unsloth/gemma-4-26B-A4B-it-GGUF.
+		got := CounterpartCandidates("gemma-4-26B-A4B-it-APEX-GGUF", "gemma-4-26B-A4B-APEX")
+
+		Expect(got).To(Equal([]string{"gemma-4-26B-A4B-it", "gemma-4-26B-A4B"}))
+	})
+
+	It("strips the MTP marker", func() {
+		got := CounterpartCandidates("Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF", "Qwopus3.6-35B-A3B-v1-APEX-MTP")
+
+		Expect(got[0]).To(Equal("Qwopus3.6-35B-A3B-v1"))
+	})
+
+	It("strips the TQ marker", func() {
+		// This is the branch that folds mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF into
+		// the qwen3.5-35b-a3b hub. Without it the probe is
+		// unsloth/Qwen3.5-35B-A3B-TQ-GGUF, which does not exist, so the family
+		// silently loses every unsloth rung.
+		got := CounterpartCandidates("Qwen3.5-35B-A3B-APEX-TQ-GGUF", "Qwen3.5-35B-A3B-APEX-TQ")
+
+		Expect(got).To(Equal([]string{"Qwen3.5-35B-A3B"}))
+	})
+
+	It("does not repeat a candidate when both derivations agree", func() {
+		got := CounterpartCandidates("Qwen3.6-35B-A3B-APEX-GGUF", "Qwen3.6-35B-A3B-APEX")
+
+		Expect(got).To(Equal([]string{"Qwen3.6-35B-A3B"}))
+	})
+})
+
+var _ = Describe("DiscoverUnslothQuants", func() {
+	It("finds flat single-file quants", func() {
+		files := []GGUFFile{
+			{Name: "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", SHA256: "a"},
+			{Name: "Qwen3.6-35B-A3B-UD-IQ1_M.gguf", SHA256: "b"},
+		}
+
+		got := DiscoverUnslothQuants(files)
+
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Quant).To(Equal("UD-Q4_K_M"))
+		Expect(got[0].Sharded).To(BeFalse())
+		Expect(got[0].Files).To(HaveLen(1))
+	})
+
+	It("collects a sharded quant from its subdirectory in shard order", func() {
+		files := []GGUFFile{
+			{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"},
+			{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"},
+		}
+
+		got := DiscoverUnslothQuants(files)
+
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Quant).To(Equal("UD-Q4_K_M"))
+		Expect(got[0].Sharded).To(BeTrue())
+		Expect(got[0].Files).To(HaveLen(2))
+		Expect(got[0].Files[0].Name).To(HaveSuffix("00001-of-00002.gguf"))
+	})
+
+	It("ignores quants outside the wanted subset", func() {
+		files := []GGUFFile{{Name: "Model-UD-IQ2_XXS.gguf", SHA256: "a"}}
+
+		Expect(DiscoverUnslothQuants(files)).To(BeEmpty())
+	})
+})

.github/ci/apexentries/verify.go

@@ -0,0 +1,312 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+)
+
+type verifyEntry struct {
+	Name      string       `yaml:"name"`
+	Tags      []string     `yaml:"tags"`
+	Variants  []VariantRef `yaml:"variants"`
+	Overrides struct {
+		// Backend scopes the checks that only hold for one engine. An entry that
+		// declares none takes its configuration from the referenced url: template,
+		// which this verifier never reads, so it cannot be judged either way.
+		Backend string   `yaml:"backend"`
+		Options []string `yaml:"options"`
+		// MMProj and DraftModel name the files that are not weights. They are
+		// the only signal for it: a drafter lands in the same models/ prefix as
+		// the weights, so the path alone cannot tell them apart.
+		MMProj     string `yaml:"mmproj"`
+		DraftModel string `yaml:"draft_model"`
+	} `yaml:"overrides"`
+	Files []struct {
+		Filename string `yaml:"filename"`
+		SHA256   string `yaml:"sha256"`
+		URI      string `yaml:"uri"`
+	} `yaml:"files"`
+}
+
+// Verify checks the invariants the variants schema and the tagging rule
+// require. It returns every problem rather than the first, so one run tells the
+// author everything that needs fixing.
+func Verify(path string) []string {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return []string{fmt.Sprintf("reading %s: %v", path, err)}
+	}
+
+	var entries []verifyEntry
+	if err := yaml.Unmarshal(raw, &entries); err != nil {
+		return []string{fmt.Sprintf("parsing %s: %v", path, err)}
+	}
+
+	var problems []string
+
+	byName := map[string]verifyEntry{}
+	for _, e := range entries {
+		if _, seen := byName[e.Name]; seen {
+			problems = append(problems, fmt.Sprintf("duplicate entry name: %s", e.Name))
+			continue
+		}
+		byName[e.Name] = e
+	}
+
+	for _, e := range entries {
+		for _, v := range e.Variants {
+			target, ok := byName[v.Model]
+			if !ok {
+				problems = append(problems, fmt.Sprintf("%s: variant %q does not exist", e.Name, v.Model))
+				continue
+			}
+			if len(target.Variants) > 0 {
+				problems = append(problems, fmt.Sprintf("%s: variant %q declares variants of its own", e.Name, v.Model))
+			}
+		}
+
+		for _, f := range e.Files {
+			if requiresSHA256(f.Filename) && f.SHA256 == "" {
+				problems = append(problems, fmt.Sprintf("%s: file %s has no sha256", e.Name, f.Filename))
+			}
+		}
+
+		problems = append(problems, checkWeightCount(e)...)
+		problems = append(problems, checkFeatureTag(e, "dflash")...)
+		problems = append(problems, checkFeatureTag(e, "mtp")...)
+	}
+
+	problems = append(problems, checkPathCollisions(entries)...)
+
+	return problems
+}
+
+// checkPathCollisions catches two different upstream files claiming one local
+// path. The install layer keys on the local filename, so whichever entry is
+// installed second either overwrites weights the first entry recorded a
+// different sha256 for or is skipped as already present. Either way some entry
+// afterwards serves bytes that do not match its own checksum, and nothing at
+// install time says so.
+//
+// This is an index-wide invariant rather than a per-entry one: neither entry is
+// wrong on its own and the collision exists only in their pairing. The usual
+// source is a path scheme built from the repo's BARE name, because two owners
+// publishing the same model name is routine for quantizers.
+//
+// Sharing a path is fine when the uri is the same, which is how several entries
+// legitimately reuse one projector. Files with no uri are skipped: there is
+// nothing to compare.
+func checkPathCollisions(entries []verifyEntry) []string {
+	type source struct{ uri, entry string }
+
+	first := map[string]source{}
+	reported := map[string]bool{}
+
+	var problems []string
+	for _, e := range entries {
+		for _, f := range e.Files {
+			if f.Filename == "" || f.URI == "" {
+				continue
+			}
+			prev, seen := first[f.Filename]
+			if !seen {
+				first[f.Filename] = source{uri: f.URI, entry: e.Name}
+				continue
+			}
+			if prev.uri == f.URI || reported[f.Filename] {
+				continue
+			}
+			// Reported once per path however many entries pile onto it, so one
+			// heavily reused filename cannot bury the rest of the report.
+			reported[f.Filename] = true
+			problems = append(problems, fmt.Sprintf(
+				"local path %s is claimed by two different uris: %s (%s) and %s (%s)",
+				f.Filename, prev.uri, prev.entry, f.URI, e.Name))
+		}
+	}
+	return problems
+}
+
+// auxiliaryExtensions are the metadata formats an entry ships beside its
+// weights, where an unverified download is a nuisance rather than a hole.
+//
+// The exclusion is stated as a list of metadata formats on purpose. Requiring
+// the checksum only on a blessed list of weight formats would silently exempt
+// every format nobody has shipped yet, and it already exempted safetensors
+// weights, which are downloaded and loaded exactly like GGUF ones.
+var auxiliaryExtensions = []string{".json", ".txt", ".md"}
+
+// requiresSHA256 reports whether an unverified download of this file would be
+// a supply-chain hole rather than a cosmetic gap.
+func requiresSHA256(filename string) bool {
+	for _, ext := range auxiliaryExtensions {
+		if strings.HasSuffix(filename, ext) {
+			return false
+		}
+	}
+	return true
+}
+
+// checkWeightCount catches an entry carrying two whole models. The flat-match
+// branch in DiscoverUnslothQuants appends every match, so a quant label that is
+// a suffix of another one (Q8_0 of UD-Q8_0) collects both files into one build
+// while the rendered model: points at only the first. The result downloads
+// twice the bytes and serves whichever file sorted first, silently.
+//
+// Shards are exempt because a sharded build is legitimately many files.
+//
+// The collision is a property of llama-cpp quant discovery, so the check is
+// scoped to that backend. Multi-component TTS, ASR and diffusion engines ship an
+// encoder, a decoder and a vocoder as one model, and there the second GGUF is
+// the design rather than a bug.
+func checkWeightCount(e verifyEntry) []string {
+	if e.Overrides.Backend != "llama-cpp" {
+		return nil
+	}
+
+	var weights []string
+	for _, f := range e.Files {
+		switch {
+		case !strings.HasSuffix(f.Filename, ".gguf"):
+		case shardRE.MatchString(f.Filename):
+		case f.Filename == e.Overrides.MMProj:
+		case f.Filename == e.Overrides.DraftModel:
+		default:
+			weights = append(weights, f.Filename)
+		}
+	}
+
+	if len(weights) > 1 {
+		return []string{fmt.Sprintf("%s: more than one weight file: %s", e.Name, strings.Join(weights, ", "))}
+	}
+	return nil
+}
+
+// checkFeatureTag enforces the rule in both directions. A tag without the
+// configuration promotes a build that is no faster; configuration without the
+// tag leaves a genuinely faster build ranked as plain.
+//
+// It only speaks about backends whose declaration it can actually read, because
+// a rule applied where the evidence is invisible reports noise rather than bugs.
+func checkFeatureTag(e verifyEntry, feature string) []string {
+	decl, configured, judgeable := featureDeclaration(e, feature)
+	if !judgeable {
+		return nil
+	}
+
+	tagged := false
+	for _, t := range e.Tags {
+		if t == feature {
+			tagged = true
+			break
+		}
+	}
+
+	switch {
+	case tagged && !configured:
+		return []string{fmt.Sprintf("%s: tagged %s but sets no %s", e.Name, feature, decl)}
+	case configured && !tagged:
+		return []string{fmt.Sprintf("%s: sets %s but is not tagged %s", e.Name, decl, feature)}
+	}
+	return nil
+}
+
+// featureDeclaration implements the per-backend table in
+// .agents/adding-gallery-models.md. It returns the declaration the backend uses
+// to configure the feature, whether the entry carries it, and whether this
+// verifier is in a position to answer at all.
+func featureDeclaration(e verifyEntry, feature string) (decl string, configured, judgeable bool) {
+	switch e.Overrides.Backend {
+	case "llama-cpp":
+		decl = "spec_type:draft-" + feature
+		for _, o := range e.Overrides.Options {
+			if strings.TrimSpace(o) == decl {
+				return decl, true, true
+			}
+		}
+		return decl, false, true
+
+	case "ds4":
+		// ds4 carries the MTP heads in the weights and turns them on with
+		// mtp_path / mtp_draft. It has no dflash counterpart, so dflash is not a
+		// question that can be asked of a ds4 entry.
+		if feature != "mtp" {
+			return "", false, false
+		}
+		decl = "mtp_path:"
+		for _, o := range e.Overrides.Options {
+			o = strings.TrimSpace(o)
+			if strings.HasPrefix(o, "mtp_path:") || strings.HasPrefix(o, "mtp_draft:") {
+				return decl, true, true
+			}
+		}
+		return decl, false, true
+
+	default:
+		// sglang configures the feature with speculative_algorithm: in the
+		// referenced gallery/*.yaml, and an entry that declares no backend takes
+		// its whole configuration from its url: template. Verify reads one index
+		// file and follows neither, so it must not judge these in either
+		// direction.
+		return "", false, false
+	}
+}
+
+// UnaccountedQuants reports a wanted quant the repo demonstrably publishes but
+// that discovery produced no build for. The layout that triggers it today is
+// root-level shards, which match neither branch of DiscoverUnslothQuants; no
+// counterpart ships that way yet, but a batch generator must not drop a build
+// with nothing said about it.
+func UnaccountedQuants(files []GGUFFile, builds []QuantBuild) []string {
+	built := map[string]bool{}
+	for _, b := range builds {
+		built[b.Quant] = true
+	}
+
+	var problems []string
+	for _, q := range WantedQuants {
+		if built[q] {
+			continue
+		}
+		for _, f := range files {
+			if filePublishesQuant(f.Name, q) {
+				problems = append(problems, fmt.Sprintf("quant %s is published upstream (%s) but produced no build", q, f.Name))
+				break
+			}
+		}
+	}
+	return problems
+}
+
+// filePublishesQuant reports whether an upstream file is a publication of
+// quant q. It anchors on the quant label the way DiscoverUnslothQuants does,
+// as the trailing token of the base name or as the sharding subdirectory, so
+// the diagnostic and the discovery it audits cannot disagree about what a file
+// is.
+//
+// An unanchored match would reproduce the very collision this diagnostic warns
+// about: Q8_0 is a substring of UD-Q8_0, so a repo publishing only UD-Q8_0
+// would be reported as publishing an unbuilt Q8_0, which it does not, and
+// UD-Q8_0 is not a wanted quant at all.
+func filePublishesQuant(name, q string) bool {
+	if strings.HasPrefix(name, q+"/") {
+		return true
+	}
+
+	base := name[strings.LastIndex(name, "/")+1:]
+	// Shard numbering sits between the quant label and the extension, so it has
+	// to come off before the label can be read as the trailing token. Root-level
+	// shards are the layout that matches neither branch of
+	// DiscoverUnslothQuants, and so the layout this diagnostic mainly catches.
+	base = shardRE.ReplaceAllString(base, ".gguf")
+
+	if !strings.HasSuffix(base, "-"+q+".gguf") {
+		return false
+	}
+	// UD- is unsloth's dynamic-quant modifier, and UD-<q> is a distinct quant
+	// label rather than a publication of <q>.
+	return !strings.HasSuffix(base, "-UD-"+q+".gguf")
+}

.github/ci/apexentries/verify_test.go

@@ -0,0 +1,480 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Verify", func() {
+	write := func(body string) string {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "index.yaml")
+		Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed())
+		return p
+	}
+
+	It("passes a sound index", func() {
+		Expect(Verify(write(`
+- name: parent
+  variants:
+    - model: child
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: child
+  files:
+    - filename: b.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(BeEmpty())
+	})
+
+	It("reports a variant pointing at a missing entry", func() {
+		Expect(Verify(write(`
+- name: parent
+  variants:
+    - model: ghost
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("ghost")))
+	})
+
+	It("reports a variant that itself declares variants", func() {
+		Expect(Verify(write(`
+- name: parent
+  variants:
+    - model: child
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: child
+  variants:
+    - model: grandchild
+  files:
+    - filename: b.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+- name: grandchild
+  files:
+    - filename: c.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+`))).To(ContainElement(ContainSubstring("declares variants of its own")))
+	})
+
+	It("reports duplicate entry names", func() {
+		Expect(Verify(write(`
+- name: dup
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: dup
+  files:
+    - filename: b.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(ContainElement(ContainSubstring("duplicate entry name")))
+	})
+
+	It("reports a file with no sha256", func() {
+		Expect(Verify(write(`
+- name: one
+  files:
+    - filename: a.gguf
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("no sha256")))
+	})
+
+	It("reports an entry tagged dflash without a matching spec_type", func() {
+		Expect(Verify(write(`
+- name: liar
+  tags:
+    - dflash
+  overrides:
+    backend: llama-cpp
+    options:
+      - use_jinja:true
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("tagged dflash")))
+	})
+
+	It("reports an entry configuring spec_type without the tag", func() {
+		Expect(Verify(write(`
+- name: shy
+  overrides:
+    backend: llama-cpp
+    options:
+      - spec_type:draft-mtp
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("not tagged mtp")))
+	})
+
+	// ds4 carries the MTP heads in the weights and names them with mtp_path, so
+	// the rule holds there in a different vocabulary rather than not at all.
+	It("reports a ds4 entry configuring mtp_path without the tag", func() {
+		Expect(Verify(write(`
+- name: ds4-shy
+  overrides:
+    backend: ds4
+    options:
+      - mtp_path:model-mtp.gguf
+      - mtp_draft:2
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("not tagged mtp")))
+	})
+
+	It("reports a ds4 entry tagged mtp that configures no mtp_path", func() {
+		Expect(Verify(write(`
+- name: ds4-liar
+  tags:
+    - mtp
+  overrides:
+    backend: ds4
+    options:
+      - context_size:4096
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(ContainElement(ContainSubstring("tagged mtp")))
+	})
+
+	It("accepts a ds4 entry that both configures mtp_path and carries the tag", func() {
+		Expect(Verify(write(`
+- name: ds4-honest
+  tags:
+    - mtp
+  overrides:
+    backend: ds4
+    options:
+      - mtp_path:model-mtp.gguf
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(BeEmpty())
+	})
+
+	// sglang declares speculative_algorithm in the referenced gallery/*.yaml,
+	// which Verify never reads, so it may not judge such an entry either way.
+	It("says nothing about an sglang entry tagged mtp", func() {
+		Expect(Verify(write(`
+- name: sglang-mtp
+  tags:
+    - mtp
+  overrides:
+    backend: sglang
+  files: []
+`))).To(BeEmpty())
+	})
+
+	It("says nothing about the tag on an entry with no declared backend", func() {
+		Expect(Verify(write(`
+- name: templated
+  tags:
+    - mtp
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+`))).To(BeEmpty())
+	})
+
+	// The flat-match branch in unsloth.go appends every match, so a repo
+	// publishing both a plain and a UD Q8_0 renders one entry holding two full
+	// models while model: points at only the first.
+	It("reports an entry holding more than one non-shard weight file", func() {
+		Expect(Verify(write(`
+- name: greedy
+  overrides:
+    backend: llama-cpp
+    options:
+      - use_jinja:true
+    parameters:
+      model: llama-cpp/models/repo/Model-Q8_0.gguf
+  files:
+    - filename: llama-cpp/models/repo/Model-Q8_0.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: llama-cpp/models/repo/Model-UD-Q8_0.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(ContainElement(ContainSubstring("more than one weight file")))
+	})
+
+	It("accepts many shards alongside an mmproj and a drafter", func() {
+		Expect(Verify(write(`
+- name: sharded
+  tags:
+    - mtp
+  overrides:
+    backend: llama-cpp
+    options:
+      - spec_type:draft-mtp
+    mmproj: llama-cpp/mmproj/repo/mm.gguf
+    draft_model: llama-cpp/models/repo/Model-draft.gguf
+  files:
+    - filename: llama-cpp/models/repo/Model-00001-of-00002.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: llama-cpp/models/repo/Model-00002-of-00002.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+    - filename: llama-cpp/mmproj/repo/mm.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+    - filename: llama-cpp/models/repo/Model-draft.gguf
+      sha256: dd
+      uri: https://example.com/d.gguf
+`))).To(BeEmpty())
+	})
+
+	// Multi-component TTS and ASR engines legitimately ship an encoder, a
+	// tokenizer, a vocoder and so on as one model, so the collision the weight
+	// count catches does not exist for them.
+	It("accepts a multi-component non-llama-cpp entry declaring five weights", func() {
+		Expect(Verify(write(`
+- name: multi
+  overrides:
+    backend: qwen3-tts-cpp
+  files:
+    - filename: talker.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: tokenizer.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+    - filename: vocoder.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+    - filename: encoder.gguf
+      sha256: dd
+      uri: https://example.com/d.gguf
+    - filename: vae.gguf
+      sha256: ee
+      uri: https://example.com/e.gguf
+`))).To(BeEmpty())
+	})
+
+	It("says nothing about the weight count of an entry with no declared backend", func() {
+		Expect(Verify(write(`
+- name: templated-weights
+  files:
+    - filename: model-Q4_K_M.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: model-mmproj-f16.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+`))).To(BeEmpty())
+	})
+
+	It("says nothing about an auxiliary metadata file carrying no sha256", func() {
+		Expect(Verify(write(`
+- name: aux
+  files:
+    - filename: a.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+    - filename: params.json
+      sha256: ""
+      uri: https://example.com/params.json
+`))).To(BeEmpty())
+	})
+
+	// safetensors weights are downloaded and loaded exactly like GGUF weights,
+	// so an unverified one is the same supply-chain hole.
+	It("reports a safetensors weight carrying no sha256", func() {
+		Expect(Verify(write(`
+- name: vae
+  files:
+    - filename: wan_2.1_vae.safetensors
+      sha256: ""
+      uri: https://example.com/vae.safetensors
+`))).To(ContainElement(ContainSubstring("no sha256")))
+	})
+
+	It("says nothing about a txt or md file carrying no sha256", func() {
+		Expect(Verify(write(`
+- name: docs
+  files:
+    - filename: notes.txt
+      sha256: ""
+      uri: https://example.com/notes.txt
+    - filename: README.md
+      sha256: ""
+      uri: https://example.com/README.md
+`))).To(BeEmpty())
+	})
+})
+
+var _ = Describe("UnaccountedQuants", func() {
+	// A quant published only as root-level shards matches neither branch in
+	// DiscoverUnslothQuants, so without this diagnostic the build would vanish
+	// from a batch run with nothing said about it.
+	It("reports a wanted quant upstream publishes but discovery dropped", func() {
+		files := []GGUFFile{
+			{Name: "Model-UD-Q4_K_M-00001-of-00003.gguf", SHA256: "aa"},
+			{Name: "Model-UD-Q4_K_M-00002-of-00003.gguf", SHA256: "bb"},
+			{Name: "Model-UD-Q4_K_M-00003-of-00003.gguf", SHA256: "cc"},
+		}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).
+			To(ContainElement(ContainSubstring("UD-Q4_K_M")))
+	})
+
+	It("says nothing when every published wanted quant produced a build", func() {
+		files := []GGUFFile{
+			{Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"},
+			{Name: "UD-Q6_K/Model-UD-Q6_K-00001-of-00002.gguf", SHA256: "bb"},
+			{Name: "UD-Q6_K/Model-UD-Q6_K-00002-of-00002.gguf", SHA256: "cc"},
+		}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
+	})
+
+	It("says nothing about a wanted quant the repo does not publish at all", func() {
+		files := []GGUFFile{{Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"}}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
+	})
+
+	// UD-Q8_0 is its own quant label and is not a wanted one. Reading it as a
+	// publication of Q8_0 is the substring collision this diagnostic exists to
+	// warn about, and subdirectory-sharded UD quants are the normal unsloth
+	// layout for large repos, so the false positive would fire on every batch.
+	It("does not read a subdirectory-sharded UD-Q8_0 as a published Q8_0", func() {
+		files := []GGUFFile{
+			{Name: "UD-Q8_0/Model-UD-Q8_0-00001-of-00002.gguf", SHA256: "aa"},
+			{Name: "UD-Q8_0/Model-UD-Q8_0-00002-of-00002.gguf", SHA256: "bb"},
+		}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
+	})
+
+	// A quant in its own subdirectory but not shard-numbered matches neither
+	// branch of DiscoverUnslothQuants, so it is genuinely published and
+	// genuinely undiscovered.
+	It("reports a wanted quant published in its own subdirectory without shard numbering", func() {
+		files := []GGUFFile{{Name: "Q8_0/Model-Q8_0.gguf", SHA256: "aa"}}
+
+		Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).
+			To(ContainElement(ContainSubstring("quant Q8_0 is published upstream")))
+	})
+
+	// builds is empty on purpose: it isolates the file-to-quant match from
+	// whatever DiscoverUnslothQuants would have made of the same file.
+	It("matches the flat single-file layout", func() {
+		files := []GGUFFile{{Name: "Model-Q8_0.gguf", SHA256: "aa"}}
+
+		Expect(UnaccountedQuants(files, nil)).
+			To(ConsistOf(ContainSubstring("quant Q8_0 is published upstream")))
+	})
+})
+
+var _ = Describe("Verify local path collisions", func() {
+	write := func(body string) string {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "index.yaml")
+		Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed())
+		return p
+	}
+
+	It("reports one local path claimed by two different uris", func() {
+		// The shape that shipped: LiquidAI and unsloth both publish
+		// LFM2.5-8B-A1B-GGUF, so a path built from the bare repo name gives both
+		// entries the same local file under two different checksums.
+		Expect(Verify(write(`
+- name: lfm2.5-8b-a1b
+  files:
+    - filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf
+      sha256: 33ab3b8c
+      uri: https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf
+- name: lfm2.5-8b-a1b-q8-0
+  files:
+    - filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf
+      sha256: ec11666b
+      uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf
+`))).To(ContainElement(SatisfyAll(
+			ContainSubstring("claimed by two different uris"),
+			ContainSubstring("lfm2.5-8b-a1b-q8-0"),
+		)))
+	})
+
+	It("accepts two entries reusing one file from the same uri", func() {
+		// Sibling builds of one repo legitimately share a projector.
+		Expect(Verify(write(`
+- name: a
+  files:
+    - filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf
+      sha256: cc
+      uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf
+- name: b
+  files:
+    - filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf
+      sha256: cc
+      uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf
+`))).To(BeEmpty())
+	})
+
+	It("reports a collision once however many entries pile onto the path", func() {
+		problems := Verify(write(`
+- name: a
+  files:
+    - filename: shared.gguf
+      sha256: aa
+      uri: https://example.com/a.gguf
+- name: b
+  files:
+    - filename: shared.gguf
+      sha256: bb
+      uri: https://example.com/b.gguf
+- name: c
+  files:
+    - filename: shared.gguf
+      sha256: cc
+      uri: https://example.com/c.gguf
+`))
+
+		var collisions int
+		for _, p := range problems {
+			if strings.Contains(p, "claimed by two different uris") {
+				collisions++
+			}
+		}
+		Expect(collisions).To(Equal(1))
+	})
+
+	It("says nothing about files that carry no uri", func() {
+		// A hand-written entry may record only a checksum. There is no upstream
+		// to compare, so the check cannot conclude anything either way.
+		Expect(Verify(write(`
+- name: a
+  files:
+    - filename: shared.gguf
+      sha256: aa
+- name: b
+  files:
+    - filename: shared.gguf
+      sha256: bb
+`))).To(BeEmpty())
+	})
+})

.github/ci/galleryedit/edit.go

@@ -0,0 +1,152 @@
+// Package galleryedit splices variant references into the LocalAI gallery index
+// as TEXT.
+//
+// Re-serialising the index through a YAML marshaller would reflow 40,000 lines,
+// drop the anchors and merge keys the gallery relies on, and produce a diff no
+// reviewer could read, which makes a pull request worthless even when the
+// content inside it is right. Every generator that adds variants to an entry the
+// gallery already ships therefore edits lines, and they share this package so
+// that two of them cannot drift apart on where a variants block belongs.
+package galleryedit
+
+import (
+	"fmt"
+	"regexp"
+	"sort"
+	"strings"
+)
+
+var (
+	entryStart   = regexp.MustCompile(`^-(?: |$)`)
+	inlineName   = regexp.MustCompile(`^- (?:&\S+ )?name:`)
+	keyName      = regexp.MustCompile(`^  name:`)
+	keyVariants  = regexp.MustCompile(`^  variants:\s*(.*)$`)
+	variantItem  = regexp.MustCompile(`^    - `)
+	unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`)
+)
+
+// Entry is the positional view of one gallery entry: what it is called and
+// which lines it occupies. Nothing about what the entry MEANS belongs here, so
+// each caller keeps its own semantic decode and only hands over the coordinates.
+type Entry struct {
+	Name string
+	// StartLine and EndLine bound the entry, zero based and half open.
+	StartLine int
+	EndLine   int
+}
+
+// Insert is one entry's pending variants addition. The caller owns the contents
+// of Variants: this package neither orders nor deduplicates them, because the
+// right order and the right dedup rule differ between generators.
+type Insert struct {
+	Entry    Entry
+	Variants []string
+}
+
+// Scan splits index text into lines and reports the line each top level list
+// item begins on.
+func Scan(text string) (lines []string, starts []int) {
+	lines = strings.Split(text, "\n")
+	for i, line := range lines {
+		if entryStart.MatchString(line) {
+			starts = append(starts, i)
+		}
+	}
+	return lines, starts
+}
+
+// Apply splices every insert into the index lines and returns the new text.
+func Apply(lines []string, inserts []Insert) ([]string, error) {
+	type edit struct {
+		at     int
+		remove int
+		insert []string
+	}
+	var edits []edit
+
+	for _, in := range inserts {
+		if len(in.Variants) == 0 {
+			continue
+		}
+
+		items := make([]string, 0, len(in.Variants))
+		for _, v := range in.Variants {
+			items = append(items, "    - model: "+QuoteName(v))
+		}
+
+		at, remove, err := insertionPoint(lines, in.Entry)
+		if err != nil {
+			return nil, err
+		}
+		block := items
+		if remove > 0 || !hasVariantsKey(lines, in.Entry) {
+			block = append([]string{"  variants:"}, items...)
+		}
+		edits = append(edits, edit{at: at, remove: remove, insert: block})
+	}
+
+	// Applying from the bottom up keeps every line number computed against the
+	// original text valid while earlier edits are still pending.
+	sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at })
+
+	out := append([]string(nil), lines...)
+	for _, e := range edits {
+		tail := append([]string(nil), out[e.at+e.remove:]...)
+		out = append(out[:e.at], append(append([]string(nil), e.insert...), tail...)...)
+	}
+	return out, nil
+}
+
+func hasVariantsKey(lines []string, e Entry) bool {
+	for i := e.StartLine; i < e.EndLine; i++ {
+		if keyVariants.MatchString(lines[i]) {
+			return true
+		}
+	}
+	return false
+}
+
+// insertionPoint reports where new variant items belong, and how many existing
+// lines the insertion replaces.
+//
+// An entry with no variants key gets one right after its name, which is where
+// the hand-written families put it. An entry with an empty "variants: []" has
+// that line replaced by a block. An entry with a block gets its items appended.
+func insertionPoint(lines []string, e Entry) (at int, remove int, err error) {
+	for i := e.StartLine; i < e.EndLine; i++ {
+		m := keyVariants.FindStringSubmatch(lines[i])
+		if m == nil {
+			continue
+		}
+		if strings.TrimSpace(m[1]) == "[]" {
+			return i, 1, nil
+		}
+		if strings.TrimSpace(m[1]) != "" {
+			return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1]))
+		}
+		last := i
+		for j := i + 1; j < e.EndLine && variantItem.MatchString(lines[j]); j++ {
+			last = j
+		}
+		return last + 1, 0, nil
+	}
+
+	if inlineName.MatchString(lines[e.StartLine]) {
+		return e.StartLine + 1, 0, nil
+	}
+	for i := e.StartLine; i < e.EndLine; i++ {
+		if keyName.MatchString(lines[i]) {
+			return i + 1, 0, nil
+		}
+	}
+	return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name)
+}
+
+// QuoteName quotes a variant reference when the name would otherwise change
+// meaning as bare YAML. Config-suffixed names carry a ":" and always need it.
+func QuoteName(name string) string {
+	if unsafeInName.MatchString(name) {
+		return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"`
+	}
+	return name
+}

.github/ci/variantproposals/edit.go

@@ -2,119 +2,41 @@ package main
 
 import (
 	"fmt"
-	"regexp"
-	"sort"
 	"strings"
-)
 
-var (
-	inlineName   = regexp.MustCompile(`^- (?:&\S+ )?name:`)
-	keyName      = regexp.MustCompile(`^  name:`)
-	keyVariants  = regexp.MustCompile(`^  variants:\s*(.*)$`)
-	variantItem  = regexp.MustCompile(`^    - `)
-	unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`)
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
 )
 
 // ApplyFamilies writes the proposed variant lists into the index text.
 //
-// The edit is textual on purpose. Re-serialising the index through a YAML
-// marshaller would reflow 40,000 lines, drop the anchors and merge keys the
-// gallery relies on, and produce a diff no reviewer could read, which would
-// make the pull request worthless even when the proposals inside it are right.
+// The line editing itself lives in galleryedit, shared with the apexentries
+// generator. Both jobs add variants to entries the gallery already ships, and a
+// second answer to "where does a variants block go" would drift from this one;
+// see that package for why the edit is textual rather than a YAML round trip.
 func ApplyFamilies(ix *Index, families []Family) ([]string, error) {
 	byName, _ := ix.ByName()
 
-	type edit struct {
-		at      int
-		remove  int
-		insert  []string
-		ordinal int
-	}
-	var edits []edit
-
+	var inserts []galleryedit.Insert
 	for _, f := range families {
 		entry, ok := byName[strings.ToLower(f.Parent)]
 		if !ok {
 			return nil, fmt.Errorf("parent %q is not in the index", f.Parent)
 		}
-		items := make([]string, 0, len(f.Proposals))
+
+		variants := make([]string, 0, len(f.Proposals))
 		for _, p := range f.Proposals {
-			items = append(items, "    - model: "+quoteName(p.Variant))
+			variants = append(variants, p.Variant)
 		}
 
-		at, remove, err := insertionPoint(ix, entry)
-		if err != nil {
-			return nil, err
-		}
-		insert := items
-		if remove > 0 || !hasVariantsKey(ix, entry) {
-			insert = append([]string{"  variants:"}, items...)
-		}
-		edits = append(edits, edit{at: at, remove: remove, insert: insert, ordinal: entry.Index})
+		inserts = append(inserts, galleryedit.Insert{
+			Entry: galleryedit.Entry{
+				Name:      entry.Name,
+				StartLine: entry.StartLine,
+				EndLine:   entry.EndLine,
+			},
+			Variants: variants,
+		})
 	}
 
-	// Applying from the bottom up keeps every line number computed against the
-	// original text valid while earlier edits are still pending.
-	sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at })
-
-	lines := append([]string(nil), ix.Lines...)
-	for _, e := range edits {
-		tail := append([]string(nil), lines[e.at+e.remove:]...)
-		lines = append(lines[:e.at], append(append([]string(nil), e.insert...), tail...)...)
-	}
-	return lines, nil
-}
-
-func hasVariantsKey(ix *Index, e *GalleryEntry) bool {
-	for i := e.StartLine; i < e.EndLine; i++ {
-		if keyVariants.MatchString(ix.Lines[i]) {
-			return true
-		}
-	}
-	return false
-}
-
-// insertionPoint reports where new variant items belong, and how many existing
-// lines the insertion replaces.
-//
-// An entry with no variants key gets one right after its name, which is where
-// the hand-written families put it. An entry with an empty "variants: []" has
-// that line replaced by a block. An entry with a block gets its items appended.
-func insertionPoint(ix *Index, e *GalleryEntry) (at int, remove int, err error) {
-	for i := e.StartLine; i < e.EndLine; i++ {
-		m := keyVariants.FindStringSubmatch(ix.Lines[i])
-		if m == nil {
-			continue
-		}
-		if strings.TrimSpace(m[1]) == "[]" {
-			return i, 1, nil
-		}
-		if strings.TrimSpace(m[1]) != "" {
-			return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1]))
-		}
-		last := i
-		for j := i + 1; j < e.EndLine && variantItem.MatchString(ix.Lines[j]); j++ {
-			last = j
-		}
-		return last + 1, 0, nil
-	}
-
-	if inlineName.MatchString(ix.Lines[e.StartLine]) {
-		return e.StartLine + 1, 0, nil
-	}
-	for i := e.StartLine; i < e.EndLine; i++ {
-		if keyName.MatchString(ix.Lines[i]) {
-			return i + 1, 0, nil
-		}
-	}
-	return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name)
-}
-
-// quoteName quotes a variant reference when the name would otherwise change
-// meaning as bare YAML. Config-suffixed names carry a ":" and always need it.
-func quoteName(name string) string {
-	if unsafeInName.MatchString(name) {
-		return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"`
-	}
-	return name
+	return galleryedit.Apply(ix.Lines, inserts)
 }

.github/ci/variantproposals/index.go

@@ -8,6 +8,8 @@ import (
 	"strings"
 
 	"gopkg.in/yaml.v3"
+
+	"github.com/mudler/LocalAI/.github/ci/galleryedit"
 )
 
 // File is the subset of a gallery file entry the proposer reads.
@@ -57,7 +59,6 @@ type Index struct {
 }
 
 var (
-	entryStart  = regexp.MustCompile(`^-(?: |$)`)
 	anchorStart = regexp.MustCompile(`^- &(\S+)`)
 	mergeStart  = regexp.MustCompile(`^- !!merge <<: \*(\S+)`)
 )
@@ -82,13 +83,7 @@ func ParseIndex(text string) (*Index, error) {
 		return nil, fmt.Errorf("decoding gallery index: %w", err)
 	}
 
-	lines := strings.Split(text, "\n")
-	var starts []int
-	for i, line := range lines {
-		if entryStart.MatchString(line) {
-			starts = append(starts, i)
-		}
-	}
+	lines, starts := galleryedit.Scan(text)
 	if len(starts) != len(entries) {
 		return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number", len(entries), len(starts))
 	}

.github/gh_curl.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+# Shared curl wrapper for the nightly dependency-bump scripts.
+#
+# The bump workflow fans out to ~25 parallel matrix jobs, each querying
+# api.github.com. Anonymous API calls are capped at 60/hour per source IP and
+# GitHub-hosted runners egress through shared NAT addresses, so a random handful
+# of jobs were getting rate-limited (HTTP 403 -> curl exit 22, empty response)
+# every single night. Authenticating with GITHUB_TOKEN lifts the ceiling to
+# 1000/hour; the retries absorb whatever transient blips remain.
+
+# Wraps curl with GitHub auth (when a token is present) plus retry/timeout
+# hardening. Callers pass their own headers and the URL.
+gh_curl() {
+    # The bump scripts run under `set -x`; without this the Authorization header
+    # would be echoed into the job log on every call.
+    local had_xtrace=0
+    case "$-" in
+        *x*) had_xtrace=1; set +x ;;
+    esac
+
+    local args=(
+        --silent --show-error --location --fail
+        # --retry-all-errors so 403 rate-limit responses are retried too; plain
+        # --retry only covers 408/429/5xx. curl honours Retry-After when sent.
+        --retry 5 --retry-delay 3 --retry-all-errors
+        --connect-timeout 15 --max-time 60
+    )
+    if [ -n "${GITHUB_TOKEN:-}" ]; then
+        args+=(--header "Authorization: Bearer ${GITHUB_TOKEN}")
+    fi
+
+    curl "${args[@]}" "$@"
+    local rc=$?
+
+    if [ "$had_xtrace" -eq 1 ]; then
+        set -x
+    fi
+    return $rc
+}

.github/workflows/build-test.yaml

@@ -6,6 +6,14 @@ on:
       - master
   pull_request:
 
+# Supersede an in-flight run when a PR gets a new push. Keyed on the PR number
+# so every push to the same PR shares a group; on a master push the key falls
+# back to github.sha (unique per commit) and cancel-in-progress is false, so
+# master runs never cancel each other -- each commit is built on its own.
+concurrency:
+  group: ci-build-test-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 jobs:
   build-test:
     runs-on: ubuntu-latest

.github/workflows/bump_deps.yaml

@@ -50,6 +50,10 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
+          - repository: "mudler/vllm.cpp"
+            variable: "VLLM_CPP_VERSION"
+            branch: "main"
+            file: "backend/go/vllm-cpp/Makefile"
           - repository: "localai-org/moss-transcribe.cpp"
             variable: "MOSS_VERSION"
             branch: "master"
@@ -110,11 +114,21 @@ jobs:
             variable: "VIBEVOICE_CPP_VERSION"
             branch: "master"
             file: "backend/go/vibevoice-cpp/Makefile"
+          - repository: "mudler/magpie-tts.cpp"
+            variable: "MAGPIETTS_CPP_VERSION"
+            branch: "main"
+            file: "backend/go/magpie-tts-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7
       - name: Bump dependencies 🔧
         id: bump
+        env:
+          # This job fans out to ~25 parallel matrix entries, all querying
+          # api.github.com from runner IPs that share the 60/hour anonymous
+          # rate limit. Authenticating raises it to 1000/hour, which is what
+          # kept a random handful of these red every night.
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_deps.sh ${{ matrix.repository }} ${{ matrix.branch }} ${{ matrix.variable }} ${{ matrix.file }}
           {
@@ -151,6 +165,8 @@ jobs:
       - uses: actions/checkout@v7
       - name: Bump vLLM cu130 wheel pin 🔧
         id: bump
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_vllm_wheel.sh vllm-project/vllm backend/python/vllm/requirements-cublas13-after.txt VLLM_VERSION
           {
@@ -187,6 +203,8 @@ jobs:
       - uses: actions/checkout@v7
       - name: Bump vllm-metal pin 🔧
         id: bump
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_vllm_metal.sh vllm-project/vllm-metal backend/python/vllm/install.sh VLLM_METAL_VERSION
           {

.github/workflows/bump_docs.yaml

@@ -15,6 +15,10 @@ jobs:
     steps:
       - uses: actions/checkout@v7
       - name: Bump dependencies 🔧
+        env:
+          # Authenticated API calls get 1000 req/hour instead of the 60/hour
+          # anonymous cap that is shared across every job on the runner's IP.
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           bash .github/bump_docs.sh ${{ matrix.repository }}
       - name: Create Pull Request

.github/workflows/ci-tools-tests.yaml

@@ -0,0 +1,39 @@
+---
+# The packages under .github/ci/ are invisible to `go list ./...`, so neither
+# `make lint` nor the repository test run ever touches them. Their specs are
+# dead weight until a workflow names each package explicitly.
+name: 'CI tool tests'
+on:
+  pull_request:
+    paths:
+      - '.github/ci/**'
+      - '.github/workflows/ci-tools-tests.yaml'
+  push:
+    branches:
+      - master
+    paths:
+      - '.github/ci/**'
+jobs:
+  ci-tools:
+    name: 'Test the .github/ci generators'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      # The discovery heuristics are the risky part of these tools. A regression
+      # produces confident, wrong gallery entries, which is worse than no tool.
+      - name: 'Test the APEX entry generator'
+        run: go test ./.github/ci/apexentries/
+
+      - name: 'Test the variant proposer'
+        run: go test ./.github/ci/variantproposals/
+
+      # Shared by both generators above. Its behaviour is exercised through their
+      # specs; this step exists so a break in the shared package fails under its
+      # own name rather than as a puzzling failure in whichever caller ran first.
+      - name: 'Test the shared gallery editor'
+        run: go test ./.github/ci/galleryedit/

.github/workflows/image-pr.yml

@@ -52,7 +52,7 @@
               tag-latest: 'false'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:22.04"
+              base-image: "ubuntu:24.04"
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'hipblas'

.github/workflows/image.yml

@@ -113,7 +113,7 @@
               tag-latest: 'auto'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:22.04"
+              base-image: "ubuntu:24.04"
               skip-drivers: 'false'
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'

.github/workflows/lint.yml

@@ -61,7 +61,7 @@ jobs:
       # The backend matrix path filter fails silently: a miss emits an empty
       # matrix, every job goes green, and the change reaches no image (#10946).
       # Its tests need only node, so they ride along with this job.
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v7
         with:
           node-version: '20'
       - name: run CI script tests

.github/workflows/secscan.yaml

@@ -7,6 +7,19 @@ on:
   schedule:
   - cron: '0 0 * * 0'
 
+# `push:` is deliberately unfiltered, so this fires on every push to every
+# branch and there is no pull_request event to key on -- the usual
+# `github.event.pull_request.number || github.sha` idiom used elsewhere would
+# key on the unique-per-commit sha and dedup nothing. Group on the ref instead
+# so successive pushes to the same feature branch supersede one another.
+#
+# Cancelling is safe here: the only output is a SARIF upload, and code scanning
+# tracks the latest result per ref, so a superseded scan has nothing to lose.
+# master is excluded anyway -- every commit on master gets its own scan.
+concurrency:
+  group: ci-secscan-${{ github.ref }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/master' }}
+
 jobs:
   tests:
     runs-on: ubuntu-latest

.github/workflows/test-extra.yml

@@ -37,6 +37,7 @@ jobs:
       sglang: ${{ steps.detect.outputs.sglang }}
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
       qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
+      magpie-tts-cpp: ${{ steps.detect.outputs.magpie-tts-cpp }}
       rfdetr-cpp: ${{ steps.detect.outputs.rfdetr-cpp }}
       locate-anything-cpp: ${{ steps.detect.outputs.locate-anything-cpp }}
       vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
@@ -866,6 +867,38 @@ jobs:
       - name: Test qwen3-tts-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/qwen3-tts-cpp test
+  tests-magpie-tts-cpp:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.magpie-tts-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
+      - name: Setup Go
+        uses: actions/setup-go@v5
+      - name: Display Go version
+        run: go version
+      - name: Proto Dependencies
+        run: |
+          # Install protoc
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Build magpie-tts-cpp
+        run: |
+          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp
+      - name: Test magpie-tts-cpp
+        run: |
+          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp test
   # Per-backend smoke for rfdetr-cpp: builds the .so + Go binary and runs
   # `make -C backend/go/rfdetr-cpp test`. test.sh fetches the small (~20 MB)
   # rfdetr-nano-q8_0 GGUF from the published mudler/rfdetr-cpp-nano HF repo

.github/workflows/yaml-check.yml

@@ -1,6 +1,11 @@
 name: 'Yamllint GitHub Actions'
 on:
   - pull_request
+
+concurrency:
+  group: ci-yamllint-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 jobs:
   yamllint:
     name: 'Yamllint'

.gitignore

@@ -111,3 +111,10 @@ core/http/react-ui/test-results/
 # the realtime-conformance gate; only the .fizz sources are authoritative.
 formal-verification/*.json
 formal-verification/out/
+
+# `go build ./.github/ci/apexentries` drops a binary of the package name into
+# whatever directory it runs in, one `git add -A` away from being committed.
+# Both paths are anchored: an unanchored `apexentries` would also match the
+# package directory itself and untrack the source.
+/apexentries
+/.github/ci/apexentries/apexentries

AGENTS.md

@@ -35,7 +35,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 
 ## Quick Reference
 
-- **Git hooks & coverage gates**: Run `make install-hooks` once per clone so the pre-commit lint + coverage gates run. **Never bypass them with `git commit --no-verify`, and never lower a coverage baseline or widen a gate's tolerance to turn a red gate green** — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
+- **Coverage gates**: Never lower a coverage baseline or widen a gate's tolerance to turn a red gate green — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
 - **Logging**: Use `github.com/mudler/xlog` (same API as slog)
 - **Go style**: Prefer `any` over `interface{}`
 - **Comments**: Explain *why*, not *what*

CONTRIBUTING.md

@@ -198,7 +198,6 @@ For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`C
 
 - Prefer modern Go idioms — for example, use `any` instead of `interface{}`.
 - Use [`golangci-lint`](https://golangci-lint.run) to catch common issues before submitting a PR.
-- Run `make install-hooks` once per clone to enable the pre-commit hook: Go changes run `make lint` + the coverage gate (`make test-coverage-check`); `core/http/react-ui/` changes run the Playwright e2e suite (`make test-ui`). Bypass a single commit with `git commit --no-verify`.
 - Use [`github.com/mudler/xlog`](https://github.com/mudler/xlog) for logging (same API as `slog`). Do not use `fmt.Println` or the standard `log` package for operational logging.
 - Use tab indentation for Go files (as defined in `.editorconfig`).
 
@@ -268,7 +267,7 @@ make test-e2e
 
 ### React UI tests and coverage
 
-The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI and pre-commit). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
+The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
 
 **If your change lowers UI coverage, raise it back by adding specs — do not widen the tolerance or hand-lower the baseline.** A *render-smoke* spec (navigate to a page, assert its header is visible) cheaply covers an entire lazy page. See `core/http/react-ui/e2e/page-render-smoke.spec.js` and the full policy in [.agents/building-and-testing.md](.agents/building-and-testing.md#react-ui-coverage).
 

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -103,7 +103,7 @@ COVERAGE_E2E_LABELS?=!real-models
 COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go
 
 
-.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check install-hooks build vendor lint lint-all
+.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
 
 all: help
 
@@ -269,8 +269,7 @@ LINT_EXCLUDE_DIRS_RE=/(backend/go/(piper|silero-vad|llm)|cmd/launcher)(/|$$)
 
 ## Set LINT_NEW_FROM to a git ref to override .golangci.yml's
 ## new-from-merge-base (origin/master). Useful from a fork clone where
-## origin/master is stale relative to the canonical repo — the pre-commit
-## hook passes the resolved upstream ref here so local lint matches CI.
+## origin/master is stale relative to the canonical repo.
 LINT_NEW_FROM?=
 lint:
 	@command -v golangci-lint >/dev/null 2>&1 || { \
@@ -289,17 +288,6 @@ lint-all:
 	}
 	golangci-lint run --new=false --new-from-merge-base= --new-from-rev= $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
 
-########################################################
-## Git hooks
-########################################################
-## Points git at the versioned .githooks/ directory so the pre-commit hook
-## (lint + coverage gate) runs locally. Run once per clone. Undo with:
-## `git config --unset core.hooksPath`. Skip a single commit with
-## `git commit --no-verify`.
-install-hooks:
-	git config core.hooksPath .githooks
-	@echo 'Installed git hooks: core.hooksPath -> .githooks (pre-commit runs lint + test-coverage-check on Go changes)'
-
 ########################################################
 ## E2E AIO tests (uses standard image with pre-configured models)
 ########################################################
@@ -637,6 +625,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/go/locate-anything-cpp test
 	$(MAKE) -C backend/go/depth-anything-cpp test
 	$(MAKE) -C backend/go/supertonic test
+	$(MAKE) -C backend/go/vllm-cpp test
 
 ##
 ## End-to-end gRPC tests that exercise a built backend container image.
@@ -1269,6 +1258,8 @@ BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
 BACKEND_MOSS_TTS_CPP = moss-tts-cpp|golang|.|false|true
+BACKEND_MAGPIE_TTS_CPP = magpie-tts-cpp|golang|.|false|true
+BACKEND_VLLM_CPP = vllm-cpp|golang|.|false|true
 BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
 BACKEND_LOCALVQE = localvqe|golang|.|false|true
@@ -1396,6 +1387,8 @@ $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TTS_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_MAGPIE_TTS_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
@@ -1415,7 +1408,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1450,7 +1443,7 @@ test-ui-e2e: build-ui-test-server
 UI_TEST_WORKERS ?=
 PLAYWRIGHT_WORKERS_FLAG = $(if $(UI_TEST_WORKERS),--workers=$(UI_TEST_WORKERS),)
 
-## Fast Playwright e2e run used by the pre-commit hook on React UI changes.
+## Fast Playwright e2e run for local React UI validation.
 ## Force-rebuilds the (non-instrumented) dist so the suite tests the working
 ## tree — not a stale dist the `react-ui` skip-guard would leave — re-embeds
 ## it into ui-test-server, and runs the specs. Uses the nix-provided browser

README.md

@@ -231,9 +231,11 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 
 | Backend | What it does |
 |---------|-------------|
+| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
 | [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
 | [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
+| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
 | [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
 | [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
 | [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |

backend/Dockerfile.golang

@@ -221,6 +221,33 @@ RUN if [ "${BACKEND}" = "crispasr" ]; then \
     apt-get clean && rm -rf /var/lib/apt/lists/*; \
 fi
 
+# sherpa-onnx links onnxruntime's CUDA execution provider, and
+# libonnxruntime_providers_cuda.so has cuDNN as a hard DT_NEEDED. The
+# onnxruntime GPU tarball does not ship cuDNN itself, so without this the
+# builder has none (the arm64 + CUDA 13 branch above is the only other place
+# that installs it) and package-gpu-libs.sh correctly refuses to produce a
+# package that references cuDNN with no cuDNN available to it.
+#
+# Installed per-backend rather than for every cublas build: the auto-detection
+# in package-gpu-libs.sh bundles only what a package actually references, so
+# the ggml backends would not grow either way, but they would all pay ~1.1 GB
+# of builder layer and registry cache for a library they never call.
+#
+# Runtime package only, no -dev: sherpa-onnx consumes onnxruntime's prebuilt
+# CUDA provider and never compiles against cuDNN headers. libcudnn9-cuda-N
+# carries the dispatcher plus all seven dlopen()ed sublibraries, which is what
+# complete_cudnn_family needs to assemble a whole bundle.
+RUN <<EOT bash
+    if [ "${BACKEND}" = "sherpa-onnx" ] && [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libcudnn9-cuda-${CUDA_MAJOR_VERSION} && \
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
 COPY . /LocalAI
 
 RUN git config --global --add safe.directory /LocalAI

backend/Dockerfile.llama-cpp

@@ -111,6 +111,10 @@ RUN make -BC /LocalAI/backend/cpp/llama-cpp package
 # ============================================================================
 FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
 
+ARG APT_MIRROR
+ENV APT_MIRROR=${APT_MIRROR}
+ARG APT_PORTS_MIRROR
+ENV APT_PORTS_MIRROR=${APT_PORTS_MIRROR}
 ARG BUILD_TYPE
 ENV BUILD_TYPE=${BUILD_TYPE}
 ARG CUDA_DOCKER_ARCH

backend/backend.proto

@@ -181,6 +181,13 @@ message ScoreRequest {
   // PredictOptions.ModelIdentity for the full rationale. Empty means "no
   // identity supplied" and backends MUST skip the check.
   string ModelIdentity = 5;
+  // Byte length of the prompt prefix that stays identical across
+  // repeated scoring calls (e.g. a classifier's option-list system
+  // prompt — everything before the per-turn probe text). Backends that
+  // snapshot state (hybrid/recurrent models cannot rewind otherwise)
+  // use it to place a reuse point exactly at the boundary, so the next
+  // call re-processes only the tokens after it. 0 means unknown.
+  int32 stable_prefix_len = 6;
 }
 
 // CandidateScore is one row in the ScoreResponse, matching by index
@@ -493,6 +500,11 @@ message ModelOptions {
   // Proxy carries the cloud-proxy backend's per-model configuration.
   // Empty for non-proxy backends.
   ProxyOptions Proxy = 74;
+
+  // EnableScore reserves backend resources for the Score RPC. It is derived
+  // from the model's explicit `known_usecases: [score]` declaration so models
+  // that never score retain their ordinary serving footprint.
+  bool EnableScore = 75;
 }
 
 // ProxyOptions configures the cloud-proxy backend. UpstreamURL and
@@ -508,6 +520,12 @@ message ProxyOptions {
   string api_key_file = 5;
   string upstream_model = 6;
   int32 request_timeout_seconds = 7;
+  // cache_prompt enables automatic Anthropic prompt-cache breakpoints
+  // (cache_control: ephemeral) on the stable prefix — system, tools, and
+  // the last message block — when translating to the Anthropic provider.
+  // Cuts input cost on repeated/agentic calls (cache read = 0.1x). Only
+  // meaningful for mode=translate + provider=anthropic; ignored otherwise.
+  bool cache_prompt = 8;
 }
 
 message Result {

backend/cpp/bonsai/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-BONSAI_VERSION?=9fcaed763ccda38ea81068ad9d7f991aaddca451
+BONSAI_VERSION?=7529fdaaf99ffdc5ca71ace9c7409a56b27ad92f
 LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp
 
 CMAKE_ARGS?=
@@ -41,6 +41,7 @@ define bonsai-build
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build llama.cpp
@@ -77,6 +78,7 @@ bonsai-cpu-all:
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build llama.cpp

backend/cpp/ds4/CMakeLists.txt

@@ -76,12 +76,13 @@ elseif(DS4_GPU STREQUAL "cpu")
     set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
 endif()
 
-# ds4.c now references ds4_distributed.c (distributed inference) and ds4_ssd.c
-# (SSD expert-cache), each split into its own translation unit upstream. Both
-# are GPU-agnostic objects shared by every GPU mode, so link them in regardless
-# of DS4_GPU.
+# Upstream splits distributed inference, tensor-parallel transport, the SSD
+# expert cache, and layer placement into GPU-agnostic translation units. Link
+# them regardless of DS4_GPU.
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_distributed.o")
+list(APPEND DS4_OBJS "${DS4_DIR}/ds4_tp.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_ssd.o")
+list(APPEND DS4_OBJS "${DS4_DIR}/ds4_layer_pack.o")
 
 add_executable(${TARGET}
     grpc-server.cpp

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=80ebbc396aee40eedc1d829222f3362d10fa4c6c
+# Upstream pin lives below as DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=80ebbc396aee40eedc1d829222f3362d10fa4c6c
+DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -18,20 +18,19 @@ UNAME_S := $(shell uname -s)
 
 CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
 
-# ds4_distributed.o and ds4_ssd.o are GPU-agnostic translation units that
-# ds4.c/ds4_cpu.o now reference (upstream split distributed inference and the
-# SSD expert-cache into their own .c files). Both objects are shared by every
-# GPU mode, so they are appended unconditionally below.
+# Upstream splits distributed inference, tensor-parallel transport, the SSD
+# expert cache, and layer placement into GPU-agnostic translation units. They
+# are shared by every GPU mode, so append them unconditionally below.
 ifeq ($(BUILD_TYPE),cublas)
     CMAKE_ARGS += -DDS4_GPU=cuda
-    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_ssd.o
+    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else ifeq ($(UNAME_S),Darwin)
     CMAKE_ARGS += -DDS4_GPU=metal
-    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_ssd.o
+    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
     # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
     CMAKE_ARGS += -DDS4_GPU=cpu
-    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_ssd.o
+    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 ifneq ($(NATIVE),true)
@@ -56,11 +55,11 @@ ds4:
 # the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
 ds4/ds4.o: ds4
 ifeq ($(BUILD_TYPE),cublas)
-	+$(MAKE) -C ds4 ds4.o ds4_cuda.o ds4_distributed.o ds4_ssd.o
+	+$(MAKE) -C ds4 ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else ifeq ($(UNAME_S),Darwin)
-	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_ssd.o
+	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
-	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_ssd.o
+	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 grpc-server: ds4/ds4.o

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=9d07d8681ece159a89fb4e16a1f9c9f3a5fac20f
+IK_LLAMA_VERSION?=b054a8b983827c01aec59d4dc273a27c492c51c4
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=571d0d540df04f25298d0e159e520d9fc62ed121
+LLAMA_VERSION?=1cbfd1988311775425d36c0ce066590f7d3049cf
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/disable-score-task.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Mark a copied gRPC server as targeting a llama.cpp fork that does not carry
+# LocalAI's slot-based Score patches. The RPC remains present in the shared
+# protobuf service, but responds with UNIMPLEMENTED instead of referencing
+# server task types and common_params fields absent from those forks.
+
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+    echo "usage: $0 <grpc-server.cpp>" >&2
+    exit 2
+fi
+
+SRC=$1
+
+if [[ ! -f "$SRC" ]]; then
+    echo "grpc-server.cpp not found at $SRC" >&2
+    exit 2
+fi
+
+if grep -q '^#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK' "$SRC"; then
+    echo "==> $SRC already disables the LocalAI score task, skipping"
+    exit 0
+fi
+
+awk '
+    !done && /^#include/ {
+        print "#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK 1"
+        print "// ^ injected by disable-score-task.sh for an unpatched llama.cpp fork"
+        print ""
+        done = 1
+    }
+    { print }
+    END {
+        if (!done) {
+            print "disable-score-task.sh: no #include anchor found" > "/dev/stderr"
+            exit 1
+        }
+    }
+' "$SRC" > "$SRC.tmp"
+mv "$SRC.tmp" "$SRC"
+
+echo "==> LocalAI score task disabled in $SRC"

backend/cpp/llama-cpp/grpc-server.cpp

@@ -52,6 +52,7 @@
 #include "common.h"
 #include "arg.h"
 #include "chat-auto-parser.h"
+#include "llama_compat.h" // fork-skew switches, generated by prepare.sh
 #include "message_content.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
@@ -151,40 +152,6 @@ static std::string base64_encode_bytes(const unsigned char* data, size_t len) {
 
 bool loaded_model; // TODO: add a mutex for this, but happens only once loading the model
 
-// Score bypasses the slot loop (see the comment on Score below) so it
-// must not run concurrently with any slot-loop RPC. These counters
-// are a defence-in-depth tripwire — ModelConfig.Validate already
-// rejects llama-cpp configs that mix score with chat/completion/
-// embeddings, so a healthy deployment never trips them. seq_cst is
-// load-bearing for the increment-then-check pattern below.
-static std::atomic<int> slot_loop_inflight{0};
-static std::atomic<int> score_inflight{0};
-
-// Increment-then-check, not check-then-increment: two simultaneous
-// racers both observe the other's increment and both abort cleanly.
-// Reversed, both could see zero and proceed.
-struct conflict_guard {
-    std::atomic<int>& self;
-    conflict_guard(const char* rpc, std::atomic<int>& self_, std::atomic<int>& other, const char* other_name)
-        : self(self_) {
-        self.fetch_add(1, std::memory_order_seq_cst);
-        int o = other.load(std::memory_order_seq_cst);
-        if (o > 0) {
-            fprintf(stderr,
-                "FATAL: %s called with %s=%d. The llama-cpp backend cannot "
-                "service Score and slot-loop RPCs concurrently — Score "
-                "bypasses the slot loop and races the llama_context. Bind "
-                "Score-using features to a model dedicated to scoring "
-                "(known_usecases: [score] with no chat/completion/embeddings).\n",
-                rpc, other_name, o);
-            std::abort();
-        }
-    }
-    ~conflict_guard() {
-        self.fetch_sub(1, std::memory_order_seq_cst);
-    }
-};
-
 static std::function<void(int)> shutdown_handler;
 static std::atomic_flag is_terminating = ATOMIC_FLAG_INIT;
 
@@ -613,6 +580,13 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
     // starts with '-'. Applied once after the loop via common_params_parse.
     std::vector<std::string> extra_argv;
 
+    // O_DIRECT intent from the `direct_io` option. Upstream folded
+    // use_mmap/use_mlock/use_direct_io into a single common_params::load_mode
+    // enum (ggml-org/llama.cpp#20834), so the three independent LocalAI settings
+    // can only be reduced to one value once all of them have been read, held
+    // here until the mmap/mlock fields arrive further down.
+    bool want_direct_io = false;
+
     auto add_device_options = [&](const std::string & devices) {
         const std::regex regex{ R"([,]+)" };
         std::sregex_token_iterator it{ devices.begin(), devices.end(), regex, -1 };
@@ -724,6 +698,22 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                     // If conversion fails, keep default value (0)
                 }
             }
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        } else if (!strcmp(optname, "n_rs_seq") || !strcmp(optname, "rs_seq")) {
+            // Recurrent-state rollback snapshots per sequence. Hybrid models
+            // (deltanet/conv layers) cannot rewind their state, so without
+            // snapshots any prompt-cache reuse that needs a rewind — e.g. a
+            // score task whose probe changed under a stable option-list
+            // prefix — falls back to a full re-prefill. Costs recurrent-state
+            // memory x (1 + N) per sequence; unsupported archs clamp to 0.
+            if (optval != NULL) {
+                try {
+                    params.n_rs_seq = std::stoi(optval_str);
+                } catch (const std::exception& e) {
+                    // If conversion fails, keep default value (0)
+                }
+            }
+#endif
         } else if (!strcmp(optname, "slot_prompt_similarity") || !strcmp(optname, "sps")) {
             if (optval != NULL) {
                 try {
@@ -868,9 +858,9 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         // --- O_DIRECT model loading (upstream --direct-io) ---
         } else if (!strcmp(optname, "direct_io") || !strcmp(optname, "use_direct_io")) {
             if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
-                params.use_direct_io = true;
+                want_direct_io = true;
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
-                params.use_direct_io = false;
+                want_direct_io = false;
             }
 
         // --- embedding normalization (upstream --embd-normalize) ---
@@ -1278,8 +1268,28 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
      lora_info.ptr = nullptr;
      params.lora_adapters.push_back(std::move(lora_info));
     }
-    params.use_mlock = request->mlock();
-    params.use_mmap = request->mmap();
+    // LocalAI keeps mmap, mlock and direct-I/O as three independent settings,
+    // while upstream now carries a single load mode. Fold them with the
+    // precedence the separate booleans used to give: direct I/O bypasses the
+    // page cache entirely, mlock implies mmap, and everything off is a plain
+    // buffered read. Forks that branched before ggml-org/llama.cpp#20834 still
+    // expose the booleans; prepare.sh probes the checkout and sets
+    // LOCALAI_LEGACY_LOAD_MODE in the generated llama_compat.h accordingly.
+#if LOCALAI_LEGACY_LOAD_MODE
+    params.use_mlock     = request->mlock();
+    params.use_mmap      = request->mmap();
+    params.use_direct_io = want_direct_io;
+#else
+    if (want_direct_io) {
+        params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO;
+    } else if (request->mlock()) {
+        params.load_mode = LLAMA_LOAD_MODE_MLOCK;
+    } else if (request->mmap()) {
+        params.load_mode = LLAMA_LOAD_MODE_MMAP;
+    } else {
+        params.load_mode = LLAMA_LOAD_MODE_NONE;
+    }
+#endif
 
     if (request->flashattention() == "on" || request->flashattention() == "enabled") {
         params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;
@@ -1364,6 +1374,17 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         }
     }
 
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+    // Score-task suffix forking: reserve seq ids (and recurrent-state cells)
+    // beyond the slots so one scoring call decodes all candidate tails in a
+    // single batch (SERVER_TASK_TYPE_SCORE, patches/). Requires the unified
+    // KV cache — with per-sequence streams the extra ids would shrink every
+    // sequence's context to n_ctx / n_seq_max. Decided after both option
+    // passes so an explicit kv_unified:false wins and disables forking.
+    params.score_enabled = request->enablescore();
+    params.n_seq_score_forks = params.score_enabled && params.kv_unified ? SERVER_SCORE_FORK_SEQS : 0;
+#endif
+
     // Terminate/pad the override vectors only after BOTH the named-option loop
     // and the generic passthrough (common_params_parse above) have pushed their
     // real entries, so back() is the null sentinel the model loader asserts on.
@@ -1450,6 +1471,16 @@ public:
         common_params params;
         params_parse(ctx_server, request, params);
 
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        if (params.score_enabled && !params.kv_unified) {
+            const std::string error_msg =
+                "Score requires the unified KV cache; remove kv_unified:false or remove score from known_usecases";
+            result->set_message(error_msg);
+            result->set_success(false);
+            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, error_msg);
+        }
+#endif
+
         common_init();
         // Ensure debug logs are enabled after common_init() sets up logging
         common_log_set_verbosity_thold(params.verbosity);
@@ -1652,7 +1683,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("PredictStream", slot_loop_inflight, score_inflight, "score_inflight");
         json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
 
@@ -2221,7 +2251,6 @@ public:
          if (params_base.model.path.empty()) {
              return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
          }
-         conflict_guard guard("Predict", slot_loop_inflight, score_inflight, "score_inflight");
          json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
         data["stream"] = false;
@@ -2755,7 +2784,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("Embedding", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
 
         body["stream"] = false;
@@ -2865,7 +2893,6 @@ public:
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "\"documents\" must be a non-empty string array");
         }
 
-        conflict_guard guard("Rerank", slot_loop_inflight, score_inflight, "score_inflight");
 
         // Create and queue the task
         auto rd = ctx_server.get_response_reader();
@@ -2942,37 +2969,16 @@ public:
     // Score returns the model's joint log-probability of each candidate
     // continuation given a shared prompt.
     //
-    // WHY bypass the slot/task queue: upstream server_context exposes
-    // get_llama_context as "main thread only" and the slot loop's
-    // update_slots() owns the context whenever a task is in flight.
-    // No public synchronization primitive is available — so Score is
-    // unsafe to call concurrently with active generation through this
-    // backend. In practice routing-classifier calls happen before the
-    // request is routed to a generation backend, so the model used
-    // for Score is typically idle. Concurrent Score calls are
-    // serialised by a local mutex; KV-cache state is isolated behind
-    // a dedicated sequence ID cleared between candidates.
-    //
-    // A patch to server-context.cpp that adds SERVER_TASK_TYPE_SCORE
-    // and routes scoring through the slot loop would be the correct
-    // long-term fix; tracked as a follow-up.
-    //
-    // Perf TODO (measured: ~450 ms warm for 3 candidates on Arch-
-    // Router-1.5B Q4_K_M + Intel SYCL): the current loop re-decodes
-    // `prompt + candidate` from scratch for every candidate, throwing
-    // away the prompt's KV cache between iterations. A smarter
-    // version would:
-    //   1. Decode just the prompt once into score_seq_id.
-    //   2. Snapshot/cp that sequence (llama_memory_seq_cp) into a
-    //      per-candidate sequence id.
-    //   3. For each candidate, decode only its tokens onto the copy
-    //      (continuing from the saved prompt state), read logits.
-    //   4. llama_memory_seq_rm the copy.
-    // Estimated speedup: 3-candidate calls 450 ms -> ~150-200 ms,
-    // 6-candidate calls 630 ms -> ~220 ms. Single source-file change,
-    // no proto / Go-side changes needed. Worth doing once routing is
-    // wired into the middleware and Score is on the hot path of every
-    // chat request.
+    // Scoring runs as a single SERVER_TASK_TYPE_SCORE task through the
+    // slot loop (added by patches/ on top of upstream server-context), so
+    // it is safe to interleave with generation on the same process and it
+    // reuses any KV prefix the slot already holds across turns. The task
+    // decodes the shared prefix (prompt + longest common candidate token
+    // prefix) once on the slot's sequence; every candidate's unique tail
+    // then rides its own forked sequence and all tails are decoded
+    // together in one batch, so a warm scoring call costs roughly one
+    // forward pass over the new prompt tokens plus one batched pass over
+    // the candidate tails.
     grpc::Status Score(ServerContext* context, const backend::ScoreRequest* request, backend::ScoreResponse* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
@@ -2981,40 +2987,21 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
+#ifdef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        (void) request;
+        (void) response;
+        return grpc::Status(grpc::StatusCode::UNIMPLEMENTED,
+            "Score is unavailable in this llama.cpp fork backend");
+#else
+        if (!params_base.score_enabled) {
+            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION,
+                "Score was not enabled when the model was loaded; add score to known_usecases");
+        }
         if (request->candidates_size() == 0) {
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "candidates must be non-empty");
         }
 
-        // Tripwire against the slot loop. Acquired before score_mutex
-        // so it fires even when this Score is queued behind another.
-        conflict_guard guard("Score", score_inflight, slot_loop_inflight, "slot_loop_inflight");
-
-        // Serialise concurrent Score calls. The slot loop is still
-        // free to race with us — see the class comment above.
-        static std::mutex score_mutex;
-        std::lock_guard<std::mutex> score_lock(score_mutex);
-
-        llama_context * lctx = ctx_server.get_llama_context();
-        if (lctx == nullptr) {
-            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "llama context unavailable (sleeping?)");
-        }
         const llama_vocab * vocab = ctx_server.impl->vocab;
-        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
-        const int32_t n_ctx = llama_n_ctx(lctx);
-        llama_memory_t mem = llama_get_memory(lctx);
-
-        // The KV-cache is sized to seq_to_stream.size() at load
-        // (typically equal to n_slots, often 1). Sequence IDs must
-        // be in [0, n_seq_max), so we can't pick a high-value
-        // "private" ID — we have to share with the slot. We clear
-        // the cache before AND after each candidate to keep
-        // scoring isolated from whatever state the slot held, and
-        // the static mutex above guarantees no other Score call is
-        // racing in the meantime. The slot loop is still free to
-        // race (see comment on this method) — Score must not run
-        // concurrently with generation through this backend.
-        const llama_seq_id score_seq_id = 0;
-        llama_memory_seq_rm(mem, score_seq_id, -1, -1);
 
         // Tokenize the shared prompt once with add_special=true so
         // BOS is prepended when the model requires it. parse_special
@@ -3023,6 +3010,15 @@ public:
         std::vector<llama_token> prompt_tokens = common_tokenize(vocab, prompt, /*add_special=*/true, /*parse_special=*/true);
         const int32_t prompt_len = (int32_t) prompt_tokens.size();
 
+        // Per candidate: full prompt+candidate token list and the
+        // divergence point, kept for piece rendering and empty-candidate
+        // handling after the task comes back.
+        std::vector<std::vector<llama_token>> cand_tokens(request->candidates_size());
+        std::vector<int32_t> cand_divergence(request->candidates_size(), 0);
+
+        // candidates that actually have tokens to score
+        std::vector<int32_t> included;
+
         for (int ci = 0; ci < request->candidates_size(); ci++) {
             const std::string & candidate_text = request->candidates(ci);
 
@@ -3039,9 +3035,135 @@ public:
                     break;
                 }
             }
+            divergence = std::min<int32_t>(divergence, (int32_t) full_tokens.size());
+
             const int32_t cand_len = (int32_t) full_tokens.size() - divergence;
+            if (cand_len > 0 && divergence < 1) {
+                // Need at least one prior token (typically BOS) to
+                // predict the first candidate token's logit. Tokeniser
+                // models without BOS + an empty prompt fall in here.
+                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
+            }
+            if (cand_len > SERVER_SCORE_MAX_CAND_TOKENS) {
+                // The context reserves logits outputs for at most this many
+                // candidate tokens per slot (server_n_outputs_max).
+                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                    "Score: candidate " + std::to_string(ci) + " is " + std::to_string(cand_len) +
+                    " tokens; the maximum is " + std::to_string(SERVER_SCORE_MAX_CAND_TOKENS));
+            }
+
+            cand_divergence[ci] = divergence;
+            cand_tokens[ci] = std::move(full_tokens);
+
+            if (cand_len > 0) {
+                included.push_back(ci);
+            }
+        }
+
+        auto rd = ctx_server.get_response_reader();
+        bool posted_task = false;
+
+        // Shared prefix bounds, needed again when stitching the results:
+        // n_shared is the longest common token prefix of the scored
+        // candidates, n_score_prompt the earliest divergence from the
+        // bare prompt (scored logprobs start there).
+        int32_t n_shared       = 0;
+        int32_t n_score_prompt = 0;
+
+        if (!included.empty()) {
+            const auto & first = cand_tokens[included[0]];
+
+            // the common prefix of a set is the shortest common prefix
+            // against any fixed member
+            n_shared = (int32_t) first.size();
+            for (int32_t ci : included) {
+                const auto & ft = cand_tokens[ci];
+                const int32_t lim = std::min<int32_t>(n_shared, (int32_t) ft.size());
+                int32_t match = 0;
+                while (match < lim && ft[match] == first[match]) {
+                    match++;
+                }
+                n_shared = match;
+            }
+
+            // below its divergence every candidate equals the prompt
+            // tokens, so n_score_prompt <= n_shared always holds
+            n_score_prompt = cand_divergence[included[0]];
+            for (int32_t ci : included) {
+                n_score_prompt = std::min(n_score_prompt, cand_divergence[ci]);
+            }
+
+            // Map the caller's stable-prefix byte length onto a token
+            // index: the last prompt token that ends at or before the
+            // boundary. A checkpoint forced there survives every future
+            // probe under the same option list, which is what keeps
+            // repeat scoring cheap on models that cannot rewind state.
+            int32_t n_stable_prompt = 0;
+            if (request->stable_prefix_len() > 0) {
+                size_t consumed = 0;
+                for (int32_t ti = 0; ti < n_score_prompt; ti++) {
+                    const size_t piece_len = common_token_to_piece(vocab, prompt_tokens[ti]).size();
+                    // BOS and other zero-length specials consume no prompt bytes
+                    if (consumed + piece_len > (size_t) request->stable_prefix_len()) {
+                        break;
+                    }
+                    consumed += piece_len;
+                    n_stable_prompt = ti + 1;
+                }
+            }
+
+            server_task task(SERVER_TASK_TYPE_SCORE);
+            task.id             = rd.queue_tasks.get_new_id();
+            task.index          = 0;
+            task.tokens         = server_tokens(llama_tokens(first.begin(), first.begin() + n_shared), false);
+            task.n_score_prompt = n_score_prompt;
+            task.n_stable_prompt = n_stable_prompt;
+            task.score_suffixes.reserve(included.size());
+            for (int32_t ci : included) {
+                task.score_suffixes.emplace_back(cand_tokens[ci].begin() + n_shared, cand_tokens[ci].end());
+            }
+
+            std::vector<server_task> tasks;
+            tasks.push_back(std::move(task));
+            rd.post_tasks(std::move(tasks));
+            posted_task = true;
+        }
+
+        // Wait for the shared-prefix and per-candidate logprob vectors.
+        // Context overflow and decode failures surface here as task errors.
+        std::vector<float> shared_logprobs;
+        std::vector<std::vector<float>> cand_logprobs;
+        if (posted_task) {
+            auto all_results = rd.wait_for_all([&context]() { return context->IsCancelled(); });
+            if (all_results.is_terminated) {
+                return grpc::Status(grpc::StatusCode::CANCELLED, "Request cancelled by client");
+            }
+            if (all_results.error) {
+                return grpc::Status(grpc::StatusCode::INTERNAL,
+                    all_results.error->to_json().value("message", "Error in receiving score results"));
+            }
+            if (all_results.results.size() != 1) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "expected a single score result");
+            }
+            auto * score_res = dynamic_cast<server_task_result_score*>(all_results.results[0].get());
+            if (score_res == nullptr) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "unexpected result type for score task");
+            }
+            shared_logprobs = std::move(score_res->shared_logprobs);
+            cand_logprobs   = std::move(score_res->cand_logprobs);
+            if (cand_logprobs.size() != included.size()) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "score result candidate count mismatch");
+            }
+        }
+
+        size_t inc = 0; // index into included / cand_logprobs
+        for (int ci = 0; ci < request->candidates_size(); ci++) {
+            const int32_t divergence = cand_divergence[ci];
+            const int32_t cand_len   = (int32_t) cand_tokens[ci].size() - divergence;
+
             backend::CandidateScore * cs = response->add_candidates();
-            cs->set_num_tokens(cand_len);
+            cs->set_num_tokens(cand_len > 0 ? cand_len : 0);
             if (cand_len <= 0) {
                 cs->set_log_prob(0.0);
                 if (request->length_normalize()) {
@@ -3049,101 +3171,57 @@ public:
                 }
                 continue;
             }
-            if (divergence < 1) {
-                // Need at least one prior token (typically BOS) to
-                // predict the first candidate token's logit. Tokeniser
-                // models without BOS + an empty prompt fall in here.
-                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
-                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
+
+            // Stitch the candidate's scored logprobs back together: the
+            // stretch inside the shared prefix (identical for every
+            // candidate) followed by its forked suffix. Suffix entries
+            // before the candidate's own divergence are prompt tokens
+            // decoded only as context — not scored.
+            std::vector<float> lp;
+            lp.reserve(cand_len);
+            for (int32_t t = divergence; t < n_shared; t++) {
+                const int32_t idx = t - n_score_prompt;
+                if (idx < 0 || idx >= (int32_t) shared_logprobs.size()) {
+                    return grpc::Status(grpc::StatusCode::INTERNAL,
+                        "Score: shared logprob index out of range for candidate " + std::to_string(ci));
+                }
+                lp.push_back(shared_logprobs[idx]);
             }
-            if ((int32_t) full_tokens.size() > n_ctx) {
-                return grpc::Status(grpc::StatusCode::OUT_OF_RANGE,
-                    "Score: prompt+candidate exceeds context size (got " +
-                    std::to_string(full_tokens.size()) + ", n_ctx=" + std::to_string(n_ctx) + ")");
+            const auto & sfx_lp = cand_logprobs[inc++];
+            for (int32_t j = std::max(0, divergence - n_shared); j < (int32_t) sfx_lp.size(); j++) {
+                lp.push_back(sfx_lp[j]);
             }
 
-            // Build a batch covering the entire prompt+candidate. We
-            // need logits at (divergence-1) onward — those are the
-            // predictions for each candidate token.
-            llama_batch batch = llama_batch_init((int32_t) full_tokens.size(), 0, 1);
-            for (int32_t i = 0; i < (int32_t) full_tokens.size(); i++) {
-                batch.token[i]    = full_tokens[i];
-                batch.pos[i]      = i;
-                batch.n_seq_id[i] = 1;
-                batch.seq_id[i][0] = score_seq_id;
-                // logits[i] is "do we want the prediction *for the
-                // next token*, computed from this position?"
-                // We want predictions for candidate tokens at
-                // positions divergence .. full_tokens.size()-1, which
-                // come from logits at positions (divergence-1) ..
-                // (full_tokens.size()-2).
-                bool need_logit = (i >= divergence - 1) && (i < (int32_t) full_tokens.size() - 1);
-                batch.logits[i] = need_logit ? 1 : 0;
-            }
-            batch.n_tokens = (int32_t) full_tokens.size();
-
-            // Decode the batch. If decode fails (e.g. KV slot
-            // exhaustion), surface as INTERNAL — the caller will
-            // typically fall back to a sampling-based classifier.
-            int decode_err = llama_decode(lctx, batch);
-            if (decode_err != 0) {
-                llama_batch_free(batch);
-                llama_memory_seq_rm(mem, score_seq_id, -1, -1);
+            if ((int32_t) lp.size() != cand_len) {
                 return grpc::Status(grpc::StatusCode::INTERNAL,
-                    "llama_decode failed during Score: " + std::to_string(decode_err));
+                    "Score: result for candidate " + std::to_string(ci) + " is missing token logprobs");
             }
 
-            // Sum log-probabilities of the actual candidate tokens.
             double total_log_prob = 0.0;
             for (int32_t k = 0; k < cand_len; k++) {
-                // The k-th candidate token sits at full_tokens index
-                // (divergence + k). Its predicting logit is at batch
-                // position (divergence + k - 1).
-                int32_t logit_pos = divergence + k - 1;
-                const float * logits = llama_get_logits_ith(lctx, logit_pos);
-                if (logits == nullptr) {
-                    llama_batch_free(batch);
-                    llama_memory_seq_rm(mem, score_seq_id, -1, -1);
+                const float token_log_prob = lp[k];
+                if (std::isnan(token_log_prob)) {
                     return grpc::Status(grpc::StatusCode::INTERNAL,
-                        "llama_get_logits_ith returned null at position " + std::to_string(logit_pos));
+                        "Score: incomplete result for candidate " + std::to_string(ci) +
+                        " at token " + std::to_string(k));
                 }
-                llama_token target_token = full_tokens[divergence + k];
-
-                // Compute log_softmax(logits)[target_token] with the
-                // max-subtraction stability trick.
-                float max_logit = logits[0];
-                for (int32_t v = 1; v < n_vocab; v++) {
-                    if (logits[v] > max_logit) max_logit = logits[v];
-                }
-                double sum_exp = 0.0;
-                for (int32_t v = 0; v < n_vocab; v++) {
-                    sum_exp += std::exp((double)(logits[v] - max_logit));
-                }
-                double token_log_prob = (double)(logits[target_token] - max_logit) - std::log(sum_exp);
-                total_log_prob += token_log_prob;
+                total_log_prob += (double) token_log_prob;
 
                 if (request->include_token_logprobs()) {
                     backend::TokenLogProb * tlp = cs->add_tokens();
-                    std::string piece = common_token_to_piece(lctx, target_token);
-                    tlp->set_token(piece);
+                    tlp->set_token(common_token_to_piece(vocab, cand_tokens[ci][divergence + k]));
                     tlp->set_log_prob(token_log_prob);
                 }
             }
 
             cs->set_log_prob(total_log_prob);
-            if (request->length_normalize() && cand_len > 0) {
+            if (request->length_normalize()) {
                 cs->set_length_normalized_log_prob(total_log_prob / (double) cand_len);
             }
-
-            llama_batch_free(batch);
-            // Drop this candidate's KV-cache contribution so the next
-            // candidate starts from a clean state. Without this, the
-            // next decode would conflict at positions 0..N-1 for our
-            // sequence ID.
-            llama_memory_seq_rm(mem, score_seq_id, -1, -1);
         }
 
         return grpc::Status::OK;
+#endif
     }
 
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
@@ -3154,7 +3232,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("TokenizeString", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
         body["stream"] = false;
 
@@ -3176,7 +3253,6 @@ public:
 
     grpc::Status GetMetrics(ServerContext* /*context*/, const backend::MetricsRequest* /*request*/, backend::MetricsResponse* response) override {
 
-        conflict_guard guard("GetMetrics", slot_loop_inflight, score_inflight, "score_inflight");
 
 // request slots data using task queue
         auto rd = ctx_server.get_response_reader();

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-chat-parser.patch

@@ -0,0 +1,225 @@
+# MiniMax-M3 chat-template parser, vendored from upstream llama.cpp PR #24523.
+#
+# Upstream has since merged the *model* half of #24523 (LLM_ARCH_MINIMAX_M3,
+# src/models/minimax-m3.cpp, the gguf-py constants and conversion/minimax.py), so
+# only the chat half is carried here: M3's namespace token "]<]minimax[>[" collides
+# with the autoparser's markup delimiters, so common/chat.cpp needs a dedicated
+# template detection + PEG parser that upstream does not have yet.
+#
+# Rebased against LLAMA_VERSION 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3, which also
+# renamed common_chat_params::thinking_end_tag to thinking_end_tags (a vector).
+# LLAMA_VERSION is auto-bumped nightly; if a bump rejects this patch, re-vendor from
+# #24523 — or, once the chat half merges upstream, delete this file.
+# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
+diff --git a/common/chat.cpp b/common/chat.cpp
+index 7a6e7238c..2dd015a2e 100644
+--- a/common/chat.cpp
++++ b/common/chat.cpp
+@@ -2121,6 +2121,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
+     return data;
+ }
+ 
++static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
++                                                             const autoparser::generation_params & inputs) {
++    common_chat_params data;
++
++    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
++    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
++    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
++    data.supports_thinking  = true;
++    data.thinking_start_tag = "<mm:think>";
++    data.thinking_end_tags  = {"</mm:think>"};
++
++    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
++    // params use the parameter name as the tag (<file_path>...</file_path>).
++    const std::string NS           = "]<]minimax[>[";
++    const std::string THINK_START  = "<mm:think>";
++    const std::string THINK_END    = "</mm:think>";
++    const std::string FC_START     = NS + "<tool_call>";
++    const std::string FC_END       = NS + "</tool_call>";
++    const std::string INVOKE_END   = NS + "</invoke>";
++
++    data.preserved_tokens   = {
++        NS,
++        "<tool_call>",
++        "</tool_call>",
++        THINK_START,
++        THINK_END,
++    };
++
++    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
++    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
++    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
++    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
++
++    const std::string GEN_PROMPT = data.generation_prompt;
++
++    if (inputs.has_continuation()) {
++        const auto & msg = inputs.continue_msg;
++
++        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
++        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
++            data.generation_prompt += THINK_END + msg.render_content();
++        }
++
++        data.prompt += data.generation_prompt;
++    }
++
++    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
++        auto generation_prompt = p.literal(GEN_PROMPT);
++        auto end = p.end();
++
++        auto reasoning = p.eps();
++        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
++        if (extract_reasoning && inputs.enable_thinking) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
++        } else if (extract_reasoning) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
++        }
++
++        if (has_response_format) {
++            auto response_format = p.rule("response-format",
++                p.literal("```json") + p.space() +
++                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
++                p.space() + p.literal("```"));
++            return generation_prompt + reasoning + response_format + end;
++        }
++
++        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
++            return generation_prompt + reasoning + p.content(p.rest()) + end;
++        }
++
++        auto tool_choice = p.choice();
++        foreach_function(inputs.tools, [&](const json & tool) {
++            const auto & function = tool.at("function");
++            std::string  name     = function.at("name");
++            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
++            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
++
++            std::set<std::string> required;
++            if (params.contains("required")) {
++                params.at("required").get_to(required);
++            }
++
++            auto schema_info = common_schema_info();
++            schema_info.resolve_refs(params);
++
++            std::vector<common_peg_parser> required_parsers;
++            std::vector<common_peg_parser> optional_parsers;
++            for (const auto & [param_name, param_schema] : props.items()) {
++                bool is_required = required.find(param_name) != required.end();
++                bool is_string   = schema_info.resolves_to_string(param_schema);
++
++                const std::string p_close = NS + "</" + param_name + ">";
++
++                auto arg = p.tool_arg(
++                    p.tool_arg_open(
++                        p.literal(NS + "<") +
++                        p.tool_arg_name(p.literal(param_name)) +
++                        p.literal(">")) +
++                    (is_string
++                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
++                                p.tool_arg_close(p.literal(p_close)), p_close)
++                         : p.tool_arg_json_value(p.schema(p.json(),
++                                                          "tool-" + name + "-arg-" + param_name + "-schema",
++                                                          param_schema, false)) +
++                           p.tool_arg_close(p.literal(p_close))));
++
++                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
++                if (is_required) {
++                    required_parsers.push_back(named_arg);
++                } else {
++                    optional_parsers.push_back(named_arg);
++                }
++            }
++
++            common_peg_parser args_seq = p.eps();
++            for (size_t i = 0; i < required_parsers.size(); i++) {
++                if (i > 0) {
++                    args_seq = args_seq + p.space();
++                }
++                args_seq = args_seq + required_parsers[i];
++            }
++
++            if (!optional_parsers.empty()) {
++                common_peg_parser any_opt = p.choice();
++                for (const auto & opt : optional_parsers) {
++                    any_opt |= opt;
++                }
++                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
++            }
++
++            common_peg_parser invoke_body = args_seq;
++            auto func_parser = p.tool(
++                p.tool_open(p.literal(NS + "<invoke name=\"") +
++                            p.tool_name(p.literal(name)) + p.literal("\">")) +
++                p.space() + invoke_body + p.space() +
++                p.tool_close(p.literal(INVOKE_END)));
++
++            tool_choice |= p.rule("tool-" + name, func_parser);
++        });
++
++        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
++
++        common_peg_parser tool_calls = p.eps();
++        if (inputs.parallel_tool_calls) {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice +
++                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
++        } else {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
++        }
++
++        if (!require_tools) {
++            tool_calls = p.optional(tool_calls);
++        }
++
++        auto content_before_tools = p.content(p.until(FC_START));
++        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
++    });
++
++    data.parser = parser.save();
++
++    if (include_grammar) {
++        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
++        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
++            foreach_function(inputs.tools, [&](const json & tool) {
++                const auto & function = tool.at("function");
++                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
++                builder.resolve_refs(schema);
++            });
++            if (has_response_format) {
++                auto schema = inputs.json_schema;
++                builder.resolve_refs(schema);
++            }
++            parser.build_grammar(builder, data.grammar_lazy);
++        });
++
++        data.grammar_triggers = {
++            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
++        };
++    }
++
++    return data;
++}
++
+ // Cohere2 MoE (a.k.a. "North Code") parser.
+ //
+ // The assistant turn is fully marker-wrapped:
+@@ -2707,6 +2892,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
+         return common_chat_params_init_gigachat_v3(tmpl, params);
+     }
+ 
++    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
++    // markup delimiters, so detect the template and use a dedicated parser.
++    if (src.find("]<]minimax[>[") != std::string::npos &&
++        src.find("<tool_call>") != std::string::npos &&
++        src.find("<invoke name=") != std::string::npos) {
++        LOG_DBG("Using specialized template: MiniMax-M3\n");
++        return common_chat_params_init_minimax_m3(tmpl, params);
++    }
++
+     // DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
+     // The template source contains the token as a variable assignment, not as a literal in markup.
+     // V3.2 names the tool call block "function_calls", V4 names it "tool_calls".

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch

@@ -1,814 +0,0 @@
-# Vendored from upstream llama.cpp PR #24523 (Preliminary MiniMax-M3 support).
-# Rebased against LLAMA_VERSION 00fa7cb284cbf133fc426733bd64238a3588a33e (also applies cleanly
-# to the later pin 505b1ed15ca80e2a19f12ff4ac365e40fb374053). LLAMA_VERSION is auto-bumped
-# nightly; if a bump rejects this patch, re-vendor from #24523 — or, once #24523 merges
-# upstream, delete this file and bump LLAMA_VERSION normally.
-# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
-diff --git a/common/chat.cpp b/common/chat.cpp
-index 22d2ee4..440be9a 100644
---- a/common/chat.cpp
-+++ b/common/chat.cpp
-@@ -2035,6 +2035,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
-     return data;
- }
- 
-+static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
-+                                                             const autoparser::generation_params & inputs) {
-+    common_chat_params data;
-+
-+    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
-+    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
-+    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
-+    data.supports_thinking  = true;
-+    data.thinking_start_tag = "<mm:think>";
-+    data.thinking_end_tag   = "</mm:think>";
-+
-+    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
-+    // params use the parameter name as the tag (<file_path>...</file_path>).
-+    const std::string NS           = "]<]minimax[>[";
-+    const std::string THINK_START  = "<mm:think>";
-+    const std::string THINK_END    = "</mm:think>";
-+    const std::string FC_START     = NS + "<tool_call>";
-+    const std::string FC_END       = NS + "</tool_call>";
-+    const std::string INVOKE_END   = NS + "</invoke>";
-+
-+    data.preserved_tokens   = {
-+        NS,
-+        "<tool_call>",
-+        "</tool_call>",
-+        THINK_START,
-+        THINK_END,
-+    };
-+
-+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
-+    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
-+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
-+
-+    const std::string GEN_PROMPT = data.generation_prompt;
-+
-+    if (inputs.has_continuation()) {
-+        const auto & msg = inputs.continue_msg;
-+
-+        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
-+        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
-+            data.generation_prompt += THINK_END + msg.render_content();
-+        }
-+
-+        data.prompt += data.generation_prompt;
-+    }
-+
-+    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
-+        auto generation_prompt = p.literal(GEN_PROMPT);
-+        auto end = p.end();
-+
-+        auto reasoning = p.eps();
-+        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
-+        if (extract_reasoning && inputs.enable_thinking) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
-+        } else if (extract_reasoning) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
-+        }
-+
-+        if (has_response_format) {
-+            auto response_format = p.rule("response-format",
-+                p.literal("```json") + p.space() +
-+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
-+                p.space() + p.literal("```"));
-+            return generation_prompt + reasoning + response_format + end;
-+        }
-+
-+        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
-+            return generation_prompt + reasoning + p.content(p.rest()) + end;
-+        }
-+
-+        auto tool_choice = p.choice();
-+        foreach_function(inputs.tools, [&](const json & tool) {
-+            const auto & function = tool.at("function");
-+            std::string  name     = function.at("name");
-+            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
-+            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
-+
-+            std::set<std::string> required;
-+            if (params.contains("required")) {
-+                params.at("required").get_to(required);
-+            }
-+
-+            auto schema_info = common_schema_info();
-+            schema_info.resolve_refs(params);
-+
-+            std::vector<common_peg_parser> required_parsers;
-+            std::vector<common_peg_parser> optional_parsers;
-+            for (const auto & [param_name, param_schema] : props.items()) {
-+                bool is_required = required.find(param_name) != required.end();
-+                bool is_string   = schema_info.resolves_to_string(param_schema);
-+
-+                const std::string p_close = NS + "</" + param_name + ">";
-+
-+                auto arg = p.tool_arg(
-+                    p.tool_arg_open(
-+                        p.literal(NS + "<") +
-+                        p.tool_arg_name(p.literal(param_name)) +
-+                        p.literal(">")) +
-+                    (is_string
-+                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
-+                                p.tool_arg_close(p.literal(p_close)), p_close)
-+                         : p.tool_arg_json_value(p.schema(p.json(),
-+                                                          "tool-" + name + "-arg-" + param_name + "-schema",
-+                                                          param_schema, false)) +
-+                           p.tool_arg_close(p.literal(p_close))));
-+
-+                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
-+                if (is_required) {
-+                    required_parsers.push_back(named_arg);
-+                } else {
-+                    optional_parsers.push_back(named_arg);
-+                }
-+            }
-+
-+            common_peg_parser args_seq = p.eps();
-+            for (size_t i = 0; i < required_parsers.size(); i++) {
-+                if (i > 0) {
-+                    args_seq = args_seq + p.space();
-+                }
-+                args_seq = args_seq + required_parsers[i];
-+            }
-+
-+            if (!optional_parsers.empty()) {
-+                common_peg_parser any_opt = p.choice();
-+                for (const auto & opt : optional_parsers) {
-+                    any_opt |= opt;
-+                }
-+                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
-+            }
-+
-+            common_peg_parser invoke_body = args_seq;
-+            auto func_parser = p.tool(
-+                p.tool_open(p.literal(NS + "<invoke name=\"") +
-+                            p.tool_name(p.literal(name)) + p.literal("\">")) +
-+                p.space() + invoke_body + p.space() +
-+                p.tool_close(p.literal(INVOKE_END)));
-+
-+            tool_choice |= p.rule("tool-" + name, func_parser);
-+        });
-+
-+        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
-+
-+        common_peg_parser tool_calls = p.eps();
-+        if (inputs.parallel_tool_calls) {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice +
-+                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
-+        } else {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
-+        }
-+
-+        if (!require_tools) {
-+            tool_calls = p.optional(tool_calls);
-+        }
-+
-+        auto content_before_tools = p.content(p.until(FC_START));
-+        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
-+    });
-+
-+    data.parser = parser.save();
-+
-+    if (include_grammar) {
-+        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
-+        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
-+            foreach_function(inputs.tools, [&](const json & tool) {
-+                const auto & function = tool.at("function");
-+                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
-+                builder.resolve_refs(schema);
-+            });
-+            if (has_response_format) {
-+                auto schema = inputs.json_schema;
-+                builder.resolve_refs(schema);
-+            }
-+            parser.build_grammar(builder, data.grammar_lazy);
-+        });
-+
-+        data.grammar_triggers = {
-+            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
-+        };
-+    }
-+
-+    return data;
-+}
-+
- // Cohere2 MoE (a.k.a. "North Code") parser.
- //
- // The assistant turn is fully marker-wrapped:
-@@ -2612,6 +2797,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
-         return common_chat_params_init_gigachat_v3(tmpl, params);
-     }
- 
-+    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
-+    // markup delimiters, so detect the template and use a dedicated parser.
-+    if (src.find("]<]minimax[>[") != std::string::npos &&
-+        src.find("<tool_call>") != std::string::npos &&
-+        src.find("<invoke name=") != std::string::npos) {
-+        LOG_DBG("Using specialized template: MiniMax-M3\n");
-+        return common_chat_params_init_minimax_m3(tmpl, params);
-+    }
-+
-     // DeepSeek V3.2 format detection: template defines dsml_token and uses it for tool calls.
-     // The template source contains the token as a variable assignment, not as a literal in markup.
-     if (src.find("dsml_token") != std::string::npos &&
-diff --git a/conversion/__init__.py b/conversion/__init__.py
-index 02ea638..71de528 100644
---- a/conversion/__init__.py
-+++ b/conversion/__init__.py
-@@ -155,6 +155,8 @@ TEXT_MODEL_MAP: dict[str, str] = {
-     "MiniCPMForCausalLM": "minicpm",
-     "MiniCPMV4_6ForConditionalGeneration": "minicpm",
-     "MiniMaxM2ForCausalLM": "minimax",
-+    "MiniMaxM3SparseForCausalLM": "minimax",
-+    "MiniMaxM3SparseForConditionalGeneration": "minimax",
-     "Ministral3ForCausalLM": "mistral3",
-     "Mistral3ForConditionalGeneration": "mistral3",
-     "MistralForCausalLM": "llama",
-diff --git a/conversion/base.py b/conversion/base.py
-index 0421aa4..224481a 100644
---- a/conversion/base.py
-+++ b/conversion/base.py
-@@ -1154,7 +1154,8 @@ class TextModel(ModelBase):
-                 or "projector." in name or "pre_mm_projector_norm" in name \
-                 or "image_newline" in name or "view_seperator" in name \
-                 or "patch_embed" in name or "patch_embedding" in name \
--                or "patch_merger." in name or "model.connector." in name:
-+                or "patch_merger." in name or "patch_merge_mlp" in name \
-+                or "model.connector." in name:
-             return None
- 
-         return super().filter_tensors(item)
-@@ -1201,7 +1202,7 @@ class TextModel(ModelBase):
-             self.gguf_writer.add_embedding_length(n_embd)
-             logger.info(f"gguf: embedding length = {n_embd}")
- 
--        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-+        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-             self.gguf_writer.add_feed_forward_length(n_ff)
-             logger.info(f"gguf: feed forward length = {n_ff}")
- 
-diff --git a/conversion/minimax.py b/conversion/minimax.py
-index 4857775..4f637f5 100644
---- a/conversion/minimax.py
-+++ b/conversion/minimax.py
-@@ -52,3 +52,67 @@ class MiniMaxM2Model(TextModel):
-             return
- 
-         yield from super().modify_tensors(data_torch, name, bid)
-+
-+
-+@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
-+class MiniMaxM3Model(TextModel):
-+    # Text-only MiniMax-M3: MiniMax-M2 GQA + DeepSeek-V3 shared/leading-dense experts (swigluoai).
-+    model_arch = gguf.MODEL_ARCH.MINIMAXM3
-+    _experts_cache: dict[int, dict[str, Tensor]] = {}
-+
-+    def set_gguf_parameters(self):
-+        # feed_forward_length comes from dense_intermediate_size (base); experts use intermediate_size.
-+        super().set_gguf_parameters()
-+
-+        self.gguf_writer.add_expert_feed_forward_length(self.find_hparam(["intermediate_size"]))
-+        self.gguf_writer.add_rope_dimension_count(self.find_hparam(["rotary_dim"]))
-+        self.gguf_writer.add_expert_shared_count(self.find_hparam(["n_shared_experts"]))
-+        self.gguf_writer.add_expert_weights_scale(self.find_hparam(["routed_scaling_factor"]))
-+        self.gguf_writer.add_expert_weights_norm(True)
-+
-+        # leading dense layers: moe_layer_freq (ints) or mlp_layer_types (Transformers 5.12, strings)
-+        moe_layer_freq = self.find_hparam(["moe_layer_freq", "mlp_layer_types"])
-+        n_dense = 0
-+        for v in moe_layer_freq:
-+            if v == 0 or v == "dense":
-+                n_dense += 1
-+            else:
-+                break
-+        self.gguf_writer.add_leading_dense_block_count(n_dense)
-+
-+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
-+        # index_* (sparse-attn indexer) tensors are preserved but unused; the loader skips them
-+        if name.startswith("language_model."):
-+            name = name[len("language_model."):]
-+
-+        # Gemma-style (1+w) RMSNorm: bake +1 in so llama.cpp can use plain RMSNorm
-+        if name.endswith("norm.weight"):
-+            data_torch = data_torch + 1.0
-+
-+        # merge routed experts (w1/w2/w3); shared_experts.* passes through to *_shexp
-+        if "block_sparse_moe.experts." in name:
-+            n_experts = self.find_hparam(["num_local_experts", "num_experts"])
-+            assert bid is not None
-+
-+            expert_cache = self._experts_cache.setdefault(bid, {})
-+            expert_cache[name] = data_torch
-+            expert_weights = ["w1", "w2", "w3"]
-+
-+            if len(expert_cache) < n_experts * len(expert_weights):
-+                return
-+
-+            for w_name in expert_weights:
-+                datas: list[Tensor] = []
-+                for xid in range(n_experts):
-+                    ename = f"model.layers.{bid}.block_sparse_moe.experts.{xid}.{w_name}.weight"
-+                    datas.append(expert_cache[ename])
-+                    del expert_cache[ename]
-+
-+                data_torch = torch.stack(datas, dim=0)
-+                merged_name = f"model.layers.{bid}.block_sparse_moe.experts.{w_name}.weight"
-+                yield from super().modify_tensors(data_torch, merged_name, bid)
-+
-+            del self._experts_cache[bid]
-+            return
-+
-+        yield from super().modify_tensors(data_torch, name, bid)
-diff --git a/gguf-py/gguf/constants.py b/gguf-py/gguf/constants.py
-index 869e436..760e3dd 100644
---- a/gguf-py/gguf/constants.py
-+++ b/gguf-py/gguf/constants.py
-@@ -525,6 +525,7 @@ class MODEL_ARCH(IntEnum):
-     APERTUS          = auto()
-     COGVLM           = auto()
-     MINIMAXM2        = auto()
-+    MINIMAXM3        = auto()
-     RND1             = auto()
-     PANGU_EMBED      = auto()
-     MISTRAL3         = auto()
-@@ -613,6 +614,10 @@ class MODEL_TENSOR(IntEnum):
-     MOE_LATENT_UP        = auto() # nemotron 3 super
-     ATTN_Q_NORM          = auto()
-     ATTN_K_NORM          = auto()
-+    ATTN_INDEX_Q         = auto() # minimax-m3 sparse-attn indexer (unused)
-+    ATTN_INDEX_K         = auto()
-+    ATTN_INDEX_Q_NORM    = auto()
-+    ATTN_INDEX_K_NORM    = auto()
-     LAYER_OUT_NORM       = auto()
-     LAYER_OUT_SCALE      = auto()
-     PER_LAYER_TOKEN_EMBD = auto() # gemma3n
-@@ -1105,6 +1110,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
-     MODEL_ARCH.GROVEMOE:         "grovemoe",
-     MODEL_ARCH.APERTUS:          "apertus",
-     MODEL_ARCH.MINIMAXM2:        "minimax-m2",
-+    MODEL_ARCH.MINIMAXM3:        "minimax-m3",
-     MODEL_ARCH.COGVLM:           "cogvlm",
-     MODEL_ARCH.RND1:             "rnd1",
-     MODEL_ARCH.PANGU_EMBED:      "pangu-embedded",
-@@ -1163,6 +1169,10 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
-     MODEL_TENSOR.ATTN_GATE:                 "blk.{bid}.attn_gate",
-     MODEL_TENSOR.ATTN_Q_NORM:               "blk.{bid}.attn_q_norm",
-     MODEL_TENSOR.ATTN_K_NORM:               "blk.{bid}.attn_k_norm",
-+    MODEL_TENSOR.ATTN_INDEX_Q:              "blk.{bid}.attn_index_q",
-+    MODEL_TENSOR.ATTN_INDEX_K:              "blk.{bid}.attn_index_k",
-+    MODEL_TENSOR.ATTN_INDEX_Q_NORM:         "blk.{bid}.attn_index_q_norm",
-+    MODEL_TENSOR.ATTN_INDEX_K_NORM:         "blk.{bid}.attn_index_k_norm",
-     MODEL_TENSOR.ATTN_OUT_NORM:             "blk.{bid}.attn_output_norm",
-     MODEL_TENSOR.ATTN_POST_NORM:            "blk.{bid}.post_attention_norm",
-     MODEL_TENSOR.FFN_GATE_INP:              "blk.{bid}.ffn_gate_inp",
-@@ -4102,6 +4112,30 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.FFN_UP_EXP,
-         MODEL_TENSOR.FFN_EXP_PROBS_B,
-     ],
-+    MODEL_ARCH.MINIMAXM3: [
-+        MODEL_TENSOR.TOKEN_EMBD,
-+        MODEL_TENSOR.OUTPUT_NORM,
-+        MODEL_TENSOR.OUTPUT,
-+        MODEL_TENSOR.ATTN_NORM,
-+        MODEL_TENSOR.ATTN_Q,
-+        MODEL_TENSOR.ATTN_Q_NORM,
-+        MODEL_TENSOR.ATTN_K,
-+        MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_V,
-+        MODEL_TENSOR.ATTN_OUT,
-+        MODEL_TENSOR.FFN_NORM,
-+        MODEL_TENSOR.FFN_GATE_INP,
-+        MODEL_TENSOR.FFN_EXP_PROBS_B,
-+        MODEL_TENSOR.FFN_GATE_EXP,
-+        MODEL_TENSOR.FFN_DOWN_EXP,
-+        MODEL_TENSOR.FFN_UP_EXP,
-+        MODEL_TENSOR.FFN_GATE_SHEXP,
-+        MODEL_TENSOR.FFN_DOWN_SHEXP,
-+        MODEL_TENSOR.FFN_UP_SHEXP,
-+        MODEL_TENSOR.FFN_GATE,
-+        MODEL_TENSOR.FFN_DOWN,
-+        MODEL_TENSOR.FFN_UP,
-+    ],
-     MODEL_ARCH.COGVLM: [
-         MODEL_TENSOR.TOKEN_EMBD,
-         MODEL_TENSOR.OUTPUT_NORM,
-@@ -4128,6 +4162,10 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.ATTN_Q_NORM,
-         MODEL_TENSOR.ATTN_K,
-         MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_Q,
-+        MODEL_TENSOR.ATTN_INDEX_K,
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM,
-         MODEL_TENSOR.ATTN_V,
-         MODEL_TENSOR.ATTN_OUT,
-         MODEL_TENSOR.FFN_NORM,
-diff --git a/gguf-py/gguf/tensor_mapping.py b/gguf-py/gguf/tensor_mapping.py
-index 9efb36f..a62040b 100644
---- a/gguf-py/gguf/tensor_mapping.py
-+++ b/gguf-py/gguf/tensor_mapping.py
-@@ -717,6 +717,22 @@ class TensorNameMap:
-             "model.layers.{bid}.attention.key_layernorm",                     # apertus
-         ),
- 
-+        MODEL_TENSOR.ATTN_INDEX_Q: (
-+            "model.layers.{bid}.self_attn.index_q_proj",                      # minimax-m3 (sparse-attn indexer)
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K: (
-+            "model.layers.{bid}.self_attn.index_k_proj",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM: (
-+            "model.layers.{bid}.self_attn.index_q_norm",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM: (
-+            "model.layers.{bid}.self_attn.index_k_norm",                      # minimax-m3
-+        ),
-+
-         MODEL_TENSOR.ROPE_FREQS: (
-             "encoder.layers.{bid}.self_attention.rotary_emb.inv_freq",  # persimmon
-         ),
-diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
-index b890e66..cb8bfc8 100644
---- a/src/llama-arch.cpp
-+++ b/src/llama-arch.cpp
-@@ -125,6 +125,7 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
-     { LLM_ARCH_GROVEMOE,         "grovemoe"         },
-     { LLM_ARCH_APERTUS,          "apertus"          },
-     { LLM_ARCH_MINIMAX_M2,       "minimax-m2"       },
-+    { LLM_ARCH_MINIMAX_M3,       "minimax-m3"       },
-     { LLM_ARCH_COGVLM,           "cogvlm"           },
-     { LLM_ARCH_RND1,             "rnd1"             },
-     { LLM_ARCH_PANGU_EMBED,      "pangu-embedded"   },
-@@ -395,6 +396,10 @@ static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
-     { LLM_TENSOR_ATTN_POST_NORM,                         "blk.%d.post_attention_norm" },
-     { LLM_TENSOR_ATTN_Q_NORM,                            "blk.%d.attn_q_norm" },
-     { LLM_TENSOR_ATTN_K_NORM,                            "blk.%d.attn_k_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_Q,                           "blk.%d.attn_index_q" },
-+    { LLM_TENSOR_ATTN_INDEX_K,                           "blk.%d.attn_index_k" },
-+    { LLM_TENSOR_ATTN_INDEX_Q_NORM,                      "blk.%d.attn_index_q_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_K_NORM,                      "blk.%d.attn_index_k_norm" },
-     { LLM_TENSOR_ATTN_GATE,                              "blk.%d.attn_gate" },
-     { LLM_TENSOR_FFN_POST_NORM,                          "blk.%d.post_ffw_norm" },
-     { LLM_TENSOR_FFN_POST_NORM_1,                        "blk.%d.post_ffw_norm_1" },
-@@ -761,6 +766,11 @@ static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
-     {LLM_TENSOR_FFN_NORM_EXPS,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_K_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-+    // minimax-m3 sparse-attn indexer: unused (GGML_OP_NONE) so the loader skips it
-+    {LLM_TENSOR_ATTN_INDEX_Q,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_Q_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-     {LLM_TENSOR_LAYER_OUT_NORM,             {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_LAYER_OUT_SCALE,            {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_A_NORM,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-@@ -998,6 +1008,7 @@ bool llm_arch_supports_sm_tensor(const llm_arch & arch) {
-         case LLM_ARCH_LFM2:
-         case LLM_ARCH_LFM2MOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_MISTRAL4:
-         case LLM_ARCH_KIMI_LINEAR:
-             return false;
-diff --git a/src/llama-arch.h b/src/llama-arch.h
-index a4f5091..2d50ead 100644
---- a/src/llama-arch.h
-+++ b/src/llama-arch.h
-@@ -144,6 +144,7 @@ enum llm_arch {
-     LLM_ARCH_TALKIE,
-     LLM_ARCH_MELLUM,
-     LLM_ARCH_EAGLE3,
-+    LLM_ARCH_MINIMAX_M3,
-     LLM_ARCH_DFLASH,
-     LLM_ARCH_UNKNOWN,
- };
-@@ -429,6 +430,10 @@ enum llm_tensor {
-     LLM_TENSOR_FFN_LATENT_UP,
-     LLM_TENSOR_ATTN_Q_NORM,
-     LLM_TENSOR_ATTN_K_NORM,
-+    LLM_TENSOR_ATTN_INDEX_Q,      // minimax-m3 sparse-attn indexer (unused)
-+    LLM_TENSOR_ATTN_INDEX_K,
-+    LLM_TENSOR_ATTN_INDEX_Q_NORM,
-+    LLM_TENSOR_ATTN_INDEX_K_NORM,
-     LLM_TENSOR_LAYER_OUT_NORM,
-     LLM_TENSOR_LAYER_OUT_SCALE,
-     LLM_TENSOR_POST_ATTN_NORM,
-diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
-index c8ecb0a..4c2c286 100644
---- a/src/llama-graph.cpp
-+++ b/src/llama-graph.cpp
-@@ -1719,6 +1719,16 @@ ggml_tensor * llm_graph_context::build_ffn(
-                 cur = ggml_reglu(ctx0, cur);
-                 cb(cur, "ffn_reglu", il);
-             } break;
-+        case LLM_FFN_SWIGLU_OAI:
-+            {
-+                // clamped SwiGLU: parallel gate path (cur=gate, tmp=up)
-+                GGML_ASSERT(gate && type_gate == LLM_FFN_PAR);
-+                constexpr float alpha = 1.702f;
-+                constexpr float limit = 7.0f;
-+                cur = ggml_swiglu_oai(ctx0, cur, tmp, alpha, limit);
-+                cb(cur, "ffn_swiglu_oai", il);
-+                type_gate = LLM_FFN_SEQ; // gate*up already fused; skip the par multiply
-+            } break;
-         default:
-             GGML_ABORT("fatal error");
-     }
-diff --git a/src/llama-graph.h b/src/llama-graph.h
-index c84cb6a..806ce7b 100644
---- a/src/llama-graph.h
-+++ b/src/llama-graph.h
-@@ -54,6 +54,7 @@ enum llm_ffn_op_type : int {
-     LLM_FFN_SWIGLU,
-     LLM_FFN_GEGLU,
-     LLM_FFN_REGLU,
-+    LLM_FFN_SWIGLU_OAI,
-     LLM_FFN_SWIGLU_OAI_MOE,
- };
- 
-diff --git a/src/llama-model.cpp b/src/llama-model.cpp
-index d874813..7bb71c0 100644
---- a/src/llama-model.cpp
-+++ b/src/llama-model.cpp
-@@ -280,6 +280,8 @@ static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params
-             return new llama_model_apertus(params);
-         case LLM_ARCH_MINIMAX_M2:
-             return new llama_model_minimax_m2(params);
-+        case LLM_ARCH_MINIMAX_M3:
-+            return new llama_model_minimax_m3(params);
-         case LLM_ARCH_COGVLM:
-             return new llama_model_cogvlm(params);
-         case LLM_ARCH_PANGU_EMBED:
-@@ -807,6 +809,7 @@ const char * llm_type_name(llm_type type) {
-         case LLM_TYPE_310B_A15B:     return "310B.A15B";
-         case LLM_TYPE_355B_A32B:     return "355B.A32B";
-         case LLM_TYPE_397B_A17B:     return "397B.A17B";
-+        case LLM_TYPE_428B_A23B:     return "428B.A23B";
-         case LLM_TYPE_685B_A37B:     return "685B.A37B";
-         case LLM_TYPE_744B_A40B:     return "744B.A40B";
-         case LLM_TYPE_E2B:           return "E2B";
-@@ -2532,6 +2535,7 @@ llama_rope_type llama_model_rope_type(const llama_model * model) {
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_APERTUS:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_COGVLM:
-         case LLM_ARCH_PANGU_EMBED:
-         case LLM_ARCH_AFMOE:
-diff --git a/src/llama-model.h b/src/llama-model.h
-index 45b054c..540e0d2 100644
---- a/src/llama-model.h
-+++ b/src/llama-model.h
-@@ -139,6 +139,7 @@ enum llm_type {
-     LLM_TYPE_310B_A15B, // /MiMo-V2-Flash
-     LLM_TYPE_355B_A32B, // GLM-4.5
-     LLM_TYPE_397B_A17B, // Qwen3.5
-+    LLM_TYPE_428B_A23B, // MiniMax M3
-     LLM_TYPE_685B_A37B, // DeepSeek V3.2
-     LLM_TYPE_744B_A40B, // GLM-5
-     LLM_TYPE_E2B,
-diff --git a/src/models/minimax-m3.cpp b/src/models/minimax-m3.cpp
-new file mode 100644
-index 0000000..137852a
---- /dev/null
-+++ b/src/models/minimax-m3.cpp
-@@ -0,0 +1,197 @@
-+#include "models.h"
-+
-+// MiniMax-M3, text-only: MiniMax-M2 GQA (per-head QK-norm, partial rotary) + DeepSeek-V3
-+// leading-dense/routed/shared experts (swigluoai). Sparse attn -> dense; vision + MTP dropped.
-+
-+void llama_model_minimax_m3::load_arch_hparams(llama_model_loader & ml) {
-+    ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
-+    ml.get_key(LLM_KV_LEADING_DENSE_BLOCK_COUNT,   hparams.n_layer_dense_lead, false);
-+    ml.get_key(LLM_KV_EXPERT_FEED_FORWARD_LENGTH,  hparams.n_ff_exp);
-+    ml.get_key(LLM_KV_EXPERT_SHARED_COUNT,         hparams.n_expert_shared);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE,        hparams.expert_weights_scale, false);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_NORM,         hparams.expert_weights_norm, false);
-+    ml.get_key(LLM_KV_EXPERT_GATING_FUNC,          hparams.expert_gating_func);
-+
-+    switch (hparams.n_layer()) {
-+        case 60: type = LLM_TYPE_428B_A23B; break;
-+        default: type = LLM_TYPE_UNKNOWN;
-+    }
-+}
-+
-+void llama_model_minimax_m3::load_arch_tensors(llama_model_loader &) {
-+    LLAMA_LOAD_LOCALS;
-+    const int64_t n_expert_shared = hparams.n_expert_shared;
-+    const int64_t n_ff_exp        = hparams.n_ff_exp;
-+
-+    tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
-+
-+    output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
-+    output      = create_tensor(tn(LLM_TENSOR_OUTPUT,      "weight"), {n_embd, n_vocab}, 0);
-+
-+    for (int i = 0; i < n_layer; ++i) {
-+        auto & layer = layers[i];
-+
-+        create_tensor_qkv(layer, i, n_embd, n_embd_head_k * n_head, n_embd_gqa, n_embd_gqa, 0);
-+        layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), { n_embd_head_k * n_head, n_embd }, 0);
-+
-+        layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
-+        // per-head QK-norm (one head_dim vector)
-+        layer.attn_q_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_NORM, "weight", i), {n_embd_head_k}, 0);
-+        layer.attn_k_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_NORM, "weight", i), {n_embd_head_k}, 0);
-+
-+        // sparse-attn indexer (unused): GGML_OP_NONE -> loader skips; NOT_REQUIRED -> older GGUFs still load;
-+        // SKIP_IF_VIRTUAL -> no-file loader (test-llama-archs) skips them too
-+        const int64_t n_index_head = 4;    // sparse_num_index_heads
-+        const int64_t d_index      = 128;  // sparse_index_dim
-+        const int idx_flags = TENSOR_NOT_REQUIRED | TENSOR_SKIP_IF_VIRTUAL;
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q,      "weight", i), {n_embd, n_index_head * d_index}, idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K,      "weight", i), {n_embd, d_index},                idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q_NORM, "weight", i), {d_index},                        idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K_NORM, "weight", i), {d_index},                        idx_flags);
-+
-+        layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
-+
-+        if (i < (int) hparams.n_layer_dense_lead) {
-+            layer.ffn_gate = create_tensor(tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd,   n_ff}, 0);
-+            layer.ffn_down = create_tensor(tn(LLM_TENSOR_FFN_DOWN, "weight", i), {  n_ff, n_embd}, 0);
-+            layer.ffn_up   = create_tensor(tn(LLM_TENSOR_FFN_UP,   "weight", i), {n_embd,   n_ff}, 0);
-+        } else {
-+            layer.ffn_gate_inp    = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP,    "weight", i), {n_embd, n_expert}, 0);
-+            layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias",   i), {n_expert}, 0);
-+            layer.ffn_gate_exps   = create_tensor(tn(LLM_TENSOR_FFN_GATE_EXPS,   "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+            layer.ffn_down_exps   = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS,   "weight", i), {n_ff_exp, n_embd, n_expert}, 0);
-+            layer.ffn_up_exps     = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,     "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+
-+            layer.ffn_gate_shexp = create_tensor(tn(LLM_TENSOR_FFN_GATE_SHEXP, "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+            layer.ffn_down_shexp = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "weight", i), {        n_ff_exp * n_expert_shared, n_embd}, 0);
-+            layer.ffn_up_shexp   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+        }
-+    }
-+}
-+
-+std::unique_ptr<llm_graph_context> llama_model_minimax_m3::build_arch_graph(const llm_graph_params & params) const {
-+    return std::make_unique<graph>(*this, params);
-+}
-+
-+llama_model_minimax_m3::graph::graph(const llama_model & model, const llm_graph_params & params) : llm_graph_context(params) {
-+    const int64_t n_embd_head = hparams.n_embd_head_v();
-+
-+    GGML_ASSERT(n_embd_head == hparams.n_embd_head_k());
-+    // partial rotary: head_dim != n_rot, so don't assert n_embd_head == n_rot
-+
-+    ggml_tensor * cur;
-+    ggml_tensor * inpL;
-+
-+    inpL = build_inp_embd(model.tok_embd);
-+
-+    ggml_tensor * inp_pos = build_inp_pos();
-+    auto inp_attn = build_attn_inp_kv();
-+    ggml_tensor * inp_out_ids = build_inp_out_ids();
-+
-+    for (int il = 0; il < n_layer; ++il) {
-+        ggml_tensor * inpSA = inpL;
-+
-+        // self-attention
-+        {
-+            cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);
-+            cb(cur, "attn_norm", il);
-+
-+            auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur,
-+                    n_embd_head, n_head, n_head_kv, il);
-+
-+            // per-head QK RMSNorm (weights include Gemma +1)
-+            Qcur = build_norm(Qcur, model.layers[il].attn_q_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Qcur, "Qcur_normed", il);
-+            Kcur = build_norm(Kcur, model.layers[il].attn_k_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Kcur, "Kcur_normed", il);
-+
-+            Qcur = ggml_rope_ext(
-+                ctx0, Qcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+            Kcur = ggml_rope_ext(
-+                ctx0, Kcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+
-+            cb(Qcur, "Qcur", il);
-+            cb(Kcur, "Kcur", il);
-+            cb(Vcur, "Vcur", il);
-+
-+            cur = build_attn(inp_attn,
-+                    model.layers[il].wo, NULL, model.layers[il].wo_s,
-+                    Qcur, Kcur, Vcur, nullptr, nullptr, nullptr, 1.0f/sqrtf(float(n_embd_head)), il);
-+        }
-+
-+        if (il == n_layer - 1 && inp_out_ids) {
-+            cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);
-+            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids);
-+        }
-+
-+        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);
-+        cb(ffn_inp, "ffn_inp", il);
-+
-+        cur = build_norm(ffn_inp, model.layers[il].ffn_norm, NULL, LLM_NORM_RMS, il);
-+        cb(cur, "ffn_norm", il);
-+
-+        if ((uint32_t) il < hparams.n_layer_dense_lead) {
-+            // leading dense
-+            cur = build_ffn(cur,
-+                    model.layers[il].ffn_up,   NULL, NULL,
-+                    model.layers[il].ffn_gate, NULL, NULL,
-+                    model.layers[il].ffn_down, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(cur, "ffn_out", il);
-+        } else {
-+            // routed experts
-+            ggml_tensor * moe_out = build_moe_ffn(cur,
-+                    model.layers[il].ffn_gate_inp,
-+                    model.layers[il].ffn_up_exps,
-+                    model.layers[il].ffn_gate_exps,
-+                    model.layers[il].ffn_down_exps,
-+                    model.layers[il].ffn_exp_probs_b,
-+                    n_expert, n_expert_used,
-+                    LLM_FFN_SWIGLU_OAI_MOE, hparams.expert_weights_norm,
-+                    hparams.expert_weights_scale,
-+                    (llama_expert_gating_func_type) hparams.expert_gating_func,
-+                    il);
-+            cb(moe_out, "ffn_moe_out", il);
-+
-+            // shared expert
-+            ggml_tensor * ffn_shexp = build_ffn(cur,
-+                    model.layers[il].ffn_up_shexp,   NULL, NULL,
-+                    model.layers[il].ffn_gate_shexp, NULL, NULL,
-+                    model.layers[il].ffn_down_shexp, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(ffn_shexp, "ffn_shexp", il);
-+
-+            cur = ggml_add(ctx0, moe_out, ffn_shexp);
-+            cb(cur, "ffn_out", il);
-+        }
-+
-+        cur = ggml_add(ctx0, cur, ffn_inp);
-+
-+        cur = build_cvec(cur, il);
-+        cb(cur, "l_out", il);
-+
-+        // input for next layer
-+        inpL = cur;
-+    }
-+
-+    cur = inpL;
-+
-+    cur = build_norm(cur, model.output_norm, NULL, LLM_NORM_RMS, -1);
-+    cb(cur, "result_norm", -1);
-+    res->t_embd = cur;
-+
-+    // lm_head
-+    cur = build_lora_mm(model.output, cur, model.output_s);
-+    cb(cur, "result_output", -1);
-+    res->t_logits = cur;
-+
-+    ggml_build_forward_expand(gf, cur);
-+}
-diff --git a/src/models/models.h b/src/models/models.h
-index 7a52e7b..5e2a826 100644
---- a/src/models/models.h
-+++ b/src/models/models.h
-@@ -1870,6 +1870,17 @@ struct llama_model_minimax_m2 : public llama_model_base {
-     std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
- };
- 
-+struct llama_model_minimax_m3 : public llama_model_base {
-+    llama_model_minimax_m3(const struct llama_model_params & params) : llama_model_base(params) {}
-+    void load_arch_hparams(llama_model_loader & ml) override;
-+    void load_arch_tensors(llama_model_loader & ml) override;
-+
-+    struct graph : public llm_graph_context {
-+        graph(const llama_model & model, const llm_graph_params & params);
-+    };
-+
-+    std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
-+};
- 
- struct llama_model_cogvlm : public llama_model_base {
-     llama_model_cogvlm(const struct llama_model_params & params) : llama_model_base(params) {}
-diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp
-index f39abe7..2085f43 100644
---- a/tests/test-llama-archs.cpp
-+++ b/tests/test-llama-archs.cpp
-@@ -352,6 +352,7 @@ static bool moe_mandatory(const llm_arch arch) {
-         case LLM_ARCH_LLADA_MOE:
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_RND1:
-         case LLM_ARCH_PADDLEOCR:
-         case LLM_ARCH_MIMO2:

backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch

@@ -0,0 +1,599 @@
+diff --git a/common/common.cpp b/common/common.cpp
+index 8f13217..fc584e1 100644
+--- a/common/common.cpp
++++ b/common/common.cpp
+@@ -1591,8 +1591,10 @@ struct llama_context_params common_context_params_to_llama(const common_params &
+     auto cparams = llama_context_default_params();
+
+     cparams.n_ctx             = params.n_ctx;
+-    cparams.n_seq_max         = params.n_parallel;
+-    cparams.n_rs_seq          = params.speculative.need_n_rs_seq();
++    // score-task forks need seq ids (and recurrent-state cells) of their
++    // own beyond the parallel slots
++    cparams.n_seq_max         = params.n_parallel + params.n_seq_score_forks;
++    cparams.n_rs_seq          = std::max(params.speculative.need_n_rs_seq(), (uint32_t) std::max(0, params.n_rs_seq));
+     cparams.n_outputs_max     = std::max(params.n_outputs_max, 0);
+     cparams.n_batch           = params.n_batch;
+     cparams.n_ubatch          = params.n_ubatch;
+diff --git a/common/common.h b/common/common.h
+index bffc176..e313bd6 100644
+--- a/common/common.h
++++ b/common/common.h
+@@ -455,6 +455,9 @@ struct common_params {
+     int32_t n_keep                =     0; // number of tokens to keep from initial prompt
+     int32_t n_chunks              =    -1; // max number of chunks to process (-1 = unlimited)
+     int32_t n_parallel            =     1; // number of parallel sequences to decode
++    int32_t n_seq_score_forks     =     0; // extra seq ids beyond n_parallel, reserved for server score-task forks
++    int32_t n_rs_seq              =     0; // recurrent-state rollback snapshots per seq (hybrid models cannot rewind without them; lets score tasks reuse a cached prompt across probe changes)
++    bool    score_enabled          = false; // reserve server resources for the Score task type
+     int32_t n_sequences           =     1; // number of sequences to decode
+     int32_t n_outputs_max         =     0; // max outputs in a batch (0 = n_batch)
+     int32_t grp_attn_n            =     1; // group-attention factor
+diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt
+index 780df32..1d2fe8f 100644
+--- a/tools/CMakeLists.txt
++++ b/tools/CMakeLists.txt
+@@ -41,3 +41,4 @@ else()
+     add_subdirectory(fit-params)
+     add_subdirectory(results)
+ endif()
++add_subdirectory(grpc-server)
+diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
+index 715477e..de5bed8 100644
+--- a/tools/server/server-context.cpp
++++ b/tools/server/server-context.cpp
+@@ -49,7 +49,16 @@ static uint32_t server_n_outputs_max(const common_params & params) {
+
+     const uint32_t n_outputs_per_seq = 1 + common_speculative_n_max(&params.speculative);
+
+-    const uint64_t n_outputs = (uint64_t) params.n_parallel * n_outputs_per_seq;
++    // score tasks (SERVER_TASK_TYPE_SCORE) output logits for every candidate
++    // token, so reserve room for a bounded candidate tail per parallel slot
++    if (!params.score_enabled) {
++        return std::max<uint32_t>(1, std::min<uint64_t>(n_batch,
++            (uint64_t) params.n_parallel * n_outputs_per_seq));
++    }
++
++    const uint32_t n_outputs_score_seq = 1 + SERVER_SCORE_MAX_CAND_TOKENS;
++
++    const uint64_t n_outputs = (uint64_t) params.n_parallel * std::max(n_outputs_per_seq, n_outputs_score_seq);
+
+     return std::max<uint32_t>(1, std::min<uint64_t>(n_batch, n_outputs));
+ }
+@@ -202,6 +211,26 @@ struct server_slot {
+
+     std::vector<completion_token_output> generated_token_probs;
+
++    // SERVER_TASK_TYPE_SCORE: shared-prefix token logprobs harvested
++    // incrementally across batch views (NaN = not yet produced)
++    std::vector<float> score_logprobs;
++
++    // SERVER_TASK_TYPE_SCORE: per-candidate suffix token logprobs; entry
++    // [c][0] comes from the last shared token's logits during prompt
++    // processing, the rest from the forked suffix decode
++    std::vector<std::vector<float>> score_cand_logprobs;
++
++    // SERVER_TASK_TYPE_SCORE: the prompt completed but some candidate has
++    // suffix tokens beyond the first, so a forked decode is still needed
++    bool score_suffix_pending = false;
++
++    // SERVER_TASK_TYPE_SCORE: where the current task's tokens diverged from
++    // the slot's previous cache. When the memory cannot rewind there and a
++    // re-prefill follows, a checkpoint at this position lets the next
++    // scoring call over the same stable prefix (e.g. a classifier's option
++    // list) resume from it instead of re-processing the whole prompt.
++    int32_t score_divergence = -1;
++
+     bool has_next_token = true;
+     bool has_new_line   = false;
+     bool truncated      = false;
+@@ -311,6 +340,10 @@ struct server_slot {
+         }
+         generated_tokens.clear();
+         generated_token_probs.clear();
++        score_logprobs.clear();
++        score_cand_logprobs.clear();
++        score_suffix_pending = false;
++        score_divergence = -1;
+         json_schema = json();
+
+         // clear speculative decoding stats
+@@ -2205,6 +2238,229 @@ private:
+         queue_results.send(std::move(res));
+     }
+
++    // log(sum(exp(logits))) with max-subtraction for stability — the
++    // log_softmax denominator shared by every token read from one output
++    static double score_log_denom(const float * logits, int32_t n_vocab) {
++        float max_logit = logits[0];
++        for (int32_t v = 1; v < n_vocab; ++v) {
++            max_logit = std::max(max_logit, logits[v]);
++        }
++        double sum_exp = 0.0;
++        for (int32_t v = 0; v < n_vocab; ++v) {
++            sum_exp += std::exp((double)(logits[v] - max_logit));
++        }
++        return (double) max_logit + std::log(sum_exp);
++    }
++
++    // Harvest logprobs for SCORE tasks from the current batch view: the
++    // shared-prefix scored tokens, and — from the last shared token's
++    // logits — the first suffix token of every candidate. The scored
++    // region can straddle ubatch boundaries for long prompts, so this
++    // accumulates view by view instead of reading everything when the
++    // prompt completes.
++    void collect_score_logprobs(server_slot & slot, const llama_batch & batch) {
++        const int32_t n_prompt = slot.task->n_score_prompt;
++        const int32_t n_total  = slot.task->n_tokens();
++        const auto &  suffixes = slot.task->score_suffixes;
++
++        const size_t n_shared_scored = (size_t) std::max(0, n_total - n_prompt);
++
++        if (slot.score_logprobs.size() != n_shared_scored) {
++            slot.score_logprobs.assign(n_shared_scored, NAN);
++        }
++        if (slot.score_cand_logprobs.size() != suffixes.size()) {
++            slot.score_cand_logprobs.resize(suffixes.size());
++            for (size_t c = 0; c < suffixes.size(); ++c) {
++                slot.score_cand_logprobs[c].assign(suffixes[c].size(), NAN);
++            }
++        }
++
++        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
++
++        for (int32_t i = 0; i < batch.n_tokens; ++i) {
++            if (!batch.logits[i] || batch.seq_id[i][0] != slot.id) {
++                continue;
++            }
++
++            // the output at position p predicts the task token at index p + 1;
++            // score tasks are text-only, so positions equal token indices
++            const int32_t target = batch.pos[i] + 1;
++            if (target < n_prompt || target > n_total) {
++                continue;
++            }
++
++            const float * logits = llama_get_logits_ith(slot.ctx_tgt, i);
++            if (logits == nullptr) {
++                SLT_ERR(slot, "failed to get logits for score target %d\n", target);
++                continue;
++            }
++
++            const double log_denom = score_log_denom(logits, n_vocab);
++
++            if (target < n_total) {
++                const llama_token tok = slot.task->tokens[target];
++                slot.score_logprobs[target - n_prompt] = (float) ((double) logits[tok] - log_denom);
++            } else {
++                // the last shared token predicts the first suffix token of
++                // every candidate
++                for (size_t c = 0; c < suffixes.size(); ++c) {
++                    if (!suffixes[c].empty()) {
++                        slot.score_cand_logprobs[c][0] = (float) ((double) logits[suffixes[c][0]] - log_denom);
++                    }
++                }
++            }
++        }
++    }
++
++    void send_score(server_slot & slot) {
++        auto res = std::make_unique<server_task_result_score>();
++        res->id              = slot.task->id;
++        res->index           = slot.task->index;
++        res->shared_logprobs = std::move(slot.score_logprobs);
++        res->cand_logprobs   = std::move(slot.score_cand_logprobs);
++
++        slot.score_logprobs.clear();
++        slot.score_cand_logprobs.clear();
++
++        SLT_DBG(slot, "sending score result, n_shared = %zu, n_cand = %zu\n",
++                res->shared_logprobs.size(), res->cand_logprobs.size());
++
++        queue_results.send(std::move(res));
++    }
++
++    // Decode the candidate suffixes of a completed score prompt: fork one
++    // sequence per candidate off the slot's shared prefix (metadata-only
++    // for the unified KV cache, copy-on-write for recurrent state) and
++    // decode all unique suffix tokens in as few llama_decode calls as the
++    // fork/batch/output budgets allow, harvesting a logprob for every
++    // suffix token that predicts a following one.
++    bool decode_score_suffixes(server_slot & slot) {
++        const auto & suffixes = slot.task->score_suffixes;
++
++        auto * mem = llama_get_memory(ctx_tgt);
++
++        // seq ids beyond the slots are reserved for score forks at context
++        // creation (common_params::n_seq_score_forks)
++        const int32_t seq_base    = (int32_t) slots.size();
++        const int32_t n_forks_max = std::min<int32_t>(SERVER_SCORE_FORK_SEQS, (int32_t) llama_n_seq_max(ctx_tgt) - seq_base);
++
++        if (n_forks_max < 1) {
++            SLT_ERR(slot, "no fork sequences reserved for score suffixes (n_seq_max = %d, n_slots = %d)\n",
++                    (int32_t) llama_n_seq_max(ctx_tgt), seq_base);
++            return false;
++        }
++
++        const int32_t   n_batch_max = llama_n_batch(ctx_tgt);
++        const int32_t   n_vocab     = llama_vocab_n_tokens(vocab);
++        const llama_pos pos0        = slot.prompt.tokens.pos_next();
++
++        std::vector<size_t> pending;
++        for (size_t c = 0; c < suffixes.size(); ++c) {
++            // single-token suffixes were fully scored from the last shared
++            // token's logits during prompt processing
++            if (suffixes[c].size() > 1) {
++                if ((int32_t) suffixes[c].size() > n_batch_max) {
++                    SLT_ERR(slot, "score suffix of candidate %zu (%zu tokens) exceeds n_batch (%d)\n",
++                            c, suffixes[c].size(), n_batch_max);
++                    return false;
++                }
++                pending.push_back(c);
++            }
++        }
++
++        size_t next = 0;
++        while (next < pending.size()) {
++            // greedy-pack candidates into one decode within the fork,
++            // batch and reserved-output budgets
++            std::vector<size_t> chunk;
++            int32_t n_tok = 0;
++            int32_t n_out = 0;
++            while (next < pending.size() && (int32_t) chunk.size() < n_forks_max) {
++                const int32_t m = (int32_t) suffixes[pending[next]].size();
++                if (!chunk.empty() && (n_tok + m > n_batch_max || n_out + m - 1 > SERVER_SCORE_MAX_CAND_TOKENS)) {
++                    break;
++                }
++                chunk.push_back(pending[next]);
++                n_tok += m;
++                n_out += m - 1;
++                next++;
++            }
++
++            llama_batch fb = llama_batch_init(n_tok, 0, 1);
++
++            for (size_t k = 0; k < chunk.size(); ++k) {
++                const llama_seq_id seq = seq_base + (llama_seq_id) k;
++                const auto & sfx = suffixes[chunk[k]];
++
++                llama_memory_seq_rm(mem, seq, -1, -1);
++                llama_memory_seq_cp(mem, slot.id, seq, -1, -1);
++
++                for (size_t j = 0; j < sfx.size(); ++j) {
++                    common_batch_add(fb, sfx[j], pos0 + (llama_pos) j, { seq }, j + 1 < sfx.size());
++                }
++            }
++
++            const int ret = llama_decode(ctx_tgt, fb);
++
++            if (ret == 0) {
++                int32_t i = 0;
++                for (size_t k = 0; k < chunk.size(); ++k) {
++                    const auto & sfx = suffixes[chunk[k]];
++                    auto & out = slot.score_cand_logprobs[chunk[k]];
++
++                    for (size_t j = 0; j < sfx.size(); ++j, ++i) {
++                        if (j + 1 >= sfx.size()) {
++                            continue; // last suffix token predicts nothing
++                        }
++                        const float * logits = llama_get_logits_ith(ctx_tgt, i);
++                        if (logits == nullptr) {
++                            SLT_ERR(slot, "failed to get logits for suffix token %zu of score candidate %zu\n", j, chunk[k]);
++                            continue;
++                        }
++                        const double log_denom = score_log_denom(logits, n_vocab);
++                        out[j + 1] = (float) ((double) logits[sfx[j + 1]] - log_denom);
++                    }
++                }
++            }
++
++            for (size_t k = 0; k < chunk.size(); ++k) {
++                llama_memory_seq_rm(mem, seq_base + (llama_seq_id) k, -1, -1);
++            }
++
++            llama_batch_free(fb);
++
++            if (ret != 0) {
++                SLT_ERR(slot, "score suffix decode failed, ret = %d\n", ret);
++                return false;
++            }
++        }
++
++        return true;
++    }
++
++    // score slots whose prompt completed this iteration decode their
++    // candidate suffixes here, after every batch view was consumed — a
++    // mid-view llama_decode would clobber logits other slots still read
++    void update_score_suffixes() {
++        for (auto & slot : slots) {
++            if (!slot.score_suffix_pending) {
++                continue;
++            }
++            slot.score_suffix_pending = false;
++
++            if (!slot.is_processing() || !slot.task || slot.task->type != SERVER_TASK_TYPE_SCORE) {
++                continue; // the task was aborted mid-iteration
++            }
++
++            if (decode_score_suffixes(slot)) {
++                send_score(slot);
++            } else {
++                send_error(slot, "failed to decode score candidate suffixes", ERROR_TYPE_SERVER);
++            }
++            slot.release();
++        }
++    }
++
+     //
+     // Functions to process the task
+     //
+@@ -2341,6 +2597,7 @@ private:
+             case SERVER_TASK_TYPE_INFILL:
+             case SERVER_TASK_TYPE_EMBEDDING:
+             case SERVER_TASK_TYPE_RERANK:
++            case SERVER_TASK_TYPE_SCORE:
+                 {
+                     // special case: if input is provided via CLI, tokenize it first
+                     // otherwise, no need to tokenize as it's already done inside the HTTP thread
+@@ -2832,6 +3089,13 @@ private:
+                 break; // stop any further processing
+             }
+         }
++
++        try {
++            update_score_suffixes();
++        } catch (const std::exception & e) {
++            SRV_ERR("update_score_suffixes() failed: %s\n", e.what());
++            abort_all_slots("update_score_suffixes() failed: " + std::string(e.what()));
++        }
+     }
+
+     void pre_decode() {
+@@ -3154,6 +3418,16 @@ private:
+                                     n_past = std::min(n_past, slot.alora_invocation_start - 1);
+                                 }
+
++                                // score tasks need the logits that predict the first candidate
++                                // token, so the last shared-prompt token must be (re-)decoded
++                                // even when the cache already covers it
++                                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
++                                    n_past = std::min(n_past, std::max(0, slot.task->n_score_prompt - 1));
++                                    // remember the divergence point before the checkpoint
++                                    // logic below possibly resets n_past to 0
++                                    slot.score_divergence = n_past;
++                                }
++
+                                 const auto n_cache_reuse = slot.task->params.n_cache_reuse;
+
+                                 const bool can_cache_reuse =
+@@ -3395,8 +3669,12 @@ private:
+
+                     bool do_checkpoint = params_base.n_ctx_checkpoints > 0;
+
+-                    // make checkpoints only for completion tasks
+-                    do_checkpoint = do_checkpoint && slot.task->type == SERVER_TASK_TYPE_COMPLETION;
++                    // make checkpoints for completion tasks, and for score tasks at the
++                    // shared-prompt boundary: models whose memory cannot be partially
++                    // rewound (SWA/hybrid/recurrent) would otherwise re-process the whole
++                    // prompt for every candidate of a scoring call
++                    do_checkpoint = do_checkpoint && (slot.task->type == SERVER_TASK_TYPE_COMPLETION ||
++                                                      slot.task->type == SERVER_TASK_TYPE_SCORE);
+
+                     // make a checkpoint of the parts of the memory that cannot be rolled back.
+                     // checkpoints are created only if:
+@@ -3463,10 +3741,17 @@ private:
+                         // embedding requires all tokens in the batch to be output;
+                         // MTP also wants logits at every prompt position so the
+                         // streaming hook can mirror t_h_nextn into ctx_dft.
++                        // score tasks need outputs at the positions that predict
++                        // each candidate token (the token at index i predicts the
++                        // task token at index i+1).
++                        const bool need_score_logit =
++                            slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                            slot.prompt.n_tokens() + 1 >= slot.task->n_score_prompt &&
++                            slot.prompt.n_tokens() + 1 <  slot.task->n_tokens();
+                         add_ok &= batch.add(slot.id,
+                             cur_tok,
+                             slot.prompt.tokens.pos_next(),
+-                            slot.need_embd());
++                            slot.need_embd() || need_score_logit);
+                         slot.prompt.tokens.push_back(cur_tok);
+
+                         slot.n_prompt_tokens_processed++;
+@@ -3481,6 +3766,32 @@ private:
+                             }
+                         }
+
++                        // score tasks: break at the shared-prompt boundary so the checkpoint
++                        // below lands exactly there — the other candidates of the same
++                        // scoring call re-process only their own tokens. Also break at the
++                        // point where this task diverged from the previous cache: after a
++                        // forced re-prefill a checkpoint there serves the next scoring call
++                        // over the same stable prefix (e.g. a classifier's option list).
++                        // The caller-declared stable-prefix boundary is the strongest of
++                        // these: a checkpoint there is at or before every future task's
++                        // divergence within the same option list, so it always survives
++                        // and always restores.
++                        if (do_checkpoint && slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                            (slot.prompt.n_tokens() == slot.task->n_score_prompt - 1 ||
++                             (slot.task->n_stable_prompt > 0 &&
++                              slot.prompt.n_tokens() == slot.task->n_stable_prompt &&
++                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1) ||
++                             (slot.prompt.n_tokens() == slot.score_divergence &&
++                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1))) {
++                            bool have_ckpt = false;
++                            for (const auto & ckpt : slot.prompt.checkpoints) {
++                                have_ckpt |= ckpt.n_tokens == slot.prompt.n_tokens();
++                            }
++                            if (!have_ckpt) {
++                                break;
++                            }
++                        }
++
+                         // process the last few tokens of the prompt separately in order to allow for a checkpoint to be created.
+                         // create checkpoints that many tokens before the end of the prompt:
+                         //  - 4 + n_ubatch
+@@ -3513,6 +3824,15 @@ private:
+                     const bool is_user_start = spans.is_user_start(n_tokens_start);
+                     const bool is_last_user_message = n_tokens_start == last_user_pos;
+
++                    // a batch starting at the score boundary or divergence point must
++                    // always checkpoint — min-step spacing would otherwise suppress it
++                    // and every candidate / next scoring call would re-process the prompt
++                    const bool is_score_boundary = slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                                                   (n_tokens_start == slot.task->n_score_prompt - 1 ||
++                                                    (slot.task->n_stable_prompt > 0 &&
++                                                     n_tokens_start == slot.task->n_stable_prompt) ||
++                                                    n_tokens_start == slot.score_divergence);
++
+                     // entire prompt has been processed
+                     if (slot.prompt.n_tokens() == slot.task->n_tokens()) {
+                         slot.state = SLOT_STATE_DONE_PROMPT;
+@@ -3528,8 +3848,8 @@ private:
+                         slot.init_sampler();
+                     } else {
+                         // skip ordinary mid-prompt checkpoints, unless the batch starts a user
+-                        // message or we are near the end of the prompt
+-                        if (!is_user_start && !near_prompt_end) {
++                        // message, the score boundary, or we are near the end of the prompt
++                        if (!is_user_start && !is_score_boundary && !near_prompt_end) {
+                             do_checkpoint = false;
+                         }
+                     }
+@@ -3546,10 +3866,10 @@ private:
+                     // do not checkpoint after mtmd chunks
+                     do_checkpoint = do_checkpoint && !has_mtmd;
+
+-                    // no need to create checkpoints that are too close together, unless it's the last user message
++                    // no need to create checkpoints that are too close together, unless it's the last user message or the score boundary
+                     do_checkpoint = do_checkpoint && (
+                             slot.prompt.checkpoints.empty() ||
+-                            is_last_user_message || near_prompt_end ||
++                            is_last_user_message || near_prompt_end || is_score_boundary ||
+                             n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
+                     SLT_DBG(slot, "main/do_checkpoint = %s, pos_min = %d, pos_max = %d\n", do_checkpoint ? "yes" : "no", pos_min, pos_max);
+
+@@ -3703,6 +4023,13 @@ private:
+                 }
+             }
+
++            // score slots harvest logprobs from every view that contains
++            // their outputs, not just the one holding the final token
++            if (slot.task && slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                (slot.state == SLOT_STATE_PROCESSING_PROMPT || slot.state == SLOT_STATE_DONE_PROMPT)) {
++                collect_score_logprobs(slot, batch_view);
++            }
++
+             if (!is_inside_view(slot.i_batch)) {
+                 // the required token not in this sub-batch, skip
+                 return;
+@@ -3724,6 +4051,25 @@ private:
+                     return;
+                 }
+
++                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
++                    // shared-prefix logprobs (and every candidate's first
++                    // suffix logprob) were accumulated per view above;
++                    // candidates with more suffix tokens still need the
++                    // forked decode at the end of update_slots()
++                    for (const auto & sfx : slot.task->score_suffixes) {
++                        if (sfx.size() > 1) {
++                            slot.score_suffix_pending = true;
++                            break;
++                        }
++                    }
++                    if (!slot.score_suffix_pending) {
++                        send_score(slot);
++                        slot.release();
++                    }
++                    slot.i_batch = -1;
++                    return;
++                }
++
+                 GGML_ASSERT(slot.task->need_sampling());
+
+                 // prompt evaluated for next-token prediction
+diff --git a/tools/server/server-task.h b/tools/server/server-task.h
+index c3eea2e..fb3c178 100644
+--- a/tools/server/server-task.h
++++ b/tools/server/server-task.h
+@@ -13,10 +13,25 @@
+
+ using json = nlohmann::ordered_json;
+
++// SERVER_TASK_TYPE_SCORE emits one logits output per candidate token (plus
++// the forced last-token output), and the context's output budget
++// (n_outputs_max) is reserved up front — so candidate length must be
++// bounded. Raising this raises the worst-case compute-buffer reservation
++// by ~n_vocab * 4 bytes per extra output.
++constexpr int32_t SERVER_SCORE_MAX_CAND_TOKENS = 64;
++
++// Maximum sequences forked off the shared prefix in one score suffix
++// decode. The context is created with this many seq ids (and
++// recurrent-state cells) beyond the parallel slots — see
++// common_params::n_seq_score_forks; candidates in excess of the budget
++// are decoded in successive chunks.
++constexpr int32_t SERVER_SCORE_FORK_SEQS = 16;
++
+ enum server_task_type {
+     SERVER_TASK_TYPE_COMPLETION,
+     SERVER_TASK_TYPE_EMBEDDING,
+     SERVER_TASK_TYPE_RERANK,
++    SERVER_TASK_TYPE_SCORE,
+     SERVER_TASK_TYPE_INFILL,
+     SERVER_TASK_TYPE_CANCEL,
+     SERVER_TASK_TYPE_CONTROL,
+@@ -153,6 +168,18 @@ struct server_task {
+     task_params   params;
+     server_tokens tokens;
+
++    // used by SERVER_TASK_TYPE_SCORE: `tokens` holds the shared prefix
++    // (prompt + longest common candidate token prefix) and logprobs are
++    // returned for its tokens from n_score_prompt onward. Each candidate's
++    // tokens beyond the shared prefix ride a forked sequence.
++    int32_t n_score_prompt = 0;
++    std::vector<llama_tokens> score_suffixes;
++    // token index where the caller-declared stable prompt prefix ends
++    // (0 = no hint): the option-list system prompt that repeats across
++    // scoring calls. A context checkpoint is forced there so models that
++    // cannot rewind state re-process only the per-call tail next time.
++    int32_t n_stable_prompt = 0;
++
+     // only used by CLI, this allow tokenizing CLI inputs on server side
+     // we need this because mtmd_context and vocab are not accessible outside of server_context
+     bool                    cli = false;
+@@ -197,6 +224,7 @@ struct server_task {
+         switch (type) {
+             case SERVER_TASK_TYPE_COMPLETION:
+             case SERVER_TASK_TYPE_INFILL:
++            case SERVER_TASK_TYPE_SCORE:
+                 return true;
+             default:
+                 return false;
+@@ -494,6 +522,25 @@ struct server_task_result_rerank : server_task_result {
+     virtual json to_json() override;
+ };
+
++struct server_task_result_score : server_task_result {
++    // log P(token | prefix) for the shared-prefix tokens after
++    // n_score_prompt, in order; NaN marks positions the decode never
++    // produced an output for
++    std::vector<float> shared_logprobs;
++
++    // per candidate: logprobs of its suffix tokens, in task order (entry
++    // 0 is the token right after the shared prefix, predicted by the last
++    // shared token's logits)
++    std::vector<std::vector<float>> cand_logprobs;
++
++    virtual json to_json() override {
++        return json {
++            {"shared_logprobs", shared_logprobs},
++            {"cand_logprobs",   cand_logprobs},
++        };
++    }
++};
++
+ struct server_task_result_error : server_task_result {
+     error_type err_type = ERROR_TYPE_SERVER;
+     std::string err_msg;

backend/cpp/llama-cpp/prepare.sh

@@ -31,6 +31,26 @@ cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 
+## Fork-skew probe. Upstream folded common_params::use_mmap / use_mlock /
+## use_direct_io into a single `load_mode` enum (ggml-org/llama.cpp#20834).
+## turboquant and bonsai compile this very same grpc-server.cpp against forks
+## that branched before that change, so the field set is decided from the
+## checkout in front of us rather than from a per-fork build flag: the flavor
+## targets disagree on whether they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and
+## probing heals itself the moment a fork rebases past the refactor.
+if grep -q "LLAMA_LOAD_MODE_MMAP" llama.cpp/include/llama.h; then
+    echo "==> llama.cpp carries the load-mode enum, using common_params::load_mode"
+    LEGACY_LOAD_MODE=0
+else
+    echo "==> llama.cpp predates the load-mode enum, using the legacy mmap/mlock/direct-io booleans"
+    LEGACY_LOAD_MODE=1
+fi
+cat > llama.cpp/tools/grpc-server/llama_compat.h <<EOF
+// Generated by backend/cpp/llama-cpp/prepare.sh. Do not edit.
+#pragma once
+#define LOCALAI_LEGACY_LOAD_MODE ${LEGACY_LOAD_MODE}
+EOF
+
 set +e
 if grep -q "grpc-server" llama.cpp/tools/CMakeLists.txt; then
     echo "grpc-server already added"

backend/cpp/turboquant/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-TURBOQUANT_VERSION?=7d9715f1f071fa07c7b2ad3dbfd320b314139e65
+TURBOQUANT_VERSION?=c26cbdffcf6fc9b7430cd6b117757e9a3f70b7ea
 LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
 
 CMAKE_ARGS?=
@@ -47,6 +47,7 @@ define turboquant-build
 	# original under backend/cpp/llama-cpp/, so the stock llama-cpp build
 	# stays compiling against vanilla upstream.
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build llama.cpp
@@ -84,6 +85,7 @@ turboquant-cpu-all:
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build purge
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build llama.cpp

backend/cpp/turboquant/patches/0001-hip-guard-copy2d-peer-fastpath.patch

@@ -1,50 +1,18 @@
 hip: port the turboquant CUDA additions that ggml's HIP shim doesn't cover
 
-The turboquant fork adds/modifies a few ggml-cuda.cu spots with CUDA APIs
-that ggml's HIP (and MUSA) compatibility layer does not provide, breaking
-the -gpu-rocm-hipblas-turboquant build:
+The turboquant fork creates backend events with plain cudaEventCreate,
+which ggml's HIP shim does not alias (it only aliases
+cudaEventCreateWithFlags). Use cudaEventCreateWithFlags(...,
+cudaEventDisableTiming), exactly as the rest of this file does.
 
-  1. ggml_cuda_copy2d_across_devices() (host-staged cross-device copy for
-     split mul_mat output) uses the CUDA 3D-peer copy APIs
-     cudaMemcpy3DPeerParms / make_cudaPitchedPtr / make_cudaExtent /
-     cudaMemcpy3DPeerAsync. HIP genuinely does not support these (see the
-     fork's own comment "HIP does not support cudaMemcpy3DPeerAsync"), so
-     guard the peer fast path with #if !defined(GGML_USE_HIP) &&
-     !defined(GGML_USE_MUSA) -- matching how the fork already guards the
-     same API for the sibling 2D copy -- and fall through to the existing
-     cudaMemcpyAsync staging fallback below (functionally identical,
-     slightly slower on multi-GPU ROCm).
-
-  2. ggml_backend_cuda_device_event_new() creates its event with plain
-     cudaEventCreate, which ggml's HIP shim does not alias (it only aliases
-     cudaEventCreateWithFlags). Use cudaEventCreateWithFlags(..., 
-     cudaEventDisableTiming) -- exactly what the rest of this file already
-     does (cf. lines ~1034, ~3461) and HIP-safe.
-
-CUDA builds are unaffected. Drop the relevant hunk once the fork HIP-ports
-these; apply-patches.sh fails fast if an anchor goes stale.
+CUDA builds are unaffected. Drop this patch once the fork HIP-ports the
+event creation; apply-patches.sh fails fast if the anchor goes stale.
 
 diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
-index 0427e6b..6352e6a 100644
+index 7d35c1a..2908acb 100644
 --- a/ggml/src/ggml-cuda/ggml-cuda.cu
 +++ b/ggml/src/ggml-cuda/ggml-cuda.cu
-@@ -1933,6 +1933,7 @@ static cudaError_t ggml_cuda_copy2d_across_devices(
-     size_t width, size_t height, cudaStream_t dst_stream, cudaStream_t src_stream) {
- 
-     const auto & info = ggml_cuda_info();
-+#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)  // 3D-peer copy types unmapped by ggml's HIP/MUSA shim; use staging fallback below
-     if (info.peer_access[src_device][dst_device]) {
-         cudaMemcpy3DPeerParms p = {};
-         p.dstDevice = dst_device;
-@@ -1942,6 +1943,7 @@ static cudaError_t ggml_cuda_copy2d_across_devices(
-         p.extent = make_cudaExtent(width, height, 1);
-         return cudaMemcpy3DPeerAsync(&p, dst_stream);
-     }
-+#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
- 
-     // Fallback: stage all rows through a single contiguous pinned buffer
-     int prev_device = ggml_cuda_get_device();
-@@ -5714,7 +5716,7 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_
+@@ -5795,7 +5795,7 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_
      ggml_cuda_set_device(dev_ctx->device);
  
      cudaEvent_t event;

backend/go/ced/Makefile

@@ -1,6 +1,6 @@
 # ced sound-classification backend Makefile.
 #
-# Upstream pin lives below as CED_VERSION?=<sha> so .github/bump_deps.sh can find
+# Upstream pin lives below as CED_VERSION?=db5aae02973a745722d6fbd2157cab1999106777
 # and update it (matches the parakeet-cpp / whisper.cpp convention).
 #
 # Local dev shortcut: symlink an out-of-tree ced.cpp shared build + header and
@@ -9,7 +9,7 @@
 #   ln -sf /path/to/ced.cpp/include/ced_capi.h .
 #   go build -o ced-grpc .
 
-CED_VERSION?=c04ac14b7992d00584d9e812c9bb6268598a6ce7
+CED_VERSION?=db5aae02973a745722d6fbd2157cab1999106777
 CED_REPO?=https://github.com/localai-org/ced.cpp
 
 GOCMD?=go

backend/go/cloud-proxy/provider_anthropic.go

@@ -32,7 +32,9 @@ import (
 type anthropicRequest struct {
 	Model         string               `json:"model"`
 	MaxTokens     int32                `json:"max_tokens"`
-	System        string               `json:"system,omitempty"`
+	// System is `any`: a bare string normally, or []anthropicSystemBlock
+	// when cache_prompt is on (the block form carries cache_control).
+	System        any                  `json:"system,omitempty"`
 	Messages      []anthropicMessage   `json:"messages"`
 	Stream        bool                 `json:"stream,omitempty"`
 	Temperature   *float64             `json:"temperature,omitempty"`
@@ -52,9 +54,30 @@ type anthropicMessage struct {
 }
 
 type anthropicTool struct {
-	Name        string          `json:"name"`
-	Description string          `json:"description,omitempty"`
-	InputSchema json.RawMessage `json:"input_schema"`
+	Name         string                 `json:"name"`
+	Description  string                 `json:"description,omitempty"`
+	InputSchema  json.RawMessage        `json:"input_schema"`
+	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
+}
+
+// anthropicCacheControl marks a prompt-cache breakpoint. Anthropic caches
+// everything up to and including a block tagged {"type":"ephemeral"} (5-min
+// TTL) and serves that prefix at the cache-read rate (0.1x input) on later
+// calls that share it — the win on agentic/multi-turn workloads.
+type anthropicCacheControl struct {
+	Type string `json:"type"` // "ephemeral"
+}
+
+// ephemeralCacheControl is the single reused breakpoint marker.
+var ephemeralCacheControl = &anthropicCacheControl{Type: "ephemeral"}
+
+// anthropicSystemBlock is the block form of the top-level system field.
+// Anthropic accepts system as a bare string OR a list of text blocks; the
+// block form is required to attach cache_control to the system prompt.
+type anthropicSystemBlock struct {
+	Type         string                 `json:"type"` // "text"
+	Text         string                 `json:"text"`
+	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
 }
 
 // anthropicToolChoice mirrors the four shapes Anthropic accepts:
@@ -81,8 +104,9 @@ type anthropicContentBlock struct {
 	// Tool-result block fields. tool_result uses `content` (not
 	// `text`) and pairs with `tool_use_id`; modelling them as
 	// distinct fields avoids ambiguity at marshal time.
-	ToolUseID     string `json:"tool_use_id,omitempty"`
-	ResultContent string `json:"content,omitempty"`
+	ToolUseID     string                 `json:"tool_use_id,omitempty"`
+	ResultContent string                 `json:"content,omitempty"`
+	CacheControl  *anthropicCacheControl `json:"cache_control,omitempty"`
 }
 
 type anthropicResponse struct {
@@ -156,6 +180,11 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 	if req.ToolChoice != nil && req.ToolChoice.Type == anthropicToolChoiceNone {
 		req.Tools, req.ToolChoice = nil, nil
 	}
+	// Prompt-cache breakpoint on the last tool: Anthropic caches the entire
+	// tool block up to the marked tool — usually a large, fully stable prefix.
+	if cfg.cachePrompt && len(req.Tools) > 0 {
+		req.Tools[len(req.Tools)-1].CacheControl = ephemeralCacheControl
+	}
 
 	var systemParts []string
 	for _, m := range opts.GetMessages() {
@@ -189,15 +218,54 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 			})
 		}
 	}
-	req.System = strings.Join(systemParts, "\n\n")
+	// System: block form (with cache_control) when caching is on, else the
+	// bare string. Only set when non-empty so `omitempty` still drops it.
+	if len(systemParts) > 0 {
+		joined := strings.Join(systemParts, "\n\n")
+		if cfg.cachePrompt {
+			req.System = []anthropicSystemBlock{{Type: "text", Text: joined, CacheControl: ephemeralCacheControl}}
+		} else {
+			req.System = joined
+		}
+	}
 
 	if len(req.Messages) == 0 && opts.GetPrompt() != "" {
 		req.Messages = []anthropicMessage{{Role: "user", Content: opts.GetPrompt()}}
 	}
 
+	// Prompt-cache breakpoint on the final message block caches the whole
+	// conversation prefix up to the newest turn. With the system + tools
+	// breakpoints above, Anthropic serves the entire stable head at the
+	// cache-read rate on the next agentic iteration (max 4 breakpoints; we
+	// use at most 3, so we never exceed the limit).
+	if cfg.cachePrompt {
+		markLastMessageCacheable(req.Messages)
+	}
+
 	return json.Marshal(req)
 }
 
+// markLastMessageCacheable tags the final block of the last message with a
+// cache_control breakpoint. String content is promoted to a single text
+// block so the marker has somewhere to attach; block content gets the marker
+// on its last element.
+func markLastMessageCacheable(msgs []anthropicMessage) {
+	if len(msgs) == 0 {
+		return
+	}
+	last := &msgs[len(msgs)-1]
+	switch c := last.Content.(type) {
+	case string:
+		if c != "" {
+			last.Content = []anthropicContentBlock{{Type: "text", Text: c, CacheControl: ephemeralCacheControl}}
+		}
+	case []anthropicContentBlock:
+		if len(c) > 0 {
+			c[len(c)-1].CacheControl = ephemeralCacheControl
+		}
+	}
+}
+
 // appendToolResult appends a tool_result block as a user message,
 // merging into a preceding user message that already carries blocks.
 // Anthropic concatenates consecutive same-role messages on its end,

backend/go/cloud-proxy/provider_anthropic_test.go

@@ -328,3 +328,62 @@ func TestBuildAnthropic_RoundTripsAssistantToolCalls(t *testing.T) {
 	g.Expect(r0["tool_use_id"]).To(Equal("call_abc"))
 	g.Expect(r0["content"]).To(Equal(`{"models":["a","b"]}`))
 }
+
+// TestPredict_Anthropic_PromptCache verifies that cache_prompt injects
+// exactly the intended cache_control breakpoints (system, last tool, last
+// message) when on, and none when off — asserting on the raw upstream body
+// because System becomes a block list that the typed struct hides.
+func TestPredict_Anthropic_PromptCache(t *testing.T) {
+	g := NewWithT(t)
+
+	// run issues one translate Predict and returns the raw body the fake
+	// Anthropic upstream received.
+	run := func(cachePrompt bool) string {
+		var rawBody string
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			b, _ := io.ReadAll(r.Body)
+			rawBody = string(b)
+			w.Header().Set("Content-Type", "application/json")
+			_, _ = io.WriteString(w, `{"id":"m","type":"message","role":"assistant","content":[{"type":"text","text":"ok"}],"model":"claude-3-5-sonnet-20241022","usage":{"input_tokens":5,"output_tokens":2}}`)
+		}))
+		defer srv.Close()
+
+		t.Setenv("CLOUD_PROXY_ANTHROPIC_FAKE", "sk-ant-fake")
+		cp := NewCloudProxy()
+		err := cp.Load(&pb.ModelOptions{
+			Model: "claude-local",
+			Proxy: &pb.ProxyOptions{
+				UpstreamUrl:   srv.URL,
+				Mode:          modeTranslate,
+				Provider:      providerAnthropic,
+				ApiKeyEnv:     "CLOUD_PROXY_ANTHROPIC_FAKE",
+				UpstreamModel: "claude-3-5-sonnet-20241022",
+				CachePrompt:   cachePrompt,
+			},
+		})
+		g.Expect(err).NotTo(HaveOccurred())
+
+		_, err = cp.Predict(&pb.PredictOptions{
+			Messages: []*pb.Message{
+				{Role: "system", Content: "be brief"},
+				{Role: "user", Content: "hello"},
+			},
+			Tools:  `[{"type":"function","function":{"name":"t","parameters":{"type":"object"}}}]`,
+			Tokens: 32,
+		})
+		g.Expect(err).NotTo(HaveOccurred())
+		return rawBody
+	}
+
+	// cache_prompt ON: three ephemeral breakpoints (system + last tool +
+	// last message), and system is emitted in block form.
+	on := run(true)
+	g.Expect(strings.Count(on, `"cache_control":{"type":"ephemeral"}`)).To(Equal(3),
+		"expected 3 breakpoints (system, tool, last message); body=%s", on)
+	g.Expect(on).To(ContainSubstring(`"system":[{"type":"text","text":"be brief"`))
+
+	// cache_prompt OFF: no breakpoints, system stays a bare string.
+	off := run(false)
+	g.Expect(off).NotTo(ContainSubstring("cache_control"))
+	g.Expect(off).To(ContainSubstring(`"system":"be brief"`))
+}

backend/go/cloud-proxy/proxy.go

@@ -48,6 +48,7 @@ type proxyConfig struct {
 	upstreamModel string
 	localModel    string // ModelOptions.Model — fallback when upstream_model is unset
 	apiKey        string // resolved at Load time
+	cachePrompt   bool   // inject Anthropic prompt-cache breakpoints (translate+anthropic)
 }
 
 func NewCloudProxy() *CloudProxy {
@@ -106,6 +107,7 @@ func (c *CloudProxy) Load(opts *pb.ModelOptions) error {
 		upstreamModel: po.GetUpstreamModel(),
 		localModel:    opts.GetModel(),
 		apiKey:        key,
+		cachePrompt:   po.GetCachePrompt(),
 	})
 	xlog.Info("cloud-proxy: ready",
 		"upstream", po.GetUpstreamUrl(),

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=5fca47ecf05cd68bb0075f8a00fe04da06f208d0
+CRISPASR_VERSION?=754b67289cf1137e3ed722885705f94132fc614f
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/depth-anything-cpp/Makefile

@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=f4e17dea695dd12ae76bea98ba58030996b98118
+DEPTHANYTHING_VERSION?=2028b47ac75a8659c6a9aa617baf09be193eb55f
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/locate-anything-cpp/Makefile

@@ -10,7 +10,7 @@ JOBS?=$(shell nproc --ignore=1)
 # this on `master` always picks up the latest C-API surface (incl. the
 # per-detection accessor functions used by golocateanythingcpp.go).
 LOCATEANYTHING_REPO?=https://github.com/mudler/locate-anything.cpp.git
-LOCATEANYTHING_VERSION?=ade2634f7f79b56121125e5885628744795a478f
+LOCATEANYTHING_VERSION?=77376ab332de918220f7a7e391542eefb5407c9f
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/magpie-tts-cpp/.gitignore

@@ -0,0 +1,6 @@
+magpie-tts-cpp
+*.so
+*.dylib
+sources/
+package/
+magpie-tts-models/

backend/go/magpie-tts-cpp/CMakeLists.txt

@@ -0,0 +1,35 @@
+cmake_minimum_required(VERSION 3.16)
+project(gomagpiettscpp LANGUAGES C CXX)
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
+set(MAGPIE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/magpie-tts.cpp)
+
+# Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
+if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
+    set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
+endif()
+
+# The magpie-tts C-API is exported directly by the upstream shared library
+# (magpie_tts_capi_* in libmagpie-tts.so, ggml statically linked inside), so
+# unlike qwen3-tts-cpp / moss-tts-cpp no local C shim is needed -- this wrapper
+# only configures the upstream project as a purego-loadable shared library.
+set(MAGPIE_SHARED       ON  CACHE BOOL "" FORCE)
+set(MAGPIE_BUILD_CLI    OFF CACHE BOOL "" FORCE)
+set(MAGPIE_BUILD_TESTS  OFF CACHE BOOL "" FORCE)
+
+# Upstream FORCE-overwrites GGML_CUDA / GGML_METAL / GGML_VULKAN / GGML_HIP from
+# its own MAGPIE_GGML_* toggles, which would silently discard the -DGGML_*=ON
+# flags the LocalAI Makefile passes for GPU BUILD_TYPEs. Translate them into the
+# MAGPIE_GGML_* vocabulary before add_subdirectory. (GGML_SYCL / GGML_BLAS and
+# the CPU ISA flags are not touched by upstream and reach ggml unchanged.)
+foreach(_be CUDA METAL VULKAN HIP)
+    if(GGML_${_be})
+        set(MAGPIE_GGML_${_be} ON CACHE BOOL "" FORCE)
+    endif()
+endforeach()
+
+add_subdirectory(${MAGPIE_DIR} magpie EXCLUDE_FROM_ALL)
+
+# Place libmagpie-tts.so at the build root, where the Makefile picks it up.
+set_target_properties(magpie-tts PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/magpie-tts-cpp/Makefile

@@ -0,0 +1,140 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+JOBS?=$(shell nproc --ignore=1)
+
+# magpie-tts.cpp version
+MAGPIETTS_REPO?=https://github.com/mudler/magpie-tts.cpp
+MAGPIETTS_CPP_VERSION?=3008ff73fc2d2da9e4d743b09350aa7023e8980c
+SO_TARGET?=libgomagpiettscpp.so
+
+CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
+
+ifeq ($(NATIVE),false)
+	CMAKE_ARGS+=-DGGML_NATIVE=OFF
+endif
+
+ifeq ($(BUILD_TYPE),cublas)
+	CMAKE_ARGS+=-DGGML_CUDA=ON
+else ifeq ($(BUILD_TYPE),openblas)
+	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
+else ifeq ($(BUILD_TYPE),clblas)
+	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
+else ifeq ($(BUILD_TYPE),hipblas)
+	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
+	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
+	ROCM_HOME ?= /opt/rocm
+	ROCM_PATH ?= /opt/rocm
+	export CXX=$(ROCM_HOME)/llvm/bin/clang++
+	export CC=$(ROCM_HOME)/llvm/bin/clang
+	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DGGML_VULKAN=ON
+else ifeq ($(OS),Darwin)
+	ifneq ($(BUILD_TYPE),metal)
+		CMAKE_ARGS+=-DGGML_METAL=OFF
+	else
+		CMAKE_ARGS+=-DGGML_METAL=ON
+		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
+	endif
+endif
+
+ifeq ($(BUILD_TYPE),sycl_f16)
+	CMAKE_ARGS+=-DGGML_SYCL=ON \
+		-DCMAKE_C_COMPILER=icx \
+		-DCMAKE_CXX_COMPILER=icpx \
+		-DGGML_SYCL_F16=ON
+endif
+
+ifeq ($(BUILD_TYPE),sycl_f32)
+	CMAKE_ARGS+=-DGGML_SYCL=ON \
+		-DCMAKE_C_COMPILER=icx \
+		-DCMAKE_CXX_COMPILER=icpx
+endif
+
+sources/magpie-tts.cpp:
+	mkdir -p sources/magpie-tts.cpp
+	cd sources/magpie-tts.cpp && \
+	git init && \
+	git remote add origin $(MAGPIETTS_REPO) && \
+	git fetch origin && \
+	git checkout $(MAGPIETTS_CPP_VERSION) && \
+	git submodule update --init --recursive --depth 1 --single-branch
+
+# Detect OS
+UNAME_S := $(shell uname -s)
+
+# Only build CPU variants on Linux
+ifeq ($(UNAME_S),Linux)
+	VARIANT_TARGETS = libgomagpiettscpp-avx.so libgomagpiettscpp-avx2.so libgomagpiettscpp-avx512.so libgomagpiettscpp-fallback.so
+else
+	# On non-Linux (e.g., Darwin), build only fallback variant (as a dylib)
+	VARIANT_TARGETS = libgomagpiettscpp-fallback.dylib
+endif
+
+magpie-tts-cpp: main.go gomagpiettscpp.go $(VARIANT_TARGETS)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o magpie-tts-cpp ./
+
+package: magpie-tts-cpp
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf libgomagpiettscpp*.so libgomagpiettscpp*.dylib package sources/magpie-tts.cpp magpie-tts-cpp
+
+purge:
+	rm -rf build*
+
+# Variants must build sequentially
+.NOTPARALLEL:
+
+# Build all variants (Linux only)
+ifeq ($(UNAME_S),Linux)
+libgomagpiettscpp-avx.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx${RESET})
+	SO_TARGET=libgomagpiettscpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx.so
+
+libgomagpiettscpp-avx2.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx2${RESET})
+	SO_TARGET=libgomagpiettscpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx2.so
+
+libgomagpiettscpp-avx512.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx512${RESET})
+	SO_TARGET=libgomagpiettscpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx512.so
+endif
+
+# Build fallback variant (all platforms)
+libgomagpiettscpp-fallback.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:fallback${RESET})
+	SO_TARGET=libgomagpiettscpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-fallback.so
+
+# Build fallback variant as a dylib (Darwin)
+libgomagpiettscpp-fallback.dylib: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:fallback (dylib)${RESET})
+	SO_TARGET=libgomagpiettscpp-fallback.dylib CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-fallback.dylib
+
+libgomagpiettscpp-custom: CMakeLists.txt
+	mkdir -p build-$(SO_TARGET) && \
+	cd build-$(SO_TARGET) && \
+	cmake .. $(CMAKE_ARGS) && \
+	cmake --build . --config Release -j$(JOBS) --target magpie-tts && \
+	cd .. && \
+	(mv build-$(SO_TARGET)/libmagpie-tts.so ./$(SO_TARGET) 2>/dev/null || \
+	 mv build-$(SO_TARGET)/libmagpie-tts.dylib ./$(SO_TARGET) 2>/dev/null)
+
+test: magpie-tts-cpp
+	@echo "Running magpie-tts-cpp tests..."
+	bash test.sh
+	@echo "magpie-tts-cpp tests completed."
+
+all: magpie-tts-cpp package

backend/go/magpie-tts-cpp/README.md

@@ -0,0 +1,69 @@
+# Magpie TTS C++ backend
+
+This backend runs NVIDIA's **Magpie TTS Multilingual 357M** GGUF through
+[magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp), a from-scratch
+C++/ggml port (model + NanoCodec + tokenizer + G2P dictionaries in one
+self-contained GGUF, no Python at inference time). It generates **22.05 kHz
+mono** speech in 5 baked voices across 9+ languages.
+
+The library is loaded via purego (cgo-less `dlopen`) exactly like
+`qwen3-tts-cpp` / `moss-tts-cpp`; the flat C-API (`magpie_tts_capi_*`) is
+exported directly by the upstream shared library, so there is no local C shim.
+
+## Model configuration
+
+The model path points at the single GGUF:
+
+```yaml
+name: magpie-tts-cpp
+backend: magpie-tts-cpp
+parameters:
+  model: magpie-tts-multilingual-357m-q8_0.gguf
+known_usecases:
+  - tts
+options:
+  - "speaker:Aria"   # optional default voice (Aria, Jason, John, Leo, Sofia, or 0-4)
+  - "language:en"    # optional default language
+```
+
+GGUFs live at
+[mudler/magpie-tts.cpp-gguf](https://huggingface.co/mudler/magpie-tts.cpp-gguf)
+(q8_0 recommended: near-lossless, ~624 MB, fastest decode).
+
+## Voices and languages
+
+Magpie has 5 baked speakers - `Aria`, `Jason`, `John`, `Leo`, `Sofia` - and no
+voice cloning. The request `voice` accepts the names case-insensitively or the
+indices `0`-`4`; empty selects Aria. Languages: `en`, `es`, `de`, `fr`, `it`,
+`pt-BR`, `hi`, `vi`, `ko`, `ar-AE`, `ar-SA`, `ar-MSA` (case-insensitive;
+default `en`).
+
+## API example
+
+```bash
+curl http://localhost:8080/v1/audio/speech \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "magpie-tts-cpp",
+    "input": "Hello world, this is a test of the text to speech system.",
+    "voice": "sofia",
+    "language": "en"
+  }' \
+  --output speech.wav
+```
+
+## Native end-to-end test
+
+The labeled test loads a real GGUF, synthesizes WAVs (verifying rate, layout
+and non-silence), and exercises the streaming path:
+
+```bash
+make -C backend/go/magpie-tts-cpp magpie-tts-cpp
+
+MAGPIETTS_MODEL=/path/to/magpie-tts-multilingual-357m-q8_0.gguf \
+MAGPIETTS_LIBRARY=backend/go/magpie-tts-cpp/libgomagpiettscpp-fallback.so \
+  go test ./backend/go/magpie-tts-cpp -ginkgo.label-filter=e2e
+```
+
+`bash test.sh` does the same and auto-downloads the q8_0 GGUF when
+`MAGPIETTS_MODEL` is unset.

backend/go/magpie-tts-cpp/audio.go

@@ -0,0 +1,93 @@
+package main
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"os"
+
+	"github.com/go-audio/audio"
+	"github.com/go-audio/wav"
+)
+
+// magpieSampleRate is the fixed Magpie TTS Multilingual (NanoCodec) output
+// rate: 22.05 kHz.
+const magpieSampleRate = 22050
+
+// magpieChannels is the fixed output layout: mono.
+const magpieChannels = 1
+
+// wavHeaderMono returns a 44-byte WAV header for a streaming 16-bit mono PCM
+// stream at 22050 Hz, with placeholder (0xFFFFFFFF) sizes since the total
+// length is unknown up front. Emitted as the first chunk of TTSStream so the
+// HTTP layer receives a self-describing WAV.
+func wavHeaderMono() []byte {
+	const blockAlign = magpieChannels * 2 // 16-bit mono
+	var buf bytes.Buffer
+	w := func(v any) { _ = binary.Write(&buf, binary.LittleEndian, v) }
+	buf.WriteString("RIFF")
+	w(uint32(0xFFFFFFFF))
+	buf.WriteString("WAVE")
+	buf.WriteString("fmt ")
+	w(uint32(16))                            // Subchunk1Size
+	w(uint16(1))                             // PCM
+	w(uint16(magpieChannels))                // mono
+	w(uint32(magpieSampleRate))              // sample rate
+	w(uint32(magpieSampleRate * blockAlign)) // byte rate = SR * blockAlign
+	w(uint16(blockAlign))                    // block align
+	w(uint16(16))                            // bits per sample
+	buf.WriteString("data")
+	w(uint32(0xFFFFFFFF))
+	return buf.Bytes()
+}
+
+// floatToPCM16LE clamps each sample to [-1,1] and encodes it as little-endian
+// signed 16-bit PCM.
+func floatToPCM16LE(samples []float32) []byte {
+	out := make([]byte, len(samples)*2)
+	for i, s := range samples {
+		if s > 1 {
+			s = 1
+		} else if s < -1 {
+			s = -1
+		}
+		v := int16(s * 32767)
+		out[i*2] = byte(v)        // #nosec G115 -- intentional little-endian split of a clamped int16
+		out[i*2+1] = byte(v >> 8) // #nosec G115 -- high byte of the same clamped int16
+	}
+	return out
+}
+
+// writeWAVMono writes float samples as a finalized 16-bit mono WAV at
+// 22050 Hz.
+func writeWAVMono(dst string, samples []float32) error {
+	f, err := os.Create(dst) // #nosec G304 -- dst is the server-chosen output path from the TTS request, not user-traversable
+	if err != nil {
+		return fmt.Errorf("magpie-tts: create %q: %w", dst, err)
+	}
+	enc := wav.NewEncoder(f, magpieSampleRate, 16, magpieChannels, 1)
+	ints := make([]int, len(samples))
+	for i, s := range samples {
+		if s > 1 {
+			s = 1
+		} else if s < -1 {
+			s = -1
+		}
+		ints[i] = int(s * 32767)
+	}
+	b := &audio.IntBuffer{
+		Format:         &audio.Format{NumChannels: magpieChannels, SampleRate: magpieSampleRate},
+		Data:           ints,
+		SourceBitDepth: 16,
+	}
+	if err := enc.Write(b); err != nil {
+		_ = enc.Close()
+		_ = f.Close()
+		return fmt.Errorf("magpie-tts: encode WAV: %w", err)
+	}
+	if err := enc.Close(); err != nil {
+		_ = f.Close()
+		return fmt.Errorf("magpie-tts: finalize WAV: %w", err)
+	}
+	return f.Close()
+}

backend/go/magpie-tts-cpp/e2e_test.go

@@ -0,0 +1,127 @@
+package main
+
+import (
+	"encoding/binary"
+	"math"
+	"os"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func ttsReq(text, voice, lang, dst string) *pb.TTSRequest {
+	r := &pb.TTSRequest{Text: text, Voice: voice, Dst: dst}
+	if lang != "" {
+		r.Language = &lang
+	}
+	return r
+}
+
+// wavRMS parses a 16-bit PCM WAV file and returns (sampleRate, channels, RMS
+// of the normalized samples).
+func wavRMS(path string) (int, int, float64) {
+	data, err := os.ReadFile(path) // #nosec G304 -- test-owned temp file
+	Expect(err).ToNot(HaveOccurred())
+	Expect(len(data)).To(BeNumerically(">", 44))
+	Expect(string(data[0:4])).To(Equal("RIFF"))
+	Expect(string(data[8:12])).To(Equal("WAVE"))
+	channels := int(binary.LittleEndian.Uint16(data[22:24]))
+	rate := int(binary.LittleEndian.Uint32(data[24:28]))
+	// Find the data chunk (go-audio writes fmt first, data after).
+	off := 12
+	for off+8 <= len(data) {
+		id := string(data[off : off+4])
+		sz := int(binary.LittleEndian.Uint32(data[off+4 : off+8]))
+		if id == "data" {
+			pcm := data[off+8:]
+			if sz < len(pcm) {
+				pcm = pcm[:sz]
+			}
+			var sum float64
+			n := len(pcm) / 2
+			for i := 0; i < n; i++ {
+				s := float64(int16(binary.LittleEndian.Uint16(pcm[i*2:]))) / 32768.0
+				sum += s * s
+			}
+			Expect(n).To(BeNumerically(">", 0))
+			return rate, channels, math.Sqrt(sum / float64(n))
+		}
+		off += 8 + sz
+	}
+	Fail("no data chunk found in " + path)
+	return 0, 0, 0
+}
+
+var _ = Describe("magpie-tts-cpp e2e", Label("e2e"), func() {
+	var (
+		loaded  bool
+		backend *MagpieTtsCpp
+	)
+
+	BeforeEach(func() {
+		modelPath := os.Getenv("MAGPIETTS_MODEL")
+		if modelPath == "" {
+			Skip("MAGPIETTS_MODEL not set; skipping e2e")
+		}
+		if !loaded {
+			lib := os.Getenv("MAGPIETTS_LIBRARY")
+			if lib == "" {
+				lib = "./libgomagpiettscpp-fallback.so"
+			}
+			h, err := purego.Dlopen(lib, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+			Expect(err).ToNot(HaveOccurred())
+			purego.RegisterLibFunc(&CppAbiVersion, h, "magpie_tts_capi_abi_version")
+			purego.RegisterLibFunc(&CppLoad, h, "magpie_tts_capi_load")
+			purego.RegisterLibFunc(&CppFree, h, "magpie_tts_capi_free")
+			purego.RegisterLibFunc(&CppSynthesize, h, "magpie_tts_capi_synthesize")
+			purego.RegisterLibFunc(&CppFreeAudio, h, "magpie_tts_capi_free_audio")
+			purego.RegisterLibFunc(&CppLastError, h, "magpie_tts_capi_last_error")
+
+			backend = &MagpieTtsCpp{}
+			Expect(backend.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
+			loaded = true
+		}
+	})
+
+	It("synthesizes a non-silent 22.05 kHz mono WAV via TTS", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		Expect(backend.TTS(ttsReq("Hello world, this is a test.", "Aria", "en", dst))).To(Succeed())
+		rate, channels, rms := wavRMS(dst)
+		Expect(rate).To(Equal(22050))
+		Expect(channels).To(Equal(1))
+		Expect(rms).To(BeNumerically(">", 0.01), "audio should not be silent")
+	})
+
+	It("accepts a case-insensitive voice and a speaker index", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		Expect(backend.TTS(ttsReq("Short test.", "sofia", "en", dst))).To(Succeed())
+		Expect(backend.TTS(ttsReq("Short test.", "1", "en", dst))).To(Succeed())
+	})
+
+	It("rejects an unknown voice before reaching the engine", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		err := backend.TTS(ttsReq("Short test.", "not-a-speaker", "en", dst))
+		Expect(err).To(MatchError(ContainSubstring("unknown voice")))
+	})
+
+	It("streams a self-describing WAV via TTSStream", func() {
+		results := make(chan []byte, 4096)
+		done := make(chan error, 1)
+		go func() { done <- backend.TTSStream(ttsReq("Hello there, streaming test.", "", "", ""), results) }()
+
+		var chunks int
+		var first []byte
+		for c := range results {
+			if chunks == 0 {
+				first = c
+			}
+			chunks++
+		}
+		Expect(<-done).ToNot(HaveOccurred())
+		Expect(chunks).To(BeNumerically(">=", 2))
+		Expect(string(first[0:4])).To(Equal("RIFF"))
+		Expect(string(first[8:12])).To(Equal("WAVE"))
+	})
+})

backend/go/magpie-tts-cpp/gomagpiettscpp.go

@@ -0,0 +1,147 @@
+package main
+
+import (
+	"fmt"
+	"path/filepath"
+	"unsafe"
+
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// capiABIVersion is the magpie_tts_capi.h surface this backend binds. Bumped
+// upstream on any breaking signature/semantics change; refuse to run on a
+// mismatch instead of crashing inside a miscompiled call.
+const capiABIVersion = 1
+
+var (
+	// magpie_tts_capi_abi_version() int
+	CppAbiVersion func() int
+	// magpie_tts_capi_load(gguf_path) -> ctx (NULL on failure)
+	CppLoad func(path string) uintptr
+	// magpie_tts_capi_free(ctx)
+	CppFree func(ctx uintptr)
+	// magpie_tts_capi_synthesize(ctx, text, language, speaker, out_n) -> float*
+	// 22050 Hz mono f32 PCM in [-1,1]; NULL on failure (see last_error).
+	CppSynthesize func(ctx uintptr, text, language, speaker string, outN unsafe.Pointer) uintptr
+	// magpie_tts_capi_free_audio(ptr)
+	CppFreeAudio func(ptr uintptr)
+	// magpie_tts_capi_last_error(ctx) -> const char* (ctx-owned, "" if none)
+	CppLastError func(ctx uintptr) string
+)
+
+// MagpieTtsCpp serves the Magpie TTS Multilingual 357M GGUF through the
+// magpie-tts.cpp C-API. The context is stateful (per-call error buffer, reused
+// graph allocator) and NOT safe for concurrent synthesize calls, so
+// base.SingleThread serializes everything behind the server-level lock.
+type MagpieTtsCpp struct {
+	base.SingleThread
+	ctx  uintptr
+	opts loadOptions
+}
+
+func (m *MagpieTtsCpp) Load(opts *pb.ModelOptions) error {
+	model := opts.ModelFile
+	if model == "" {
+		model = opts.ModelPath
+	}
+	if !filepath.IsAbs(model) && opts.ModelPath != "" {
+		model = filepath.Join(opts.ModelPath, model)
+	}
+
+	m.opts = parseOptions(opts.Options)
+
+	if abi := CppAbiVersion(); abi != capiABIVersion {
+		return fmt.Errorf("magpie-tts: C-API ABI mismatch: library reports v%d, backend built for v%d", abi, capiABIVersion)
+	}
+
+	xlog.Info("[magpie-tts-cpp] Load", "model", model)
+
+	ctx := CppLoad(model)
+	if ctx == 0 {
+		// Load failures have no context to query last_error on; the C side
+		// logs the reason to stderr.
+		return fmt.Errorf("magpie-tts: failed to load model %q", model)
+	}
+	m.ctx = ctx
+	return nil
+}
+
+// lastError surfaces the context's last error, falling back to a generic
+// message when the C side left it empty.
+func (m *MagpieTtsCpp) lastError() string {
+	if m.ctx == 0 {
+		return "no model loaded"
+	}
+	if e := CppLastError(m.ctx); e != "" {
+		return e
+	}
+	return "unknown error"
+}
+
+// synthesize runs one C-API synthesis and copies the PCM out of C memory.
+func (m *MagpieTtsCpp) synthesize(req *pb.TTSRequest) ([]float32, error) {
+	if m.ctx == 0 {
+		return nil, fmt.Errorf("magpie-tts: no model loaded")
+	}
+	if req.Text == "" {
+		return nil, fmt.Errorf("magpie-tts: TTS requires text")
+	}
+	speaker, err := resolveSpeaker(req.Voice, m.opts.speaker)
+	if err != nil {
+		return nil, err
+	}
+	lang := resolveLanguage(req.Language, m.opts.language)
+
+	var n int32
+	ptr := CppSynthesize(m.ctx, req.Text, lang, speaker, unsafe.Pointer(&n)) // #nosec G103 -- out-param for the purego-bound C-API
+	if ptr == 0 {
+		return nil, fmt.Errorf("magpie-tts: synthesis failed: %s", m.lastError())
+	}
+	// Register the free as soon as we own a non-null buffer, so the n<=0 guard
+	// below cannot leak it (defensive: the C contract returns NULL on failure).
+	defer CppFreeAudio(ptr)
+	if n <= 0 {
+		return nil, fmt.Errorf("magpie-tts: synthesis produced no samples")
+	}
+	//nolint:govet // C-allocated PCM, copied out before free
+	src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) // #nosec G103 -- C-allocated PCM, copied out before free
+	out := make([]float32, int(n))
+	copy(out, src)
+	return out, nil
+}
+
+func (m *MagpieTtsCpp) TTS(req *pb.TTSRequest) error {
+	if req.Dst == "" {
+		return fmt.Errorf("magpie-tts: TTS requires a destination path")
+	}
+	samples, err := m.synthesize(req)
+	if err != nil {
+		return err
+	}
+	return writeWAVMono(req.Dst, samples)
+}
+
+// TTSStream synthesizes one-shot (the magpie C-API has no streaming call) and
+// then emits a self-describing mono WAV: a header chunk followed by the PCM in
+// fixed-size slices, so the HTTP layer still receives a streamed WAV (the gRPC
+// TTSStream path never sets Message, so the backend owns the header - see
+// core/backend/tts.go:ModelTTSStream).
+func (m *MagpieTtsCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
+	defer close(results)
+	samples, err := m.synthesize(req)
+	if err != nil {
+		return err
+	}
+	results <- wavHeaderMono()
+	const sampleChunk = 4096 // mono samples per emitted chunk
+	for off := 0; off < len(samples); off += sampleChunk {
+		end := off + sampleChunk
+		if end > len(samples) {
+			end = len(samples)
+		}
+		results <- floatToPCM16LE(samples[off:end])
+	}
+	return nil
+}

backend/go/magpie-tts-cpp/magpiettscpp_test.go

@@ -0,0 +1,115 @@
+package main
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestMagpieTtsCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "magpie-tts-cpp suite")
+}
+
+var _ = Describe("resolveSpeaker", func() {
+	It("canonicalizes case-insensitive names", func() {
+		for in, want := range map[string]string{
+			"aria": "Aria", "JASON": "Jason", "john": "John",
+			"Leo": "Leo", "sofia": "Sofia",
+		} {
+			got, err := resolveSpeaker(in, "")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(got).To(Equal(want))
+		}
+	})
+	It("accepts indices 0-4", func() {
+		for in, want := range map[string]string{
+			"0": "Aria", "1": "Jason", "2": "John", "3": "Leo", "4": "Sofia",
+		} {
+			got, err := resolveSpeaker(in, "")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(got).To(Equal(want))
+		}
+	})
+	It("selects the engine default on empty", func() {
+		got, err := resolveSpeaker("", "")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(BeEmpty())
+	})
+	It("falls back to the model-level default speaker", func() {
+		got, err := resolveSpeaker("", "sofia")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("Sofia"))
+	})
+	It("prefers the request voice over the fallback", func() {
+		got, err := resolveSpeaker("leo", "sofia")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("Leo"))
+	})
+	It("rejects out-of-range indices", func() {
+		_, err := resolveSpeaker("5", "")
+		Expect(err).To(HaveOccurred())
+		_, err = resolveSpeaker("-1", "")
+		Expect(err).To(HaveOccurred())
+	})
+	It("rejects unknown names with the valid choices", func() {
+		_, err := resolveSpeaker("serena", "")
+		Expect(err).To(MatchError(ContainSubstring("Aria, Jason, John, Leo, Sofia")))
+	})
+})
+
+var _ = Describe("resolveLanguage", func() {
+	strp := func(s string) *string { return &s }
+
+	It("defaults to empty (engine picks en)", func() {
+		Expect(resolveLanguage(nil, "")).To(BeEmpty())
+	})
+	It("canonicalizes case for known codes", func() {
+		Expect(resolveLanguage(strp("EN"), "")).To(Equal("en"))
+		Expect(resolveLanguage(strp("pt-br"), "")).To(Equal("pt-BR"))
+		Expect(resolveLanguage(strp("AR-MSA"), "")).To(Equal("ar-MSA"))
+	})
+	It("falls back to the model-level default language", func() {
+		Expect(resolveLanguage(nil, "de")).To(Equal("de"))
+		Expect(resolveLanguage(strp(""), "PT-BR")).To(Equal("pt-BR"))
+	})
+	It("prefers the request language over the fallback", func() {
+		Expect(resolveLanguage(strp("it"), "de")).To(Equal("it"))
+	})
+	It("passes unknown codes through verbatim", func() {
+		Expect(resolveLanguage(strp("zh"), "")).To(Equal("zh"))
+	})
+})
+
+var _ = Describe("parseOptions", func() {
+	It("reads speaker and language defaults", func() {
+		o := parseOptions([]string{"speaker:Jason", "language:de"})
+		Expect(o.speaker).To(Equal("Jason"))
+		Expect(o.language).To(Equal("de"))
+	})
+	It("accepts the voice/lang aliases and ignores unknown keys", func() {
+		o := parseOptions([]string{"voice: sofia ", "lang: pt-BR", "bogus:1", "novalue"})
+		Expect(o.speaker).To(Equal("sofia"))
+		Expect(o.language).To(Equal("pt-BR"))
+	})
+})
+
+var _ = Describe("audio encoding", func() {
+	It("emits a well-formed streaming mono WAV header", func() {
+		h := wavHeaderMono()
+		Expect(h).To(HaveLen(44))
+		Expect(string(h[0:4])).To(Equal("RIFF"))
+		Expect(string(h[8:12])).To(Equal("WAVE"))
+		// channels (offset 22) == 1, sample rate (offset 24) == 22050
+		Expect(int(h[22]) | int(h[23])<<8).To(Equal(1))
+		Expect(int(h[24]) | int(h[25])<<8 | int(h[26])<<16 | int(h[27])<<24).To(Equal(22050))
+	})
+	It("clamps float PCM to int16", func() {
+		b := floatToPCM16LE([]float32{2, -2, 0})
+		Expect(b).To(HaveLen(6))
+		Expect(int16(uint16(b[0]) | uint16(b[1])<<8)).To(Equal(int16(32767)))
+		Expect(int16(uint16(b[2]) | uint16(b[3])<<8)).To(Equal(int16(-32767)))
+		Expect(int16(uint16(b[4]) | uint16(b[5])<<8)).To(Equal(int16(0)))
+	})
+})

backend/go/magpie-tts-cpp/main.go

@@ -0,0 +1,54 @@
+package main
+
+// Note: this is started internally by LocalAI and a server is allocated for each model
+import (
+	"flag"
+	"os"
+	"runtime"
+
+	"github.com/ebitengine/purego"
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+type LibFuncs struct {
+	FuncPtr any
+	Name    string
+}
+
+func main() {
+	libName := os.Getenv("MAGPIETTS_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "./libgomagpiettscpp-fallback.dylib"
+		} else {
+			libName = "./libgomagpiettscpp-fallback.so"
+		}
+	}
+
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		panic(err)
+	}
+
+	libFuncs := []LibFuncs{
+		{&CppAbiVersion, "magpie_tts_capi_abi_version"},
+		{&CppLoad, "magpie_tts_capi_load"},
+		{&CppFree, "magpie_tts_capi_free"},
+		{&CppSynthesize, "magpie_tts_capi_synthesize"},
+		{&CppFreeAudio, "magpie_tts_capi_free_audio"},
+		{&CppLastError, "magpie_tts_capi_last_error"},
+	}
+	for _, lf := range libFuncs {
+		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &MagpieTtsCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/magpie-tts-cpp/options.go

@@ -0,0 +1,108 @@
+package main
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+)
+
+// loadOptions holds the parsed model-level options. Magpie is a single
+// self-contained GGUF (model + codec + tokenizer + G2P dictionaries), so the
+// options only cover synthesis defaults.
+type loadOptions struct {
+	// speaker is the default baked speaker when a request has no voice.
+	speaker string
+	// language is the default language when a request has none ("" = engine
+	// default, which is "en").
+	language string
+}
+
+func splitOption(o string) (key, value string, ok bool) {
+	i := strings.Index(o, ":")
+	if i < 0 {
+		return "", "", false
+	}
+	return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true
+}
+
+// parseOptions reads the backend "key:value" option slice. Unknown keys are
+// ignored.
+func parseOptions(opts []string) loadOptions {
+	var o loadOptions
+	for _, oo := range opts {
+		key, value, ok := splitOption(oo)
+		if !ok {
+			continue
+		}
+		switch key {
+		case "speaker", "voice":
+			o.speaker = value
+		case "language", "lang":
+			o.language = value
+		}
+	}
+	return o
+}
+
+// magpieSpeakers are the baked speakers of Magpie TTS Multilingual 357M, in
+// index order (the engine matches names exactly, so the Go side canonicalizes
+// case-insensitive names and 0-4 indices to these strings).
+var magpieSpeakers = []string{"Aria", "Jason", "John", "Leo", "Sofia"}
+
+// resolveSpeaker maps the request voice (falling back to the model-level
+// default) onto a canonical baked speaker name. Accepted forms:
+// case-insensitive names (aria, JASON, ...) and indices 0-4. Empty selects the
+// engine default (speaker 0, Aria). Anything else is rejected with the valid
+// choices, instead of surfacing the engine's late error.
+func resolveSpeaker(voice, fallback string) (string, error) {
+	v := strings.TrimSpace(voice)
+	if v == "" {
+		v = strings.TrimSpace(fallback)
+	}
+	if v == "" {
+		return "", nil
+	}
+	if idx, err := strconv.Atoi(v); err == nil {
+		if idx < 0 || idx >= len(magpieSpeakers) {
+			return "", fmt.Errorf("magpie-tts: speaker index %d out of range 0-%d", idx, len(magpieSpeakers)-1)
+		}
+		return magpieSpeakers[idx], nil
+	}
+	for _, s := range magpieSpeakers {
+		if strings.EqualFold(s, v) {
+			return s, nil
+		}
+	}
+	return "", fmt.Errorf("magpie-tts: unknown voice %q (valid: %s, or 0-%d)",
+		voice, strings.Join(magpieSpeakers, ", "), len(magpieSpeakers)-1)
+}
+
+// magpieLanguages are the canonical language codes the tokenizer's language
+// map knows (exact-match on the C side), keyed by their lowercase form so
+// requests can be case-insensitive.
+var magpieLanguages = map[string]string{
+	"en": "en", "es": "es", "de": "de", "fr": "fr", "it": "it",
+	"pt-br": "pt-BR", "hi": "hi", "vi": "vi", "ko": "ko",
+	"ar-ae": "ar-AE", "ar-sa": "ar-SA", "ar-msa": "ar-MSA",
+}
+
+// resolveLanguage picks the request language, else the model-level default,
+// else "" (the engine defaults to "en"), canonicalizing case for the known
+// codes. Unknown codes pass through verbatim so the engine reports them with
+// its own exact-vocabulary error.
+func resolveLanguage(reqLang *string, fallback string) string {
+	l := ""
+	if reqLang != nil {
+		l = strings.TrimSpace(*reqLang)
+	}
+	if l == "" {
+		l = strings.TrimSpace(fallback)
+	}
+	if l == "" {
+		return ""
+	}
+	if canon, ok := magpieLanguages[strings.ToLower(l)]; ok {
+		return canon
+	}
+	return l
+}

backend/go/magpie-tts-cpp/package.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/magpie-tts-cpp $CURDIR/package/
+cp -fv $CURDIR/libgomagpiettscpp-*.so $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/libgomagpiettscpp-*.dylib $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/magpie-tts-cpp/run.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+echo "CPU info:"
+if [ "$(uname)" != "Darwin" ]; then
+	grep -e "model\sname" /proc/cpuinfo | head -1
+	grep -e "flags" /proc/cpuinfo | head -1
+fi
+
+if [ "$(uname)" = "Darwin" ]; then
+	# macOS: single dylib variant (Metal or Accelerate)
+	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.dylib"
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.so"
+
+	if grep -q -e "\savx\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX    found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx.so"
+		fi
+	fi
+
+	if grep -q -e "\savx2\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX2   found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx2.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx2.so"
+		fi
+	fi
+
+	# Check avx 512
+	if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX512F found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx512.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx512.so"
+		fi
+	fi
+
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+fi
+
+export MAGPIETTS_LIBRARY=$LIBRARY
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LIBRARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/magpie-tts-cpp "$@"
+fi
+
+echo "Using library: $LIBRARY"
+exec "$CURDIR"/magpie-tts-cpp "$@"

backend/go/magpie-tts-cpp/test.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+echo "Running magpie-tts-cpp backend tests..."
+
+# Auto-download the q8_0 GGUF only when MAGPIETTS_MODEL is not set.
+if [ -z "$MAGPIETTS_MODEL" ]; then
+    MODEL_DIR="./magpie-tts-models"
+    mkdir -p "$MODEL_DIR"
+    REPO_ID="mudler/magpie-tts.cpp-gguf"
+    BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
+    FILE="magpie-tts-multilingual-357m-q8_0.gguf"
+    dest="${MODEL_DIR}/${FILE}"
+    if [ -f "${dest}" ]; then
+        echo "  [skip] ${FILE}"
+    else
+        echo "  [download] ${FILE}..."
+        curl -L -o "${dest}" "${BASE_URL}/${FILE}" --progress-bar
+    fi
+    export MAGPIETTS_MODEL="${dest}"
+fi
+
+go test -v -timeout 1200s .
+
+echo "All magpie-tts-cpp tests passed."

backend/go/omnivoice-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # omnivoice.cpp version
 OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=f39cc4a3af988091f662313b336dddf8c83a3fb5
+OMNIVOICE_VERSION?=4f33af825d66e6ef1cb185e87b4589cacf747291
 SO_TARGET?=libgomnivoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/parakeet-cpp/Makefile

@@ -1,6 +1,6 @@
 # parakeet-cpp backend Makefile.
 #
-# Upstream pin lives below as PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
+# Upstream pin lives below as PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
 # (.github/bump_deps.sh) can find and update it - matches the
 # whisper.cpp / ds4 / vibevoice-cpp convention.
 #
@@ -15,7 +15,7 @@
 # That's what the L0 smoke test uses. The default target below does the
 # proper clone-at-pin + cmake build so CI doesn't need a side-checkout.
 
-PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
+PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
 PARAKEET_REPO?=https://github.com/mudler/parakeet.cpp
 
 GOCMD?=go

backend/go/qwen3-tts-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # qwentts.cpp version
 QWEN3TTS_REPO?=https://github.com/ServeurpersoCom/qwentts.cpp
-QWEN3TTS_CPP_VERSION?=e93292bee1778854ab7dcb2d325ffe531fef910f
+QWEN3TTS_CPP_VERSION?=35ebe5376b82a0a59d008586d55bbe623d449011
 SO_TARGET?=libgoqwen3ttscpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=ea4e566ccffa10f853ecc3f29e74b1820bc91beb
+STABLEDIFFUSION_GGML_VERSION?=22516991cbdf725e69b0b4a87e52ca16cce07c2d
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/vllm-cpp/.gitignore

@@ -0,0 +1,6 @@
+sources/
+build/
+package/
+vllm-cpp
+libvllm.so
+libvllm.dylib

backend/go/vllm-cpp/Makefile

@@ -0,0 +1,102 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+# nproc doesn't exist on the macOS runners: an empty JOBS turns `-j$(JOBS)`
+# into bare `-j` (unlimited clang jobs), which swap-thrashes the 3-core Mac
+# until the 6h GHA timeout. Fall back to sysctl there, then to a constant.
+JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+# vllm.cpp version
+VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
+VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845
+
+# The backend consumes only the stable C ABI (libvllm + include/vllm.h), so the
+# server, examples and tests of the engine are never built here.
+CMAKE_ARGS+=-DVLLM_CPP_SERVER=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_BUILD_EXAMPLES=OFF
+CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
+
+# vllm.cpp sets no global -march: SIMD tiers are per-file with runtime dispatch,
+# so ONE portable library serves every CPU of the target arch (unlike the
+# ggml-based backends and their avx/avx2/avx512 variant builds).
+UNAME_M := $(shell uname -m)
+
+ifeq ($(BUILD_TYPE),cublas)
+	# Blackwell-family targets only: other CUDA arches are build-supported
+	# upstream but have no runtime-proven fast path. amd64 gets the consumer
+	# (120a) + GB10 (121a) fat binary; arm64 CUDA (l4t-style images, DGX
+	# Spark) is GB10 only. Triton-AOT GDN cubins are vendored per-arch, no
+	# Python needed to consume them.
+	ifeq ($(UNAME_M),x86_64)
+		# NO -DVLLM_CPP_TRITON on fat builds: the vendored Triton-AOT cubin
+		# trees are per-arch and the engine refuses a multi-arch build unless
+		# pinned to one tree (unsound for the other arch). The non-AOT GDN
+		# path serves the fat binary; single-arch builds keep the cubins.
+		#
+		# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks
+		# compute_121a (GB10) and its ptxas rejects the sm_120a NVFP4 MMA
+		# kernels ("Vector type too large"), so no cuda-12 variant is shipped.
+		ifeq ($(CUDA_MAJOR_VERSION),12)
+      $(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
+		endif
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=120a;121a"
+	else
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
+	endif
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF
+else ifeq ($(BUILD_TYPE),metal)
+	CMAKE_ARGS+=-DVLLM_CPP_METAL=ON
+else
+	CMAKE_ARGS+=-DVLLM_CPP_CUDA=OFF
+endif
+
+UNAME_S := $(shell uname -s)
+ifeq ($(UNAME_S),Darwin)
+	LIB=libvllm.dylib
+else
+	LIB=libvllm.so
+endif
+
+sources/vllm.cpp:
+	mkdir -p sources/vllm.cpp
+	cd sources/vllm.cpp && \
+	git init && \
+	git remote add origin $(VLLM_CPP_REPO) && \
+	git fetch --depth 1 origin $(VLLM_CPP_VERSION) && \
+	git checkout FETCH_HEAD
+
+$(LIB): sources/vllm.cpp
+	mkdir -p build && \
+	cd build && \
+	cmake ../sources/vllm.cpp $(CMAKE_ARGS) && \
+	cmake --build . --config Release -j$(JOBS) --target vllm_shared
+	cp -fL build/$(LIB) ./$(LIB)
+
+vllm-cpp: main.go govllmcpp.go backend.go options.go $(LIB)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vllm-cpp ./
+
+package: vllm-cpp
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf libvllm.so libvllm.dylib package sources/vllm.cpp vllm-cpp
+
+purge:
+	rm -rf build
+
+.NOTPARALLEL:
+
+# The unit specs are pure Go (struct mirrors, option mapping, load
+# validation): no libvllm build is needed. The e2e specs skip unless
+# VLLM_CPP_MODEL points at a real model (then build the lib first).
+test:
+	@echo "Running vllm-cpp tests..."
+	bash test.sh
+	@echo "vllm-cpp tests completed."
+
+all: vllm-cpp package

backend/go/vllm-cpp/README.md

@@ -0,0 +1,45 @@
+# vllm-cpp backend
+
+LocalAI text-generation backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
+the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching,
+safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
+inference time.
+
+The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
+ABI v2) through purego:
+
+- `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
+  directory (`config.json` + safetensors). `context_size` maps to
+  `max_model_len`; `options: ["block_size:<n>", "num_blocks:<n>",
+  "max_num_seqs:<n>"]` size the KV cache and scheduler admission.
+- `Predict` -> `vllm_complete` (blocking).
+- `PredictStream` -> `vllm_complete_stream`; concurrent gRPC requests batch
+  continuously in the engine's shared AsyncLLM scheduler.
+- Chat / tool calling rides the SAME code path as the llama.cpp autoparser:
+  with `use_tokenizer_template: true` the backend implements
+  `PredictRich`/`PredictStreamRich` over the ABI v3 chat entry points
+  (`vllm_chat` / `vllm_chat_stream`). The ENGINE applies the model's chat
+  template (GGUF `tokenizer.chat_template` or `tokenizer_config.json`),
+  decides when a tool call engages (`tool_choice: auto` lowers to a LAZY
+  structural-tag decode constraint; `required`/named force one), parses tool
+  calls with its streaming Hermes-style parser, and the backend maps each
+  `chat.completion.chunk` onto `ChatDelta`/`ToolCallDelta` protos.
+- Without structured messages the plain path applies:
+  `PredictOptions.Grammar` -> the ABI's `structured_grammar` (GBNF) for
+  LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
+  choice constraints are also exposed by the ABI.
+
+Model config example:
+
+```yaml
+name: qwen3-vllm
+backend: vllm-cpp
+context_size: 8192
+parameters:
+  model: Qwen3-4B   # model dir (safetensors) or .gguf file
+options:
+- max_num_seqs:16
+```
+
+Testing: `make test` runs the unit specs; export `VLLM_CPP_MODEL=<model>` (and
+optionally `VLLM_CPP_LIBRARY=<libvllm path>`) to enable the e2e specs.

backend/go/vllm-cpp/backend.go

@@ -0,0 +1,245 @@
+package main
+
+// LocalAI gRPC backend over the vllm.cpp C ABI.
+//
+// Predict maps to the blocking vllm_complete; PredictStream maps to
+// vllm_complete_stream, whose per-delta C callback bridges into the gRPC
+// stream channel. Concurrent calls are intentional: every completion entry
+// point submits into the engine's shared AsyncLLM scheduler, so parallel
+// LocalAI requests batch continuously inside the engine (the reason this
+// backend embeds base.Base and not base.SingleThread).
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+type VllmCpp struct {
+	base.Base
+
+	engine uintptr
+	opts   loadOptions
+}
+
+// Stream registry: the per-request bridge between the C token callback and
+// the gRPC stream channel, keyed by an integer handle round-tripped through
+// the C user_data pointer (never a Go pointer across the ABI). The host gRPC
+// server drains the channel even after a client disconnect, so sends here
+// cannot wedge the engine's delivery loop.
+var (
+	streamsMu   sync.Mutex
+	streams     = map[uintptr]chan string{}
+	streamNext  uintptr
+	tokenCbOnce sync.Once
+	tokenCbPtr  uintptr
+)
+
+// tokenCallback is the single C-shared callback for every stream; it
+// dispatches on the user_data handle. Returning 0 aborts the in-flight
+// request (vllm_token_callback contract).
+func tokenCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	streamsMu.Lock()
+	results := streams[userData]
+	streamsMu.Unlock()
+	if results == nil {
+		return 0 // unknown request: stop generation.
+	}
+	if text := goString(delta); text != "" {
+		results <- text
+	}
+	return 1
+}
+
+func registerStream(results chan string) uintptr {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	streamNext++
+	streams[streamNext] = results
+	return streamNext
+}
+
+func unregisterStream(h uintptr) {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	delete(streams, h)
+}
+
+// validModelPath enforces the greedy-probe rule: when a model config has no
+// explicit backend, the loader probes every backend with the model name, so
+// Load must refuse anything vllm.cpp cannot serve (a GGUF file, or a HF-style
+// directory with config.json + safetensors).
+func validModelPath(model string) error {
+	info, err := os.Stat(model)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: model path %q not found: %w", model, err)
+	}
+	if info.IsDir() {
+		if _, err := os.Stat(filepath.Join(model, "config.json")); err != nil {
+			return fmt.Errorf("vllm-cpp: model dir %q has no config.json", model)
+		}
+		return nil
+	}
+	if strings.EqualFold(filepath.Ext(model), ".gguf") {
+		return nil
+	}
+	return fmt.Errorf("vllm-cpp: model %q is neither a .gguf file nor a config.json model dir", model)
+}
+
+func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
+	model := opts.ModelFile
+	if model == "" {
+		model = opts.ModelPath
+	}
+	if !filepath.IsAbs(model) && opts.ModelPath != "" {
+		model = filepath.Join(opts.ModelPath, model)
+	}
+	if err := validModelPath(model); err != nil {
+		return err
+	}
+
+	v.opts = parseOptions(opts)
+
+	mp := defaultModelParams()
+	if v.opts.blockSize > 0 {
+		mp.BlockSize = v.opts.blockSize
+	}
+	if v.opts.numBlocks > 0 {
+		mp.NumBlocks = v.opts.numBlocks
+	}
+	if opts.ContextSize > 0 {
+		mp.MaxModelLen = opts.ContextSize
+	}
+	if v.opts.maxNumSeqs > 0 {
+		mp.MaxNumSeqs = v.opts.maxNumSeqs
+	}
+
+	modelC := cString(model)
+	mp.ModelPath = uintptr(unsafe.Pointer(&modelC[0])) // #nosec G103 -- borrowed by C for the load call only
+	var toolParserC, reasoningParserC []byte
+	if v.opts.toolParser != "" {
+		toolParserC = cString(v.opts.toolParser)
+		mp.ToolParser = uintptr(unsafe.Pointer(&toolParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+	if v.opts.reasoningParser != "" {
+		reasoningParserC = cString(v.opts.reasoningParser)
+		mp.ReasoningParser = uintptr(unsafe.Pointer(&reasoningParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+
+	xlog.Info("[vllm-cpp] Load", "model", model, "engine", vllmVersion(),
+		"blockSize", mp.BlockSize, "numBlocks", mp.NumBlocks,
+		"maxModelLen", mp.MaxModelLen, "maxNumSeqs", mp.MaxNumSeqs)
+
+	var engine uintptr
+	rc := vllmEngineLoad(unsafe.Pointer(&mp), unsafe.Pointer(&engine)) // #nosec G103 -- POD out-params
+	runtime.KeepAlive(modelC)
+	runtime.KeepAlive(toolParserC)
+	runtime.KeepAlive(reasoningParserC)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: engine load failed: %s", vllmLastError())
+	}
+	v.engine = engine
+	return nil
+}
+
+func (v *VllmCpp) Free() error {
+	if v.engine != 0 {
+		vllmEngineFree(v.engine)
+		v.engine = 0
+	}
+	return nil
+}
+
+// samplingFromPredict lowers PredictOptions into the C sampling POD plus the
+// backing buffers that must stay alive for the duration of the C call.
+func samplingFromPredict(opts *pb.PredictOptions) (sp cSamplingParams, keep []any) {
+	sp = defaultSamplingParams()
+	sp.Temperature = opts.Temperature
+	if opts.TopP > 0 {
+		sp.TopP = opts.TopP
+	}
+	if opts.TopK > 0 {
+		sp.TopK = opts.TopK
+	}
+	if opts.MinP > 0 {
+		sp.MinP = opts.MinP
+	}
+	if opts.Tokens > 0 {
+		sp.MaxTokens = opts.Tokens
+	} else {
+		sp.MaxTokens = 0 // unbounded; the engine caps at max_model_len.
+	}
+	if opts.Seed > 0 {
+		sp.Seed = uint64(opts.Seed)
+		sp.HasSeed = 1
+	}
+	sp.PresencePenalty = opts.PresencePenalty
+	sp.FrequencyPenalty = opts.FrequencyPenalty
+	if opts.Penalty > 0 {
+		sp.RepetitionPenalty = opts.Penalty
+	}
+	if opts.IgnoreEOS {
+		sp.IgnoreEOS = 1
+	}
+	if len(opts.StopPrompts) > 0 {
+		ptrs, backing := cStringArray(opts.StopPrompts)
+		sp.Stop = uintptr(unsafe.Pointer(&ptrs[0])) // #nosec G103 -- borrowed by C for the call only
+		sp.NStop = int32(len(ptrs))
+		keep = append(keep, ptrs, backing)
+	}
+	if opts.Grammar != "" {
+		g := cString(opts.Grammar)
+		sp.StructuredGrammar = uintptr(unsafe.Pointer(&g[0])) // #nosec G103 -- borrowed by C for the call only
+		keep = append(keep, g)
+	}
+	return sp, keep
+}
+
+func (v *VllmCpp) Predict(opts *pb.PredictOptions) (string, error) {
+	if v.engine == 0 {
+		return "", fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	sp, keep := samplingFromPredict(opts)
+	var out cCompletion
+	rc := vllmComplete(v.engine, opts.Prompt, unsafe.Pointer(&sp), unsafe.Pointer(&out)) // #nosec G103 -- POD in/out params
+	runtime.KeepAlive(keep)
+	if rc != vllmOK {
+		return "", fmt.Errorf("vllm-cpp: completion failed: %s", vllmLastError())
+	}
+	text := goString(out.Text)
+	vllmCompletionFree(unsafe.Pointer(&out)) // #nosec G103 -- frees out.Text
+	return text, nil
+}
+
+func (v *VllmCpp) PredictStream(opts *pb.PredictOptions, results chan string) error {
+	if v.engine == 0 {
+		close(results)
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	tokenCbOnce.Do(func() {
+		tokenCbPtr = purego.NewCallback(tokenCallback)
+	})
+
+	sp, keep := samplingFromPredict(opts)
+	handle := registerStream(results)
+
+	go func() {
+		defer close(results)
+		defer unregisterStream(handle)
+		rc := vllmCompleteStream(v.engine, opts.Prompt, unsafe.Pointer(&sp), tokenCbPtr, handle) // #nosec G103 -- POD in-params
+		runtime.KeepAlive(keep)
+		if rc != vllmOK {
+			xlog.Error("[vllm-cpp] stream failed", "error", vllmLastError())
+		}
+	}()
+	return nil
+}

backend/go/vllm-cpp/chat.go

@@ -0,0 +1,289 @@
+package main
+
+// The rich chat path (AIModelRich): rides the ENGINE's serving pipeline via
+// the ABI v3 chat entry points, exactly like the llama-cpp autoparser flow.
+// The engine applies the model's chat template, decides when a tool call
+// engages (tool_choice auto lowers to a LAZY structural-tag decode
+// constraint), parses tool calls with its streaming-stateful Hermes-style
+// parser, and hands back chat.completion.chunk JSON that this file maps 1:1
+// onto pb.Reply ChatDelta / ToolCallDelta.
+
+import (
+	"encoding/json"
+	"fmt"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// useChatPath reports whether the request should go through the engine-side
+// chat pipeline: the model config asked for backend-side templating and the
+// host handed us structured messages.
+func useChatPath(opts *pb.PredictOptions) bool {
+	return opts.UseTokenizerTemplate && len(opts.Messages) > 0
+}
+
+// chatRequestJSON lowers PredictOptions into one OpenAI chat-completions
+// request object for the ABI (the engine ignores `model`/`stream`).
+func chatRequestJSON(opts *pb.PredictOptions, stream bool) (string, error) {
+	messages := make([]map[string]any, 0, len(opts.Messages))
+	for _, m := range opts.Messages {
+		msg := map[string]any{"role": m.Role, "content": m.Content}
+		if m.ToolCalls != "" {
+			var toolCalls any
+			if err := json.Unmarshal([]byte(m.ToolCalls), &toolCalls); err == nil {
+				msg["tool_calls"] = toolCalls
+			}
+		}
+		// Multi-turn tool identity + prior reasoning: a role="tool" reply
+		// carries the id (and optionally the name) of the assistant call it
+		// answers, and assistant history may carry its reasoning span. The
+		// engine's template context needs all three or a second turn after
+		// tool execution is malformed.
+		if m.ToolCallId != "" {
+			msg["tool_call_id"] = m.ToolCallId
+		}
+		if m.Name != "" {
+			msg["name"] = m.Name
+		}
+		if m.ReasoningContent != "" {
+			msg["reasoning"] = m.ReasoningContent
+		}
+		messages = append(messages, msg)
+	}
+	req := map[string]any{"messages": messages}
+
+	if opts.Tools != "" {
+		var tools any
+		if err := json.Unmarshal([]byte(opts.Tools), &tools); err != nil {
+			return "", fmt.Errorf("vllm-cpp: tools is not valid JSON: %w", err)
+		}
+		req["tools"] = tools
+	}
+	if opts.ToolChoice != "" {
+		var choice any
+		// ToolChoice arrives either as a bare string ("auto"/"required"/"none")
+		// or as the OpenAI named-function JSON object.
+		if err := json.Unmarshal([]byte(opts.ToolChoice), &choice); err == nil {
+			req["tool_choice"] = choice
+		} else {
+			req["tool_choice"] = opts.ToolChoice
+		}
+	}
+
+	req["temperature"] = opts.Temperature
+	if opts.TopP > 0 {
+		req["top_p"] = opts.TopP
+	}
+	if opts.TopK > 0 {
+		req["top_k"] = opts.TopK
+	}
+	if opts.Tokens > 0 {
+		req["max_tokens"] = opts.Tokens
+	}
+	if opts.Seed > 0 {
+		req["seed"] = opts.Seed
+	}
+	if len(opts.StopPrompts) > 0 {
+		req["stop"] = opts.StopPrompts
+	}
+	if opts.PresencePenalty != 0 {
+		req["presence_penalty"] = opts.PresencePenalty
+	}
+	if opts.FrequencyPenalty != 0 {
+		req["frequency_penalty"] = opts.FrequencyPenalty
+	}
+	if stream {
+		// The engine's request parser validates stream_options against the
+		// stream flag at parse time (before the ABI entry point forces it),
+		// so state the intent explicitly.
+		req["stream"] = true
+		req["stream_options"] = map[string]any{"include_usage": true}
+	}
+
+	b, err := json.Marshal(req)
+	if err != nil {
+		return "", err
+	}
+	return string(b), nil
+}
+
+// chatChunk is the subset of an OpenAI chat.completion(.chunk) object the
+// backend consumes.
+type chatChunk struct {
+	Object  string `json:"object"`
+	Choices []struct {
+		Delta        *chatDelta `json:"delta"`   // streaming chunks
+		Message      *chatDelta `json:"message"` // non-stream response
+		FinishReason string     `json:"finish_reason"`
+	} `json:"choices"`
+	Usage *struct {
+		PromptTokens     int32 `json:"prompt_tokens"`
+		CompletionTokens int32 `json:"completion_tokens"`
+	} `json:"usage"`
+}
+
+type chatDelta struct {
+	Content          string `json:"content"`
+	ReasoningContent string `json:"reasoning"`
+	ToolCalls        []struct {
+		Index    int32  `json:"index"`
+		ID       string `json:"id"`
+		Function struct {
+			Name      string `json:"name"`
+			Arguments string `json:"arguments"`
+		} `json:"function"`
+	} `json:"tool_calls"`
+}
+
+// toReply maps one parsed chunk onto a pb.Reply carrying the content bytes
+// plus the structured ChatDelta (the host prefers ChatDeltas when present).
+func (c *chatChunk) toReply() *pb.Reply {
+	reply := &pb.Reply{}
+	if c.Usage != nil {
+		reply.PromptTokens = c.Usage.PromptTokens
+		reply.Tokens = c.Usage.CompletionTokens
+	}
+	if len(c.Choices) == 0 {
+		return reply
+	}
+	d := c.Choices[0].Delta
+	if d == nil {
+		d = c.Choices[0].Message
+	}
+	if d == nil {
+		return reply
+	}
+	delta := &pb.ChatDelta{
+		Content:          d.Content,
+		ReasoningContent: d.ReasoningContent,
+	}
+	for _, tc := range d.ToolCalls {
+		delta.ToolCalls = append(delta.ToolCalls, &pb.ToolCallDelta{
+			Index:     tc.Index,
+			Id:        tc.ID,
+			Name:      tc.Function.Name,
+			Arguments: tc.Function.Arguments,
+		})
+	}
+	reply.Message = []byte(d.Content)
+	if delta.Content != "" || delta.ReasoningContent != "" ||
+		len(delta.ToolCalls) > 0 {
+		reply.ChatDeltas = []*pb.ChatDelta{delta}
+	}
+	return reply
+}
+
+// Chat-stream registry: chunk JSON arrives on the engine's delivery thread
+// through one shared C callback; the integer handle in user_data selects the
+// destination channel (never a Go pointer across the ABI).
+var (
+	chatStreamsMu  sync.Mutex
+	chatStreams    = map[uintptr]chan<- *pb.Reply{}
+	chatStreamNext uintptr
+	chatCbOnce     sync.Once
+	chatCbPtr      uintptr
+)
+
+func chatCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	chatStreamsMu.Lock()
+	results := chatStreams[userData]
+	chatStreamsMu.Unlock()
+	if results == nil {
+		return 0
+	}
+	_ = finished // the terminal call carries an empty delta; nothing to emit.
+	payload := goString(delta)
+	if payload == "" {
+		return 1
+	}
+	var chunk chatChunk
+	if err := json.Unmarshal([]byte(payload), &chunk); err != nil {
+		xlog.Error("[vllm-cpp] unparseable chat chunk", "error", err)
+		return 1
+	}
+	results <- chunk.toReply()
+	return 1
+}
+
+func registerChatStream(results chan<- *pb.Reply) uintptr {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	chatStreamNext++
+	chatStreams[chatStreamNext] = results
+	return chatStreamNext
+}
+
+func unregisterChatStream(h uintptr) {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	delete(chatStreams, h)
+}
+
+// PredictRich implements the non-streaming rich path. Without structured
+// messages it falls back to the plain Predict flow (LocalAI-side templating,
+// optional grammar constraint).
+func (v *VllmCpp) PredictRich(opts *pb.PredictOptions) (*pb.Reply, error) {
+	if !useChatPath(opts) {
+		text, err := v.Predict(opts)
+		if err != nil {
+			return nil, err
+		}
+		return &pb.Reply{Message: []byte(text)}, nil
+	}
+	if v.engine == 0 {
+		return nil, fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, false)
+	if err != nil {
+		return nil, err
+	}
+	var out uintptr
+	rc := vllmChat(v.engine, request, unsafe.Pointer(&out)) // #nosec G103 -- char** out-param
+	if rc != vllmOK {
+		return nil, fmt.Errorf("vllm-cpp: chat failed: %s", vllmLastError())
+	}
+	payload := goString(out)
+	vllmStringFree(out)
+	var response chatChunk
+	if err := json.Unmarshal([]byte(payload), &response); err != nil {
+		return nil, fmt.Errorf("vllm-cpp: unparseable chat response: %w", err)
+	}
+	return response.toReply(), nil
+}
+
+// PredictStreamRich implements the streaming rich path. Contract: send into
+// the channel and return when finished; the host closes the channel.
+func (v *VllmCpp) PredictStreamRich(opts *pb.PredictOptions, results chan<- *pb.Reply) error {
+	if !useChatPath(opts) {
+		// Legacy bridge: run the plain stream and wrap deltas.
+		plain := make(chan string)
+		if err := v.PredictStream(opts, plain); err != nil {
+			return err
+		}
+		for delta := range plain {
+			results <- &pb.Reply{Message: []byte(delta)}
+		}
+		return nil
+	}
+	if v.engine == 0 {
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, true)
+	if err != nil {
+		return err
+	}
+	chatCbOnce.Do(func() {
+		chatCbPtr = purego.NewCallback(chatCallback)
+	})
+	handle := registerChatStream(results)
+	defer unregisterChatStream(handle)
+	rc := vllmChatStream(v.engine, request, chatCbPtr, handle)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: chat stream failed: %s", vllmLastError())
+	}
+	return nil
+}

backend/go/vllm-cpp/chat_test.go

@@ -0,0 +1,158 @@
+package main
+
+import (
+	"encoding/json"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("useChatPath", func() {
+	It("requires tokenizer templating AND structured messages", func() {
+		Expect(useChatPath(&pb.PredictOptions{})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{UseTokenizerTemplate: true})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages:             []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeTrue())
+	})
+})
+
+var _ = Describe("chatRequestJSON", func() {
+	It("lowers messages, tools, tool_choice and sampling onto one request", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "system", Content: "be brief"},
+				{Role: "user", Content: "weather in Rome?"},
+			},
+			Tools:       `[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object"}}}]`,
+			ToolChoice:  "required",
+			Temperature: 0.2,
+			TopP:        0.9,
+			Tokens:      64,
+			StopPrompts: []string{"<|im_end|>"},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req["messages"]).To(HaveLen(2))
+		Expect(req["tools"]).To(HaveLen(1))
+		Expect(req["tool_choice"]).To(Equal("required"))
+		Expect(req["max_tokens"]).To(BeNumerically("==", 64))
+		Expect(req["top_p"]).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(req["stop"]).To(ConsistOf("<|im_end|>"))
+		Expect(req).NotTo(HaveKey("stream_options"))
+	})
+
+	It("parses a named-function tool_choice object and asks for stream usage", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			Messages:   []*pb.Message{{Role: "user", Content: "hi"}},
+			ToolChoice: `{"type":"function","function":{"name":"get_weather"}}`,
+		}, true)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		choice, ok := req["tool_choice"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		Expect(choice["type"]).To(Equal("function"))
+		Expect(req["stream_options"]).To(HaveKeyWithValue("include_usage", true))
+	})
+
+	It("rejects malformed tools JSON", func() {
+		_, err := chatRequestJSON(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+			Tools:    "{not json",
+		}, false)
+		Expect(err).To(HaveOccurred())
+	})
+})
+
+var _ = Describe("chatChunk.toReply", func() {
+	It("maps a streaming tool-call delta onto ChatDelta/ToolCallDelta", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{
+			"tool_calls":[{"index":0,"id":"call_1","function":{"name":"get_weather","arguments":"{\"city\":"}}]
+		},"finish_reason":null}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].ToolCalls).To(HaveLen(1))
+		tc := reply.ChatDeltas[0].ToolCalls[0]
+		Expect(tc.Name).To(Equal("get_weather"))
+		Expect(tc.Id).To(Equal("call_1"))
+		Expect(tc.Arguments).To(Equal(`{"city":`))
+	})
+
+	It("maps a non-stream response message and usage", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion","choices":[{"message":{
+			"role":"assistant","content":"Sunny."},"finish_reason":"stop"}],
+			"usage":{"prompt_tokens":12,"completion_tokens":3}}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(string(reply.Message)).To(Equal("Sunny."))
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].Content).To(Equal("Sunny."))
+		Expect(reply.PromptTokens).To(BeNumerically("==", 12))
+		Expect(reply.Tokens).To(BeNumerically("==", 3))
+	})
+
+	It("emits no ChatDelta for an empty role-only chunk", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":""}}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+		Expect(chunk.toReply().ChatDeltas).To(BeEmpty())
+	})
+})
+
+var _ = Describe("chatRequestJSON multi-turn tool round trip", func() {
+	It("forwards tool_call_id, name, reasoning and assistant tool_calls", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "What is the weather in Rome?"},
+				{
+					Role:             "assistant",
+					ReasoningContent: "need the weather tool",
+					ToolCalls:        `[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Rome\"}"}}]`,
+				},
+				{Role: "tool", ToolCallId: "call_1", Name: "get_weather", Content: `{"temp": 21}`},
+			},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req struct {
+			Messages []map[string]any `json:"messages"`
+		}
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req.Messages).To(HaveLen(3))
+
+		assistant := req.Messages[1]
+		Expect(assistant["reasoning"]).To(Equal("need the weather tool"))
+		calls, ok := assistant["tool_calls"].([]any)
+		Expect(ok).To(BeTrue())
+		Expect(calls).To(HaveLen(1))
+		call := calls[0].(map[string]any)
+		Expect(call["id"]).To(Equal("call_1"))
+
+		tool := req.Messages[2]
+		Expect(tool["role"]).To(Equal("tool"))
+		Expect(tool["tool_call_id"]).To(Equal("call_1"))
+		Expect(tool["name"]).To(Equal("get_weather"))
+		Expect(tool["content"]).To(Equal(`{"temp": 21}`))
+
+		user := req.Messages[0]
+		Expect(user).NotTo(HaveKey("tool_call_id"))
+		Expect(user).NotTo(HaveKey("name"))
+		Expect(user).NotTo(HaveKey("reasoning"))
+	})
+})

backend/go/vllm-cpp/e2e_test.go

@@ -0,0 +1,281 @@
+package main
+
+// E2E over a real model + the built libvllm. Gated on VLLM_CPP_MODEL (a .gguf
+// file or a safetensors model dir): without it the suite skips, so CI runs
+// only the unit specs. test.sh auto-downloads a small GGUF when the gate is
+// unset and the download is allowed.
+
+import (
+	"encoding/json"
+	"os"
+	"runtime"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("vllm-cpp e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	It("refuses a foreign model artefact", func() {
+		other := &VllmCpp{}
+		Expect(other.Load(&pb.ModelOptions{ModelFile: "/nonexistent/foreign.bin"})).NotTo(Succeed())
+	})
+
+	It("completes a prompt (greedy)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "The capital of France is",
+			Tokens:      16,
+			Temperature: 0,
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(BeEmpty())
+	})
+
+	It("is deterministic under greedy decoding", func() {
+		opts := &pb.PredictOptions{Prompt: "1 2 3 4", Tokens: 8, Temperature: 0}
+		a, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		b, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(a).To(Equal(b))
+	})
+
+	It("streams deltas that concatenate to the blocking result", func() {
+		opts := &pb.PredictOptions{Prompt: "Count: one two", Tokens: 12, Temperature: 0}
+		blocking, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan string)
+		Expect(backend.PredictStream(opts, results)).To(Succeed())
+		var sb strings.Builder
+		for delta := range results {
+			sb.WriteString(delta)
+		}
+		Expect(sb.String()).To(Equal(blocking))
+	})
+
+	It("honors stop words", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "a b c d e f",
+			Tokens:      64,
+			Temperature: 0,
+			StopPrompts: []string{"g"},
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(ContainSubstring("g h"))
+	})
+
+	It("constrains generation with a GBNF grammar (tool-call path)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "Answer strictly yes or no: is water wet?",
+			Tokens:      4,
+			Temperature: 0,
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).To(Or(HavePrefix("yes"), HavePrefix("no")))
+	})
+
+	It("serves concurrent streams", func() {
+		const n = 4
+		type result struct {
+			text string
+			err  error
+		}
+		done := make(chan result, n)
+		for i := 0; i < n; i++ {
+			go func() {
+				results := make(chan string)
+				err := backend.PredictStream(&pb.PredictOptions{
+					Prompt: "Hello", Tokens: 8, Temperature: 0,
+				}, results)
+				var sb strings.Builder
+				for delta := range results {
+					sb.WriteString(delta)
+				}
+				done <- result{sb.String(), err}
+			}()
+		}
+		for i := 0; i < n; i++ {
+			r := <-done
+			Expect(r.err).NotTo(HaveOccurred())
+			Expect(r.text).NotTo(BeEmpty())
+		}
+	})
+})
+
+var _ = Describe("vllm-cpp chat e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	weatherTools := `[{"type":"function","function":{"name":"get_weather",` +
+		`"description":"Get the current weather for a city.",` +
+		`"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]`
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping chat e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	chatOpts := func() *pb.PredictOptions {
+		return &pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "Reply with one short sentence: what is the capital of France?"},
+			},
+			Tokens:      512,
+			Temperature: 0,
+		}
+	}
+
+	It("answers a plain chat turn through the engine-side template", func() {
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(BeEmpty())
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+	})
+
+	It("splits reasoning from content engine-side (auto-detected from the template)", func() {
+		// The Qwen3.5 chat template carries <think>, so the engine auto-selects
+		// the think_auto reasoning parser: a markerless answer stays pure
+		// content; if the model DOES think, the block arrives as
+		// ReasoningContent and never leaks into Message.
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(ContainSubstring("<think>"))
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+		for _, d := range reply.ChatDeltas {
+			Expect(d.ReasoningContent).NotTo(ContainSubstring("Paris"),
+				"the user-visible answer must not be swallowed into reasoning")
+		}
+	})
+
+	It("streams chat deltas that concatenate to the blocking answer", func() {
+		blocking, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan *pb.Reply, 64)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(chatOpts(), results)
+			close(results)
+		}()
+		var sb strings.Builder
+		for r := range results {
+			sb.WriteString(string(r.Message))
+		}
+		Expect(<-done).To(Succeed())
+		Expect(sb.String()).To(Equal(string(blocking.Message)))
+	})
+
+	It("emits a parsed tool call when tool_choice requires it", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "What is the weather in Rome right now?"},
+		}
+		opts.Tools = weatherTools
+		opts.ToolChoice = "required"
+		opts.Tokens = 256
+
+		reply, err := backend.PredictRich(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(reply.ChatDeltas).NotTo(BeEmpty())
+		var name, args string
+		for _, d := range reply.ChatDeltas {
+			for _, tc := range d.ToolCalls {
+				if tc.Name != "" {
+					name = tc.Name
+				}
+				args += tc.Arguments
+			}
+		}
+		Expect(name).To(Equal("get_weather"))
+		var parsed map[string]any
+		Expect(json.Unmarshal([]byte(args), &parsed)).To(Succeed(),
+			"tool arguments must be valid JSON: %q", args)
+		Expect(parsed).To(HaveKey("city"))
+	})
+
+	It("lets the engine decide on auto tool choice and streams tool deltas", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "Use the get_weather tool to check the weather in Rome."},
+		}
+		opts.Tools = weatherTools
+		opts.Tokens = 512
+
+		results := make(chan *pb.Reply, 128)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(opts, results)
+			close(results)
+		}()
+		sawToolDelta := false
+		for r := range results {
+			for _, d := range r.ChatDeltas {
+				if len(d.ToolCalls) > 0 {
+					sawToolDelta = true
+				}
+			}
+		}
+		Expect(<-done).To(Succeed())
+		// tool_choice auto is a LAZY constraint: the model may or may not call.
+		// With an explicit instruction the gate model reliably does; treat a
+		// no-call run as a soft signal rather than a hard failure only if the
+		// engine produced SOME output.
+		Expect(sawToolDelta).To(BeTrue(), "expected the engine to engage the tool")
+	})
+})

backend/go/vllm-cpp/govllmcpp.go

@@ -0,0 +1,182 @@
+package main
+
+// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v2).
+//
+// The structs below are hand-mirrored PODs of the C declarations, with
+// explicit padding so the Go layout matches the C layout on linux/darwin
+// amd64+arm64. Struct-by-value entry points (the *_default helpers) are NOT
+// bound - purego's struct-return support is platform-dependent - so the
+// defaults are replicated here and guarded by the vllm_abi_version check at
+// startup: a library whose ABI differs from what these mirrors were written
+// against is refused before any request runs.
+
+import (
+	"fmt"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+)
+
+// abiVersion is the VLLM_ABI_VERSION this file mirrors (vllm.h).
+const abiVersion = 5
+
+// vllm_status (vllm.h).
+const (
+	vllmOK = 0
+)
+
+// cModelParams mirrors vllm_model_params.
+type cModelParams struct {
+	ModelPath           uintptr // const char*
+	TokenizerConfigPath uintptr // const char*
+	BlockSize           int32
+	NumBlocks           int32
+	MaxModelLen         int32
+	MaxNumSeqs          int32
+	ToolParser          uintptr // const char*; NULL = auto-detect (ABI v4)
+	ReasoningParser     uintptr // const char*; NULL = auto-detect (ABI v5)
+}
+
+// cSamplingParams mirrors vllm_sampling_params (ABI v2, structured fields
+// included). Padding matches the C compiler's: the uint64 seed is 8-aligned,
+// and each pointer following an int32 is 8-aligned.
+type cSamplingParams struct {
+	Temperature          float32
+	TopP                 float32
+	TopK                 int32
+	MinP                 float32
+	MaxTokens            int32
+	_                    [4]byte
+	Seed                 uint64
+	HasSeed              int32
+	PresencePenalty      float32
+	FrequencyPenalty     float32
+	RepetitionPenalty    float32
+	MinTokens            int32
+	IgnoreEOS            int32
+	Stop                 uintptr // const char* const*
+	NStop                int32
+	_                    [4]byte
+	StructuredJSON       uintptr // const char*
+	StructuredRegex      uintptr // const char*
+	StructuredChoice     uintptr // const char* const*
+	NStructuredChoice    int32
+	_                    [4]byte
+	StructuredGrammar    uintptr // const char*
+	StructuredJSONObject int32
+	_                    [4]byte
+}
+
+// cCompletion mirrors vllm_completion.
+type cCompletion struct {
+	Text             uintptr // char*, caller-owned
+	FinishReason     uintptr // const char*, library-owned
+	PromptTokens     int32
+	CompletionTokens int32
+}
+
+// defaultSamplingParams mirrors vllm_sampling_params_default().
+func defaultSamplingParams() cSamplingParams {
+	return cSamplingParams{
+		Temperature:       1.0,
+		TopP:              1.0,
+		MaxTokens:         16,
+		RepetitionPenalty: 1.0,
+	}
+}
+
+// defaultModelParams mirrors vllm_model_params_default().
+func defaultModelParams() cModelParams {
+	return cModelParams{
+		BlockSize:  32,
+		NumBlocks:  256,
+		MaxNumSeqs: 8,
+	}
+}
+
+var (
+	vllmEngineLoad     func(params, out unsafe.Pointer) int32
+	vllmEngineFree     func(engine uintptr)
+	vllmComplete       func(engine uintptr, prompt string, params, out unsafe.Pointer) int32
+	vllmCompleteStream func(engine uintptr, prompt string, params unsafe.Pointer, cb uintptr, userData uintptr) int32
+	vllmChat           func(engine uintptr, requestJSON string, out unsafe.Pointer) int32
+	vllmChatStream     func(engine uintptr, requestJSON string, cb uintptr, userData uintptr) int32
+	vllmStringFree     func(s uintptr)
+	vllmCompletionFree func(out unsafe.Pointer)
+	vllmLastError      func() string
+	vllmVersion        func() string
+	vllmABIVersion     func() int32
+)
+
+type libFunc struct {
+	ptr  any
+	name string
+}
+
+// registerLib dlopens libvllm and binds the C ABI, refusing an ABI-version
+// mismatch (the struct mirrors above would be undefined behavior against a
+// different layout).
+func registerLib(libName string) error {
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: dlopen %s: %w", libName, err)
+	}
+	for _, lf := range []libFunc{
+		{&vllmEngineLoad, "vllm_engine_load"},
+		{&vllmEngineFree, "vllm_engine_free"},
+		{&vllmComplete, "vllm_complete"},
+		{&vllmCompleteStream, "vllm_complete_stream"},
+		{&vllmChat, "vllm_chat"},
+		{&vllmChatStream, "vllm_chat_stream"},
+		{&vllmStringFree, "vllm_string_free"},
+		{&vllmCompletionFree, "vllm_completion_free"},
+		{&vllmLastError, "vllm_last_error"},
+		{&vllmVersion, "vllm_version"},
+		{&vllmABIVersion, "vllm_abi_version"},
+	} {
+		purego.RegisterLibFunc(lf.ptr, lib, lf.name)
+	}
+	if v := vllmABIVersion(); v != abiVersion {
+		return fmt.Errorf("vllm-cpp: ABI mismatch: library reports v%d, backend built against v%d", v, abiVersion)
+	}
+	return nil
+}
+
+// cString returns a NUL-terminated byte slice for s. The backing array may be
+// passed to C for the duration of a call (the ABI borrows and copies); keep it
+// alive across the call with runtime.KeepAlive.
+func cString(s string) []byte {
+	b := make([]byte, len(s)+1)
+	copy(b, s)
+	return b
+}
+
+// cStringArray builds a NULL-free array of C-string pointers plus the backing
+// buffers that must stay alive for the duration of the C call.
+func cStringArray(ss []string) (ptrs []uintptr, backing [][]byte) {
+	backing = make([][]byte, 0, len(ss))
+	ptrs = make([]uintptr, 0, len(ss))
+	for _, s := range ss {
+		b := cString(s)
+		backing = append(backing, b)
+		ptrs = append(ptrs, uintptr(unsafe.Pointer(&b[0]))) // #nosec G103 -- borrowed by C for the call only
+	}
+	return ptrs, backing
+}
+
+// goString copies a NUL-terminated C string.
+func goString(p uintptr) string {
+	if p == 0 {
+		return ""
+	}
+	//nolint:govet // C-owned pointer handed over by purego, valid for this call
+	base := unsafe.Pointer(p) // #nosec G103 -- C-owned, copied out immediately
+	n := 0
+	for *(*byte)(unsafe.Add(base, n)) != 0 {
+		n++
+	}
+	if n == 0 {
+		return ""
+	}
+	return string(unsafe.Slice((*byte)(base), n))
+}

backend/go/vllm-cpp/main.go

@@ -0,0 +1,35 @@
+package main
+
+// Note: this is started internally by LocalAI and a server is allocated for each model
+import (
+	"flag"
+	"os"
+	"runtime"
+
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+func main() {
+	libName := os.Getenv("VLLM_CPP_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "./libvllm.dylib"
+		} else {
+			libName = "./libvllm.so"
+		}
+	}
+
+	if err := registerLib(libName); err != nil {
+		panic(err)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &VllmCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/vllm-cpp/options.go

@@ -0,0 +1,54 @@
+package main
+
+// Engine-sizing knobs carried through the model config's free-form
+// `options:` list ("key:value" entries), mirroring how the other in-house
+// backends pass engine-specific settings that have no proto field.
+
+import (
+	"strconv"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+type loadOptions struct {
+	blockSize  int32 // KV block size (tokens/block); engine default 32.
+	numBlocks  int32 // KV blocks to allocate; engine default 256.
+	maxNumSeqs int32 // max concurrent sequences; engine default 8.
+	// Engine-side parser selection (ABI v4/v5). Empty = the engine
+	// auto-detects from the chat template; "none" disables the reasoning
+	// split; unknown names fail the first chat call.
+	toolParser      string
+	reasoningParser string
+}
+
+func parseOptions(opts *pb.ModelOptions) loadOptions {
+	lo := loadOptions{}
+	for _, o := range opts.GetOptions() {
+		k, v, found := strings.Cut(o, ":")
+		if !found {
+			continue
+		}
+		switch strings.TrimSpace(k) {
+		case "block_size":
+			lo.blockSize = parseInt32(v, lo.blockSize)
+		case "num_blocks":
+			lo.numBlocks = parseInt32(v, lo.numBlocks)
+		case "max_num_seqs":
+			lo.maxNumSeqs = parseInt32(v, lo.maxNumSeqs)
+		case "tool_parser":
+			lo.toolParser = strings.TrimSpace(v)
+		case "reasoning_parser":
+			lo.reasoningParser = strings.TrimSpace(v)
+		}
+	}
+	return lo
+}
+
+func parseInt32(s string, fallback int32) int32 {
+	n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 32)
+	if err != nil || n <= 0 {
+		return fallback
+	}
+	return int32(n)
+}

backend/go/vllm-cpp/package.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/vllm-cpp $CURDIR/package/
+cp -fLv $CURDIR/libvllm.so $CURDIR/package/ 2>/dev/null || true
+cp -fLv $CURDIR/libvllm.dylib $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/vllm-cpp/run.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+# vllm.cpp ships ONE portable library per platform (SIMD tiers are per-file
+# with runtime dispatch), so there is no per-CPU variant probing here.
+if [ "$(uname)" = "Darwin" ]; then
+	LIBRARY="$CURDIR/libvllm.dylib"
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	LIBRARY="$CURDIR/libvllm.so"
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+fi
+
+export VLLM_CPP_LIBRARY=$LIBRARY
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LIBRARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/vllm-cpp "$@"
+fi
+
+echo "Using library: $LIBRARY"
+exec "$CURDIR"/vllm-cpp "$@"

backend/go/vllm-cpp/test.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+echo "Running vllm-cpp backend tests..."
+
+# Unit specs always run (struct-mirror layout, option/sampling mapping, load
+# validation). The e2e specs need a real model: set VLLM_CPP_MODEL to a .gguf
+# file or a safetensors model dir to enable them (see e2e_test.go).
+go test -v -timeout 1200s .
+
+echo "All vllm-cpp tests passed."

backend/go/vllm-cpp/vllmcpp_test.go

@@ -0,0 +1,162 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestVllmCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "vllm-cpp suite")
+}
+
+// The Go POD mirrors must match the C struct layout of vllm.h (ABI v2)
+// byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
+// amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
+var _ = Describe("C ABI struct mirrors", func() {
+	It("cModelParams matches vllm_model_params", func() {
+		var p cModelParams
+		Expect(unsafe.Offsetof(p.ModelPath)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TokenizerConfigPath)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.BlockSize)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.NumBlocks)).To(Equal(uintptr(20)))
+		Expect(unsafe.Offsetof(p.MaxModelLen)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.MaxNumSeqs)).To(Equal(uintptr(28)))
+		Expect(unsafe.Offsetof(p.ToolParser)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.ReasoningParser)).To(Equal(uintptr(40)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(48)))
+	})
+
+	It("cSamplingParams matches vllm_sampling_params (ABI v2)", func() {
+		var p cSamplingParams
+		Expect(unsafe.Offsetof(p.Temperature)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TopP)).To(Equal(uintptr(4)))
+		Expect(unsafe.Offsetof(p.TopK)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.MinP)).To(Equal(uintptr(12)))
+		Expect(unsafe.Offsetof(p.MaxTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.Seed)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.HasSeed)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.PresencePenalty)).To(Equal(uintptr(36)))
+		Expect(unsafe.Offsetof(p.FrequencyPenalty)).To(Equal(uintptr(40)))
+		Expect(unsafe.Offsetof(p.RepetitionPenalty)).To(Equal(uintptr(44)))
+		Expect(unsafe.Offsetof(p.MinTokens)).To(Equal(uintptr(48)))
+		Expect(unsafe.Offsetof(p.IgnoreEOS)).To(Equal(uintptr(52)))
+		Expect(unsafe.Offsetof(p.Stop)).To(Equal(uintptr(56)))
+		Expect(unsafe.Offsetof(p.NStop)).To(Equal(uintptr(64)))
+		Expect(unsafe.Offsetof(p.StructuredJSON)).To(Equal(uintptr(72)))
+		Expect(unsafe.Offsetof(p.StructuredRegex)).To(Equal(uintptr(80)))
+		Expect(unsafe.Offsetof(p.StructuredChoice)).To(Equal(uintptr(88)))
+		Expect(unsafe.Offsetof(p.NStructuredChoice)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.StructuredGrammar)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.StructuredJSONObject)).To(Equal(uintptr(112)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
+	})
+
+	It("cCompletion matches vllm_completion", func() {
+		var c cCompletion
+		Expect(unsafe.Offsetof(c.Text)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(c.FinishReason)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(c.PromptTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(c.CompletionTokens)).To(Equal(uintptr(20)))
+		Expect(unsafe.Sizeof(c)).To(Equal(uintptr(24)))
+	})
+})
+
+var _ = Describe("parseOptions", func() {
+	It("extracts the engine sizing knobs", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:64", "num_blocks:512", "max_num_seqs:32", "unknown:ignored",
+		}})
+		Expect(lo.blockSize).To(Equal(int32(64)))
+		Expect(lo.numBlocks).To(Equal(int32(512)))
+		Expect(lo.maxNumSeqs).To(Equal(int32(32)))
+	})
+	It("ignores malformed and non-positive values", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:abc", "num_blocks:-1", "max_num_seqs", "block_size:0",
+		}})
+		Expect(lo).To(Equal(loadOptions{}))
+	})
+})
+
+var _ = Describe("samplingFromPredict", func() {
+	It("maps the sampling fields onto the C POD", func() {
+		sp, _ := samplingFromPredict(&pb.PredictOptions{
+			Temperature:      0.7,
+			TopP:             0.9,
+			TopK:             40,
+			MinP:             0.05,
+			Tokens:           128,
+			Seed:             42,
+			Penalty:          1.1,
+			PresencePenalty:  0.5,
+			FrequencyPenalty: 0.25,
+			IgnoreEOS:        true,
+		})
+		Expect(sp.Temperature).To(BeNumerically("~", 0.7, 1e-6))
+		Expect(sp.TopP).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(sp.TopK).To(Equal(int32(40)))
+		Expect(sp.MinP).To(BeNumerically("~", 0.05, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(128)))
+		Expect(sp.HasSeed).To(Equal(int32(1)))
+		Expect(sp.Seed).To(Equal(uint64(42)))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.1, 1e-6))
+		Expect(sp.PresencePenalty).To(BeNumerically("~", 0.5, 1e-6))
+		Expect(sp.FrequencyPenalty).To(BeNumerically("~", 0.25, 1e-6))
+		Expect(sp.IgnoreEOS).To(Equal(int32(1)))
+	})
+
+	It("keeps the engine defaults for unset fields and stays unseeded", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{})
+		Expect(sp.TopP).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(0))) // unbounded, engine-capped.
+		Expect(sp.HasSeed).To(Equal(int32(0)))
+		Expect(sp.Stop).To(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).To(Equal(uintptr(0)))
+		Expect(keep).To(BeEmpty())
+	})
+
+	It("wires stop prompts and the grammar constraint", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{
+			StopPrompts: []string{"</s>", "\n\n"},
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(sp.NStop).To(Equal(int32(2)))
+		Expect(sp.Stop).NotTo(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).NotTo(Equal(uintptr(0)))
+		Expect(keep).NotTo(BeEmpty())
+	})
+})
+
+var _ = Describe("validModelPath", func() {
+	It("accepts a .gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "model.gguf")
+		Expect(os.WriteFile(p, []byte("GGUF"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).To(Succeed())
+	})
+	It("accepts a directory with config.json", func() {
+		dir := GinkgoT().TempDir()
+		Expect(os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o600)).To(Succeed())
+		Expect(validModelPath(dir)).To(Succeed())
+	})
+	It("refuses a directory without config.json (greedy-probe rule)", func() {
+		Expect(validModelPath(GinkgoT().TempDir())).NotTo(Succeed())
+	})
+	It("refuses a non-gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "weights.bin")
+		Expect(os.WriteFile(p, []byte("x"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).NotTo(Succeed())
+	})
+	It("refuses a missing path", func() {
+		Expect(validModelPath("/nonexistent/model.gguf")).NotTo(Succeed())
+	})
+})

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=080bbbe85230f624f0b52127f1ae1218247989f9
+WHISPER_CPP_VERSION?=97c56f1dc1d1100a9d859c865a20c82d22f823ed
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/index.yaml

@@ -156,6 +156,44 @@
     nvidia-cuda-12: "cuda12-whisper"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-whisper"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-whisper"
+- &vllm-cpp
+  name: "vllm-cpp"
+  alias: "vllm-cpp"
+  license: apache-2.0
+  description: |
+    vllm.cpp is a from-scratch C++20 port of vLLM created and maintained by the LocalAI team.
+    It mirrors vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching,
+    scheduler, sampler) on a portable tensor runtime with no Python, PyTorch or ggml at
+    inference time. It loads Hugging Face safetensors and GGUF checkpoints, supports
+    structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine,
+    and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan.
+  urls:
+    - https://github.com/mudler/vllm.cpp
+  tags:
+    - text-to-text
+    - LLM
+    - CPU
+    - GPU
+    - CUDA
+    - metal
+  capabilities:
+    default: "cpu-vllm-cpp"
+    nvidia: "cuda13-vllm-cpp"
+    metal: "metal-vllm-cpp"
+    vulkan: "vulkan-vllm-cpp"
+    nvidia-cuda-13: "cuda13-vllm-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp"
+- !!merge <<: *vllm-cpp
+  name: "vllm-cpp-development"
+  capabilities:
+    default: "cpu-vllm-cpp-development"
+    nvidia: "cuda13-vllm-cpp-development"
+    metal: "metal-vllm-cpp-development"
+    vulkan: "vulkan-vllm-cpp-development"
+    nvidia-cuda-13: "cuda13-vllm-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp-development"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp-development"
 - &crispasr
   name: "crispasr"
   alias: "crispasr"
@@ -1171,6 +1209,48 @@
     nvidia-l4t: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-moss-tts-cpp-development"
+- &magpiettscpp
+  name: "magpie-tts-cpp"
+  description: |
+    Magpie TTS C++ backend using GGML (magpie-tts.cpp). Native C++
+    text-to-speech for NVIDIA's Magpie TTS Multilingual 357M model (encoder +
+    autoregressive decoder over NanoCodec tokens), running from a single
+    self-contained GGUF (model, codec, tokenizer, G2P dictionaries) with no
+    Python at inference time. 22.05kHz mono output, 5 baked voices (Aria,
+    Jason, John, Leo, Sofia), 9+ languages.
+  urls:
+    - https://github.com/mudler/magpie-tts.cpp
+    - https://huggingface.co/mudler/magpie-tts.cpp-gguf
+  tags:
+    - text-to-speech
+    - tts
+  alias: "magpie-tts-cpp"
+  capabilities:
+    default: "cpu-magpie-tts-cpp"
+    nvidia: "cuda12-magpie-tts-cpp"
+    nvidia-cuda-13: "cuda13-magpie-tts-cpp"
+    nvidia-cuda-12: "cuda12-magpie-tts-cpp"
+    intel: "intel-sycl-f16-magpie-tts-cpp"
+    metal: "metal-magpie-tts-cpp"
+    amd: "rocm-magpie-tts-cpp"
+    vulkan: "vulkan-magpie-tts-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
+- !!merge <<: *magpiettscpp
+  name: "magpie-tts-cpp-development"
+  capabilities:
+    default: "cpu-magpie-tts-cpp-development"
+    nvidia: "cuda12-magpie-tts-cpp-development"
+    nvidia-cuda-13: "cuda13-magpie-tts-cpp-development"
+    nvidia-cuda-12: "cuda12-magpie-tts-cpp-development"
+    intel: "intel-sycl-f16-magpie-tts-cpp-development"
+    metal: "metal-magpie-tts-cpp-development"
+    amd: "rocm-magpie-tts-cpp-development"
+    vulkan: "vulkan-magpie-tts-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
 - &omnivoicecpp
   name: "omnivoice-cpp"
   description: |
@@ -1375,6 +1455,7 @@
   alias: "kokoro"
   name: "kokoro"
   capabilities:
+    default: "cpu-kokoro"
     nvidia: "cuda12-kokoro"
     intel: "intel-kokoro"
     amd: "rocm-kokoro"
@@ -4754,6 +4835,107 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp
+## magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "nvidia-l4t-arm64-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cpu-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "metal-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "metal-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cpu-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda12-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "rocm-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f32-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f16-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "vulkan-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "vulkan-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda12-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "rocm-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f32-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f16-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp
 ## omnivoice-cpp
 - !!merge <<: *omnivoicecpp
   name: "omnivoice-cpp-development"
@@ -5005,11 +5187,22 @@
 - !!merge <<: *kokoro
   name: "kokoro-development"
   capabilities:
+    default: "cpu-kokoro-development"
     nvidia: "cuda12-kokoro-development"
     intel: "intel-kokoro-development"
     amd: "rocm-kokoro-development"
     nvidia-l4t: "nvidia-l4t-kokoro-development"
     metal: "metal-kokoro-development"
+- !!merge <<: *kokoro
+  name: "cpu-kokoro"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-kokoro"
+  mirrors:
+    - localai/localai-backends:latest-cpu-kokoro
+- !!merge <<: *kokoro
+  name: "cpu-kokoro-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-kokoro"
+  mirrors:
+    - localai/localai-backends:master-cpu-kokoro
 - !!merge <<: *kokoro
   name: "cuda12-kokoro-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-kokoro"
@@ -6430,3 +6623,53 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-supertonic"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-supertonic
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-vllm-cpp

backend/python/ace-step/backend.py

@@ -401,16 +401,16 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 "sample_query": request.text or "",
                 "sample_mode": True,
                 "thinking": True,
-                "vocal_language": request.language or request.GetLanguage() or "en",
+                "vocal_language": request.language or "en",
                 "instrumental": request.instrumental if request.HasField("instrumental") else False,
             }
         else:
-            caption = request.caption or request.GetCaption() or request.text
+            caption = request.caption or request.text
             payload = {
                 "prompt": caption,
                 "lyrics": request.lyrics or request.lyrics or "",
                 "thinking": request.think if request.HasField("think") else False,
-                "vocal_language": request.language or request.GetLanguage() or "en",
+                "vocal_language": request.language or "en",
             }
             if request.HasField("bpm"):
                 payload["bpm"] = request.bpm