docs: clarify model configuration precedence

Assisted-by: Codex:gpt-5 [Codex]

.agents/building-and-testing.md

@@ -28,7 +28,6 @@ The core Go suites (`./pkg`, `./core`, plus the in-process integration suite `./
   - **Build tags (`COVERAGE_TAGS`, passed via `GINKGO_TAGS`):** defaults to `debug auth`. The `auth` tag is required to compile the real (sqlite-backed) auth implementation and its ~150 `//go:build auth` tests — without it those files aren't built, the tests don't run, and the gate scores auth against a stub (~3.7% instead of ~38%). If you add new tag-gated tests, extend `COVERAGE_TAGS` or they won't count (and likely won't run in CI at all).
 - `make test-coverage-check` — runs `test-coverage`, then `scripts/coverage-check.sh` fails the build if total coverage is **below** the committed baseline in `coverage-baseline.txt`. The Linux job in `.github/workflows/test.yml` runs this instead of `make test`.
 - `make test-coverage-baseline` — regenerates and overwrites `coverage-baseline.txt` from the current run.
-- `make install-hooks` — sets `core.hooksPath` to the versioned `.githooks/`, whose `pre-commit` runs checks scoped to what's staged: Go changes → `make lint` + `make test-coverage-check`; `core/http/react-ui/` changes → `make test-ui-coverage-check` (Playwright e2e + UI coverage gate). A commit touching neither is skipped; bypass with `git commit --no-verify`. The hook resolves golangci-lint's new-from base to `upstream/master` → `origin/master` → `master`, so it works from a fork clone where `origin/master` is stale (passed to `make lint` via `LINT_NEW_FROM`).
 
 ### React UI coverage
 
@@ -38,12 +37,11 @@ The React UI (`core/http/react-ui/`) has **no component/unit tests** — its onl
 - **Browser:** the flake dev shell ships `chromium` and exports `PLAYWRIGHT_CHROMIUM_PATH`; `playwright.config.js` uses it via `launchOptions.executablePath`, and the Makefile skips `playwright install` when it's set. This avoids Playwright's downloaded browser, which can't resolve system libs (`libglib-2.0`, …) on NixOS. In CI (no `PLAYWRIGHT_CHROMIUM_PATH`) the Makefile falls back to `playwright install --with-deps chromium`.
 - The app is a React SPA, so coverage accumulates across in-app navigation within a test; a full `page.goto`/reload resets it.
 - `.nycrc.json` uses `all: true`, so **every `src/**` file is in the report**, including 0%-coverage ones — that's how you spot features with no test at all (sort the HTML report or `coverage-summary.json` by line% ascending). 
-- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`) and pre-commit on `core/http/react-ui/` changes.
+- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`).
 - **Why it has a tolerance (unlike the strict Go gate):** UI e2e coverage is *non-deterministic*. Specs that assert on state and end while async/lazy render work is still in flight collect those lines only when the render beats the coverage teardown — so the total drifts with machine speed/load (a fast local box reads higher than a slow CI runner), diffusely across many specs. The tolerance absorbs that drift, so set the baseline *below* the slow-CI floor, never to a fast-local `make test-ui-coverage-baseline` number, or CI flaps.
 - **Raising coverage is cheap:** a *render-smoke* spec (navigate to a route, assert its header renders) mounts a lazy page and runs its full render + initial effects, capturing most of its lines in a few lines of test — see `e2e/page-render-smoke.spec.js`. Auth is disabled in the test server (`isAdmin=true`), so `RequireAdmin`/`RequireFeature` routes render without a mock. The most *deterministic* win is removing a race: make a spec `await` a rendered element before ending (see `e2e/agents.spec.js` → AgentCreate) so its lines count every run.
 
 Rules (both gates):
-- **Install the hooks:** `make install-hooks` once per clone so lint + coverage run pre-commit. Don't lean on CI for what the hook catches.
-- **Don't work around the gate:** never `git commit --no-verify`, and never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
+- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
 - If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
 - The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.

.docker/llama-cpp-compile.sh

@@ -28,6 +28,10 @@ if [ -z "${BUILD_TYPE:-}" ]; then
   # variants with it (the host never *selects* SME unless it has it, but every variant must
   # still compile).
   if [ "${TARGETARCH}" = "arm64" ]; then
+    # The prebuilt base inherits default ports.ubuntu.com sources; honor the
+    # APT_*_MIRROR build args here like the from-source path does, so this
+    # apt step survives a mirror outage.
+    sh /LocalAI/.docker/apt-mirror.sh || true
     apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
     export CC=gcc-14 CXX=g++-14
   fi

.githooks/pre-commit

@@ -1,72 +0,0 @@
-#!/usr/bin/env sh
-#
-# LocalAI pre-commit hook. Install it (once per clone) with:
-#
-#     make install-hooks
-#
-# Runs only the checks relevant to what's staged:
-#   - Go files          -> make lint + make test-coverage-check
-#   - core/http/react-ui -> make test-ui-coverage-check (Playwright e2e + gate)
-#   - realtime state machines / specs -> make test-realtime-conformance
-#       (respcoord/**, turncoord/**, or formal-verification/** -- a pure .fizz
-#        spec edit must still re-verify the design, detected separately from Go)
-# A commit touching none of these is skipped entirely (other docs/YAML can't
-# change lint findings, Go coverage, the UI, or the realtime conformance gate).
-#
-# To bypass for a single commit (e.g. a WIP checkpoint): git commit --no-verify
-set -eu
-
-repo_root="$(git rev-parse --show-toplevel)"
-cd "$repo_root"
-
-staged="$(git diff --cached --name-only --diff-filter=ACMRD)"
-
-go_changed=0
-ui_changed=0
-rt_changed=0
-if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
-if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
-if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
-
-if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
-	echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
-	exit 0
-fi
-
-if [ "$go_changed" -eq 1 ]; then
-	# Resolve the ref golangci-lint's new-from-merge-base should compare
-	# against. .golangci.yml pins origin/master, which is correct in CI
-	# (origin == the canonical repo) but wrong from a fork clone, where
-	# origin/master lags behind and lint would report the whole upstream
-	# backlog. Prefer upstream/master, then origin/master, then master.
-	lint_base=""
-	for ref in upstream/master origin/master master; do
-		if git rev-parse --verify --quiet "${ref}^{commit}" >/dev/null 2>&1; then
-			lint_base="$ref"
-			break
-		fi
-	done
-
-	echo "pre-commit ▶ golangci-lint (make lint${lint_base:+, new-from $lint_base})"
-	make lint LINT_NEW_FROM="$lint_base"
-
-	echo "pre-commit ▶ coverage gate (make test-coverage-check) — builds and runs the"
-	echo "             pkg/core suites plus tests/e2e; can take a few minutes."
-	make test-coverage-check
-fi
-
-if [ "$ui_changed" -eq 1 ]; then
-	echo "pre-commit ▶ React UI e2e + coverage gate (make test-ui-coverage-check) —"
-	echo "             rebuilds the UI + ui-test-server, runs the Playwright specs, and"
-	echo "             fails if line coverage regressed; can take a couple of minutes."
-	make test-ui-coverage-check
-fi
-
-if [ "$rt_changed" -eq 1 ]; then
-	echo "pre-commit ▶ realtime state-machine conformance (make test-realtime-conformance) —"
-	echo "             Go transition/rapid tests under -race + FizzBee model check of the"
-	echo "             authoritative specs. Fail-closed: needs FizzBee (make install-fizzbee)."
-	make test-realtime-conformance
-fi
-
-echo "pre-commit ✓ all relevant checks passed"

.github/backend-matrix.yml

@@ -66,6 +66,34 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-kokoro'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'true'
+    backend: "kokoro"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-kokoro'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'true'
+    backend: "kokoro"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -871,6 +899,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1935,6 +1976,32 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2000,6 +2067,32 @@ include:
     backend: "moss-tts-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp'
+    base-image: "ubuntu:24.04"
+    ubuntu-version: '2404'
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2963,6 +3056,19 @@ include:
     dockerfile: "./backend/Dockerfile.privacy-filter"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # Vulkan: base-grpc-vulkan-amd64 carries the SDK. arm64 vulkan is a one-line
   # add once amd64 is proven in CI.
   - build-type: 'vulkan'
@@ -4583,6 +4689,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4597,7 +4717,50 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # vllm-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # omnivoice-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-magpie-tts-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4652,6 +4815,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f32'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f32-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4691,6 +4867,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f16'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f16-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4732,6 +4921,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4774,6 +4977,20 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4814,6 +5031,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2204'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-arm64-magpie-tts-cpp'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2204'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -4853,6 +5083,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-magpie-tts-cpp'
+    base-image: "rocm/dev-ubuntu-24.04:6.4.4"
+    runs-on: 'ubuntu-latest'
+    skip-drivers: 'false'
+    backend: "magpie-tts-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -5774,6 +6017,14 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-moss-tts-cpp"
     build-type: "metal"
     lang: "go"
+  - backend: "magpie-tts-cpp"
+    tag-suffix: "-metal-darwin-arm64-magpie-tts-cpp"
+    build-type: "metal"
+    lang: "go"
+  - backend: "vllm-cpp"
+    tag-suffix: "-metal-darwin-arm64-vllm-cpp"
+    build-type: "metal"
+    lang: "go"
   - backend: "omnivoice-cpp"
     tag-suffix: "-metal-darwin-arm64-omnivoice-cpp"
     build-type: "metal"

.github/workflows/bump_deps.yaml

@@ -50,6 +50,10 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
+          - repository: "mudler/vllm.cpp"
+            variable: "VLLM_CPP_VERSION"
+            branch: "main"
+            file: "backend/go/vllm-cpp/Makefile"
           - repository: "localai-org/moss-transcribe.cpp"
             variable: "MOSS_VERSION"
             branch: "master"
@@ -110,6 +114,10 @@ jobs:
             variable: "VIBEVOICE_CPP_VERSION"
             branch: "master"
             file: "backend/go/vibevoice-cpp/Makefile"
+          - repository: "mudler/magpie-tts.cpp"
+            variable: "MAGPIETTS_CPP_VERSION"
+            branch: "main"
+            file: "backend/go/magpie-tts-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7

.github/workflows/image-pr.yml

@@ -52,7 +52,7 @@
               tag-latest: 'false'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:22.04"
+              base-image: "ubuntu:24.04"
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'hipblas'

.github/workflows/image.yml

@@ -113,7 +113,7 @@
               tag-latest: 'auto'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:22.04"
+              base-image: "ubuntu:24.04"
               skip-drivers: 'false'
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'

.github/workflows/lint.yml

@@ -61,7 +61,7 @@ jobs:
       # The backend matrix path filter fails silently: a miss emits an empty
       # matrix, every job goes green, and the change reaches no image (#10946).
       # Its tests need only node, so they ride along with this job.
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v7
         with:
           node-version: '20'
       - name: run CI script tests

.github/workflows/test-extra.yml

@@ -37,6 +37,7 @@ jobs:
       sglang: ${{ steps.detect.outputs.sglang }}
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
       qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
+      magpie-tts-cpp: ${{ steps.detect.outputs.magpie-tts-cpp }}
       rfdetr-cpp: ${{ steps.detect.outputs.rfdetr-cpp }}
       locate-anything-cpp: ${{ steps.detect.outputs.locate-anything-cpp }}
       vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
@@ -866,6 +867,38 @@ jobs:
       - name: Test qwen3-tts-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/qwen3-tts-cpp test
+  tests-magpie-tts-cpp:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.magpie-tts-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
+      - name: Setup Go
+        uses: actions/setup-go@v5
+      - name: Display Go version
+        run: go version
+      - name: Proto Dependencies
+        run: |
+          # Install protoc
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Build magpie-tts-cpp
+        run: |
+          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp
+      - name: Test magpie-tts-cpp
+        run: |
+          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp test
   # Per-backend smoke for rfdetr-cpp: builds the .so + Go binary and runs
   # `make -C backend/go/rfdetr-cpp test`. test.sh fetches the small (~20 MB)
   # rfdetr-nano-q8_0 GGUF from the published mudler/rfdetr-cpp-nano HF repo

AGENTS.md

@@ -35,7 +35,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 
 ## Quick Reference
 
-- **Git hooks & coverage gates**: Run `make install-hooks` once per clone so the pre-commit lint + coverage gates run. **Never bypass them with `git commit --no-verify`, and never lower a coverage baseline or widen a gate's tolerance to turn a red gate green** — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
+- **Coverage gates**: Never lower a coverage baseline or widen a gate's tolerance to turn a red gate green — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
 - **Logging**: Use `github.com/mudler/xlog` (same API as slog)
 - **Go style**: Prefer `any` over `interface{}`
 - **Comments**: Explain *why*, not *what*

CONTRIBUTING.md

@@ -198,7 +198,6 @@ For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`C
 
 - Prefer modern Go idioms — for example, use `any` instead of `interface{}`.
 - Use [`golangci-lint`](https://golangci-lint.run) to catch common issues before submitting a PR.
-- Run `make install-hooks` once per clone to enable the pre-commit hook: Go changes run `make lint` + the coverage gate (`make test-coverage-check`); `core/http/react-ui/` changes run the Playwright e2e suite (`make test-ui`). Bypass a single commit with `git commit --no-verify`.
 - Use [`github.com/mudler/xlog`](https://github.com/mudler/xlog) for logging (same API as `slog`). Do not use `fmt.Println` or the standard `log` package for operational logging.
 - Use tab indentation for Go files (as defined in `.editorconfig`).
 
@@ -268,7 +267,7 @@ make test-e2e
 
 ### React UI tests and coverage
 
-The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI and pre-commit). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
+The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
 
 **If your change lowers UI coverage, raise it back by adding specs — do not widen the tolerance or hand-lower the baseline.** A *render-smoke* spec (navigate to a page, assert its header is visible) cheaply covers an entire lazy page. See `core/http/react-ui/e2e/page-render-smoke.spec.js` and the full policy in [.agents/building-and-testing.md](.agents/building-and-testing.md#react-ui-coverage).
 

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -103,7 +103,7 @@ COVERAGE_E2E_LABELS?=!real-models
 COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go
 
 
-.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check install-hooks build vendor lint lint-all
+.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
 
 all: help
 
@@ -269,8 +269,7 @@ LINT_EXCLUDE_DIRS_RE=/(backend/go/(piper|silero-vad|llm)|cmd/launcher)(/|$$)
 
 ## Set LINT_NEW_FROM to a git ref to override .golangci.yml's
 ## new-from-merge-base (origin/master). Useful from a fork clone where
-## origin/master is stale relative to the canonical repo — the pre-commit
-## hook passes the resolved upstream ref here so local lint matches CI.
+## origin/master is stale relative to the canonical repo.
 LINT_NEW_FROM?=
 lint:
 	@command -v golangci-lint >/dev/null 2>&1 || { \
@@ -289,17 +288,6 @@ lint-all:
 	}
 	golangci-lint run --new=false --new-from-merge-base= --new-from-rev= $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
 
-########################################################
-## Git hooks
-########################################################
-## Points git at the versioned .githooks/ directory so the pre-commit hook
-## (lint + coverage gate) runs locally. Run once per clone. Undo with:
-## `git config --unset core.hooksPath`. Skip a single commit with
-## `git commit --no-verify`.
-install-hooks:
-	git config core.hooksPath .githooks
-	@echo 'Installed git hooks: core.hooksPath -> .githooks (pre-commit runs lint + test-coverage-check on Go changes)'
-
 ########################################################
 ## E2E AIO tests (uses standard image with pre-configured models)
 ########################################################
@@ -637,6 +625,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/go/locate-anything-cpp test
 	$(MAKE) -C backend/go/depth-anything-cpp test
 	$(MAKE) -C backend/go/supertonic test
+	$(MAKE) -C backend/go/vllm-cpp test
 
 ##
 ## End-to-end gRPC tests that exercise a built backend container image.
@@ -1269,6 +1258,8 @@ BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
 BACKEND_MOSS_TTS_CPP = moss-tts-cpp|golang|.|false|true
+BACKEND_MAGPIE_TTS_CPP = magpie-tts-cpp|golang|.|false|true
+BACKEND_VLLM_CPP = vllm-cpp|golang|.|false|true
 BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
 BACKEND_LOCALVQE = localvqe|golang|.|false|true
@@ -1396,6 +1387,8 @@ $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TTS_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_MAGPIE_TTS_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
@@ -1415,7 +1408,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1450,7 +1443,7 @@ test-ui-e2e: build-ui-test-server
 UI_TEST_WORKERS ?=
 PLAYWRIGHT_WORKERS_FLAG = $(if $(UI_TEST_WORKERS),--workers=$(UI_TEST_WORKERS),)
 
-## Fast Playwright e2e run used by the pre-commit hook on React UI changes.
+## Fast Playwright e2e run for local React UI validation.
 ## Force-rebuilds the (non-instrumented) dist so the suite tests the working
 ## tree — not a stale dist the `react-ui` skip-guard would leave — re-embeds
 ## it into ui-test-server, and runs the specs. Uses the nix-provided browser

README.md

@@ -231,9 +231,11 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 
 | Backend | What it does |
 |---------|-------------|
+| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
 | [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
 | [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
+| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
 | [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
 | [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
 | [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |

backend/Dockerfile.golang

@@ -221,6 +221,33 @@ RUN if [ "${BACKEND}" = "crispasr" ]; then \
     apt-get clean && rm -rf /var/lib/apt/lists/*; \
 fi
 
+# sherpa-onnx links onnxruntime's CUDA execution provider, and
+# libonnxruntime_providers_cuda.so has cuDNN as a hard DT_NEEDED. The
+# onnxruntime GPU tarball does not ship cuDNN itself, so without this the
+# builder has none (the arm64 + CUDA 13 branch above is the only other place
+# that installs it) and package-gpu-libs.sh correctly refuses to produce a
+# package that references cuDNN with no cuDNN available to it.
+#
+# Installed per-backend rather than for every cublas build: the auto-detection
+# in package-gpu-libs.sh bundles only what a package actually references, so
+# the ggml backends would not grow either way, but they would all pay ~1.1 GB
+# of builder layer and registry cache for a library they never call.
+#
+# Runtime package only, no -dev: sherpa-onnx consumes onnxruntime's prebuilt
+# CUDA provider and never compiles against cuDNN headers. libcudnn9-cuda-N
+# carries the dispatcher plus all seven dlopen()ed sublibraries, which is what
+# complete_cudnn_family needs to assemble a whole bundle.
+RUN <<EOT bash
+    if [ "${BACKEND}" = "sherpa-onnx" ] && [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libcudnn9-cuda-${CUDA_MAJOR_VERSION} && \
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
 COPY . /LocalAI
 
 RUN git config --global --add safe.directory /LocalAI

backend/Dockerfile.llama-cpp

@@ -111,6 +111,10 @@ RUN make -BC /LocalAI/backend/cpp/llama-cpp package
 # ============================================================================
 FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
 
+ARG APT_MIRROR
+ENV APT_MIRROR=${APT_MIRROR}
+ARG APT_PORTS_MIRROR
+ENV APT_PORTS_MIRROR=${APT_PORTS_MIRROR}
 ARG BUILD_TYPE
 ENV BUILD_TYPE=${BUILD_TYPE}
 ARG CUDA_DOCKER_ARCH

backend/backend.proto

@@ -181,6 +181,13 @@ message ScoreRequest {
   // PredictOptions.ModelIdentity for the full rationale. Empty means "no
   // identity supplied" and backends MUST skip the check.
   string ModelIdentity = 5;
+  // Byte length of the prompt prefix that stays identical across
+  // repeated scoring calls (e.g. a classifier's option-list system
+  // prompt — everything before the per-turn probe text). Backends that
+  // snapshot state (hybrid/recurrent models cannot rewind otherwise)
+  // use it to place a reuse point exactly at the boundary, so the next
+  // call re-processes only the tokens after it. 0 means unknown.
+  int32 stable_prefix_len = 6;
 }
 
 // CandidateScore is one row in the ScoreResponse, matching by index
@@ -493,6 +500,11 @@ message ModelOptions {
   // Proxy carries the cloud-proxy backend's per-model configuration.
   // Empty for non-proxy backends.
   ProxyOptions Proxy = 74;
+
+  // EnableScore reserves backend resources for the Score RPC. It is derived
+  // from the model's explicit `known_usecases: [score]` declaration so models
+  // that never score retain their ordinary serving footprint.
+  bool EnableScore = 75;
 }
 
 // ProxyOptions configures the cloud-proxy backend. UpstreamURL and
@@ -508,6 +520,12 @@ message ProxyOptions {
   string api_key_file = 5;
   string upstream_model = 6;
   int32 request_timeout_seconds = 7;
+  // cache_prompt enables automatic Anthropic prompt-cache breakpoints
+  // (cache_control: ephemeral) on the stable prefix — system, tools, and
+  // the last message block — when translating to the Anthropic provider.
+  // Cuts input cost on repeated/agentic calls (cache read = 0.1x). Only
+  // meaningful for mode=translate + provider=anthropic; ignored otherwise.
+  bool cache_prompt = 8;
 }
 
 message Result {

backend/cpp/bonsai/Makefile

@@ -41,6 +41,7 @@ define bonsai-build
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build llama.cpp
@@ -77,6 +78,7 @@ bonsai-cpu-all:
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build llama.cpp

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=efdadd41e20134af4f3381e1ed90e96fe4faef6f
+# Upstream pin lives below as DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=efdadd41e20134af4f3381e1ed90e96fe4faef6f
+DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=e5357286c0d433cd4384e82ed7e2b6d655f57087
+IK_LLAMA_VERSION?=b054a8b983827c01aec59d4dc273a27c492c51c4
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=571d0d540df04f25298d0e159e520d9fc62ed121
+LLAMA_VERSION?=1cbfd1988311775425d36c0ce066590f7d3049cf
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/disable-score-task.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Mark a copied gRPC server as targeting a llama.cpp fork that does not carry
+# LocalAI's slot-based Score patches. The RPC remains present in the shared
+# protobuf service, but responds with UNIMPLEMENTED instead of referencing
+# server task types and common_params fields absent from those forks.
+
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+    echo "usage: $0 <grpc-server.cpp>" >&2
+    exit 2
+fi
+
+SRC=$1
+
+if [[ ! -f "$SRC" ]]; then
+    echo "grpc-server.cpp not found at $SRC" >&2
+    exit 2
+fi
+
+if grep -q '^#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK' "$SRC"; then
+    echo "==> $SRC already disables the LocalAI score task, skipping"
+    exit 0
+fi
+
+awk '
+    !done && /^#include/ {
+        print "#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK 1"
+        print "// ^ injected by disable-score-task.sh for an unpatched llama.cpp fork"
+        print ""
+        done = 1
+    }
+    { print }
+    END {
+        if (!done) {
+            print "disable-score-task.sh: no #include anchor found" > "/dev/stderr"
+            exit 1
+        }
+    }
+' "$SRC" > "$SRC.tmp"
+mv "$SRC.tmp" "$SRC"
+
+echo "==> LocalAI score task disabled in $SRC"

backend/cpp/llama-cpp/grpc-server.cpp

@@ -52,6 +52,7 @@
 #include "common.h"
 #include "arg.h"
 #include "chat-auto-parser.h"
+#include "llama_compat.h" // fork-skew switches, generated by prepare.sh
 #include "message_content.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
@@ -151,40 +152,6 @@ static std::string base64_encode_bytes(const unsigned char* data, size_t len) {
 
 bool loaded_model; // TODO: add a mutex for this, but happens only once loading the model
 
-// Score bypasses the slot loop (see the comment on Score below) so it
-// must not run concurrently with any slot-loop RPC. These counters
-// are a defence-in-depth tripwire — ModelConfig.Validate already
-// rejects llama-cpp configs that mix score with chat/completion/
-// embeddings, so a healthy deployment never trips them. seq_cst is
-// load-bearing for the increment-then-check pattern below.
-static std::atomic<int> slot_loop_inflight{0};
-static std::atomic<int> score_inflight{0};
-
-// Increment-then-check, not check-then-increment: two simultaneous
-// racers both observe the other's increment and both abort cleanly.
-// Reversed, both could see zero and proceed.
-struct conflict_guard {
-    std::atomic<int>& self;
-    conflict_guard(const char* rpc, std::atomic<int>& self_, std::atomic<int>& other, const char* other_name)
-        : self(self_) {
-        self.fetch_add(1, std::memory_order_seq_cst);
-        int o = other.load(std::memory_order_seq_cst);
-        if (o > 0) {
-            fprintf(stderr,
-                "FATAL: %s called with %s=%d. The llama-cpp backend cannot "
-                "service Score and slot-loop RPCs concurrently — Score "
-                "bypasses the slot loop and races the llama_context. Bind "
-                "Score-using features to a model dedicated to scoring "
-                "(known_usecases: [score] with no chat/completion/embeddings).\n",
-                rpc, other_name, o);
-            std::abort();
-        }
-    }
-    ~conflict_guard() {
-        self.fetch_sub(1, std::memory_order_seq_cst);
-    }
-};
-
 static std::function<void(int)> shutdown_handler;
 static std::atomic_flag is_terminating = ATOMIC_FLAG_INIT;
 
@@ -613,6 +580,13 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
     // starts with '-'. Applied once after the loop via common_params_parse.
     std::vector<std::string> extra_argv;
 
+    // O_DIRECT intent from the `direct_io` option. Upstream folded
+    // use_mmap/use_mlock/use_direct_io into a single common_params::load_mode
+    // enum (ggml-org/llama.cpp#20834), so the three independent LocalAI settings
+    // can only be reduced to one value once all of them have been read, held
+    // here until the mmap/mlock fields arrive further down.
+    bool want_direct_io = false;
+
     auto add_device_options = [&](const std::string & devices) {
         const std::regex regex{ R"([,]+)" };
         std::sregex_token_iterator it{ devices.begin(), devices.end(), regex, -1 };
@@ -724,6 +698,22 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                     // If conversion fails, keep default value (0)
                 }
             }
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        } else if (!strcmp(optname, "n_rs_seq") || !strcmp(optname, "rs_seq")) {
+            // Recurrent-state rollback snapshots per sequence. Hybrid models
+            // (deltanet/conv layers) cannot rewind their state, so without
+            // snapshots any prompt-cache reuse that needs a rewind — e.g. a
+            // score task whose probe changed under a stable option-list
+            // prefix — falls back to a full re-prefill. Costs recurrent-state
+            // memory x (1 + N) per sequence; unsupported archs clamp to 0.
+            if (optval != NULL) {
+                try {
+                    params.n_rs_seq = std::stoi(optval_str);
+                } catch (const std::exception& e) {
+                    // If conversion fails, keep default value (0)
+                }
+            }
+#endif
         } else if (!strcmp(optname, "slot_prompt_similarity") || !strcmp(optname, "sps")) {
             if (optval != NULL) {
                 try {
@@ -868,9 +858,9 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         // --- O_DIRECT model loading (upstream --direct-io) ---
         } else if (!strcmp(optname, "direct_io") || !strcmp(optname, "use_direct_io")) {
             if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
-                params.use_direct_io = true;
+                want_direct_io = true;
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
-                params.use_direct_io = false;
+                want_direct_io = false;
             }
 
         // --- embedding normalization (upstream --embd-normalize) ---
@@ -1278,8 +1268,28 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
      lora_info.ptr = nullptr;
      params.lora_adapters.push_back(std::move(lora_info));
     }
-    params.use_mlock = request->mlock();
-    params.use_mmap = request->mmap();
+    // LocalAI keeps mmap, mlock and direct-I/O as three independent settings,
+    // while upstream now carries a single load mode. Fold them with the
+    // precedence the separate booleans used to give: direct I/O bypasses the
+    // page cache entirely, mlock implies mmap, and everything off is a plain
+    // buffered read. Forks that branched before ggml-org/llama.cpp#20834 still
+    // expose the booleans; prepare.sh probes the checkout and sets
+    // LOCALAI_LEGACY_LOAD_MODE in the generated llama_compat.h accordingly.
+#if LOCALAI_LEGACY_LOAD_MODE
+    params.use_mlock     = request->mlock();
+    params.use_mmap      = request->mmap();
+    params.use_direct_io = want_direct_io;
+#else
+    if (want_direct_io) {
+        params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO;
+    } else if (request->mlock()) {
+        params.load_mode = LLAMA_LOAD_MODE_MLOCK;
+    } else if (request->mmap()) {
+        params.load_mode = LLAMA_LOAD_MODE_MMAP;
+    } else {
+        params.load_mode = LLAMA_LOAD_MODE_NONE;
+    }
+#endif
 
     if (request->flashattention() == "on" || request->flashattention() == "enabled") {
         params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;
@@ -1364,6 +1374,17 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         }
     }
 
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+    // Score-task suffix forking: reserve seq ids (and recurrent-state cells)
+    // beyond the slots so one scoring call decodes all candidate tails in a
+    // single batch (SERVER_TASK_TYPE_SCORE, patches/). Requires the unified
+    // KV cache — with per-sequence streams the extra ids would shrink every
+    // sequence's context to n_ctx / n_seq_max. Decided after both option
+    // passes so an explicit kv_unified:false wins and disables forking.
+    params.score_enabled = request->enablescore();
+    params.n_seq_score_forks = params.score_enabled && params.kv_unified ? SERVER_SCORE_FORK_SEQS : 0;
+#endif
+
     // Terminate/pad the override vectors only after BOTH the named-option loop
     // and the generic passthrough (common_params_parse above) have pushed their
     // real entries, so back() is the null sentinel the model loader asserts on.
@@ -1450,6 +1471,16 @@ public:
         common_params params;
         params_parse(ctx_server, request, params);
 
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        if (params.score_enabled && !params.kv_unified) {
+            const std::string error_msg =
+                "Score requires the unified KV cache; remove kv_unified:false or remove score from known_usecases";
+            result->set_message(error_msg);
+            result->set_success(false);
+            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, error_msg);
+        }
+#endif
+
         common_init();
         // Ensure debug logs are enabled after common_init() sets up logging
         common_log_set_verbosity_thold(params.verbosity);
@@ -1652,7 +1683,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("PredictStream", slot_loop_inflight, score_inflight, "score_inflight");
         json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
 
@@ -2221,7 +2251,6 @@ public:
          if (params_base.model.path.empty()) {
              return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
          }
-         conflict_guard guard("Predict", slot_loop_inflight, score_inflight, "score_inflight");
          json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
         data["stream"] = false;
@@ -2755,7 +2784,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("Embedding", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
 
         body["stream"] = false;
@@ -2865,7 +2893,6 @@ public:
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "\"documents\" must be a non-empty string array");
         }
 
-        conflict_guard guard("Rerank", slot_loop_inflight, score_inflight, "score_inflight");
 
         // Create and queue the task
         auto rd = ctx_server.get_response_reader();
@@ -2942,37 +2969,16 @@ public:
     // Score returns the model's joint log-probability of each candidate
     // continuation given a shared prompt.
     //
-    // WHY bypass the slot/task queue: upstream server_context exposes
-    // get_llama_context as "main thread only" and the slot loop's
-    // update_slots() owns the context whenever a task is in flight.
-    // No public synchronization primitive is available — so Score is
-    // unsafe to call concurrently with active generation through this
-    // backend. In practice routing-classifier calls happen before the
-    // request is routed to a generation backend, so the model used
-    // for Score is typically idle. Concurrent Score calls are
-    // serialised by a local mutex; KV-cache state is isolated behind
-    // a dedicated sequence ID cleared between candidates.
-    //
-    // A patch to server-context.cpp that adds SERVER_TASK_TYPE_SCORE
-    // and routes scoring through the slot loop would be the correct
-    // long-term fix; tracked as a follow-up.
-    //
-    // Perf TODO (measured: ~450 ms warm for 3 candidates on Arch-
-    // Router-1.5B Q4_K_M + Intel SYCL): the current loop re-decodes
-    // `prompt + candidate` from scratch for every candidate, throwing
-    // away the prompt's KV cache between iterations. A smarter
-    // version would:
-    //   1. Decode just the prompt once into score_seq_id.
-    //   2. Snapshot/cp that sequence (llama_memory_seq_cp) into a
-    //      per-candidate sequence id.
-    //   3. For each candidate, decode only its tokens onto the copy
-    //      (continuing from the saved prompt state), read logits.
-    //   4. llama_memory_seq_rm the copy.
-    // Estimated speedup: 3-candidate calls 450 ms -> ~150-200 ms,
-    // 6-candidate calls 630 ms -> ~220 ms. Single source-file change,
-    // no proto / Go-side changes needed. Worth doing once routing is
-    // wired into the middleware and Score is on the hot path of every
-    // chat request.
+    // Scoring runs as a single SERVER_TASK_TYPE_SCORE task through the
+    // slot loop (added by patches/ on top of upstream server-context), so
+    // it is safe to interleave with generation on the same process and it
+    // reuses any KV prefix the slot already holds across turns. The task
+    // decodes the shared prefix (prompt + longest common candidate token
+    // prefix) once on the slot's sequence; every candidate's unique tail
+    // then rides its own forked sequence and all tails are decoded
+    // together in one batch, so a warm scoring call costs roughly one
+    // forward pass over the new prompt tokens plus one batched pass over
+    // the candidate tails.
     grpc::Status Score(ServerContext* context, const backend::ScoreRequest* request, backend::ScoreResponse* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
@@ -2981,40 +2987,21 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
+#ifdef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        (void) request;
+        (void) response;
+        return grpc::Status(grpc::StatusCode::UNIMPLEMENTED,
+            "Score is unavailable in this llama.cpp fork backend");
+#else
+        if (!params_base.score_enabled) {
+            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION,
+                "Score was not enabled when the model was loaded; add score to known_usecases");
+        }
         if (request->candidates_size() == 0) {
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "candidates must be non-empty");
         }
 
-        // Tripwire against the slot loop. Acquired before score_mutex
-        // so it fires even when this Score is queued behind another.
-        conflict_guard guard("Score", score_inflight, slot_loop_inflight, "slot_loop_inflight");
-
-        // Serialise concurrent Score calls. The slot loop is still
-        // free to race with us — see the class comment above.
-        static std::mutex score_mutex;
-        std::lock_guard<std::mutex> score_lock(score_mutex);
-
-        llama_context * lctx = ctx_server.get_llama_context();
-        if (lctx == nullptr) {
-            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "llama context unavailable (sleeping?)");
-        }
         const llama_vocab * vocab = ctx_server.impl->vocab;
-        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
-        const int32_t n_ctx = llama_n_ctx(lctx);
-        llama_memory_t mem = llama_get_memory(lctx);
-
-        // The KV-cache is sized to seq_to_stream.size() at load
-        // (typically equal to n_slots, often 1). Sequence IDs must
-        // be in [0, n_seq_max), so we can't pick a high-value
-        // "private" ID — we have to share with the slot. We clear
-        // the cache before AND after each candidate to keep
-        // scoring isolated from whatever state the slot held, and
-        // the static mutex above guarantees no other Score call is
-        // racing in the meantime. The slot loop is still free to
-        // race (see comment on this method) — Score must not run
-        // concurrently with generation through this backend.
-        const llama_seq_id score_seq_id = 0;
-        llama_memory_seq_rm(mem, score_seq_id, -1, -1);
 
         // Tokenize the shared prompt once with add_special=true so
         // BOS is prepended when the model requires it. parse_special
@@ -3023,6 +3010,15 @@ public:
         std::vector<llama_token> prompt_tokens = common_tokenize(vocab, prompt, /*add_special=*/true, /*parse_special=*/true);
         const int32_t prompt_len = (int32_t) prompt_tokens.size();
 
+        // Per candidate: full prompt+candidate token list and the
+        // divergence point, kept for piece rendering and empty-candidate
+        // handling after the task comes back.
+        std::vector<std::vector<llama_token>> cand_tokens(request->candidates_size());
+        std::vector<int32_t> cand_divergence(request->candidates_size(), 0);
+
+        // candidates that actually have tokens to score
+        std::vector<int32_t> included;
+
         for (int ci = 0; ci < request->candidates_size(); ci++) {
             const std::string & candidate_text = request->candidates(ci);
 
@@ -3039,9 +3035,135 @@ public:
                     break;
                 }
             }
+            divergence = std::min<int32_t>(divergence, (int32_t) full_tokens.size());
+
             const int32_t cand_len = (int32_t) full_tokens.size() - divergence;
+            if (cand_len > 0 && divergence < 1) {
+                // Need at least one prior token (typically BOS) to
+                // predict the first candidate token's logit. Tokeniser
+                // models without BOS + an empty prompt fall in here.
+                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
+            }
+            if (cand_len > SERVER_SCORE_MAX_CAND_TOKENS) {
+                // The context reserves logits outputs for at most this many
+                // candidate tokens per slot (server_n_outputs_max).
+                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                    "Score: candidate " + std::to_string(ci) + " is " + std::to_string(cand_len) +
+                    " tokens; the maximum is " + std::to_string(SERVER_SCORE_MAX_CAND_TOKENS));
+            }
+
+            cand_divergence[ci] = divergence;
+            cand_tokens[ci] = std::move(full_tokens);
+
+            if (cand_len > 0) {
+                included.push_back(ci);
+            }
+        }
+
+        auto rd = ctx_server.get_response_reader();
+        bool posted_task = false;
+
+        // Shared prefix bounds, needed again when stitching the results:
+        // n_shared is the longest common token prefix of the scored
+        // candidates, n_score_prompt the earliest divergence from the
+        // bare prompt (scored logprobs start there).
+        int32_t n_shared       = 0;
+        int32_t n_score_prompt = 0;
+
+        if (!included.empty()) {
+            const auto & first = cand_tokens[included[0]];
+
+            // the common prefix of a set is the shortest common prefix
+            // against any fixed member
+            n_shared = (int32_t) first.size();
+            for (int32_t ci : included) {
+                const auto & ft = cand_tokens[ci];
+                const int32_t lim = std::min<int32_t>(n_shared, (int32_t) ft.size());
+                int32_t match = 0;
+                while (match < lim && ft[match] == first[match]) {
+                    match++;
+                }
+                n_shared = match;
+            }
+
+            // below its divergence every candidate equals the prompt
+            // tokens, so n_score_prompt <= n_shared always holds
+            n_score_prompt = cand_divergence[included[0]];
+            for (int32_t ci : included) {
+                n_score_prompt = std::min(n_score_prompt, cand_divergence[ci]);
+            }
+
+            // Map the caller's stable-prefix byte length onto a token
+            // index: the last prompt token that ends at or before the
+            // boundary. A checkpoint forced there survives every future
+            // probe under the same option list, which is what keeps
+            // repeat scoring cheap on models that cannot rewind state.
+            int32_t n_stable_prompt = 0;
+            if (request->stable_prefix_len() > 0) {
+                size_t consumed = 0;
+                for (int32_t ti = 0; ti < n_score_prompt; ti++) {
+                    const size_t piece_len = common_token_to_piece(vocab, prompt_tokens[ti]).size();
+                    // BOS and other zero-length specials consume no prompt bytes
+                    if (consumed + piece_len > (size_t) request->stable_prefix_len()) {
+                        break;
+                    }
+                    consumed += piece_len;
+                    n_stable_prompt = ti + 1;
+                }
+            }
+
+            server_task task(SERVER_TASK_TYPE_SCORE);
+            task.id             = rd.queue_tasks.get_new_id();
+            task.index          = 0;
+            task.tokens         = server_tokens(llama_tokens(first.begin(), first.begin() + n_shared), false);
+            task.n_score_prompt = n_score_prompt;
+            task.n_stable_prompt = n_stable_prompt;
+            task.score_suffixes.reserve(included.size());
+            for (int32_t ci : included) {
+                task.score_suffixes.emplace_back(cand_tokens[ci].begin() + n_shared, cand_tokens[ci].end());
+            }
+
+            std::vector<server_task> tasks;
+            tasks.push_back(std::move(task));
+            rd.post_tasks(std::move(tasks));
+            posted_task = true;
+        }
+
+        // Wait for the shared-prefix and per-candidate logprob vectors.
+        // Context overflow and decode failures surface here as task errors.
+        std::vector<float> shared_logprobs;
+        std::vector<std::vector<float>> cand_logprobs;
+        if (posted_task) {
+            auto all_results = rd.wait_for_all([&context]() { return context->IsCancelled(); });
+            if (all_results.is_terminated) {
+                return grpc::Status(grpc::StatusCode::CANCELLED, "Request cancelled by client");
+            }
+            if (all_results.error) {
+                return grpc::Status(grpc::StatusCode::INTERNAL,
+                    all_results.error->to_json().value("message", "Error in receiving score results"));
+            }
+            if (all_results.results.size() != 1) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "expected a single score result");
+            }
+            auto * score_res = dynamic_cast<server_task_result_score*>(all_results.results[0].get());
+            if (score_res == nullptr) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "unexpected result type for score task");
+            }
+            shared_logprobs = std::move(score_res->shared_logprobs);
+            cand_logprobs   = std::move(score_res->cand_logprobs);
+            if (cand_logprobs.size() != included.size()) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "score result candidate count mismatch");
+            }
+        }
+
+        size_t inc = 0; // index into included / cand_logprobs
+        for (int ci = 0; ci < request->candidates_size(); ci++) {
+            const int32_t divergence = cand_divergence[ci];
+            const int32_t cand_len   = (int32_t) cand_tokens[ci].size() - divergence;
+
             backend::CandidateScore * cs = response->add_candidates();
-            cs->set_num_tokens(cand_len);
+            cs->set_num_tokens(cand_len > 0 ? cand_len : 0);
             if (cand_len <= 0) {
                 cs->set_log_prob(0.0);
                 if (request->length_normalize()) {
@@ -3049,101 +3171,57 @@ public:
                 }
                 continue;
             }
-            if (divergence < 1) {
-                // Need at least one prior token (typically BOS) to
-                // predict the first candidate token's logit. Tokeniser
-                // models without BOS + an empty prompt fall in here.
-                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
-                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
+
+            // Stitch the candidate's scored logprobs back together: the
+            // stretch inside the shared prefix (identical for every
+            // candidate) followed by its forked suffix. Suffix entries
+            // before the candidate's own divergence are prompt tokens
+            // decoded only as context — not scored.
+            std::vector<float> lp;
+            lp.reserve(cand_len);
+            for (int32_t t = divergence; t < n_shared; t++) {
+                const int32_t idx = t - n_score_prompt;
+                if (idx < 0 || idx >= (int32_t) shared_logprobs.size()) {
+                    return grpc::Status(grpc::StatusCode::INTERNAL,
+                        "Score: shared logprob index out of range for candidate " + std::to_string(ci));
+                }
+                lp.push_back(shared_logprobs[idx]);
             }
-            if ((int32_t) full_tokens.size() > n_ctx) {
-                return grpc::Status(grpc::StatusCode::OUT_OF_RANGE,
-                    "Score: prompt+candidate exceeds context size (got " +
-                    std::to_string(full_tokens.size()) + ", n_ctx=" + std::to_string(n_ctx) + ")");
+            const auto & sfx_lp = cand_logprobs[inc++];
+            for (int32_t j = std::max(0, divergence - n_shared); j < (int32_t) sfx_lp.size(); j++) {
+                lp.push_back(sfx_lp[j]);
             }
 
-            // Build a batch covering the entire prompt+candidate. We
-            // need logits at (divergence-1) onward — those are the
-            // predictions for each candidate token.
-            llama_batch batch = llama_batch_init((int32_t) full_tokens.size(), 0, 1);
-            for (int32_t i = 0; i < (int32_t) full_tokens.size(); i++) {
-                batch.token[i]    = full_tokens[i];
-                batch.pos[i]      = i;
-                batch.n_seq_id[i] = 1;
-                batch.seq_id[i][0] = score_seq_id;
-                // logits[i] is "do we want the prediction *for the
-                // next token*, computed from this position?"
-                // We want predictions for candidate tokens at
-                // positions divergence .. full_tokens.size()-1, which
-                // come from logits at positions (divergence-1) ..
-                // (full_tokens.size()-2).
-                bool need_logit = (i >= divergence - 1) && (i < (int32_t) full_tokens.size() - 1);
-                batch.logits[i] = need_logit ? 1 : 0;
-            }
-            batch.n_tokens = (int32_t) full_tokens.size();
-
-            // Decode the batch. If decode fails (e.g. KV slot
-            // exhaustion), surface as INTERNAL — the caller will
-            // typically fall back to a sampling-based classifier.
-            int decode_err = llama_decode(lctx, batch);
-            if (decode_err != 0) {
-                llama_batch_free(batch);
-                llama_memory_seq_rm(mem, score_seq_id, -1, -1);
+            if ((int32_t) lp.size() != cand_len) {
                 return grpc::Status(grpc::StatusCode::INTERNAL,
-                    "llama_decode failed during Score: " + std::to_string(decode_err));
+                    "Score: result for candidate " + std::to_string(ci) + " is missing token logprobs");
             }
 
-            // Sum log-probabilities of the actual candidate tokens.
             double total_log_prob = 0.0;
             for (int32_t k = 0; k < cand_len; k++) {
-                // The k-th candidate token sits at full_tokens index
-                // (divergence + k). Its predicting logit is at batch
-                // position (divergence + k - 1).
-                int32_t logit_pos = divergence + k - 1;
-                const float * logits = llama_get_logits_ith(lctx, logit_pos);
-                if (logits == nullptr) {
-                    llama_batch_free(batch);
-                    llama_memory_seq_rm(mem, score_seq_id, -1, -1);
+                const float token_log_prob = lp[k];
+                if (std::isnan(token_log_prob)) {
                     return grpc::Status(grpc::StatusCode::INTERNAL,
-                        "llama_get_logits_ith returned null at position " + std::to_string(logit_pos));
+                        "Score: incomplete result for candidate " + std::to_string(ci) +
+                        " at token " + std::to_string(k));
                 }
-                llama_token target_token = full_tokens[divergence + k];
-
-                // Compute log_softmax(logits)[target_token] with the
-                // max-subtraction stability trick.
-                float max_logit = logits[0];
-                for (int32_t v = 1; v < n_vocab; v++) {
-                    if (logits[v] > max_logit) max_logit = logits[v];
-                }
-                double sum_exp = 0.0;
-                for (int32_t v = 0; v < n_vocab; v++) {
-                    sum_exp += std::exp((double)(logits[v] - max_logit));
-                }
-                double token_log_prob = (double)(logits[target_token] - max_logit) - std::log(sum_exp);
-                total_log_prob += token_log_prob;
+                total_log_prob += (double) token_log_prob;
 
                 if (request->include_token_logprobs()) {
                     backend::TokenLogProb * tlp = cs->add_tokens();
-                    std::string piece = common_token_to_piece(lctx, target_token);
-                    tlp->set_token(piece);
+                    tlp->set_token(common_token_to_piece(vocab, cand_tokens[ci][divergence + k]));
                     tlp->set_log_prob(token_log_prob);
                 }
             }
 
             cs->set_log_prob(total_log_prob);
-            if (request->length_normalize() && cand_len > 0) {
+            if (request->length_normalize()) {
                 cs->set_length_normalized_log_prob(total_log_prob / (double) cand_len);
             }
-
-            llama_batch_free(batch);
-            // Drop this candidate's KV-cache contribution so the next
-            // candidate starts from a clean state. Without this, the
-            // next decode would conflict at positions 0..N-1 for our
-            // sequence ID.
-            llama_memory_seq_rm(mem, score_seq_id, -1, -1);
         }
 
         return grpc::Status::OK;
+#endif
     }
 
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
@@ -3154,7 +3232,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("TokenizeString", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
         body["stream"] = false;
 
@@ -3176,7 +3253,6 @@ public:
 
     grpc::Status GetMetrics(ServerContext* /*context*/, const backend::MetricsRequest* /*request*/, backend::MetricsResponse* response) override {
 
-        conflict_guard guard("GetMetrics", slot_loop_inflight, score_inflight, "score_inflight");
 
 // request slots data using task queue
         auto rd = ctx_server.get_response_reader();

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-chat-parser.patch

@@ -0,0 +1,225 @@
+# MiniMax-M3 chat-template parser, vendored from upstream llama.cpp PR #24523.
+#
+# Upstream has since merged the *model* half of #24523 (LLM_ARCH_MINIMAX_M3,
+# src/models/minimax-m3.cpp, the gguf-py constants and conversion/minimax.py), so
+# only the chat half is carried here: M3's namespace token "]<]minimax[>[" collides
+# with the autoparser's markup delimiters, so common/chat.cpp needs a dedicated
+# template detection + PEG parser that upstream does not have yet.
+#
+# Rebased against LLAMA_VERSION 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3, which also
+# renamed common_chat_params::thinking_end_tag to thinking_end_tags (a vector).
+# LLAMA_VERSION is auto-bumped nightly; if a bump rejects this patch, re-vendor from
+# #24523 — or, once the chat half merges upstream, delete this file.
+# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
+diff --git a/common/chat.cpp b/common/chat.cpp
+index 7a6e7238c..2dd015a2e 100644
+--- a/common/chat.cpp
++++ b/common/chat.cpp
+@@ -2121,6 +2121,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
+     return data;
+ }
+ 
++static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
++                                                             const autoparser::generation_params & inputs) {
++    common_chat_params data;
++
++    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
++    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
++    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
++    data.supports_thinking  = true;
++    data.thinking_start_tag = "<mm:think>";
++    data.thinking_end_tags  = {"</mm:think>"};
++
++    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
++    // params use the parameter name as the tag (<file_path>...</file_path>).
++    const std::string NS           = "]<]minimax[>[";
++    const std::string THINK_START  = "<mm:think>";
++    const std::string THINK_END    = "</mm:think>";
++    const std::string FC_START     = NS + "<tool_call>";
++    const std::string FC_END       = NS + "</tool_call>";
++    const std::string INVOKE_END   = NS + "</invoke>";
++
++    data.preserved_tokens   = {
++        NS,
++        "<tool_call>",
++        "</tool_call>",
++        THINK_START,
++        THINK_END,
++    };
++
++    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
++    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
++    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
++    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
++
++    const std::string GEN_PROMPT = data.generation_prompt;
++
++    if (inputs.has_continuation()) {
++        const auto & msg = inputs.continue_msg;
++
++        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
++        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
++            data.generation_prompt += THINK_END + msg.render_content();
++        }
++
++        data.prompt += data.generation_prompt;
++    }
++
++    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
++        auto generation_prompt = p.literal(GEN_PROMPT);
++        auto end = p.end();
++
++        auto reasoning = p.eps();
++        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
++        if (extract_reasoning && inputs.enable_thinking) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
++        } else if (extract_reasoning) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
++        }
++
++        if (has_response_format) {
++            auto response_format = p.rule("response-format",
++                p.literal("```json") + p.space() +
++                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
++                p.space() + p.literal("```"));
++            return generation_prompt + reasoning + response_format + end;
++        }
++
++        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
++            return generation_prompt + reasoning + p.content(p.rest()) + end;
++        }
++
++        auto tool_choice = p.choice();
++        foreach_function(inputs.tools, [&](const json & tool) {
++            const auto & function = tool.at("function");
++            std::string  name     = function.at("name");
++            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
++            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
++
++            std::set<std::string> required;
++            if (params.contains("required")) {
++                params.at("required").get_to(required);
++            }
++
++            auto schema_info = common_schema_info();
++            schema_info.resolve_refs(params);
++
++            std::vector<common_peg_parser> required_parsers;
++            std::vector<common_peg_parser> optional_parsers;
++            for (const auto & [param_name, param_schema] : props.items()) {
++                bool is_required = required.find(param_name) != required.end();
++                bool is_string   = schema_info.resolves_to_string(param_schema);
++
++                const std::string p_close = NS + "</" + param_name + ">";
++
++                auto arg = p.tool_arg(
++                    p.tool_arg_open(
++                        p.literal(NS + "<") +
++                        p.tool_arg_name(p.literal(param_name)) +
++                        p.literal(">")) +
++                    (is_string
++                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
++                                p.tool_arg_close(p.literal(p_close)), p_close)
++                         : p.tool_arg_json_value(p.schema(p.json(),
++                                                          "tool-" + name + "-arg-" + param_name + "-schema",
++                                                          param_schema, false)) +
++                           p.tool_arg_close(p.literal(p_close))));
++
++                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
++                if (is_required) {
++                    required_parsers.push_back(named_arg);
++                } else {
++                    optional_parsers.push_back(named_arg);
++                }
++            }
++
++            common_peg_parser args_seq = p.eps();
++            for (size_t i = 0; i < required_parsers.size(); i++) {
++                if (i > 0) {
++                    args_seq = args_seq + p.space();
++                }
++                args_seq = args_seq + required_parsers[i];
++            }
++
++            if (!optional_parsers.empty()) {
++                common_peg_parser any_opt = p.choice();
++                for (const auto & opt : optional_parsers) {
++                    any_opt |= opt;
++                }
++                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
++            }
++
++            common_peg_parser invoke_body = args_seq;
++            auto func_parser = p.tool(
++                p.tool_open(p.literal(NS + "<invoke name=\"") +
++                            p.tool_name(p.literal(name)) + p.literal("\">")) +
++                p.space() + invoke_body + p.space() +
++                p.tool_close(p.literal(INVOKE_END)));
++
++            tool_choice |= p.rule("tool-" + name, func_parser);
++        });
++
++        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
++
++        common_peg_parser tool_calls = p.eps();
++        if (inputs.parallel_tool_calls) {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice +
++                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
++        } else {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
++        }
++
++        if (!require_tools) {
++            tool_calls = p.optional(tool_calls);
++        }
++
++        auto content_before_tools = p.content(p.until(FC_START));
++        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
++    });
++
++    data.parser = parser.save();
++
++    if (include_grammar) {
++        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
++        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
++            foreach_function(inputs.tools, [&](const json & tool) {
++                const auto & function = tool.at("function");
++                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
++                builder.resolve_refs(schema);
++            });
++            if (has_response_format) {
++                auto schema = inputs.json_schema;
++                builder.resolve_refs(schema);
++            }
++            parser.build_grammar(builder, data.grammar_lazy);
++        });
++
++        data.grammar_triggers = {
++            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
++        };
++    }
++
++    return data;
++}
++
+ // Cohere2 MoE (a.k.a. "North Code") parser.
+ //
+ // The assistant turn is fully marker-wrapped:
+@@ -2707,6 +2892,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
+         return common_chat_params_init_gigachat_v3(tmpl, params);
+     }
+ 
++    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
++    // markup delimiters, so detect the template and use a dedicated parser.
++    if (src.find("]<]minimax[>[") != std::string::npos &&
++        src.find("<tool_call>") != std::string::npos &&
++        src.find("<invoke name=") != std::string::npos) {
++        LOG_DBG("Using specialized template: MiniMax-M3\n");
++        return common_chat_params_init_minimax_m3(tmpl, params);
++    }
++
+     // DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
+     // The template source contains the token as a variable assignment, not as a literal in markup.
+     // V3.2 names the tool call block "function_calls", V4 names it "tool_calls".

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch

@@ -1,814 +0,0 @@
-# Vendored from upstream llama.cpp PR #24523 (Preliminary MiniMax-M3 support).
-# Rebased against LLAMA_VERSION 00fa7cb284cbf133fc426733bd64238a3588a33e (also applies cleanly
-# to the later pin 505b1ed15ca80e2a19f12ff4ac365e40fb374053). LLAMA_VERSION is auto-bumped
-# nightly; if a bump rejects this patch, re-vendor from #24523 — or, once #24523 merges
-# upstream, delete this file and bump LLAMA_VERSION normally.
-# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
-diff --git a/common/chat.cpp b/common/chat.cpp
-index 22d2ee4..440be9a 100644
---- a/common/chat.cpp
-+++ b/common/chat.cpp
-@@ -2035,6 +2035,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
-     return data;
- }
- 
-+static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
-+                                                             const autoparser::generation_params & inputs) {
-+    common_chat_params data;
-+
-+    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
-+    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
-+    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
-+    data.supports_thinking  = true;
-+    data.thinking_start_tag = "<mm:think>";
-+    data.thinking_end_tag   = "</mm:think>";
-+
-+    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
-+    // params use the parameter name as the tag (<file_path>...</file_path>).
-+    const std::string NS           = "]<]minimax[>[";
-+    const std::string THINK_START  = "<mm:think>";
-+    const std::string THINK_END    = "</mm:think>";
-+    const std::string FC_START     = NS + "<tool_call>";
-+    const std::string FC_END       = NS + "</tool_call>";
-+    const std::string INVOKE_END   = NS + "</invoke>";
-+
-+    data.preserved_tokens   = {
-+        NS,
-+        "<tool_call>",
-+        "</tool_call>",
-+        THINK_START,
-+        THINK_END,
-+    };
-+
-+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
-+    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
-+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
-+
-+    const std::string GEN_PROMPT = data.generation_prompt;
-+
-+    if (inputs.has_continuation()) {
-+        const auto & msg = inputs.continue_msg;
-+
-+        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
-+        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
-+            data.generation_prompt += THINK_END + msg.render_content();
-+        }
-+
-+        data.prompt += data.generation_prompt;
-+    }
-+
-+    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
-+        auto generation_prompt = p.literal(GEN_PROMPT);
-+        auto end = p.end();
-+
-+        auto reasoning = p.eps();
-+        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
-+        if (extract_reasoning && inputs.enable_thinking) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
-+        } else if (extract_reasoning) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
-+        }
-+
-+        if (has_response_format) {
-+            auto response_format = p.rule("response-format",
-+                p.literal("```json") + p.space() +
-+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
-+                p.space() + p.literal("```"));
-+            return generation_prompt + reasoning + response_format + end;
-+        }
-+
-+        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
-+            return generation_prompt + reasoning + p.content(p.rest()) + end;
-+        }
-+
-+        auto tool_choice = p.choice();
-+        foreach_function(inputs.tools, [&](const json & tool) {
-+            const auto & function = tool.at("function");
-+            std::string  name     = function.at("name");
-+            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
-+            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
-+
-+            std::set<std::string> required;
-+            if (params.contains("required")) {
-+                params.at("required").get_to(required);
-+            }
-+
-+            auto schema_info = common_schema_info();
-+            schema_info.resolve_refs(params);
-+
-+            std::vector<common_peg_parser> required_parsers;
-+            std::vector<common_peg_parser> optional_parsers;
-+            for (const auto & [param_name, param_schema] : props.items()) {
-+                bool is_required = required.find(param_name) != required.end();
-+                bool is_string   = schema_info.resolves_to_string(param_schema);
-+
-+                const std::string p_close = NS + "</" + param_name + ">";
-+
-+                auto arg = p.tool_arg(
-+                    p.tool_arg_open(
-+                        p.literal(NS + "<") +
-+                        p.tool_arg_name(p.literal(param_name)) +
-+                        p.literal(">")) +
-+                    (is_string
-+                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
-+                                p.tool_arg_close(p.literal(p_close)), p_close)
-+                         : p.tool_arg_json_value(p.schema(p.json(),
-+                                                          "tool-" + name + "-arg-" + param_name + "-schema",
-+                                                          param_schema, false)) +
-+                           p.tool_arg_close(p.literal(p_close))));
-+
-+                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
-+                if (is_required) {
-+                    required_parsers.push_back(named_arg);
-+                } else {
-+                    optional_parsers.push_back(named_arg);
-+                }
-+            }
-+
-+            common_peg_parser args_seq = p.eps();
-+            for (size_t i = 0; i < required_parsers.size(); i++) {
-+                if (i > 0) {
-+                    args_seq = args_seq + p.space();
-+                }
-+                args_seq = args_seq + required_parsers[i];
-+            }
-+
-+            if (!optional_parsers.empty()) {
-+                common_peg_parser any_opt = p.choice();
-+                for (const auto & opt : optional_parsers) {
-+                    any_opt |= opt;
-+                }
-+                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
-+            }
-+
-+            common_peg_parser invoke_body = args_seq;
-+            auto func_parser = p.tool(
-+                p.tool_open(p.literal(NS + "<invoke name=\"") +
-+                            p.tool_name(p.literal(name)) + p.literal("\">")) +
-+                p.space() + invoke_body + p.space() +
-+                p.tool_close(p.literal(INVOKE_END)));
-+
-+            tool_choice |= p.rule("tool-" + name, func_parser);
-+        });
-+
-+        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
-+
-+        common_peg_parser tool_calls = p.eps();
-+        if (inputs.parallel_tool_calls) {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice +
-+                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
-+        } else {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
-+        }
-+
-+        if (!require_tools) {
-+            tool_calls = p.optional(tool_calls);
-+        }
-+
-+        auto content_before_tools = p.content(p.until(FC_START));
-+        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
-+    });
-+
-+    data.parser = parser.save();
-+
-+    if (include_grammar) {
-+        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
-+        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
-+            foreach_function(inputs.tools, [&](const json & tool) {
-+                const auto & function = tool.at("function");
-+                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
-+                builder.resolve_refs(schema);
-+            });
-+            if (has_response_format) {
-+                auto schema = inputs.json_schema;
-+                builder.resolve_refs(schema);
-+            }
-+            parser.build_grammar(builder, data.grammar_lazy);
-+        });
-+
-+        data.grammar_triggers = {
-+            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
-+        };
-+    }
-+
-+    return data;
-+}
-+
- // Cohere2 MoE (a.k.a. "North Code") parser.
- //
- // The assistant turn is fully marker-wrapped:
-@@ -2612,6 +2797,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
-         return common_chat_params_init_gigachat_v3(tmpl, params);
-     }
- 
-+    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
-+    // markup delimiters, so detect the template and use a dedicated parser.
-+    if (src.find("]<]minimax[>[") != std::string::npos &&
-+        src.find("<tool_call>") != std::string::npos &&
-+        src.find("<invoke name=") != std::string::npos) {
-+        LOG_DBG("Using specialized template: MiniMax-M3\n");
-+        return common_chat_params_init_minimax_m3(tmpl, params);
-+    }
-+
-     // DeepSeek V3.2 format detection: template defines dsml_token and uses it for tool calls.
-     // The template source contains the token as a variable assignment, not as a literal in markup.
-     if (src.find("dsml_token") != std::string::npos &&
-diff --git a/conversion/__init__.py b/conversion/__init__.py
-index 02ea638..71de528 100644
---- a/conversion/__init__.py
-+++ b/conversion/__init__.py
-@@ -155,6 +155,8 @@ TEXT_MODEL_MAP: dict[str, str] = {
-     "MiniCPMForCausalLM": "minicpm",
-     "MiniCPMV4_6ForConditionalGeneration": "minicpm",
-     "MiniMaxM2ForCausalLM": "minimax",
-+    "MiniMaxM3SparseForCausalLM": "minimax",
-+    "MiniMaxM3SparseForConditionalGeneration": "minimax",
-     "Ministral3ForCausalLM": "mistral3",
-     "Mistral3ForConditionalGeneration": "mistral3",
-     "MistralForCausalLM": "llama",
-diff --git a/conversion/base.py b/conversion/base.py
-index 0421aa4..224481a 100644
---- a/conversion/base.py
-+++ b/conversion/base.py
-@@ -1154,7 +1154,8 @@ class TextModel(ModelBase):
-                 or "projector." in name or "pre_mm_projector_norm" in name \
-                 or "image_newline" in name or "view_seperator" in name \
-                 or "patch_embed" in name or "patch_embedding" in name \
--                or "patch_merger." in name or "model.connector." in name:
-+                or "patch_merger." in name or "patch_merge_mlp" in name \
-+                or "model.connector." in name:
-             return None
- 
-         return super().filter_tensors(item)
-@@ -1201,7 +1202,7 @@ class TextModel(ModelBase):
-             self.gguf_writer.add_embedding_length(n_embd)
-             logger.info(f"gguf: embedding length = {n_embd}")
- 
--        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-+        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-             self.gguf_writer.add_feed_forward_length(n_ff)
-             logger.info(f"gguf: feed forward length = {n_ff}")
- 
-diff --git a/conversion/minimax.py b/conversion/minimax.py
-index 4857775..4f637f5 100644
---- a/conversion/minimax.py
-+++ b/conversion/minimax.py
-@@ -52,3 +52,67 @@ class MiniMaxM2Model(TextModel):
-             return
- 
-         yield from super().modify_tensors(data_torch, name, bid)
-+
-+
-+@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
-+class MiniMaxM3Model(TextModel):
-+    # Text-only MiniMax-M3: MiniMax-M2 GQA + DeepSeek-V3 shared/leading-dense experts (swigluoai).
-+    model_arch = gguf.MODEL_ARCH.MINIMAXM3
-+    _experts_cache: dict[int, dict[str, Tensor]] = {}
-+
-+    def set_gguf_parameters(self):
-+        # feed_forward_length comes from dense_intermediate_size (base); experts use intermediate_size.
-+        super().set_gguf_parameters()
-+
-+        self.gguf_writer.add_expert_feed_forward_length(self.find_hparam(["intermediate_size"]))
-+        self.gguf_writer.add_rope_dimension_count(self.find_hparam(["rotary_dim"]))
-+        self.gguf_writer.add_expert_shared_count(self.find_hparam(["n_shared_experts"]))
-+        self.gguf_writer.add_expert_weights_scale(self.find_hparam(["routed_scaling_factor"]))
-+        self.gguf_writer.add_expert_weights_norm(True)
-+
-+        # leading dense layers: moe_layer_freq (ints) or mlp_layer_types (Transformers 5.12, strings)
-+        moe_layer_freq = self.find_hparam(["moe_layer_freq", "mlp_layer_types"])
-+        n_dense = 0
-+        for v in moe_layer_freq:
-+            if v == 0 or v == "dense":
-+                n_dense += 1
-+            else:
-+                break
-+        self.gguf_writer.add_leading_dense_block_count(n_dense)
-+
-+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
-+        # index_* (sparse-attn indexer) tensors are preserved but unused; the loader skips them
-+        if name.startswith("language_model."):
-+            name = name[len("language_model."):]
-+
-+        # Gemma-style (1+w) RMSNorm: bake +1 in so llama.cpp can use plain RMSNorm
-+        if name.endswith("norm.weight"):
-+            data_torch = data_torch + 1.0
-+
-+        # merge routed experts (w1/w2/w3); shared_experts.* passes through to *_shexp
-+        if "block_sparse_moe.experts." in name:
-+            n_experts = self.find_hparam(["num_local_experts", "num_experts"])
-+            assert bid is not None
-+
-+            expert_cache = self._experts_cache.setdefault(bid, {})
-+            expert_cache[name] = data_torch
-+            expert_weights = ["w1", "w2", "w3"]
-+
-+            if len(expert_cache) < n_experts * len(expert_weights):
-+                return
-+
-+            for w_name in expert_weights:
-+                datas: list[Tensor] = []
-+                for xid in range(n_experts):
-+                    ename = f"model.layers.{bid}.block_sparse_moe.experts.{xid}.{w_name}.weight"
-+                    datas.append(expert_cache[ename])
-+                    del expert_cache[ename]
-+
-+                data_torch = torch.stack(datas, dim=0)
-+                merged_name = f"model.layers.{bid}.block_sparse_moe.experts.{w_name}.weight"
-+                yield from super().modify_tensors(data_torch, merged_name, bid)
-+
-+            del self._experts_cache[bid]
-+            return
-+
-+        yield from super().modify_tensors(data_torch, name, bid)
-diff --git a/gguf-py/gguf/constants.py b/gguf-py/gguf/constants.py
-index 869e436..760e3dd 100644
---- a/gguf-py/gguf/constants.py
-+++ b/gguf-py/gguf/constants.py
-@@ -525,6 +525,7 @@ class MODEL_ARCH(IntEnum):
-     APERTUS          = auto()
-     COGVLM           = auto()
-     MINIMAXM2        = auto()
-+    MINIMAXM3        = auto()
-     RND1             = auto()
-     PANGU_EMBED      = auto()
-     MISTRAL3         = auto()
-@@ -613,6 +614,10 @@ class MODEL_TENSOR(IntEnum):
-     MOE_LATENT_UP        = auto() # nemotron 3 super
-     ATTN_Q_NORM          = auto()
-     ATTN_K_NORM          = auto()
-+    ATTN_INDEX_Q         = auto() # minimax-m3 sparse-attn indexer (unused)
-+    ATTN_INDEX_K         = auto()
-+    ATTN_INDEX_Q_NORM    = auto()
-+    ATTN_INDEX_K_NORM    = auto()
-     LAYER_OUT_NORM       = auto()
-     LAYER_OUT_SCALE      = auto()
-     PER_LAYER_TOKEN_EMBD = auto() # gemma3n
-@@ -1105,6 +1110,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
-     MODEL_ARCH.GROVEMOE:         "grovemoe",
-     MODEL_ARCH.APERTUS:          "apertus",
-     MODEL_ARCH.MINIMAXM2:        "minimax-m2",
-+    MODEL_ARCH.MINIMAXM3:        "minimax-m3",
-     MODEL_ARCH.COGVLM:           "cogvlm",
-     MODEL_ARCH.RND1:             "rnd1",
-     MODEL_ARCH.PANGU_EMBED:      "pangu-embedded",
-@@ -1163,6 +1169,10 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
-     MODEL_TENSOR.ATTN_GATE:                 "blk.{bid}.attn_gate",
-     MODEL_TENSOR.ATTN_Q_NORM:               "blk.{bid}.attn_q_norm",
-     MODEL_TENSOR.ATTN_K_NORM:               "blk.{bid}.attn_k_norm",
-+    MODEL_TENSOR.ATTN_INDEX_Q:              "blk.{bid}.attn_index_q",
-+    MODEL_TENSOR.ATTN_INDEX_K:              "blk.{bid}.attn_index_k",
-+    MODEL_TENSOR.ATTN_INDEX_Q_NORM:         "blk.{bid}.attn_index_q_norm",
-+    MODEL_TENSOR.ATTN_INDEX_K_NORM:         "blk.{bid}.attn_index_k_norm",
-     MODEL_TENSOR.ATTN_OUT_NORM:             "blk.{bid}.attn_output_norm",
-     MODEL_TENSOR.ATTN_POST_NORM:            "blk.{bid}.post_attention_norm",
-     MODEL_TENSOR.FFN_GATE_INP:              "blk.{bid}.ffn_gate_inp",
-@@ -4102,6 +4112,30 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.FFN_UP_EXP,
-         MODEL_TENSOR.FFN_EXP_PROBS_B,
-     ],
-+    MODEL_ARCH.MINIMAXM3: [
-+        MODEL_TENSOR.TOKEN_EMBD,
-+        MODEL_TENSOR.OUTPUT_NORM,
-+        MODEL_TENSOR.OUTPUT,
-+        MODEL_TENSOR.ATTN_NORM,
-+        MODEL_TENSOR.ATTN_Q,
-+        MODEL_TENSOR.ATTN_Q_NORM,
-+        MODEL_TENSOR.ATTN_K,
-+        MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_V,
-+        MODEL_TENSOR.ATTN_OUT,
-+        MODEL_TENSOR.FFN_NORM,
-+        MODEL_TENSOR.FFN_GATE_INP,
-+        MODEL_TENSOR.FFN_EXP_PROBS_B,
-+        MODEL_TENSOR.FFN_GATE_EXP,
-+        MODEL_TENSOR.FFN_DOWN_EXP,
-+        MODEL_TENSOR.FFN_UP_EXP,
-+        MODEL_TENSOR.FFN_GATE_SHEXP,
-+        MODEL_TENSOR.FFN_DOWN_SHEXP,
-+        MODEL_TENSOR.FFN_UP_SHEXP,
-+        MODEL_TENSOR.FFN_GATE,
-+        MODEL_TENSOR.FFN_DOWN,
-+        MODEL_TENSOR.FFN_UP,
-+    ],
-     MODEL_ARCH.COGVLM: [
-         MODEL_TENSOR.TOKEN_EMBD,
-         MODEL_TENSOR.OUTPUT_NORM,
-@@ -4128,6 +4162,10 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.ATTN_Q_NORM,
-         MODEL_TENSOR.ATTN_K,
-         MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_Q,
-+        MODEL_TENSOR.ATTN_INDEX_K,
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM,
-         MODEL_TENSOR.ATTN_V,
-         MODEL_TENSOR.ATTN_OUT,
-         MODEL_TENSOR.FFN_NORM,
-diff --git a/gguf-py/gguf/tensor_mapping.py b/gguf-py/gguf/tensor_mapping.py
-index 9efb36f..a62040b 100644
---- a/gguf-py/gguf/tensor_mapping.py
-+++ b/gguf-py/gguf/tensor_mapping.py
-@@ -717,6 +717,22 @@ class TensorNameMap:
-             "model.layers.{bid}.attention.key_layernorm",                     # apertus
-         ),
- 
-+        MODEL_TENSOR.ATTN_INDEX_Q: (
-+            "model.layers.{bid}.self_attn.index_q_proj",                      # minimax-m3 (sparse-attn indexer)
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K: (
-+            "model.layers.{bid}.self_attn.index_k_proj",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM: (
-+            "model.layers.{bid}.self_attn.index_q_norm",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM: (
-+            "model.layers.{bid}.self_attn.index_k_norm",                      # minimax-m3
-+        ),
-+
-         MODEL_TENSOR.ROPE_FREQS: (
-             "encoder.layers.{bid}.self_attention.rotary_emb.inv_freq",  # persimmon
-         ),
-diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
-index b890e66..cb8bfc8 100644
---- a/src/llama-arch.cpp
-+++ b/src/llama-arch.cpp
-@@ -125,6 +125,7 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
-     { LLM_ARCH_GROVEMOE,         "grovemoe"         },
-     { LLM_ARCH_APERTUS,          "apertus"          },
-     { LLM_ARCH_MINIMAX_M2,       "minimax-m2"       },
-+    { LLM_ARCH_MINIMAX_M3,       "minimax-m3"       },
-     { LLM_ARCH_COGVLM,           "cogvlm"           },
-     { LLM_ARCH_RND1,             "rnd1"             },
-     { LLM_ARCH_PANGU_EMBED,      "pangu-embedded"   },
-@@ -395,6 +396,10 @@ static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
-     { LLM_TENSOR_ATTN_POST_NORM,                         "blk.%d.post_attention_norm" },
-     { LLM_TENSOR_ATTN_Q_NORM,                            "blk.%d.attn_q_norm" },
-     { LLM_TENSOR_ATTN_K_NORM,                            "blk.%d.attn_k_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_Q,                           "blk.%d.attn_index_q" },
-+    { LLM_TENSOR_ATTN_INDEX_K,                           "blk.%d.attn_index_k" },
-+    { LLM_TENSOR_ATTN_INDEX_Q_NORM,                      "blk.%d.attn_index_q_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_K_NORM,                      "blk.%d.attn_index_k_norm" },
-     { LLM_TENSOR_ATTN_GATE,                              "blk.%d.attn_gate" },
-     { LLM_TENSOR_FFN_POST_NORM,                          "blk.%d.post_ffw_norm" },
-     { LLM_TENSOR_FFN_POST_NORM_1,                        "blk.%d.post_ffw_norm_1" },
-@@ -761,6 +766,11 @@ static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
-     {LLM_TENSOR_FFN_NORM_EXPS,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_K_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-+    // minimax-m3 sparse-attn indexer: unused (GGML_OP_NONE) so the loader skips it
-+    {LLM_TENSOR_ATTN_INDEX_Q,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_Q_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-     {LLM_TENSOR_LAYER_OUT_NORM,             {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_LAYER_OUT_SCALE,            {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_A_NORM,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-@@ -998,6 +1008,7 @@ bool llm_arch_supports_sm_tensor(const llm_arch & arch) {
-         case LLM_ARCH_LFM2:
-         case LLM_ARCH_LFM2MOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_MISTRAL4:
-         case LLM_ARCH_KIMI_LINEAR:
-             return false;
-diff --git a/src/llama-arch.h b/src/llama-arch.h
-index a4f5091..2d50ead 100644
---- a/src/llama-arch.h
-+++ b/src/llama-arch.h
-@@ -144,6 +144,7 @@ enum llm_arch {
-     LLM_ARCH_TALKIE,
-     LLM_ARCH_MELLUM,
-     LLM_ARCH_EAGLE3,
-+    LLM_ARCH_MINIMAX_M3,
-     LLM_ARCH_DFLASH,
-     LLM_ARCH_UNKNOWN,
- };
-@@ -429,6 +430,10 @@ enum llm_tensor {
-     LLM_TENSOR_FFN_LATENT_UP,
-     LLM_TENSOR_ATTN_Q_NORM,
-     LLM_TENSOR_ATTN_K_NORM,
-+    LLM_TENSOR_ATTN_INDEX_Q,      // minimax-m3 sparse-attn indexer (unused)
-+    LLM_TENSOR_ATTN_INDEX_K,
-+    LLM_TENSOR_ATTN_INDEX_Q_NORM,
-+    LLM_TENSOR_ATTN_INDEX_K_NORM,
-     LLM_TENSOR_LAYER_OUT_NORM,
-     LLM_TENSOR_LAYER_OUT_SCALE,
-     LLM_TENSOR_POST_ATTN_NORM,
-diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
-index c8ecb0a..4c2c286 100644
---- a/src/llama-graph.cpp
-+++ b/src/llama-graph.cpp
-@@ -1719,6 +1719,16 @@ ggml_tensor * llm_graph_context::build_ffn(
-                 cur = ggml_reglu(ctx0, cur);
-                 cb(cur, "ffn_reglu", il);
-             } break;
-+        case LLM_FFN_SWIGLU_OAI:
-+            {
-+                // clamped SwiGLU: parallel gate path (cur=gate, tmp=up)
-+                GGML_ASSERT(gate && type_gate == LLM_FFN_PAR);
-+                constexpr float alpha = 1.702f;
-+                constexpr float limit = 7.0f;
-+                cur = ggml_swiglu_oai(ctx0, cur, tmp, alpha, limit);
-+                cb(cur, "ffn_swiglu_oai", il);
-+                type_gate = LLM_FFN_SEQ; // gate*up already fused; skip the par multiply
-+            } break;
-         default:
-             GGML_ABORT("fatal error");
-     }
-diff --git a/src/llama-graph.h b/src/llama-graph.h
-index c84cb6a..806ce7b 100644
---- a/src/llama-graph.h
-+++ b/src/llama-graph.h
-@@ -54,6 +54,7 @@ enum llm_ffn_op_type : int {
-     LLM_FFN_SWIGLU,
-     LLM_FFN_GEGLU,
-     LLM_FFN_REGLU,
-+    LLM_FFN_SWIGLU_OAI,
-     LLM_FFN_SWIGLU_OAI_MOE,
- };
- 
-diff --git a/src/llama-model.cpp b/src/llama-model.cpp
-index d874813..7bb71c0 100644
---- a/src/llama-model.cpp
-+++ b/src/llama-model.cpp
-@@ -280,6 +280,8 @@ static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params
-             return new llama_model_apertus(params);
-         case LLM_ARCH_MINIMAX_M2:
-             return new llama_model_minimax_m2(params);
-+        case LLM_ARCH_MINIMAX_M3:
-+            return new llama_model_minimax_m3(params);
-         case LLM_ARCH_COGVLM:
-             return new llama_model_cogvlm(params);
-         case LLM_ARCH_PANGU_EMBED:
-@@ -807,6 +809,7 @@ const char * llm_type_name(llm_type type) {
-         case LLM_TYPE_310B_A15B:     return "310B.A15B";
-         case LLM_TYPE_355B_A32B:     return "355B.A32B";
-         case LLM_TYPE_397B_A17B:     return "397B.A17B";
-+        case LLM_TYPE_428B_A23B:     return "428B.A23B";
-         case LLM_TYPE_685B_A37B:     return "685B.A37B";
-         case LLM_TYPE_744B_A40B:     return "744B.A40B";
-         case LLM_TYPE_E2B:           return "E2B";
-@@ -2532,6 +2535,7 @@ llama_rope_type llama_model_rope_type(const llama_model * model) {
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_APERTUS:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_COGVLM:
-         case LLM_ARCH_PANGU_EMBED:
-         case LLM_ARCH_AFMOE:
-diff --git a/src/llama-model.h b/src/llama-model.h
-index 45b054c..540e0d2 100644
---- a/src/llama-model.h
-+++ b/src/llama-model.h
-@@ -139,6 +139,7 @@ enum llm_type {
-     LLM_TYPE_310B_A15B, // /MiMo-V2-Flash
-     LLM_TYPE_355B_A32B, // GLM-4.5
-     LLM_TYPE_397B_A17B, // Qwen3.5
-+    LLM_TYPE_428B_A23B, // MiniMax M3
-     LLM_TYPE_685B_A37B, // DeepSeek V3.2
-     LLM_TYPE_744B_A40B, // GLM-5
-     LLM_TYPE_E2B,
-diff --git a/src/models/minimax-m3.cpp b/src/models/minimax-m3.cpp
-new file mode 100644
-index 0000000..137852a
---- /dev/null
-+++ b/src/models/minimax-m3.cpp
-@@ -0,0 +1,197 @@
-+#include "models.h"
-+
-+// MiniMax-M3, text-only: MiniMax-M2 GQA (per-head QK-norm, partial rotary) + DeepSeek-V3
-+// leading-dense/routed/shared experts (swigluoai). Sparse attn -> dense; vision + MTP dropped.
-+
-+void llama_model_minimax_m3::load_arch_hparams(llama_model_loader & ml) {
-+    ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
-+    ml.get_key(LLM_KV_LEADING_DENSE_BLOCK_COUNT,   hparams.n_layer_dense_lead, false);
-+    ml.get_key(LLM_KV_EXPERT_FEED_FORWARD_LENGTH,  hparams.n_ff_exp);
-+    ml.get_key(LLM_KV_EXPERT_SHARED_COUNT,         hparams.n_expert_shared);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE,        hparams.expert_weights_scale, false);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_NORM,         hparams.expert_weights_norm, false);
-+    ml.get_key(LLM_KV_EXPERT_GATING_FUNC,          hparams.expert_gating_func);
-+
-+    switch (hparams.n_layer()) {
-+        case 60: type = LLM_TYPE_428B_A23B; break;
-+        default: type = LLM_TYPE_UNKNOWN;
-+    }
-+}
-+
-+void llama_model_minimax_m3::load_arch_tensors(llama_model_loader &) {
-+    LLAMA_LOAD_LOCALS;
-+    const int64_t n_expert_shared = hparams.n_expert_shared;
-+    const int64_t n_ff_exp        = hparams.n_ff_exp;
-+
-+    tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
-+
-+    output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
-+    output      = create_tensor(tn(LLM_TENSOR_OUTPUT,      "weight"), {n_embd, n_vocab}, 0);
-+
-+    for (int i = 0; i < n_layer; ++i) {
-+        auto & layer = layers[i];
-+
-+        create_tensor_qkv(layer, i, n_embd, n_embd_head_k * n_head, n_embd_gqa, n_embd_gqa, 0);
-+        layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), { n_embd_head_k * n_head, n_embd }, 0);
-+
-+        layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
-+        // per-head QK-norm (one head_dim vector)
-+        layer.attn_q_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_NORM, "weight", i), {n_embd_head_k}, 0);
-+        layer.attn_k_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_NORM, "weight", i), {n_embd_head_k}, 0);
-+
-+        // sparse-attn indexer (unused): GGML_OP_NONE -> loader skips; NOT_REQUIRED -> older GGUFs still load;
-+        // SKIP_IF_VIRTUAL -> no-file loader (test-llama-archs) skips them too
-+        const int64_t n_index_head = 4;    // sparse_num_index_heads
-+        const int64_t d_index      = 128;  // sparse_index_dim
-+        const int idx_flags = TENSOR_NOT_REQUIRED | TENSOR_SKIP_IF_VIRTUAL;
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q,      "weight", i), {n_embd, n_index_head * d_index}, idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K,      "weight", i), {n_embd, d_index},                idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q_NORM, "weight", i), {d_index},                        idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K_NORM, "weight", i), {d_index},                        idx_flags);
-+
-+        layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
-+
-+        if (i < (int) hparams.n_layer_dense_lead) {
-+            layer.ffn_gate = create_tensor(tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd,   n_ff}, 0);
-+            layer.ffn_down = create_tensor(tn(LLM_TENSOR_FFN_DOWN, "weight", i), {  n_ff, n_embd}, 0);
-+            layer.ffn_up   = create_tensor(tn(LLM_TENSOR_FFN_UP,   "weight", i), {n_embd,   n_ff}, 0);
-+        } else {
-+            layer.ffn_gate_inp    = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP,    "weight", i), {n_embd, n_expert}, 0);
-+            layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias",   i), {n_expert}, 0);
-+            layer.ffn_gate_exps   = create_tensor(tn(LLM_TENSOR_FFN_GATE_EXPS,   "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+            layer.ffn_down_exps   = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS,   "weight", i), {n_ff_exp, n_embd, n_expert}, 0);
-+            layer.ffn_up_exps     = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,     "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+
-+            layer.ffn_gate_shexp = create_tensor(tn(LLM_TENSOR_FFN_GATE_SHEXP, "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+            layer.ffn_down_shexp = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "weight", i), {        n_ff_exp * n_expert_shared, n_embd}, 0);
-+            layer.ffn_up_shexp   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+        }
-+    }
-+}
-+
-+std::unique_ptr<llm_graph_context> llama_model_minimax_m3::build_arch_graph(const llm_graph_params & params) const {
-+    return std::make_unique<graph>(*this, params);
-+}
-+
-+llama_model_minimax_m3::graph::graph(const llama_model & model, const llm_graph_params & params) : llm_graph_context(params) {
-+    const int64_t n_embd_head = hparams.n_embd_head_v();
-+
-+    GGML_ASSERT(n_embd_head == hparams.n_embd_head_k());
-+    // partial rotary: head_dim != n_rot, so don't assert n_embd_head == n_rot
-+
-+    ggml_tensor * cur;
-+    ggml_tensor * inpL;
-+
-+    inpL = build_inp_embd(model.tok_embd);
-+
-+    ggml_tensor * inp_pos = build_inp_pos();
-+    auto inp_attn = build_attn_inp_kv();
-+    ggml_tensor * inp_out_ids = build_inp_out_ids();
-+
-+    for (int il = 0; il < n_layer; ++il) {
-+        ggml_tensor * inpSA = inpL;
-+
-+        // self-attention
-+        {
-+            cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);
-+            cb(cur, "attn_norm", il);
-+
-+            auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur,
-+                    n_embd_head, n_head, n_head_kv, il);
-+
-+            // per-head QK RMSNorm (weights include Gemma +1)
-+            Qcur = build_norm(Qcur, model.layers[il].attn_q_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Qcur, "Qcur_normed", il);
-+            Kcur = build_norm(Kcur, model.layers[il].attn_k_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Kcur, "Kcur_normed", il);
-+
-+            Qcur = ggml_rope_ext(
-+                ctx0, Qcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+            Kcur = ggml_rope_ext(
-+                ctx0, Kcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+
-+            cb(Qcur, "Qcur", il);
-+            cb(Kcur, "Kcur", il);
-+            cb(Vcur, "Vcur", il);
-+
-+            cur = build_attn(inp_attn,
-+                    model.layers[il].wo, NULL, model.layers[il].wo_s,
-+                    Qcur, Kcur, Vcur, nullptr, nullptr, nullptr, 1.0f/sqrtf(float(n_embd_head)), il);
-+        }
-+
-+        if (il == n_layer - 1 && inp_out_ids) {
-+            cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);
-+            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids);
-+        }
-+
-+        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);
-+        cb(ffn_inp, "ffn_inp", il);
-+
-+        cur = build_norm(ffn_inp, model.layers[il].ffn_norm, NULL, LLM_NORM_RMS, il);
-+        cb(cur, "ffn_norm", il);
-+
-+        if ((uint32_t) il < hparams.n_layer_dense_lead) {
-+            // leading dense
-+            cur = build_ffn(cur,
-+                    model.layers[il].ffn_up,   NULL, NULL,
-+                    model.layers[il].ffn_gate, NULL, NULL,
-+                    model.layers[il].ffn_down, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(cur, "ffn_out", il);
-+        } else {
-+            // routed experts
-+            ggml_tensor * moe_out = build_moe_ffn(cur,
-+                    model.layers[il].ffn_gate_inp,
-+                    model.layers[il].ffn_up_exps,
-+                    model.layers[il].ffn_gate_exps,
-+                    model.layers[il].ffn_down_exps,
-+                    model.layers[il].ffn_exp_probs_b,
-+                    n_expert, n_expert_used,
-+                    LLM_FFN_SWIGLU_OAI_MOE, hparams.expert_weights_norm,
-+                    hparams.expert_weights_scale,
-+                    (llama_expert_gating_func_type) hparams.expert_gating_func,
-+                    il);
-+            cb(moe_out, "ffn_moe_out", il);
-+
-+            // shared expert
-+            ggml_tensor * ffn_shexp = build_ffn(cur,
-+                    model.layers[il].ffn_up_shexp,   NULL, NULL,
-+                    model.layers[il].ffn_gate_shexp, NULL, NULL,
-+                    model.layers[il].ffn_down_shexp, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(ffn_shexp, "ffn_shexp", il);
-+
-+            cur = ggml_add(ctx0, moe_out, ffn_shexp);
-+            cb(cur, "ffn_out", il);
-+        }
-+
-+        cur = ggml_add(ctx0, cur, ffn_inp);
-+
-+        cur = build_cvec(cur, il);
-+        cb(cur, "l_out", il);
-+
-+        // input for next layer
-+        inpL = cur;
-+    }
-+
-+    cur = inpL;
-+
-+    cur = build_norm(cur, model.output_norm, NULL, LLM_NORM_RMS, -1);
-+    cb(cur, "result_norm", -1);
-+    res->t_embd = cur;
-+
-+    // lm_head
-+    cur = build_lora_mm(model.output, cur, model.output_s);
-+    cb(cur, "result_output", -1);
-+    res->t_logits = cur;
-+
-+    ggml_build_forward_expand(gf, cur);
-+}
-diff --git a/src/models/models.h b/src/models/models.h
-index 7a52e7b..5e2a826 100644
---- a/src/models/models.h
-+++ b/src/models/models.h
-@@ -1870,6 +1870,17 @@ struct llama_model_minimax_m2 : public llama_model_base {
-     std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
- };
- 
-+struct llama_model_minimax_m3 : public llama_model_base {
-+    llama_model_minimax_m3(const struct llama_model_params & params) : llama_model_base(params) {}
-+    void load_arch_hparams(llama_model_loader & ml) override;
-+    void load_arch_tensors(llama_model_loader & ml) override;
-+
-+    struct graph : public llm_graph_context {
-+        graph(const llama_model & model, const llm_graph_params & params);
-+    };
-+
-+    std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
-+};
- 
- struct llama_model_cogvlm : public llama_model_base {
-     llama_model_cogvlm(const struct llama_model_params & params) : llama_model_base(params) {}
-diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp
-index f39abe7..2085f43 100644
---- a/tests/test-llama-archs.cpp
-+++ b/tests/test-llama-archs.cpp
-@@ -352,6 +352,7 @@ static bool moe_mandatory(const llm_arch arch) {
-         case LLM_ARCH_LLADA_MOE:
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_RND1:
-         case LLM_ARCH_PADDLEOCR:
-         case LLM_ARCH_MIMO2:

backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch

@@ -0,0 +1,599 @@
+diff --git a/common/common.cpp b/common/common.cpp
+index 8f13217..fc584e1 100644
+--- a/common/common.cpp
++++ b/common/common.cpp
+@@ -1591,8 +1591,10 @@ struct llama_context_params common_context_params_to_llama(const common_params &
+     auto cparams = llama_context_default_params();
+
+     cparams.n_ctx             = params.n_ctx;
+-    cparams.n_seq_max         = params.n_parallel;
+-    cparams.n_rs_seq          = params.speculative.need_n_rs_seq();
++    // score-task forks need seq ids (and recurrent-state cells) of their
++    // own beyond the parallel slots
++    cparams.n_seq_max         = params.n_parallel + params.n_seq_score_forks;
++    cparams.n_rs_seq          = std::max(params.speculative.need_n_rs_seq(), (uint32_t) std::max(0, params.n_rs_seq));
+     cparams.n_outputs_max     = std::max(params.n_outputs_max, 0);
+     cparams.n_batch           = params.n_batch;
+     cparams.n_ubatch          = params.n_ubatch;
+diff --git a/common/common.h b/common/common.h
+index bffc176..e313bd6 100644
+--- a/common/common.h
++++ b/common/common.h
+@@ -455,6 +455,9 @@ struct common_params {
+     int32_t n_keep                =     0; // number of tokens to keep from initial prompt
+     int32_t n_chunks              =    -1; // max number of chunks to process (-1 = unlimited)
+     int32_t n_parallel            =     1; // number of parallel sequences to decode
++    int32_t n_seq_score_forks     =     0; // extra seq ids beyond n_parallel, reserved for server score-task forks
++    int32_t n_rs_seq              =     0; // recurrent-state rollback snapshots per seq (hybrid models cannot rewind without them; lets score tasks reuse a cached prompt across probe changes)
++    bool    score_enabled          = false; // reserve server resources for the Score task type
+     int32_t n_sequences           =     1; // number of sequences to decode
+     int32_t n_outputs_max         =     0; // max outputs in a batch (0 = n_batch)
+     int32_t grp_attn_n            =     1; // group-attention factor
+diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt
+index 780df32..1d2fe8f 100644
+--- a/tools/CMakeLists.txt
++++ b/tools/CMakeLists.txt
+@@ -41,3 +41,4 @@ else()
+     add_subdirectory(fit-params)
+     add_subdirectory(results)
+ endif()
++add_subdirectory(grpc-server)
+diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
+index 715477e..de5bed8 100644
+--- a/tools/server/server-context.cpp
++++ b/tools/server/server-context.cpp
+@@ -49,7 +49,16 @@ static uint32_t server_n_outputs_max(const common_params & params) {
+
+     const uint32_t n_outputs_per_seq = 1 + common_speculative_n_max(&params.speculative);
+
+-    const uint64_t n_outputs = (uint64_t) params.n_parallel * n_outputs_per_seq;
++    // score tasks (SERVER_TASK_TYPE_SCORE) output logits for every candidate
++    // token, so reserve room for a bounded candidate tail per parallel slot
++    if (!params.score_enabled) {
++        return std::max<uint32_t>(1, std::min<uint64_t>(n_batch,
++            (uint64_t) params.n_parallel * n_outputs_per_seq));
++    }
++
++    const uint32_t n_outputs_score_seq = 1 + SERVER_SCORE_MAX_CAND_TOKENS;
++
++    const uint64_t n_outputs = (uint64_t) params.n_parallel * std::max(n_outputs_per_seq, n_outputs_score_seq);
+
+     return std::max<uint32_t>(1, std::min<uint64_t>(n_batch, n_outputs));
+ }
+@@ -202,6 +211,26 @@ struct server_slot {
+
+     std::vector<completion_token_output> generated_token_probs;
+
++    // SERVER_TASK_TYPE_SCORE: shared-prefix token logprobs harvested
++    // incrementally across batch views (NaN = not yet produced)
++    std::vector<float> score_logprobs;
++
++    // SERVER_TASK_TYPE_SCORE: per-candidate suffix token logprobs; entry
++    // [c][0] comes from the last shared token's logits during prompt
++    // processing, the rest from the forked suffix decode
++    std::vector<std::vector<float>> score_cand_logprobs;
++
++    // SERVER_TASK_TYPE_SCORE: the prompt completed but some candidate has
++    // suffix tokens beyond the first, so a forked decode is still needed
++    bool score_suffix_pending = false;
++
++    // SERVER_TASK_TYPE_SCORE: where the current task's tokens diverged from
++    // the slot's previous cache. When the memory cannot rewind there and a
++    // re-prefill follows, a checkpoint at this position lets the next
++    // scoring call over the same stable prefix (e.g. a classifier's option
++    // list) resume from it instead of re-processing the whole prompt.
++    int32_t score_divergence = -1;
++
+     bool has_next_token = true;
+     bool has_new_line   = false;
+     bool truncated      = false;
+@@ -311,6 +340,10 @@ struct server_slot {
+         }
+         generated_tokens.clear();
+         generated_token_probs.clear();
++        score_logprobs.clear();
++        score_cand_logprobs.clear();
++        score_suffix_pending = false;
++        score_divergence = -1;
+         json_schema = json();
+
+         // clear speculative decoding stats
+@@ -2205,6 +2238,229 @@ private:
+         queue_results.send(std::move(res));
+     }
+
++    // log(sum(exp(logits))) with max-subtraction for stability — the
++    // log_softmax denominator shared by every token read from one output
++    static double score_log_denom(const float * logits, int32_t n_vocab) {
++        float max_logit = logits[0];
++        for (int32_t v = 1; v < n_vocab; ++v) {
++            max_logit = std::max(max_logit, logits[v]);
++        }
++        double sum_exp = 0.0;
++        for (int32_t v = 0; v < n_vocab; ++v) {
++            sum_exp += std::exp((double)(logits[v] - max_logit));
++        }
++        return (double) max_logit + std::log(sum_exp);
++    }
++
++    // Harvest logprobs for SCORE tasks from the current batch view: the
++    // shared-prefix scored tokens, and — from the last shared token's
++    // logits — the first suffix token of every candidate. The scored
++    // region can straddle ubatch boundaries for long prompts, so this
++    // accumulates view by view instead of reading everything when the
++    // prompt completes.
++    void collect_score_logprobs(server_slot & slot, const llama_batch & batch) {
++        const int32_t n_prompt = slot.task->n_score_prompt;
++        const int32_t n_total  = slot.task->n_tokens();
++        const auto &  suffixes = slot.task->score_suffixes;
++
++        const size_t n_shared_scored = (size_t) std::max(0, n_total - n_prompt);
++
++        if (slot.score_logprobs.size() != n_shared_scored) {
++            slot.score_logprobs.assign(n_shared_scored, NAN);
++        }
++        if (slot.score_cand_logprobs.size() != suffixes.size()) {
++            slot.score_cand_logprobs.resize(suffixes.size());
++            for (size_t c = 0; c < suffixes.size(); ++c) {
++                slot.score_cand_logprobs[c].assign(suffixes[c].size(), NAN);
++            }
++        }
++
++        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
++
++        for (int32_t i = 0; i < batch.n_tokens; ++i) {
++            if (!batch.logits[i] || batch.seq_id[i][0] != slot.id) {
++                continue;
++            }
++
++            // the output at position p predicts the task token at index p + 1;
++            // score tasks are text-only, so positions equal token indices
++            const int32_t target = batch.pos[i] + 1;
++            if (target < n_prompt || target > n_total) {
++                continue;
++            }
++
++            const float * logits = llama_get_logits_ith(slot.ctx_tgt, i);
++            if (logits == nullptr) {
++                SLT_ERR(slot, "failed to get logits for score target %d\n", target);
++                continue;
++            }
++
++            const double log_denom = score_log_denom(logits, n_vocab);
++
++            if (target < n_total) {
++                const llama_token tok = slot.task->tokens[target];
++                slot.score_logprobs[target - n_prompt] = (float) ((double) logits[tok] - log_denom);
++            } else {
++                // the last shared token predicts the first suffix token of
++                // every candidate
++                for (size_t c = 0; c < suffixes.size(); ++c) {
++                    if (!suffixes[c].empty()) {
++                        slot.score_cand_logprobs[c][0] = (float) ((double) logits[suffixes[c][0]] - log_denom);
++                    }
++                }
++            }
++        }
++    }
++
++    void send_score(server_slot & slot) {
++        auto res = std::make_unique<server_task_result_score>();
++        res->id              = slot.task->id;
++        res->index           = slot.task->index;
++        res->shared_logprobs = std::move(slot.score_logprobs);
++        res->cand_logprobs   = std::move(slot.score_cand_logprobs);
++
++        slot.score_logprobs.clear();
++        slot.score_cand_logprobs.clear();
++
++        SLT_DBG(slot, "sending score result, n_shared = %zu, n_cand = %zu\n",
++                res->shared_logprobs.size(), res->cand_logprobs.size());
++
++        queue_results.send(std::move(res));
++    }
++
++    // Decode the candidate suffixes of a completed score prompt: fork one
++    // sequence per candidate off the slot's shared prefix (metadata-only
++    // for the unified KV cache, copy-on-write for recurrent state) and
++    // decode all unique suffix tokens in as few llama_decode calls as the
++    // fork/batch/output budgets allow, harvesting a logprob for every
++    // suffix token that predicts a following one.
++    bool decode_score_suffixes(server_slot & slot) {
++        const auto & suffixes = slot.task->score_suffixes;
++
++        auto * mem = llama_get_memory(ctx_tgt);
++
++        // seq ids beyond the slots are reserved for score forks at context
++        // creation (common_params::n_seq_score_forks)
++        const int32_t seq_base    = (int32_t) slots.size();
++        const int32_t n_forks_max = std::min<int32_t>(SERVER_SCORE_FORK_SEQS, (int32_t) llama_n_seq_max(ctx_tgt) - seq_base);
++
++        if (n_forks_max < 1) {
++            SLT_ERR(slot, "no fork sequences reserved for score suffixes (n_seq_max = %d, n_slots = %d)\n",
++                    (int32_t) llama_n_seq_max(ctx_tgt), seq_base);
++            return false;
++        }
++
++        const int32_t   n_batch_max = llama_n_batch(ctx_tgt);
++        const int32_t   n_vocab     = llama_vocab_n_tokens(vocab);
++        const llama_pos pos0        = slot.prompt.tokens.pos_next();
++
++        std::vector<size_t> pending;
++        for (size_t c = 0; c < suffixes.size(); ++c) {
++            // single-token suffixes were fully scored from the last shared
++            // token's logits during prompt processing
++            if (suffixes[c].size() > 1) {
++                if ((int32_t) suffixes[c].size() > n_batch_max) {
++                    SLT_ERR(slot, "score suffix of candidate %zu (%zu tokens) exceeds n_batch (%d)\n",
++                            c, suffixes[c].size(), n_batch_max);
++                    return false;
++                }
++                pending.push_back(c);
++            }
++        }
++
++        size_t next = 0;
++        while (next < pending.size()) {
++            // greedy-pack candidates into one decode within the fork,
++            // batch and reserved-output budgets
++            std::vector<size_t> chunk;
++            int32_t n_tok = 0;
++            int32_t n_out = 0;
++            while (next < pending.size() && (int32_t) chunk.size() < n_forks_max) {
++                const int32_t m = (int32_t) suffixes[pending[next]].size();
++                if (!chunk.empty() && (n_tok + m > n_batch_max || n_out + m - 1 > SERVER_SCORE_MAX_CAND_TOKENS)) {
++                    break;
++                }
++                chunk.push_back(pending[next]);
++                n_tok += m;
++                n_out += m - 1;
++                next++;
++            }
++
++            llama_batch fb = llama_batch_init(n_tok, 0, 1);
++
++            for (size_t k = 0; k < chunk.size(); ++k) {
++                const llama_seq_id seq = seq_base + (llama_seq_id) k;
++                const auto & sfx = suffixes[chunk[k]];
++
++                llama_memory_seq_rm(mem, seq, -1, -1);
++                llama_memory_seq_cp(mem, slot.id, seq, -1, -1);
++
++                for (size_t j = 0; j < sfx.size(); ++j) {
++                    common_batch_add(fb, sfx[j], pos0 + (llama_pos) j, { seq }, j + 1 < sfx.size());
++                }
++            }
++
++            const int ret = llama_decode(ctx_tgt, fb);
++
++            if (ret == 0) {
++                int32_t i = 0;
++                for (size_t k = 0; k < chunk.size(); ++k) {
++                    const auto & sfx = suffixes[chunk[k]];
++                    auto & out = slot.score_cand_logprobs[chunk[k]];
++
++                    for (size_t j = 0; j < sfx.size(); ++j, ++i) {
++                        if (j + 1 >= sfx.size()) {
++                            continue; // last suffix token predicts nothing
++                        }
++                        const float * logits = llama_get_logits_ith(ctx_tgt, i);
++                        if (logits == nullptr) {
++                            SLT_ERR(slot, "failed to get logits for suffix token %zu of score candidate %zu\n", j, chunk[k]);
++                            continue;
++                        }
++                        const double log_denom = score_log_denom(logits, n_vocab);
++                        out[j + 1] = (float) ((double) logits[sfx[j + 1]] - log_denom);
++                    }
++                }
++            }
++
++            for (size_t k = 0; k < chunk.size(); ++k) {
++                llama_memory_seq_rm(mem, seq_base + (llama_seq_id) k, -1, -1);
++            }
++
++            llama_batch_free(fb);
++
++            if (ret != 0) {
++                SLT_ERR(slot, "score suffix decode failed, ret = %d\n", ret);
++                return false;
++            }
++        }
++
++        return true;
++    }
++
++    // score slots whose prompt completed this iteration decode their
++    // candidate suffixes here, after every batch view was consumed — a
++    // mid-view llama_decode would clobber logits other slots still read
++    void update_score_suffixes() {
++        for (auto & slot : slots) {
++            if (!slot.score_suffix_pending) {
++                continue;
++            }
++            slot.score_suffix_pending = false;
++
++            if (!slot.is_processing() || !slot.task || slot.task->type != SERVER_TASK_TYPE_SCORE) {
++                continue; // the task was aborted mid-iteration
++            }
++
++            if (decode_score_suffixes(slot)) {
++                send_score(slot);
++            } else {
++                send_error(slot, "failed to decode score candidate suffixes", ERROR_TYPE_SERVER);
++            }
++            slot.release();
++        }
++    }
++
+     //
+     // Functions to process the task
+     //
+@@ -2341,6 +2597,7 @@ private:
+             case SERVER_TASK_TYPE_INFILL:
+             case SERVER_TASK_TYPE_EMBEDDING:
+             case SERVER_TASK_TYPE_RERANK:
++            case SERVER_TASK_TYPE_SCORE:
+                 {
+                     // special case: if input is provided via CLI, tokenize it first
+                     // otherwise, no need to tokenize as it's already done inside the HTTP thread
+@@ -2832,6 +3089,13 @@ private:
+                 break; // stop any further processing
+             }
+         }
++
++        try {
++            update_score_suffixes();
++        } catch (const std::exception & e) {
++            SRV_ERR("update_score_suffixes() failed: %s\n", e.what());
++            abort_all_slots("update_score_suffixes() failed: " + std::string(e.what()));
++        }
+     }
+
+     void pre_decode() {
+@@ -3154,6 +3418,16 @@ private:
+                                     n_past = std::min(n_past, slot.alora_invocation_start - 1);
+                                 }
+
++                                // score tasks need the logits that predict the first candidate
++                                // token, so the last shared-prompt token must be (re-)decoded
++                                // even when the cache already covers it
++                                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
++                                    n_past = std::min(n_past, std::max(0, slot.task->n_score_prompt - 1));
++                                    // remember the divergence point before the checkpoint
++                                    // logic below possibly resets n_past to 0
++                                    slot.score_divergence = n_past;
++                                }
++
+                                 const auto n_cache_reuse = slot.task->params.n_cache_reuse;
+
+                                 const bool can_cache_reuse =
+@@ -3395,8 +3669,12 @@ private:
+
+                     bool do_checkpoint = params_base.n_ctx_checkpoints > 0;
+
+-                    // make checkpoints only for completion tasks
+-                    do_checkpoint = do_checkpoint && slot.task->type == SERVER_TASK_TYPE_COMPLETION;
++                    // make checkpoints for completion tasks, and for score tasks at the
++                    // shared-prompt boundary: models whose memory cannot be partially
++                    // rewound (SWA/hybrid/recurrent) would otherwise re-process the whole
++                    // prompt for every candidate of a scoring call
++                    do_checkpoint = do_checkpoint && (slot.task->type == SERVER_TASK_TYPE_COMPLETION ||
++                                                      slot.task->type == SERVER_TASK_TYPE_SCORE);
+
+                     // make a checkpoint of the parts of the memory that cannot be rolled back.
+                     // checkpoints are created only if:
+@@ -3463,10 +3741,17 @@ private:
+                         // embedding requires all tokens in the batch to be output;
+                         // MTP also wants logits at every prompt position so the
+                         // streaming hook can mirror t_h_nextn into ctx_dft.
++                        // score tasks need outputs at the positions that predict
++                        // each candidate token (the token at index i predicts the
++                        // task token at index i+1).
++                        const bool need_score_logit =
++                            slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                            slot.prompt.n_tokens() + 1 >= slot.task->n_score_prompt &&
++                            slot.prompt.n_tokens() + 1 <  slot.task->n_tokens();
+                         add_ok &= batch.add(slot.id,
+                             cur_tok,
+                             slot.prompt.tokens.pos_next(),
+-                            slot.need_embd());
++                            slot.need_embd() || need_score_logit);
+                         slot.prompt.tokens.push_back(cur_tok);
+
+                         slot.n_prompt_tokens_processed++;
+@@ -3481,6 +3766,32 @@ private:
+                             }
+                         }
+
++                        // score tasks: break at the shared-prompt boundary so the checkpoint
++                        // below lands exactly there — the other candidates of the same
++                        // scoring call re-process only their own tokens. Also break at the
++                        // point where this task diverged from the previous cache: after a
++                        // forced re-prefill a checkpoint there serves the next scoring call
++                        // over the same stable prefix (e.g. a classifier's option list).
++                        // The caller-declared stable-prefix boundary is the strongest of
++                        // these: a checkpoint there is at or before every future task's
++                        // divergence within the same option list, so it always survives
++                        // and always restores.
++                        if (do_checkpoint && slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                            (slot.prompt.n_tokens() == slot.task->n_score_prompt - 1 ||
++                             (slot.task->n_stable_prompt > 0 &&
++                              slot.prompt.n_tokens() == slot.task->n_stable_prompt &&
++                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1) ||
++                             (slot.prompt.n_tokens() == slot.score_divergence &&
++                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1))) {
++                            bool have_ckpt = false;
++                            for (const auto & ckpt : slot.prompt.checkpoints) {
++                                have_ckpt |= ckpt.n_tokens == slot.prompt.n_tokens();
++                            }
++                            if (!have_ckpt) {
++                                break;
++                            }
++                        }
++
+                         // process the last few tokens of the prompt separately in order to allow for a checkpoint to be created.
+                         // create checkpoints that many tokens before the end of the prompt:
+                         //  - 4 + n_ubatch
+@@ -3513,6 +3824,15 @@ private:
+                     const bool is_user_start = spans.is_user_start(n_tokens_start);
+                     const bool is_last_user_message = n_tokens_start == last_user_pos;
+
++                    // a batch starting at the score boundary or divergence point must
++                    // always checkpoint — min-step spacing would otherwise suppress it
++                    // and every candidate / next scoring call would re-process the prompt
++                    const bool is_score_boundary = slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                                                   (n_tokens_start == slot.task->n_score_prompt - 1 ||
++                                                    (slot.task->n_stable_prompt > 0 &&
++                                                     n_tokens_start == slot.task->n_stable_prompt) ||
++                                                    n_tokens_start == slot.score_divergence);
++
+                     // entire prompt has been processed
+                     if (slot.prompt.n_tokens() == slot.task->n_tokens()) {
+                         slot.state = SLOT_STATE_DONE_PROMPT;
+@@ -3528,8 +3848,8 @@ private:
+                         slot.init_sampler();
+                     } else {
+                         // skip ordinary mid-prompt checkpoints, unless the batch starts a user
+-                        // message or we are near the end of the prompt
+-                        if (!is_user_start && !near_prompt_end) {
++                        // message, the score boundary, or we are near the end of the prompt
++                        if (!is_user_start && !is_score_boundary && !near_prompt_end) {
+                             do_checkpoint = false;
+                         }
+                     }
+@@ -3546,10 +3866,10 @@ private:
+                     // do not checkpoint after mtmd chunks
+                     do_checkpoint = do_checkpoint && !has_mtmd;
+
+-                    // no need to create checkpoints that are too close together, unless it's the last user message
++                    // no need to create checkpoints that are too close together, unless it's the last user message or the score boundary
+                     do_checkpoint = do_checkpoint && (
+                             slot.prompt.checkpoints.empty() ||
+-                            is_last_user_message || near_prompt_end ||
++                            is_last_user_message || near_prompt_end || is_score_boundary ||
+                             n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
+                     SLT_DBG(slot, "main/do_checkpoint = %s, pos_min = %d, pos_max = %d\n", do_checkpoint ? "yes" : "no", pos_min, pos_max);
+
+@@ -3703,6 +4023,13 @@ private:
+                 }
+             }
+
++            // score slots harvest logprobs from every view that contains
++            // their outputs, not just the one holding the final token
++            if (slot.task && slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                (slot.state == SLOT_STATE_PROCESSING_PROMPT || slot.state == SLOT_STATE_DONE_PROMPT)) {
++                collect_score_logprobs(slot, batch_view);
++            }
++
+             if (!is_inside_view(slot.i_batch)) {
+                 // the required token not in this sub-batch, skip
+                 return;
+@@ -3724,6 +4051,25 @@ private:
+                     return;
+                 }
+
++                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
++                    // shared-prefix logprobs (and every candidate's first
++                    // suffix logprob) were accumulated per view above;
++                    // candidates with more suffix tokens still need the
++                    // forked decode at the end of update_slots()
++                    for (const auto & sfx : slot.task->score_suffixes) {
++                        if (sfx.size() > 1) {
++                            slot.score_suffix_pending = true;
++                            break;
++                        }
++                    }
++                    if (!slot.score_suffix_pending) {
++                        send_score(slot);
++                        slot.release();
++                    }
++                    slot.i_batch = -1;
++                    return;
++                }
++
+                 GGML_ASSERT(slot.task->need_sampling());
+
+                 // prompt evaluated for next-token prediction
+diff --git a/tools/server/server-task.h b/tools/server/server-task.h
+index c3eea2e..fb3c178 100644
+--- a/tools/server/server-task.h
++++ b/tools/server/server-task.h
+@@ -13,10 +13,25 @@
+
+ using json = nlohmann::ordered_json;
+
++// SERVER_TASK_TYPE_SCORE emits one logits output per candidate token (plus
++// the forced last-token output), and the context's output budget
++// (n_outputs_max) is reserved up front — so candidate length must be
++// bounded. Raising this raises the worst-case compute-buffer reservation
++// by ~n_vocab * 4 bytes per extra output.
++constexpr int32_t SERVER_SCORE_MAX_CAND_TOKENS = 64;
++
++// Maximum sequences forked off the shared prefix in one score suffix
++// decode. The context is created with this many seq ids (and
++// recurrent-state cells) beyond the parallel slots — see
++// common_params::n_seq_score_forks; candidates in excess of the budget
++// are decoded in successive chunks.
++constexpr int32_t SERVER_SCORE_FORK_SEQS = 16;
++
+ enum server_task_type {
+     SERVER_TASK_TYPE_COMPLETION,
+     SERVER_TASK_TYPE_EMBEDDING,
+     SERVER_TASK_TYPE_RERANK,
++    SERVER_TASK_TYPE_SCORE,
+     SERVER_TASK_TYPE_INFILL,
+     SERVER_TASK_TYPE_CANCEL,
+     SERVER_TASK_TYPE_CONTROL,
+@@ -153,6 +168,18 @@ struct server_task {
+     task_params   params;
+     server_tokens tokens;
+
++    // used by SERVER_TASK_TYPE_SCORE: `tokens` holds the shared prefix
++    // (prompt + longest common candidate token prefix) and logprobs are
++    // returned for its tokens from n_score_prompt onward. Each candidate's
++    // tokens beyond the shared prefix ride a forked sequence.
++    int32_t n_score_prompt = 0;
++    std::vector<llama_tokens> score_suffixes;
++    // token index where the caller-declared stable prompt prefix ends
++    // (0 = no hint): the option-list system prompt that repeats across
++    // scoring calls. A context checkpoint is forced there so models that
++    // cannot rewind state re-process only the per-call tail next time.
++    int32_t n_stable_prompt = 0;
++
+     // only used by CLI, this allow tokenizing CLI inputs on server side
+     // we need this because mtmd_context and vocab are not accessible outside of server_context
+     bool                    cli = false;
+@@ -197,6 +224,7 @@ struct server_task {
+         switch (type) {
+             case SERVER_TASK_TYPE_COMPLETION:
+             case SERVER_TASK_TYPE_INFILL:
++            case SERVER_TASK_TYPE_SCORE:
+                 return true;
+             default:
+                 return false;
+@@ -494,6 +522,25 @@ struct server_task_result_rerank : server_task_result {
+     virtual json to_json() override;
+ };
+
++struct server_task_result_score : server_task_result {
++    // log P(token | prefix) for the shared-prefix tokens after
++    // n_score_prompt, in order; NaN marks positions the decode never
++    // produced an output for
++    std::vector<float> shared_logprobs;
++
++    // per candidate: logprobs of its suffix tokens, in task order (entry
++    // 0 is the token right after the shared prefix, predicted by the last
++    // shared token's logits)
++    std::vector<std::vector<float>> cand_logprobs;
++
++    virtual json to_json() override {
++        return json {
++            {"shared_logprobs", shared_logprobs},
++            {"cand_logprobs",   cand_logprobs},
++        };
++    }
++};
++
+ struct server_task_result_error : server_task_result {
+     error_type err_type = ERROR_TYPE_SERVER;
+     std::string err_msg;

backend/cpp/llama-cpp/prepare.sh

@@ -31,6 +31,26 @@ cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 
+## Fork-skew probe. Upstream folded common_params::use_mmap / use_mlock /
+## use_direct_io into a single `load_mode` enum (ggml-org/llama.cpp#20834).
+## turboquant and bonsai compile this very same grpc-server.cpp against forks
+## that branched before that change, so the field set is decided from the
+## checkout in front of us rather than from a per-fork build flag: the flavor
+## targets disagree on whether they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and
+## probing heals itself the moment a fork rebases past the refactor.
+if grep -q "LLAMA_LOAD_MODE_MMAP" llama.cpp/include/llama.h; then
+    echo "==> llama.cpp carries the load-mode enum, using common_params::load_mode"
+    LEGACY_LOAD_MODE=0
+else
+    echo "==> llama.cpp predates the load-mode enum, using the legacy mmap/mlock/direct-io booleans"
+    LEGACY_LOAD_MODE=1
+fi
+cat > llama.cpp/tools/grpc-server/llama_compat.h <<EOF
+// Generated by backend/cpp/llama-cpp/prepare.sh. Do not edit.
+#pragma once
+#define LOCALAI_LEGACY_LOAD_MODE ${LEGACY_LOAD_MODE}
+EOF
+
 set +e
 if grep -q "grpc-server" llama.cpp/tools/CMakeLists.txt; then
     echo "grpc-server already added"

backend/cpp/turboquant/Makefile

@@ -47,6 +47,7 @@ define turboquant-build
 	# original under backend/cpp/llama-cpp/, so the stock llama-cpp build
 	# stays compiling against vanilla upstream.
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build llama.cpp
@@ -84,6 +85,7 @@ turboquant-cpu-all:
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build purge
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build llama.cpp

backend/go/cloud-proxy/provider_anthropic.go

@@ -32,7 +32,9 @@ import (
 type anthropicRequest struct {
 	Model         string               `json:"model"`
 	MaxTokens     int32                `json:"max_tokens"`
-	System        string               `json:"system,omitempty"`
+	// System is `any`: a bare string normally, or []anthropicSystemBlock
+	// when cache_prompt is on (the block form carries cache_control).
+	System        any                  `json:"system,omitempty"`
 	Messages      []anthropicMessage   `json:"messages"`
 	Stream        bool                 `json:"stream,omitempty"`
 	Temperature   *float64             `json:"temperature,omitempty"`
@@ -52,9 +54,30 @@ type anthropicMessage struct {
 }
 
 type anthropicTool struct {
-	Name        string          `json:"name"`
-	Description string          `json:"description,omitempty"`
-	InputSchema json.RawMessage `json:"input_schema"`
+	Name         string                 `json:"name"`
+	Description  string                 `json:"description,omitempty"`
+	InputSchema  json.RawMessage        `json:"input_schema"`
+	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
+}
+
+// anthropicCacheControl marks a prompt-cache breakpoint. Anthropic caches
+// everything up to and including a block tagged {"type":"ephemeral"} (5-min
+// TTL) and serves that prefix at the cache-read rate (0.1x input) on later
+// calls that share it — the win on agentic/multi-turn workloads.
+type anthropicCacheControl struct {
+	Type string `json:"type"` // "ephemeral"
+}
+
+// ephemeralCacheControl is the single reused breakpoint marker.
+var ephemeralCacheControl = &anthropicCacheControl{Type: "ephemeral"}
+
+// anthropicSystemBlock is the block form of the top-level system field.
+// Anthropic accepts system as a bare string OR a list of text blocks; the
+// block form is required to attach cache_control to the system prompt.
+type anthropicSystemBlock struct {
+	Type         string                 `json:"type"` // "text"
+	Text         string                 `json:"text"`
+	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
 }
 
 // anthropicToolChoice mirrors the four shapes Anthropic accepts:
@@ -81,8 +104,9 @@ type anthropicContentBlock struct {
 	// Tool-result block fields. tool_result uses `content` (not
 	// `text`) and pairs with `tool_use_id`; modelling them as
 	// distinct fields avoids ambiguity at marshal time.
-	ToolUseID     string `json:"tool_use_id,omitempty"`
-	ResultContent string `json:"content,omitempty"`
+	ToolUseID     string                 `json:"tool_use_id,omitempty"`
+	ResultContent string                 `json:"content,omitempty"`
+	CacheControl  *anthropicCacheControl `json:"cache_control,omitempty"`
 }
 
 type anthropicResponse struct {
@@ -156,6 +180,11 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 	if req.ToolChoice != nil && req.ToolChoice.Type == anthropicToolChoiceNone {
 		req.Tools, req.ToolChoice = nil, nil
 	}
+	// Prompt-cache breakpoint on the last tool: Anthropic caches the entire
+	// tool block up to the marked tool — usually a large, fully stable prefix.
+	if cfg.cachePrompt && len(req.Tools) > 0 {
+		req.Tools[len(req.Tools)-1].CacheControl = ephemeralCacheControl
+	}
 
 	var systemParts []string
 	for _, m := range opts.GetMessages() {
@@ -189,15 +218,54 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 			})
 		}
 	}
-	req.System = strings.Join(systemParts, "\n\n")
+	// System: block form (with cache_control) when caching is on, else the
+	// bare string. Only set when non-empty so `omitempty` still drops it.
+	if len(systemParts) > 0 {
+		joined := strings.Join(systemParts, "\n\n")
+		if cfg.cachePrompt {
+			req.System = []anthropicSystemBlock{{Type: "text", Text: joined, CacheControl: ephemeralCacheControl}}
+		} else {
+			req.System = joined
+		}
+	}
 
 	if len(req.Messages) == 0 && opts.GetPrompt() != "" {
 		req.Messages = []anthropicMessage{{Role: "user", Content: opts.GetPrompt()}}
 	}
 
+	// Prompt-cache breakpoint on the final message block caches the whole
+	// conversation prefix up to the newest turn. With the system + tools
+	// breakpoints above, Anthropic serves the entire stable head at the
+	// cache-read rate on the next agentic iteration (max 4 breakpoints; we
+	// use at most 3, so we never exceed the limit).
+	if cfg.cachePrompt {
+		markLastMessageCacheable(req.Messages)
+	}
+
 	return json.Marshal(req)
 }
 
+// markLastMessageCacheable tags the final block of the last message with a
+// cache_control breakpoint. String content is promoted to a single text
+// block so the marker has somewhere to attach; block content gets the marker
+// on its last element.
+func markLastMessageCacheable(msgs []anthropicMessage) {
+	if len(msgs) == 0 {
+		return
+	}
+	last := &msgs[len(msgs)-1]
+	switch c := last.Content.(type) {
+	case string:
+		if c != "" {
+			last.Content = []anthropicContentBlock{{Type: "text", Text: c, CacheControl: ephemeralCacheControl}}
+		}
+	case []anthropicContentBlock:
+		if len(c) > 0 {
+			c[len(c)-1].CacheControl = ephemeralCacheControl
+		}
+	}
+}
+
 // appendToolResult appends a tool_result block as a user message,
 // merging into a preceding user message that already carries blocks.
 // Anthropic concatenates consecutive same-role messages on its end,

backend/go/cloud-proxy/provider_anthropic_test.go

@@ -328,3 +328,62 @@ func TestBuildAnthropic_RoundTripsAssistantToolCalls(t *testing.T) {
 	g.Expect(r0["tool_use_id"]).To(Equal("call_abc"))
 	g.Expect(r0["content"]).To(Equal(`{"models":["a","b"]}`))
 }
+
+// TestPredict_Anthropic_PromptCache verifies that cache_prompt injects
+// exactly the intended cache_control breakpoints (system, last tool, last
+// message) when on, and none when off — asserting on the raw upstream body
+// because System becomes a block list that the typed struct hides.
+func TestPredict_Anthropic_PromptCache(t *testing.T) {
+	g := NewWithT(t)
+
+	// run issues one translate Predict and returns the raw body the fake
+	// Anthropic upstream received.
+	run := func(cachePrompt bool) string {
+		var rawBody string
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			b, _ := io.ReadAll(r.Body)
+			rawBody = string(b)
+			w.Header().Set("Content-Type", "application/json")
+			_, _ = io.WriteString(w, `{"id":"m","type":"message","role":"assistant","content":[{"type":"text","text":"ok"}],"model":"claude-3-5-sonnet-20241022","usage":{"input_tokens":5,"output_tokens":2}}`)
+		}))
+		defer srv.Close()
+
+		t.Setenv("CLOUD_PROXY_ANTHROPIC_FAKE", "sk-ant-fake")
+		cp := NewCloudProxy()
+		err := cp.Load(&pb.ModelOptions{
+			Model: "claude-local",
+			Proxy: &pb.ProxyOptions{
+				UpstreamUrl:   srv.URL,
+				Mode:          modeTranslate,
+				Provider:      providerAnthropic,
+				ApiKeyEnv:     "CLOUD_PROXY_ANTHROPIC_FAKE",
+				UpstreamModel: "claude-3-5-sonnet-20241022",
+				CachePrompt:   cachePrompt,
+			},
+		})
+		g.Expect(err).NotTo(HaveOccurred())
+
+		_, err = cp.Predict(&pb.PredictOptions{
+			Messages: []*pb.Message{
+				{Role: "system", Content: "be brief"},
+				{Role: "user", Content: "hello"},
+			},
+			Tools:  `[{"type":"function","function":{"name":"t","parameters":{"type":"object"}}}]`,
+			Tokens: 32,
+		})
+		g.Expect(err).NotTo(HaveOccurred())
+		return rawBody
+	}
+
+	// cache_prompt ON: three ephemeral breakpoints (system + last tool +
+	// last message), and system is emitted in block form.
+	on := run(true)
+	g.Expect(strings.Count(on, `"cache_control":{"type":"ephemeral"}`)).To(Equal(3),
+		"expected 3 breakpoints (system, tool, last message); body=%s", on)
+	g.Expect(on).To(ContainSubstring(`"system":[{"type":"text","text":"be brief"`))
+
+	// cache_prompt OFF: no breakpoints, system stays a bare string.
+	off := run(false)
+	g.Expect(off).NotTo(ContainSubstring("cache_control"))
+	g.Expect(off).To(ContainSubstring(`"system":"be brief"`))
+}

backend/go/cloud-proxy/proxy.go

@@ -48,6 +48,7 @@ type proxyConfig struct {
 	upstreamModel string
 	localModel    string // ModelOptions.Model — fallback when upstream_model is unset
 	apiKey        string // resolved at Load time
+	cachePrompt   bool   // inject Anthropic prompt-cache breakpoints (translate+anthropic)
 }
 
 func NewCloudProxy() *CloudProxy {
@@ -106,6 +107,7 @@ func (c *CloudProxy) Load(opts *pb.ModelOptions) error {
 		upstreamModel: po.GetUpstreamModel(),
 		localModel:    opts.GetModel(),
 		apiKey:        key,
+		cachePrompt:   po.GetCachePrompt(),
 	})
 	xlog.Info("cloud-proxy: ready",
 		"upstream", po.GetUpstreamUrl(),

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=3ab5f4ac13685966b47cc75dc7fd02f3c4a51beb
+CRISPASR_VERSION?=754b67289cf1137e3ed722885705f94132fc614f
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/depth-anything-cpp/Makefile

@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=f4e17dea695dd12ae76bea98ba58030996b98118
+DEPTHANYTHING_VERSION?=2028b47ac75a8659c6a9aa617baf09be193eb55f
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/magpie-tts-cpp/.gitignore

@@ -0,0 +1,6 @@
+magpie-tts-cpp
+*.so
+*.dylib
+sources/
+package/
+magpie-tts-models/

backend/go/magpie-tts-cpp/CMakeLists.txt

@@ -0,0 +1,35 @@
+cmake_minimum_required(VERSION 3.16)
+project(gomagpiettscpp LANGUAGES C CXX)
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
+set(MAGPIE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/magpie-tts.cpp)
+
+# Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
+if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
+    set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
+endif()
+
+# The magpie-tts C-API is exported directly by the upstream shared library
+# (magpie_tts_capi_* in libmagpie-tts.so, ggml statically linked inside), so
+# unlike qwen3-tts-cpp / moss-tts-cpp no local C shim is needed -- this wrapper
+# only configures the upstream project as a purego-loadable shared library.
+set(MAGPIE_SHARED       ON  CACHE BOOL "" FORCE)
+set(MAGPIE_BUILD_CLI    OFF CACHE BOOL "" FORCE)
+set(MAGPIE_BUILD_TESTS  OFF CACHE BOOL "" FORCE)
+
+# Upstream FORCE-overwrites GGML_CUDA / GGML_METAL / GGML_VULKAN / GGML_HIP from
+# its own MAGPIE_GGML_* toggles, which would silently discard the -DGGML_*=ON
+# flags the LocalAI Makefile passes for GPU BUILD_TYPEs. Translate them into the
+# MAGPIE_GGML_* vocabulary before add_subdirectory. (GGML_SYCL / GGML_BLAS and
+# the CPU ISA flags are not touched by upstream and reach ggml unchanged.)
+foreach(_be CUDA METAL VULKAN HIP)
+    if(GGML_${_be})
+        set(MAGPIE_GGML_${_be} ON CACHE BOOL "" FORCE)
+    endif()
+endforeach()
+
+add_subdirectory(${MAGPIE_DIR} magpie EXCLUDE_FROM_ALL)
+
+# Place libmagpie-tts.so at the build root, where the Makefile picks it up.
+set_target_properties(magpie-tts PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/magpie-tts-cpp/Makefile

@@ -0,0 +1,140 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+JOBS?=$(shell nproc --ignore=1)
+
+# magpie-tts.cpp version
+MAGPIETTS_REPO?=https://github.com/mudler/magpie-tts.cpp
+MAGPIETTS_CPP_VERSION?=3008ff73fc2d2da9e4d743b09350aa7023e8980c
+SO_TARGET?=libgomagpiettscpp.so
+
+CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
+
+ifeq ($(NATIVE),false)
+	CMAKE_ARGS+=-DGGML_NATIVE=OFF
+endif
+
+ifeq ($(BUILD_TYPE),cublas)
+	CMAKE_ARGS+=-DGGML_CUDA=ON
+else ifeq ($(BUILD_TYPE),openblas)
+	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
+else ifeq ($(BUILD_TYPE),clblas)
+	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
+else ifeq ($(BUILD_TYPE),hipblas)
+	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
+	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
+	ROCM_HOME ?= /opt/rocm
+	ROCM_PATH ?= /opt/rocm
+	export CXX=$(ROCM_HOME)/llvm/bin/clang++
+	export CC=$(ROCM_HOME)/llvm/bin/clang
+	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DGGML_VULKAN=ON
+else ifeq ($(OS),Darwin)
+	ifneq ($(BUILD_TYPE),metal)
+		CMAKE_ARGS+=-DGGML_METAL=OFF
+	else
+		CMAKE_ARGS+=-DGGML_METAL=ON
+		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
+	endif
+endif
+
+ifeq ($(BUILD_TYPE),sycl_f16)
+	CMAKE_ARGS+=-DGGML_SYCL=ON \
+		-DCMAKE_C_COMPILER=icx \
+		-DCMAKE_CXX_COMPILER=icpx \
+		-DGGML_SYCL_F16=ON
+endif
+
+ifeq ($(BUILD_TYPE),sycl_f32)
+	CMAKE_ARGS+=-DGGML_SYCL=ON \
+		-DCMAKE_C_COMPILER=icx \
+		-DCMAKE_CXX_COMPILER=icpx
+endif
+
+sources/magpie-tts.cpp:
+	mkdir -p sources/magpie-tts.cpp
+	cd sources/magpie-tts.cpp && \
+	git init && \
+	git remote add origin $(MAGPIETTS_REPO) && \
+	git fetch origin && \
+	git checkout $(MAGPIETTS_CPP_VERSION) && \
+	git submodule update --init --recursive --depth 1 --single-branch
+
+# Detect OS
+UNAME_S := $(shell uname -s)
+
+# Only build CPU variants on Linux
+ifeq ($(UNAME_S),Linux)
+	VARIANT_TARGETS = libgomagpiettscpp-avx.so libgomagpiettscpp-avx2.so libgomagpiettscpp-avx512.so libgomagpiettscpp-fallback.so
+else
+	# On non-Linux (e.g., Darwin), build only fallback variant (as a dylib)
+	VARIANT_TARGETS = libgomagpiettscpp-fallback.dylib
+endif
+
+magpie-tts-cpp: main.go gomagpiettscpp.go $(VARIANT_TARGETS)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o magpie-tts-cpp ./
+
+package: magpie-tts-cpp
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf libgomagpiettscpp*.so libgomagpiettscpp*.dylib package sources/magpie-tts.cpp magpie-tts-cpp
+
+purge:
+	rm -rf build*
+
+# Variants must build sequentially
+.NOTPARALLEL:
+
+# Build all variants (Linux only)
+ifeq ($(UNAME_S),Linux)
+libgomagpiettscpp-avx.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx${RESET})
+	SO_TARGET=libgomagpiettscpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx.so
+
+libgomagpiettscpp-avx2.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx2${RESET})
+	SO_TARGET=libgomagpiettscpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx2.so
+
+libgomagpiettscpp-avx512.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:avx512${RESET})
+	SO_TARGET=libgomagpiettscpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-avx512.so
+endif
+
+# Build fallback variant (all platforms)
+libgomagpiettscpp-fallback.so: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:fallback${RESET})
+	SO_TARGET=libgomagpiettscpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-fallback.so
+
+# Build fallback variant as a dylib (Darwin)
+libgomagpiettscpp-fallback.dylib: sources/magpie-tts.cpp
+	$(info ${GREEN}I magpie-tts-cpp build info:fallback (dylib)${RESET})
+	SO_TARGET=libgomagpiettscpp-fallback.dylib CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
+	rm -rf build-libgomagpiettscpp-fallback.dylib
+
+libgomagpiettscpp-custom: CMakeLists.txt
+	mkdir -p build-$(SO_TARGET) && \
+	cd build-$(SO_TARGET) && \
+	cmake .. $(CMAKE_ARGS) && \
+	cmake --build . --config Release -j$(JOBS) --target magpie-tts && \
+	cd .. && \
+	(mv build-$(SO_TARGET)/libmagpie-tts.so ./$(SO_TARGET) 2>/dev/null || \
+	 mv build-$(SO_TARGET)/libmagpie-tts.dylib ./$(SO_TARGET) 2>/dev/null)
+
+test: magpie-tts-cpp
+	@echo "Running magpie-tts-cpp tests..."
+	bash test.sh
+	@echo "magpie-tts-cpp tests completed."
+
+all: magpie-tts-cpp package

backend/go/magpie-tts-cpp/README.md

@@ -0,0 +1,69 @@
+# Magpie TTS C++ backend
+
+This backend runs NVIDIA's **Magpie TTS Multilingual 357M** GGUF through
+[magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp), a from-scratch
+C++/ggml port (model + NanoCodec + tokenizer + G2P dictionaries in one
+self-contained GGUF, no Python at inference time). It generates **22.05 kHz
+mono** speech in 5 baked voices across 9+ languages.
+
+The library is loaded via purego (cgo-less `dlopen`) exactly like
+`qwen3-tts-cpp` / `moss-tts-cpp`; the flat C-API (`magpie_tts_capi_*`) is
+exported directly by the upstream shared library, so there is no local C shim.
+
+## Model configuration
+
+The model path points at the single GGUF:
+
+```yaml
+name: magpie-tts-cpp
+backend: magpie-tts-cpp
+parameters:
+  model: magpie-tts-multilingual-357m-q8_0.gguf
+known_usecases:
+  - tts
+options:
+  - "speaker:Aria"   # optional default voice (Aria, Jason, John, Leo, Sofia, or 0-4)
+  - "language:en"    # optional default language
+```
+
+GGUFs live at
+[mudler/magpie-tts.cpp-gguf](https://huggingface.co/mudler/magpie-tts.cpp-gguf)
+(q8_0 recommended: near-lossless, ~624 MB, fastest decode).
+
+## Voices and languages
+
+Magpie has 5 baked speakers - `Aria`, `Jason`, `John`, `Leo`, `Sofia` - and no
+voice cloning. The request `voice` accepts the names case-insensitively or the
+indices `0`-`4`; empty selects Aria. Languages: `en`, `es`, `de`, `fr`, `it`,
+`pt-BR`, `hi`, `vi`, `ko`, `ar-AE`, `ar-SA`, `ar-MSA` (case-insensitive;
+default `en`).
+
+## API example
+
+```bash
+curl http://localhost:8080/v1/audio/speech \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "magpie-tts-cpp",
+    "input": "Hello world, this is a test of the text to speech system.",
+    "voice": "sofia",
+    "language": "en"
+  }' \
+  --output speech.wav
+```
+
+## Native end-to-end test
+
+The labeled test loads a real GGUF, synthesizes WAVs (verifying rate, layout
+and non-silence), and exercises the streaming path:
+
+```bash
+make -C backend/go/magpie-tts-cpp magpie-tts-cpp
+
+MAGPIETTS_MODEL=/path/to/magpie-tts-multilingual-357m-q8_0.gguf \
+MAGPIETTS_LIBRARY=backend/go/magpie-tts-cpp/libgomagpiettscpp-fallback.so \
+  go test ./backend/go/magpie-tts-cpp -ginkgo.label-filter=e2e
+```
+
+`bash test.sh` does the same and auto-downloads the q8_0 GGUF when
+`MAGPIETTS_MODEL` is unset.

backend/go/magpie-tts-cpp/audio.go

@@ -0,0 +1,93 @@
+package main
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"os"
+
+	"github.com/go-audio/audio"
+	"github.com/go-audio/wav"
+)
+
+// magpieSampleRate is the fixed Magpie TTS Multilingual (NanoCodec) output
+// rate: 22.05 kHz.
+const magpieSampleRate = 22050
+
+// magpieChannels is the fixed output layout: mono.
+const magpieChannels = 1
+
+// wavHeaderMono returns a 44-byte WAV header for a streaming 16-bit mono PCM
+// stream at 22050 Hz, with placeholder (0xFFFFFFFF) sizes since the total
+// length is unknown up front. Emitted as the first chunk of TTSStream so the
+// HTTP layer receives a self-describing WAV.
+func wavHeaderMono() []byte {
+	const blockAlign = magpieChannels * 2 // 16-bit mono
+	var buf bytes.Buffer
+	w := func(v any) { _ = binary.Write(&buf, binary.LittleEndian, v) }
+	buf.WriteString("RIFF")
+	w(uint32(0xFFFFFFFF))
+	buf.WriteString("WAVE")
+	buf.WriteString("fmt ")
+	w(uint32(16))                            // Subchunk1Size
+	w(uint16(1))                             // PCM
+	w(uint16(magpieChannels))                // mono
+	w(uint32(magpieSampleRate))              // sample rate
+	w(uint32(magpieSampleRate * blockAlign)) // byte rate = SR * blockAlign
+	w(uint16(blockAlign))                    // block align
+	w(uint16(16))                            // bits per sample
+	buf.WriteString("data")
+	w(uint32(0xFFFFFFFF))
+	return buf.Bytes()
+}
+
+// floatToPCM16LE clamps each sample to [-1,1] and encodes it as little-endian
+// signed 16-bit PCM.
+func floatToPCM16LE(samples []float32) []byte {
+	out := make([]byte, len(samples)*2)
+	for i, s := range samples {
+		if s > 1 {
+			s = 1
+		} else if s < -1 {
+			s = -1
+		}
+		v := int16(s * 32767)
+		out[i*2] = byte(v)        // #nosec G115 -- intentional little-endian split of a clamped int16
+		out[i*2+1] = byte(v >> 8) // #nosec G115 -- high byte of the same clamped int16
+	}
+	return out
+}
+
+// writeWAVMono writes float samples as a finalized 16-bit mono WAV at
+// 22050 Hz.
+func writeWAVMono(dst string, samples []float32) error {
+	f, err := os.Create(dst) // #nosec G304 -- dst is the server-chosen output path from the TTS request, not user-traversable
+	if err != nil {
+		return fmt.Errorf("magpie-tts: create %q: %w", dst, err)
+	}
+	enc := wav.NewEncoder(f, magpieSampleRate, 16, magpieChannels, 1)
+	ints := make([]int, len(samples))
+	for i, s := range samples {
+		if s > 1 {
+			s = 1
+		} else if s < -1 {
+			s = -1
+		}
+		ints[i] = int(s * 32767)
+	}
+	b := &audio.IntBuffer{
+		Format:         &audio.Format{NumChannels: magpieChannels, SampleRate: magpieSampleRate},
+		Data:           ints,
+		SourceBitDepth: 16,
+	}
+	if err := enc.Write(b); err != nil {
+		_ = enc.Close()
+		_ = f.Close()
+		return fmt.Errorf("magpie-tts: encode WAV: %w", err)
+	}
+	if err := enc.Close(); err != nil {
+		_ = f.Close()
+		return fmt.Errorf("magpie-tts: finalize WAV: %w", err)
+	}
+	return f.Close()
+}

backend/go/magpie-tts-cpp/e2e_test.go

@@ -0,0 +1,127 @@
+package main
+
+import (
+	"encoding/binary"
+	"math"
+	"os"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func ttsReq(text, voice, lang, dst string) *pb.TTSRequest {
+	r := &pb.TTSRequest{Text: text, Voice: voice, Dst: dst}
+	if lang != "" {
+		r.Language = &lang
+	}
+	return r
+}
+
+// wavRMS parses a 16-bit PCM WAV file and returns (sampleRate, channels, RMS
+// of the normalized samples).
+func wavRMS(path string) (int, int, float64) {
+	data, err := os.ReadFile(path) // #nosec G304 -- test-owned temp file
+	Expect(err).ToNot(HaveOccurred())
+	Expect(len(data)).To(BeNumerically(">", 44))
+	Expect(string(data[0:4])).To(Equal("RIFF"))
+	Expect(string(data[8:12])).To(Equal("WAVE"))
+	channels := int(binary.LittleEndian.Uint16(data[22:24]))
+	rate := int(binary.LittleEndian.Uint32(data[24:28]))
+	// Find the data chunk (go-audio writes fmt first, data after).
+	off := 12
+	for off+8 <= len(data) {
+		id := string(data[off : off+4])
+		sz := int(binary.LittleEndian.Uint32(data[off+4 : off+8]))
+		if id == "data" {
+			pcm := data[off+8:]
+			if sz < len(pcm) {
+				pcm = pcm[:sz]
+			}
+			var sum float64
+			n := len(pcm) / 2
+			for i := 0; i < n; i++ {
+				s := float64(int16(binary.LittleEndian.Uint16(pcm[i*2:]))) / 32768.0
+				sum += s * s
+			}
+			Expect(n).To(BeNumerically(">", 0))
+			return rate, channels, math.Sqrt(sum / float64(n))
+		}
+		off += 8 + sz
+	}
+	Fail("no data chunk found in " + path)
+	return 0, 0, 0
+}
+
+var _ = Describe("magpie-tts-cpp e2e", Label("e2e"), func() {
+	var (
+		loaded  bool
+		backend *MagpieTtsCpp
+	)
+
+	BeforeEach(func() {
+		modelPath := os.Getenv("MAGPIETTS_MODEL")
+		if modelPath == "" {
+			Skip("MAGPIETTS_MODEL not set; skipping e2e")
+		}
+		if !loaded {
+			lib := os.Getenv("MAGPIETTS_LIBRARY")
+			if lib == "" {
+				lib = "./libgomagpiettscpp-fallback.so"
+			}
+			h, err := purego.Dlopen(lib, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+			Expect(err).ToNot(HaveOccurred())
+			purego.RegisterLibFunc(&CppAbiVersion, h, "magpie_tts_capi_abi_version")
+			purego.RegisterLibFunc(&CppLoad, h, "magpie_tts_capi_load")
+			purego.RegisterLibFunc(&CppFree, h, "magpie_tts_capi_free")
+			purego.RegisterLibFunc(&CppSynthesize, h, "magpie_tts_capi_synthesize")
+			purego.RegisterLibFunc(&CppFreeAudio, h, "magpie_tts_capi_free_audio")
+			purego.RegisterLibFunc(&CppLastError, h, "magpie_tts_capi_last_error")
+
+			backend = &MagpieTtsCpp{}
+			Expect(backend.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
+			loaded = true
+		}
+	})
+
+	It("synthesizes a non-silent 22.05 kHz mono WAV via TTS", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		Expect(backend.TTS(ttsReq("Hello world, this is a test.", "Aria", "en", dst))).To(Succeed())
+		rate, channels, rms := wavRMS(dst)
+		Expect(rate).To(Equal(22050))
+		Expect(channels).To(Equal(1))
+		Expect(rms).To(BeNumerically(">", 0.01), "audio should not be silent")
+	})
+
+	It("accepts a case-insensitive voice and a speaker index", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		Expect(backend.TTS(ttsReq("Short test.", "sofia", "en", dst))).To(Succeed())
+		Expect(backend.TTS(ttsReq("Short test.", "1", "en", dst))).To(Succeed())
+	})
+
+	It("rejects an unknown voice before reaching the engine", func() {
+		dst := GinkgoT().TempDir() + "/out.wav"
+		err := backend.TTS(ttsReq("Short test.", "not-a-speaker", "en", dst))
+		Expect(err).To(MatchError(ContainSubstring("unknown voice")))
+	})
+
+	It("streams a self-describing WAV via TTSStream", func() {
+		results := make(chan []byte, 4096)
+		done := make(chan error, 1)
+		go func() { done <- backend.TTSStream(ttsReq("Hello there, streaming test.", "", "", ""), results) }()
+
+		var chunks int
+		var first []byte
+		for c := range results {
+			if chunks == 0 {
+				first = c
+			}
+			chunks++
+		}
+		Expect(<-done).ToNot(HaveOccurred())
+		Expect(chunks).To(BeNumerically(">=", 2))
+		Expect(string(first[0:4])).To(Equal("RIFF"))
+		Expect(string(first[8:12])).To(Equal("WAVE"))
+	})
+})

backend/go/magpie-tts-cpp/gomagpiettscpp.go

@@ -0,0 +1,147 @@
+package main
+
+import (
+	"fmt"
+	"path/filepath"
+	"unsafe"
+
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// capiABIVersion is the magpie_tts_capi.h surface this backend binds. Bumped
+// upstream on any breaking signature/semantics change; refuse to run on a
+// mismatch instead of crashing inside a miscompiled call.
+const capiABIVersion = 1
+
+var (
+	// magpie_tts_capi_abi_version() int
+	CppAbiVersion func() int
+	// magpie_tts_capi_load(gguf_path) -> ctx (NULL on failure)
+	CppLoad func(path string) uintptr
+	// magpie_tts_capi_free(ctx)
+	CppFree func(ctx uintptr)
+	// magpie_tts_capi_synthesize(ctx, text, language, speaker, out_n) -> float*
+	// 22050 Hz mono f32 PCM in [-1,1]; NULL on failure (see last_error).
+	CppSynthesize func(ctx uintptr, text, language, speaker string, outN unsafe.Pointer) uintptr
+	// magpie_tts_capi_free_audio(ptr)
+	CppFreeAudio func(ptr uintptr)
+	// magpie_tts_capi_last_error(ctx) -> const char* (ctx-owned, "" if none)
+	CppLastError func(ctx uintptr) string
+)
+
+// MagpieTtsCpp serves the Magpie TTS Multilingual 357M GGUF through the
+// magpie-tts.cpp C-API. The context is stateful (per-call error buffer, reused
+// graph allocator) and NOT safe for concurrent synthesize calls, so
+// base.SingleThread serializes everything behind the server-level lock.
+type MagpieTtsCpp struct {
+	base.SingleThread
+	ctx  uintptr
+	opts loadOptions
+}
+
+func (m *MagpieTtsCpp) Load(opts *pb.ModelOptions) error {
+	model := opts.ModelFile
+	if model == "" {
+		model = opts.ModelPath
+	}
+	if !filepath.IsAbs(model) && opts.ModelPath != "" {
+		model = filepath.Join(opts.ModelPath, model)
+	}
+
+	m.opts = parseOptions(opts.Options)
+
+	if abi := CppAbiVersion(); abi != capiABIVersion {
+		return fmt.Errorf("magpie-tts: C-API ABI mismatch: library reports v%d, backend built for v%d", abi, capiABIVersion)
+	}
+
+	xlog.Info("[magpie-tts-cpp] Load", "model", model)
+
+	ctx := CppLoad(model)
+	if ctx == 0 {
+		// Load failures have no context to query last_error on; the C side
+		// logs the reason to stderr.
+		return fmt.Errorf("magpie-tts: failed to load model %q", model)
+	}
+	m.ctx = ctx
+	return nil
+}
+
+// lastError surfaces the context's last error, falling back to a generic
+// message when the C side left it empty.
+func (m *MagpieTtsCpp) lastError() string {
+	if m.ctx == 0 {
+		return "no model loaded"
+	}
+	if e := CppLastError(m.ctx); e != "" {
+		return e
+	}
+	return "unknown error"
+}
+
+// synthesize runs one C-API synthesis and copies the PCM out of C memory.
+func (m *MagpieTtsCpp) synthesize(req *pb.TTSRequest) ([]float32, error) {
+	if m.ctx == 0 {
+		return nil, fmt.Errorf("magpie-tts: no model loaded")
+	}
+	if req.Text == "" {
+		return nil, fmt.Errorf("magpie-tts: TTS requires text")
+	}
+	speaker, err := resolveSpeaker(req.Voice, m.opts.speaker)
+	if err != nil {
+		return nil, err
+	}
+	lang := resolveLanguage(req.Language, m.opts.language)
+
+	var n int32
+	ptr := CppSynthesize(m.ctx, req.Text, lang, speaker, unsafe.Pointer(&n)) // #nosec G103 -- out-param for the purego-bound C-API
+	if ptr == 0 {
+		return nil, fmt.Errorf("magpie-tts: synthesis failed: %s", m.lastError())
+	}
+	// Register the free as soon as we own a non-null buffer, so the n<=0 guard
+	// below cannot leak it (defensive: the C contract returns NULL on failure).
+	defer CppFreeAudio(ptr)
+	if n <= 0 {
+		return nil, fmt.Errorf("magpie-tts: synthesis produced no samples")
+	}
+	//nolint:govet // C-allocated PCM, copied out before free
+	src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) // #nosec G103 -- C-allocated PCM, copied out before free
+	out := make([]float32, int(n))
+	copy(out, src)
+	return out, nil
+}
+
+func (m *MagpieTtsCpp) TTS(req *pb.TTSRequest) error {
+	if req.Dst == "" {
+		return fmt.Errorf("magpie-tts: TTS requires a destination path")
+	}
+	samples, err := m.synthesize(req)
+	if err != nil {
+		return err
+	}
+	return writeWAVMono(req.Dst, samples)
+}
+
+// TTSStream synthesizes one-shot (the magpie C-API has no streaming call) and
+// then emits a self-describing mono WAV: a header chunk followed by the PCM in
+// fixed-size slices, so the HTTP layer still receives a streamed WAV (the gRPC
+// TTSStream path never sets Message, so the backend owns the header - see
+// core/backend/tts.go:ModelTTSStream).
+func (m *MagpieTtsCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
+	defer close(results)
+	samples, err := m.synthesize(req)
+	if err != nil {
+		return err
+	}
+	results <- wavHeaderMono()
+	const sampleChunk = 4096 // mono samples per emitted chunk
+	for off := 0; off < len(samples); off += sampleChunk {
+		end := off + sampleChunk
+		if end > len(samples) {
+			end = len(samples)
+		}
+		results <- floatToPCM16LE(samples[off:end])
+	}
+	return nil
+}

backend/go/magpie-tts-cpp/magpiettscpp_test.go

@@ -0,0 +1,115 @@
+package main
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestMagpieTtsCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "magpie-tts-cpp suite")
+}
+
+var _ = Describe("resolveSpeaker", func() {
+	It("canonicalizes case-insensitive names", func() {
+		for in, want := range map[string]string{
+			"aria": "Aria", "JASON": "Jason", "john": "John",
+			"Leo": "Leo", "sofia": "Sofia",
+		} {
+			got, err := resolveSpeaker(in, "")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(got).To(Equal(want))
+		}
+	})
+	It("accepts indices 0-4", func() {
+		for in, want := range map[string]string{
+			"0": "Aria", "1": "Jason", "2": "John", "3": "Leo", "4": "Sofia",
+		} {
+			got, err := resolveSpeaker(in, "")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(got).To(Equal(want))
+		}
+	})
+	It("selects the engine default on empty", func() {
+		got, err := resolveSpeaker("", "")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(BeEmpty())
+	})
+	It("falls back to the model-level default speaker", func() {
+		got, err := resolveSpeaker("", "sofia")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("Sofia"))
+	})
+	It("prefers the request voice over the fallback", func() {
+		got, err := resolveSpeaker("leo", "sofia")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("Leo"))
+	})
+	It("rejects out-of-range indices", func() {
+		_, err := resolveSpeaker("5", "")
+		Expect(err).To(HaveOccurred())
+		_, err = resolveSpeaker("-1", "")
+		Expect(err).To(HaveOccurred())
+	})
+	It("rejects unknown names with the valid choices", func() {
+		_, err := resolveSpeaker("serena", "")
+		Expect(err).To(MatchError(ContainSubstring("Aria, Jason, John, Leo, Sofia")))
+	})
+})
+
+var _ = Describe("resolveLanguage", func() {
+	strp := func(s string) *string { return &s }
+
+	It("defaults to empty (engine picks en)", func() {
+		Expect(resolveLanguage(nil, "")).To(BeEmpty())
+	})
+	It("canonicalizes case for known codes", func() {
+		Expect(resolveLanguage(strp("EN"), "")).To(Equal("en"))
+		Expect(resolveLanguage(strp("pt-br"), "")).To(Equal("pt-BR"))
+		Expect(resolveLanguage(strp("AR-MSA"), "")).To(Equal("ar-MSA"))
+	})
+	It("falls back to the model-level default language", func() {
+		Expect(resolveLanguage(nil, "de")).To(Equal("de"))
+		Expect(resolveLanguage(strp(""), "PT-BR")).To(Equal("pt-BR"))
+	})
+	It("prefers the request language over the fallback", func() {
+		Expect(resolveLanguage(strp("it"), "de")).To(Equal("it"))
+	})
+	It("passes unknown codes through verbatim", func() {
+		Expect(resolveLanguage(strp("zh"), "")).To(Equal("zh"))
+	})
+})
+
+var _ = Describe("parseOptions", func() {
+	It("reads speaker and language defaults", func() {
+		o := parseOptions([]string{"speaker:Jason", "language:de"})
+		Expect(o.speaker).To(Equal("Jason"))
+		Expect(o.language).To(Equal("de"))
+	})
+	It("accepts the voice/lang aliases and ignores unknown keys", func() {
+		o := parseOptions([]string{"voice: sofia ", "lang: pt-BR", "bogus:1", "novalue"})
+		Expect(o.speaker).To(Equal("sofia"))
+		Expect(o.language).To(Equal("pt-BR"))
+	})
+})
+
+var _ = Describe("audio encoding", func() {
+	It("emits a well-formed streaming mono WAV header", func() {
+		h := wavHeaderMono()
+		Expect(h).To(HaveLen(44))
+		Expect(string(h[0:4])).To(Equal("RIFF"))
+		Expect(string(h[8:12])).To(Equal("WAVE"))
+		// channels (offset 22) == 1, sample rate (offset 24) == 22050
+		Expect(int(h[22]) | int(h[23])<<8).To(Equal(1))
+		Expect(int(h[24]) | int(h[25])<<8 | int(h[26])<<16 | int(h[27])<<24).To(Equal(22050))
+	})
+	It("clamps float PCM to int16", func() {
+		b := floatToPCM16LE([]float32{2, -2, 0})
+		Expect(b).To(HaveLen(6))
+		Expect(int16(uint16(b[0]) | uint16(b[1])<<8)).To(Equal(int16(32767)))
+		Expect(int16(uint16(b[2]) | uint16(b[3])<<8)).To(Equal(int16(-32767)))
+		Expect(int16(uint16(b[4]) | uint16(b[5])<<8)).To(Equal(int16(0)))
+	})
+})

backend/go/magpie-tts-cpp/main.go

@@ -0,0 +1,54 @@
+package main
+
+// Note: this is started internally by LocalAI and a server is allocated for each model
+import (
+	"flag"
+	"os"
+	"runtime"
+
+	"github.com/ebitengine/purego"
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+type LibFuncs struct {
+	FuncPtr any
+	Name    string
+}
+
+func main() {
+	libName := os.Getenv("MAGPIETTS_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "./libgomagpiettscpp-fallback.dylib"
+		} else {
+			libName = "./libgomagpiettscpp-fallback.so"
+		}
+	}
+
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		panic(err)
+	}
+
+	libFuncs := []LibFuncs{
+		{&CppAbiVersion, "magpie_tts_capi_abi_version"},
+		{&CppLoad, "magpie_tts_capi_load"},
+		{&CppFree, "magpie_tts_capi_free"},
+		{&CppSynthesize, "magpie_tts_capi_synthesize"},
+		{&CppFreeAudio, "magpie_tts_capi_free_audio"},
+		{&CppLastError, "magpie_tts_capi_last_error"},
+	}
+	for _, lf := range libFuncs {
+		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &MagpieTtsCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/magpie-tts-cpp/options.go

@@ -0,0 +1,108 @@
+package main
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+)
+
+// loadOptions holds the parsed model-level options. Magpie is a single
+// self-contained GGUF (model + codec + tokenizer + G2P dictionaries), so the
+// options only cover synthesis defaults.
+type loadOptions struct {
+	// speaker is the default baked speaker when a request has no voice.
+	speaker string
+	// language is the default language when a request has none ("" = engine
+	// default, which is "en").
+	language string
+}
+
+func splitOption(o string) (key, value string, ok bool) {
+	i := strings.Index(o, ":")
+	if i < 0 {
+		return "", "", false
+	}
+	return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true
+}
+
+// parseOptions reads the backend "key:value" option slice. Unknown keys are
+// ignored.
+func parseOptions(opts []string) loadOptions {
+	var o loadOptions
+	for _, oo := range opts {
+		key, value, ok := splitOption(oo)
+		if !ok {
+			continue
+		}
+		switch key {
+		case "speaker", "voice":
+			o.speaker = value
+		case "language", "lang":
+			o.language = value
+		}
+	}
+	return o
+}
+
+// magpieSpeakers are the baked speakers of Magpie TTS Multilingual 357M, in
+// index order (the engine matches names exactly, so the Go side canonicalizes
+// case-insensitive names and 0-4 indices to these strings).
+var magpieSpeakers = []string{"Aria", "Jason", "John", "Leo", "Sofia"}
+
+// resolveSpeaker maps the request voice (falling back to the model-level
+// default) onto a canonical baked speaker name. Accepted forms:
+// case-insensitive names (aria, JASON, ...) and indices 0-4. Empty selects the
+// engine default (speaker 0, Aria). Anything else is rejected with the valid
+// choices, instead of surfacing the engine's late error.
+func resolveSpeaker(voice, fallback string) (string, error) {
+	v := strings.TrimSpace(voice)
+	if v == "" {
+		v = strings.TrimSpace(fallback)
+	}
+	if v == "" {
+		return "", nil
+	}
+	if idx, err := strconv.Atoi(v); err == nil {
+		if idx < 0 || idx >= len(magpieSpeakers) {
+			return "", fmt.Errorf("magpie-tts: speaker index %d out of range 0-%d", idx, len(magpieSpeakers)-1)
+		}
+		return magpieSpeakers[idx], nil
+	}
+	for _, s := range magpieSpeakers {
+		if strings.EqualFold(s, v) {
+			return s, nil
+		}
+	}
+	return "", fmt.Errorf("magpie-tts: unknown voice %q (valid: %s, or 0-%d)",
+		voice, strings.Join(magpieSpeakers, ", "), len(magpieSpeakers)-1)
+}
+
+// magpieLanguages are the canonical language codes the tokenizer's language
+// map knows (exact-match on the C side), keyed by their lowercase form so
+// requests can be case-insensitive.
+var magpieLanguages = map[string]string{
+	"en": "en", "es": "es", "de": "de", "fr": "fr", "it": "it",
+	"pt-br": "pt-BR", "hi": "hi", "vi": "vi", "ko": "ko",
+	"ar-ae": "ar-AE", "ar-sa": "ar-SA", "ar-msa": "ar-MSA",
+}
+
+// resolveLanguage picks the request language, else the model-level default,
+// else "" (the engine defaults to "en"), canonicalizing case for the known
+// codes. Unknown codes pass through verbatim so the engine reports them with
+// its own exact-vocabulary error.
+func resolveLanguage(reqLang *string, fallback string) string {
+	l := ""
+	if reqLang != nil {
+		l = strings.TrimSpace(*reqLang)
+	}
+	if l == "" {
+		l = strings.TrimSpace(fallback)
+	}
+	if l == "" {
+		return ""
+	}
+	if canon, ok := magpieLanguages[strings.ToLower(l)]; ok {
+		return canon
+	}
+	return l
+}

backend/go/magpie-tts-cpp/package.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/magpie-tts-cpp $CURDIR/package/
+cp -fv $CURDIR/libgomagpiettscpp-*.so $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/libgomagpiettscpp-*.dylib $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/magpie-tts-cpp/run.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+echo "CPU info:"
+if [ "$(uname)" != "Darwin" ]; then
+	grep -e "model\sname" /proc/cpuinfo | head -1
+	grep -e "flags" /proc/cpuinfo | head -1
+fi
+
+if [ "$(uname)" = "Darwin" ]; then
+	# macOS: single dylib variant (Metal or Accelerate)
+	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.dylib"
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.so"
+
+	if grep -q -e "\savx\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX    found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx.so"
+		fi
+	fi
+
+	if grep -q -e "\savx2\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX2   found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx2.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx2.so"
+		fi
+	fi
+
+	# Check avx 512
+	if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
+		echo "CPU:    AVX512F found OK"
+		if [ -e "$CURDIR"/libgomagpiettscpp-avx512.so ]; then
+			LIBRARY="$CURDIR/libgomagpiettscpp-avx512.so"
+		fi
+	fi
+
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+fi
+
+export MAGPIETTS_LIBRARY=$LIBRARY
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LIBRARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/magpie-tts-cpp "$@"
+fi
+
+echo "Using library: $LIBRARY"
+exec "$CURDIR"/magpie-tts-cpp "$@"

backend/go/magpie-tts-cpp/test.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+echo "Running magpie-tts-cpp backend tests..."
+
+# Auto-download the q8_0 GGUF only when MAGPIETTS_MODEL is not set.
+if [ -z "$MAGPIETTS_MODEL" ]; then
+    MODEL_DIR="./magpie-tts-models"
+    mkdir -p "$MODEL_DIR"
+    REPO_ID="mudler/magpie-tts.cpp-gguf"
+    BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
+    FILE="magpie-tts-multilingual-357m-q8_0.gguf"
+    dest="${MODEL_DIR}/${FILE}"
+    if [ -f "${dest}" ]; then
+        echo "  [skip] ${FILE}"
+    else
+        echo "  [download] ${FILE}..."
+        curl -L -o "${dest}" "${BASE_URL}/${FILE}" --progress-bar
+    fi
+    export MAGPIETTS_MODEL="${dest}"
+fi
+
+go test -v -timeout 1200s .
+
+echo "All magpie-tts-cpp tests passed."

backend/go/parakeet-cpp/Makefile

@@ -1,6 +1,6 @@
 # parakeet-cpp backend Makefile.
 #
-# Upstream pin lives below as PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
+# Upstream pin lives below as PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
 # (.github/bump_deps.sh) can find and update it - matches the
 # whisper.cpp / ds4 / vibevoice-cpp convention.
 #
@@ -15,7 +15,7 @@
 # That's what the L0 smoke test uses. The default target below does the
 # proper clone-at-pin + cmake build so CI doesn't need a side-checkout.
 
-PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
+PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
 PARAKEET_REPO?=https://github.com/mudler/parakeet.cpp
 
 GOCMD?=go

backend/go/qwen3-tts-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # qwentts.cpp version
 QWEN3TTS_REPO?=https://github.com/ServeurpersoCom/qwentts.cpp
-QWEN3TTS_CPP_VERSION?=82cd05b9f3a175612dc89fd6943e610fab096ef5
+QWEN3TTS_CPP_VERSION?=35ebe5376b82a0a59d008586d55bbe623d449011
 SO_TARGET?=libgoqwen3ttscpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=8a51eb92848c1327a5aaeff5ad81a7a9a2435255
+STABLEDIFFUSION_GGML_VERSION?=22516991cbdf725e69b0b4a87e52ca16cce07c2d
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/vllm-cpp/.gitignore

@@ -0,0 +1,6 @@
+sources/
+build/
+package/
+vllm-cpp
+libvllm.so
+libvllm.dylib

backend/go/vllm-cpp/Makefile

@@ -0,0 +1,102 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+# nproc doesn't exist on the macOS runners: an empty JOBS turns `-j$(JOBS)`
+# into bare `-j` (unlimited clang jobs), which swap-thrashes the 3-core Mac
+# until the 6h GHA timeout. Fall back to sysctl there, then to a constant.
+JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+# vllm.cpp version
+VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
+VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845
+
+# The backend consumes only the stable C ABI (libvllm + include/vllm.h), so the
+# server, examples and tests of the engine are never built here.
+CMAKE_ARGS+=-DVLLM_CPP_SERVER=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_BUILD_EXAMPLES=OFF
+CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
+
+# vllm.cpp sets no global -march: SIMD tiers are per-file with runtime dispatch,
+# so ONE portable library serves every CPU of the target arch (unlike the
+# ggml-based backends and their avx/avx2/avx512 variant builds).
+UNAME_M := $(shell uname -m)
+
+ifeq ($(BUILD_TYPE),cublas)
+	# Blackwell-family targets only: other CUDA arches are build-supported
+	# upstream but have no runtime-proven fast path. amd64 gets the consumer
+	# (120a) + GB10 (121a) fat binary; arm64 CUDA (l4t-style images, DGX
+	# Spark) is GB10 only. Triton-AOT GDN cubins are vendored per-arch, no
+	# Python needed to consume them.
+	ifeq ($(UNAME_M),x86_64)
+		# NO -DVLLM_CPP_TRITON on fat builds: the vendored Triton-AOT cubin
+		# trees are per-arch and the engine refuses a multi-arch build unless
+		# pinned to one tree (unsound for the other arch). The non-AOT GDN
+		# path serves the fat binary; single-arch builds keep the cubins.
+		#
+		# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks
+		# compute_121a (GB10) and its ptxas rejects the sm_120a NVFP4 MMA
+		# kernels ("Vector type too large"), so no cuda-12 variant is shipped.
+		ifeq ($(CUDA_MAJOR_VERSION),12)
+      $(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
+		endif
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=120a;121a"
+	else
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
+	endif
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF
+else ifeq ($(BUILD_TYPE),metal)
+	CMAKE_ARGS+=-DVLLM_CPP_METAL=ON
+else
+	CMAKE_ARGS+=-DVLLM_CPP_CUDA=OFF
+endif
+
+UNAME_S := $(shell uname -s)
+ifeq ($(UNAME_S),Darwin)
+	LIB=libvllm.dylib
+else
+	LIB=libvllm.so
+endif
+
+sources/vllm.cpp:
+	mkdir -p sources/vllm.cpp
+	cd sources/vllm.cpp && \
+	git init && \
+	git remote add origin $(VLLM_CPP_REPO) && \
+	git fetch --depth 1 origin $(VLLM_CPP_VERSION) && \
+	git checkout FETCH_HEAD
+
+$(LIB): sources/vllm.cpp
+	mkdir -p build && \
+	cd build && \
+	cmake ../sources/vllm.cpp $(CMAKE_ARGS) && \
+	cmake --build . --config Release -j$(JOBS) --target vllm_shared
+	cp -fL build/$(LIB) ./$(LIB)
+
+vllm-cpp: main.go govllmcpp.go backend.go options.go $(LIB)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vllm-cpp ./
+
+package: vllm-cpp
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf libvllm.so libvllm.dylib package sources/vllm.cpp vllm-cpp
+
+purge:
+	rm -rf build
+
+.NOTPARALLEL:
+
+# The unit specs are pure Go (struct mirrors, option mapping, load
+# validation): no libvllm build is needed. The e2e specs skip unless
+# VLLM_CPP_MODEL points at a real model (then build the lib first).
+test:
+	@echo "Running vllm-cpp tests..."
+	bash test.sh
+	@echo "vllm-cpp tests completed."
+
+all: vllm-cpp package

backend/go/vllm-cpp/README.md

@@ -0,0 +1,45 @@
+# vllm-cpp backend
+
+LocalAI text-generation backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
+the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching,
+safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
+inference time.
+
+The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
+ABI v2) through purego:
+
+- `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
+  directory (`config.json` + safetensors). `context_size` maps to
+  `max_model_len`; `options: ["block_size:<n>", "num_blocks:<n>",
+  "max_num_seqs:<n>"]` size the KV cache and scheduler admission.
+- `Predict` -> `vllm_complete` (blocking).
+- `PredictStream` -> `vllm_complete_stream`; concurrent gRPC requests batch
+  continuously in the engine's shared AsyncLLM scheduler.
+- Chat / tool calling rides the SAME code path as the llama.cpp autoparser:
+  with `use_tokenizer_template: true` the backend implements
+  `PredictRich`/`PredictStreamRich` over the ABI v3 chat entry points
+  (`vllm_chat` / `vllm_chat_stream`). The ENGINE applies the model's chat
+  template (GGUF `tokenizer.chat_template` or `tokenizer_config.json`),
+  decides when a tool call engages (`tool_choice: auto` lowers to a LAZY
+  structural-tag decode constraint; `required`/named force one), parses tool
+  calls with its streaming Hermes-style parser, and the backend maps each
+  `chat.completion.chunk` onto `ChatDelta`/`ToolCallDelta` protos.
+- Without structured messages the plain path applies:
+  `PredictOptions.Grammar` -> the ABI's `structured_grammar` (GBNF) for
+  LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
+  choice constraints are also exposed by the ABI.
+
+Model config example:
+
+```yaml
+name: qwen3-vllm
+backend: vllm-cpp
+context_size: 8192
+parameters:
+  model: Qwen3-4B   # model dir (safetensors) or .gguf file
+options:
+- max_num_seqs:16
+```
+
+Testing: `make test` runs the unit specs; export `VLLM_CPP_MODEL=<model>` (and
+optionally `VLLM_CPP_LIBRARY=<libvllm path>`) to enable the e2e specs.

backend/go/vllm-cpp/backend.go

@@ -0,0 +1,245 @@
+package main
+
+// LocalAI gRPC backend over the vllm.cpp C ABI.
+//
+// Predict maps to the blocking vllm_complete; PredictStream maps to
+// vllm_complete_stream, whose per-delta C callback bridges into the gRPC
+// stream channel. Concurrent calls are intentional: every completion entry
+// point submits into the engine's shared AsyncLLM scheduler, so parallel
+// LocalAI requests batch continuously inside the engine (the reason this
+// backend embeds base.Base and not base.SingleThread).
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+type VllmCpp struct {
+	base.Base
+
+	engine uintptr
+	opts   loadOptions
+}
+
+// Stream registry: the per-request bridge between the C token callback and
+// the gRPC stream channel, keyed by an integer handle round-tripped through
+// the C user_data pointer (never a Go pointer across the ABI). The host gRPC
+// server drains the channel even after a client disconnect, so sends here
+// cannot wedge the engine's delivery loop.
+var (
+	streamsMu   sync.Mutex
+	streams     = map[uintptr]chan string{}
+	streamNext  uintptr
+	tokenCbOnce sync.Once
+	tokenCbPtr  uintptr
+)
+
+// tokenCallback is the single C-shared callback for every stream; it
+// dispatches on the user_data handle. Returning 0 aborts the in-flight
+// request (vllm_token_callback contract).
+func tokenCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	streamsMu.Lock()
+	results := streams[userData]
+	streamsMu.Unlock()
+	if results == nil {
+		return 0 // unknown request: stop generation.
+	}
+	if text := goString(delta); text != "" {
+		results <- text
+	}
+	return 1
+}
+
+func registerStream(results chan string) uintptr {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	streamNext++
+	streams[streamNext] = results
+	return streamNext
+}
+
+func unregisterStream(h uintptr) {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	delete(streams, h)
+}
+
+// validModelPath enforces the greedy-probe rule: when a model config has no
+// explicit backend, the loader probes every backend with the model name, so
+// Load must refuse anything vllm.cpp cannot serve (a GGUF file, or a HF-style
+// directory with config.json + safetensors).
+func validModelPath(model string) error {
+	info, err := os.Stat(model)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: model path %q not found: %w", model, err)
+	}
+	if info.IsDir() {
+		if _, err := os.Stat(filepath.Join(model, "config.json")); err != nil {
+			return fmt.Errorf("vllm-cpp: model dir %q has no config.json", model)
+		}
+		return nil
+	}
+	if strings.EqualFold(filepath.Ext(model), ".gguf") {
+		return nil
+	}
+	return fmt.Errorf("vllm-cpp: model %q is neither a .gguf file nor a config.json model dir", model)
+}
+
+func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
+	model := opts.ModelFile
+	if model == "" {
+		model = opts.ModelPath
+	}
+	if !filepath.IsAbs(model) && opts.ModelPath != "" {
+		model = filepath.Join(opts.ModelPath, model)
+	}
+	if err := validModelPath(model); err != nil {
+		return err
+	}
+
+	v.opts = parseOptions(opts)
+
+	mp := defaultModelParams()
+	if v.opts.blockSize > 0 {
+		mp.BlockSize = v.opts.blockSize
+	}
+	if v.opts.numBlocks > 0 {
+		mp.NumBlocks = v.opts.numBlocks
+	}
+	if opts.ContextSize > 0 {
+		mp.MaxModelLen = opts.ContextSize
+	}
+	if v.opts.maxNumSeqs > 0 {
+		mp.MaxNumSeqs = v.opts.maxNumSeqs
+	}
+
+	modelC := cString(model)
+	mp.ModelPath = uintptr(unsafe.Pointer(&modelC[0])) // #nosec G103 -- borrowed by C for the load call only
+	var toolParserC, reasoningParserC []byte
+	if v.opts.toolParser != "" {
+		toolParserC = cString(v.opts.toolParser)
+		mp.ToolParser = uintptr(unsafe.Pointer(&toolParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+	if v.opts.reasoningParser != "" {
+		reasoningParserC = cString(v.opts.reasoningParser)
+		mp.ReasoningParser = uintptr(unsafe.Pointer(&reasoningParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+
+	xlog.Info("[vllm-cpp] Load", "model", model, "engine", vllmVersion(),
+		"blockSize", mp.BlockSize, "numBlocks", mp.NumBlocks,
+		"maxModelLen", mp.MaxModelLen, "maxNumSeqs", mp.MaxNumSeqs)
+
+	var engine uintptr
+	rc := vllmEngineLoad(unsafe.Pointer(&mp), unsafe.Pointer(&engine)) // #nosec G103 -- POD out-params
+	runtime.KeepAlive(modelC)
+	runtime.KeepAlive(toolParserC)
+	runtime.KeepAlive(reasoningParserC)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: engine load failed: %s", vllmLastError())
+	}
+	v.engine = engine
+	return nil
+}
+
+func (v *VllmCpp) Free() error {
+	if v.engine != 0 {
+		vllmEngineFree(v.engine)
+		v.engine = 0
+	}
+	return nil
+}
+
+// samplingFromPredict lowers PredictOptions into the C sampling POD plus the
+// backing buffers that must stay alive for the duration of the C call.
+func samplingFromPredict(opts *pb.PredictOptions) (sp cSamplingParams, keep []any) {
+	sp = defaultSamplingParams()
+	sp.Temperature = opts.Temperature
+	if opts.TopP > 0 {
+		sp.TopP = opts.TopP
+	}
+	if opts.TopK > 0 {
+		sp.TopK = opts.TopK
+	}
+	if opts.MinP > 0 {
+		sp.MinP = opts.MinP
+	}
+	if opts.Tokens > 0 {
+		sp.MaxTokens = opts.Tokens
+	} else {
+		sp.MaxTokens = 0 // unbounded; the engine caps at max_model_len.
+	}
+	if opts.Seed > 0 {
+		sp.Seed = uint64(opts.Seed)
+		sp.HasSeed = 1
+	}
+	sp.PresencePenalty = opts.PresencePenalty
+	sp.FrequencyPenalty = opts.FrequencyPenalty
+	if opts.Penalty > 0 {
+		sp.RepetitionPenalty = opts.Penalty
+	}
+	if opts.IgnoreEOS {
+		sp.IgnoreEOS = 1
+	}
+	if len(opts.StopPrompts) > 0 {
+		ptrs, backing := cStringArray(opts.StopPrompts)
+		sp.Stop = uintptr(unsafe.Pointer(&ptrs[0])) // #nosec G103 -- borrowed by C for the call only
+		sp.NStop = int32(len(ptrs))
+		keep = append(keep, ptrs, backing)
+	}
+	if opts.Grammar != "" {
+		g := cString(opts.Grammar)
+		sp.StructuredGrammar = uintptr(unsafe.Pointer(&g[0])) // #nosec G103 -- borrowed by C for the call only
+		keep = append(keep, g)
+	}
+	return sp, keep
+}
+
+func (v *VllmCpp) Predict(opts *pb.PredictOptions) (string, error) {
+	if v.engine == 0 {
+		return "", fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	sp, keep := samplingFromPredict(opts)
+	var out cCompletion
+	rc := vllmComplete(v.engine, opts.Prompt, unsafe.Pointer(&sp), unsafe.Pointer(&out)) // #nosec G103 -- POD in/out params
+	runtime.KeepAlive(keep)
+	if rc != vllmOK {
+		return "", fmt.Errorf("vllm-cpp: completion failed: %s", vllmLastError())
+	}
+	text := goString(out.Text)
+	vllmCompletionFree(unsafe.Pointer(&out)) // #nosec G103 -- frees out.Text
+	return text, nil
+}
+
+func (v *VllmCpp) PredictStream(opts *pb.PredictOptions, results chan string) error {
+	if v.engine == 0 {
+		close(results)
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	tokenCbOnce.Do(func() {
+		tokenCbPtr = purego.NewCallback(tokenCallback)
+	})
+
+	sp, keep := samplingFromPredict(opts)
+	handle := registerStream(results)
+
+	go func() {
+		defer close(results)
+		defer unregisterStream(handle)
+		rc := vllmCompleteStream(v.engine, opts.Prompt, unsafe.Pointer(&sp), tokenCbPtr, handle) // #nosec G103 -- POD in-params
+		runtime.KeepAlive(keep)
+		if rc != vllmOK {
+			xlog.Error("[vllm-cpp] stream failed", "error", vllmLastError())
+		}
+	}()
+	return nil
+}

backend/go/vllm-cpp/chat.go

@@ -0,0 +1,289 @@
+package main
+
+// The rich chat path (AIModelRich): rides the ENGINE's serving pipeline via
+// the ABI v3 chat entry points, exactly like the llama-cpp autoparser flow.
+// The engine applies the model's chat template, decides when a tool call
+// engages (tool_choice auto lowers to a LAZY structural-tag decode
+// constraint), parses tool calls with its streaming-stateful Hermes-style
+// parser, and hands back chat.completion.chunk JSON that this file maps 1:1
+// onto pb.Reply ChatDelta / ToolCallDelta.
+
+import (
+	"encoding/json"
+	"fmt"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// useChatPath reports whether the request should go through the engine-side
+// chat pipeline: the model config asked for backend-side templating and the
+// host handed us structured messages.
+func useChatPath(opts *pb.PredictOptions) bool {
+	return opts.UseTokenizerTemplate && len(opts.Messages) > 0
+}
+
+// chatRequestJSON lowers PredictOptions into one OpenAI chat-completions
+// request object for the ABI (the engine ignores `model`/`stream`).
+func chatRequestJSON(opts *pb.PredictOptions, stream bool) (string, error) {
+	messages := make([]map[string]any, 0, len(opts.Messages))
+	for _, m := range opts.Messages {
+		msg := map[string]any{"role": m.Role, "content": m.Content}
+		if m.ToolCalls != "" {
+			var toolCalls any
+			if err := json.Unmarshal([]byte(m.ToolCalls), &toolCalls); err == nil {
+				msg["tool_calls"] = toolCalls
+			}
+		}
+		// Multi-turn tool identity + prior reasoning: a role="tool" reply
+		// carries the id (and optionally the name) of the assistant call it
+		// answers, and assistant history may carry its reasoning span. The
+		// engine's template context needs all three or a second turn after
+		// tool execution is malformed.
+		if m.ToolCallId != "" {
+			msg["tool_call_id"] = m.ToolCallId
+		}
+		if m.Name != "" {
+			msg["name"] = m.Name
+		}
+		if m.ReasoningContent != "" {
+			msg["reasoning"] = m.ReasoningContent
+		}
+		messages = append(messages, msg)
+	}
+	req := map[string]any{"messages": messages}
+
+	if opts.Tools != "" {
+		var tools any
+		if err := json.Unmarshal([]byte(opts.Tools), &tools); err != nil {
+			return "", fmt.Errorf("vllm-cpp: tools is not valid JSON: %w", err)
+		}
+		req["tools"] = tools
+	}
+	if opts.ToolChoice != "" {
+		var choice any
+		// ToolChoice arrives either as a bare string ("auto"/"required"/"none")
+		// or as the OpenAI named-function JSON object.
+		if err := json.Unmarshal([]byte(opts.ToolChoice), &choice); err == nil {
+			req["tool_choice"] = choice
+		} else {
+			req["tool_choice"] = opts.ToolChoice
+		}
+	}
+
+	req["temperature"] = opts.Temperature
+	if opts.TopP > 0 {
+		req["top_p"] = opts.TopP
+	}
+	if opts.TopK > 0 {
+		req["top_k"] = opts.TopK
+	}
+	if opts.Tokens > 0 {
+		req["max_tokens"] = opts.Tokens
+	}
+	if opts.Seed > 0 {
+		req["seed"] = opts.Seed
+	}
+	if len(opts.StopPrompts) > 0 {
+		req["stop"] = opts.StopPrompts
+	}
+	if opts.PresencePenalty != 0 {
+		req["presence_penalty"] = opts.PresencePenalty
+	}
+	if opts.FrequencyPenalty != 0 {
+		req["frequency_penalty"] = opts.FrequencyPenalty
+	}
+	if stream {
+		// The engine's request parser validates stream_options against the
+		// stream flag at parse time (before the ABI entry point forces it),
+		// so state the intent explicitly.
+		req["stream"] = true
+		req["stream_options"] = map[string]any{"include_usage": true}
+	}
+
+	b, err := json.Marshal(req)
+	if err != nil {
+		return "", err
+	}
+	return string(b), nil
+}
+
+// chatChunk is the subset of an OpenAI chat.completion(.chunk) object the
+// backend consumes.
+type chatChunk struct {
+	Object  string `json:"object"`
+	Choices []struct {
+		Delta        *chatDelta `json:"delta"`   // streaming chunks
+		Message      *chatDelta `json:"message"` // non-stream response
+		FinishReason string     `json:"finish_reason"`
+	} `json:"choices"`
+	Usage *struct {
+		PromptTokens     int32 `json:"prompt_tokens"`
+		CompletionTokens int32 `json:"completion_tokens"`
+	} `json:"usage"`
+}
+
+type chatDelta struct {
+	Content          string `json:"content"`
+	ReasoningContent string `json:"reasoning"`
+	ToolCalls        []struct {
+		Index    int32  `json:"index"`
+		ID       string `json:"id"`
+		Function struct {
+			Name      string `json:"name"`
+			Arguments string `json:"arguments"`
+		} `json:"function"`
+	} `json:"tool_calls"`
+}
+
+// toReply maps one parsed chunk onto a pb.Reply carrying the content bytes
+// plus the structured ChatDelta (the host prefers ChatDeltas when present).
+func (c *chatChunk) toReply() *pb.Reply {
+	reply := &pb.Reply{}
+	if c.Usage != nil {
+		reply.PromptTokens = c.Usage.PromptTokens
+		reply.Tokens = c.Usage.CompletionTokens
+	}
+	if len(c.Choices) == 0 {
+		return reply
+	}
+	d := c.Choices[0].Delta
+	if d == nil {
+		d = c.Choices[0].Message
+	}
+	if d == nil {
+		return reply
+	}
+	delta := &pb.ChatDelta{
+		Content:          d.Content,
+		ReasoningContent: d.ReasoningContent,
+	}
+	for _, tc := range d.ToolCalls {
+		delta.ToolCalls = append(delta.ToolCalls, &pb.ToolCallDelta{
+			Index:     tc.Index,
+			Id:        tc.ID,
+			Name:      tc.Function.Name,
+			Arguments: tc.Function.Arguments,
+		})
+	}
+	reply.Message = []byte(d.Content)
+	if delta.Content != "" || delta.ReasoningContent != "" ||
+		len(delta.ToolCalls) > 0 {
+		reply.ChatDeltas = []*pb.ChatDelta{delta}
+	}
+	return reply
+}
+
+// Chat-stream registry: chunk JSON arrives on the engine's delivery thread
+// through one shared C callback; the integer handle in user_data selects the
+// destination channel (never a Go pointer across the ABI).
+var (
+	chatStreamsMu  sync.Mutex
+	chatStreams    = map[uintptr]chan<- *pb.Reply{}
+	chatStreamNext uintptr
+	chatCbOnce     sync.Once
+	chatCbPtr      uintptr
+)
+
+func chatCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	chatStreamsMu.Lock()
+	results := chatStreams[userData]
+	chatStreamsMu.Unlock()
+	if results == nil {
+		return 0
+	}
+	_ = finished // the terminal call carries an empty delta; nothing to emit.
+	payload := goString(delta)
+	if payload == "" {
+		return 1
+	}
+	var chunk chatChunk
+	if err := json.Unmarshal([]byte(payload), &chunk); err != nil {
+		xlog.Error("[vllm-cpp] unparseable chat chunk", "error", err)
+		return 1
+	}
+	results <- chunk.toReply()
+	return 1
+}
+
+func registerChatStream(results chan<- *pb.Reply) uintptr {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	chatStreamNext++
+	chatStreams[chatStreamNext] = results
+	return chatStreamNext
+}
+
+func unregisterChatStream(h uintptr) {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	delete(chatStreams, h)
+}
+
+// PredictRich implements the non-streaming rich path. Without structured
+// messages it falls back to the plain Predict flow (LocalAI-side templating,
+// optional grammar constraint).
+func (v *VllmCpp) PredictRich(opts *pb.PredictOptions) (*pb.Reply, error) {
+	if !useChatPath(opts) {
+		text, err := v.Predict(opts)
+		if err != nil {
+			return nil, err
+		}
+		return &pb.Reply{Message: []byte(text)}, nil
+	}
+	if v.engine == 0 {
+		return nil, fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, false)
+	if err != nil {
+		return nil, err
+	}
+	var out uintptr
+	rc := vllmChat(v.engine, request, unsafe.Pointer(&out)) // #nosec G103 -- char** out-param
+	if rc != vllmOK {
+		return nil, fmt.Errorf("vllm-cpp: chat failed: %s", vllmLastError())
+	}
+	payload := goString(out)
+	vllmStringFree(out)
+	var response chatChunk
+	if err := json.Unmarshal([]byte(payload), &response); err != nil {
+		return nil, fmt.Errorf("vllm-cpp: unparseable chat response: %w", err)
+	}
+	return response.toReply(), nil
+}
+
+// PredictStreamRich implements the streaming rich path. Contract: send into
+// the channel and return when finished; the host closes the channel.
+func (v *VllmCpp) PredictStreamRich(opts *pb.PredictOptions, results chan<- *pb.Reply) error {
+	if !useChatPath(opts) {
+		// Legacy bridge: run the plain stream and wrap deltas.
+		plain := make(chan string)
+		if err := v.PredictStream(opts, plain); err != nil {
+			return err
+		}
+		for delta := range plain {
+			results <- &pb.Reply{Message: []byte(delta)}
+		}
+		return nil
+	}
+	if v.engine == 0 {
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, true)
+	if err != nil {
+		return err
+	}
+	chatCbOnce.Do(func() {
+		chatCbPtr = purego.NewCallback(chatCallback)
+	})
+	handle := registerChatStream(results)
+	defer unregisterChatStream(handle)
+	rc := vllmChatStream(v.engine, request, chatCbPtr, handle)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: chat stream failed: %s", vllmLastError())
+	}
+	return nil
+}

backend/go/vllm-cpp/chat_test.go

@@ -0,0 +1,158 @@
+package main
+
+import (
+	"encoding/json"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("useChatPath", func() {
+	It("requires tokenizer templating AND structured messages", func() {
+		Expect(useChatPath(&pb.PredictOptions{})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{UseTokenizerTemplate: true})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages:             []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeTrue())
+	})
+})
+
+var _ = Describe("chatRequestJSON", func() {
+	It("lowers messages, tools, tool_choice and sampling onto one request", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "system", Content: "be brief"},
+				{Role: "user", Content: "weather in Rome?"},
+			},
+			Tools:       `[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object"}}}]`,
+			ToolChoice:  "required",
+			Temperature: 0.2,
+			TopP:        0.9,
+			Tokens:      64,
+			StopPrompts: []string{"<|im_end|>"},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req["messages"]).To(HaveLen(2))
+		Expect(req["tools"]).To(HaveLen(1))
+		Expect(req["tool_choice"]).To(Equal("required"))
+		Expect(req["max_tokens"]).To(BeNumerically("==", 64))
+		Expect(req["top_p"]).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(req["stop"]).To(ConsistOf("<|im_end|>"))
+		Expect(req).NotTo(HaveKey("stream_options"))
+	})
+
+	It("parses a named-function tool_choice object and asks for stream usage", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			Messages:   []*pb.Message{{Role: "user", Content: "hi"}},
+			ToolChoice: `{"type":"function","function":{"name":"get_weather"}}`,
+		}, true)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		choice, ok := req["tool_choice"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		Expect(choice["type"]).To(Equal("function"))
+		Expect(req["stream_options"]).To(HaveKeyWithValue("include_usage", true))
+	})
+
+	It("rejects malformed tools JSON", func() {
+		_, err := chatRequestJSON(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+			Tools:    "{not json",
+		}, false)
+		Expect(err).To(HaveOccurred())
+	})
+})
+
+var _ = Describe("chatChunk.toReply", func() {
+	It("maps a streaming tool-call delta onto ChatDelta/ToolCallDelta", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{
+			"tool_calls":[{"index":0,"id":"call_1","function":{"name":"get_weather","arguments":"{\"city\":"}}]
+		},"finish_reason":null}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].ToolCalls).To(HaveLen(1))
+		tc := reply.ChatDeltas[0].ToolCalls[0]
+		Expect(tc.Name).To(Equal("get_weather"))
+		Expect(tc.Id).To(Equal("call_1"))
+		Expect(tc.Arguments).To(Equal(`{"city":`))
+	})
+
+	It("maps a non-stream response message and usage", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion","choices":[{"message":{
+			"role":"assistant","content":"Sunny."},"finish_reason":"stop"}],
+			"usage":{"prompt_tokens":12,"completion_tokens":3}}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(string(reply.Message)).To(Equal("Sunny."))
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].Content).To(Equal("Sunny."))
+		Expect(reply.PromptTokens).To(BeNumerically("==", 12))
+		Expect(reply.Tokens).To(BeNumerically("==", 3))
+	})
+
+	It("emits no ChatDelta for an empty role-only chunk", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":""}}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+		Expect(chunk.toReply().ChatDeltas).To(BeEmpty())
+	})
+})
+
+var _ = Describe("chatRequestJSON multi-turn tool round trip", func() {
+	It("forwards tool_call_id, name, reasoning and assistant tool_calls", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "What is the weather in Rome?"},
+				{
+					Role:             "assistant",
+					ReasoningContent: "need the weather tool",
+					ToolCalls:        `[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Rome\"}"}}]`,
+				},
+				{Role: "tool", ToolCallId: "call_1", Name: "get_weather", Content: `{"temp": 21}`},
+			},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req struct {
+			Messages []map[string]any `json:"messages"`
+		}
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req.Messages).To(HaveLen(3))
+
+		assistant := req.Messages[1]
+		Expect(assistant["reasoning"]).To(Equal("need the weather tool"))
+		calls, ok := assistant["tool_calls"].([]any)
+		Expect(ok).To(BeTrue())
+		Expect(calls).To(HaveLen(1))
+		call := calls[0].(map[string]any)
+		Expect(call["id"]).To(Equal("call_1"))
+
+		tool := req.Messages[2]
+		Expect(tool["role"]).To(Equal("tool"))
+		Expect(tool["tool_call_id"]).To(Equal("call_1"))
+		Expect(tool["name"]).To(Equal("get_weather"))
+		Expect(tool["content"]).To(Equal(`{"temp": 21}`))
+
+		user := req.Messages[0]
+		Expect(user).NotTo(HaveKey("tool_call_id"))
+		Expect(user).NotTo(HaveKey("name"))
+		Expect(user).NotTo(HaveKey("reasoning"))
+	})
+})

backend/go/vllm-cpp/e2e_test.go

@@ -0,0 +1,281 @@
+package main
+
+// E2E over a real model + the built libvllm. Gated on VLLM_CPP_MODEL (a .gguf
+// file or a safetensors model dir): without it the suite skips, so CI runs
+// only the unit specs. test.sh auto-downloads a small GGUF when the gate is
+// unset and the download is allowed.
+
+import (
+	"encoding/json"
+	"os"
+	"runtime"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("vllm-cpp e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	It("refuses a foreign model artefact", func() {
+		other := &VllmCpp{}
+		Expect(other.Load(&pb.ModelOptions{ModelFile: "/nonexistent/foreign.bin"})).NotTo(Succeed())
+	})
+
+	It("completes a prompt (greedy)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "The capital of France is",
+			Tokens:      16,
+			Temperature: 0,
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(BeEmpty())
+	})
+
+	It("is deterministic under greedy decoding", func() {
+		opts := &pb.PredictOptions{Prompt: "1 2 3 4", Tokens: 8, Temperature: 0}
+		a, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		b, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(a).To(Equal(b))
+	})
+
+	It("streams deltas that concatenate to the blocking result", func() {
+		opts := &pb.PredictOptions{Prompt: "Count: one two", Tokens: 12, Temperature: 0}
+		blocking, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan string)
+		Expect(backend.PredictStream(opts, results)).To(Succeed())
+		var sb strings.Builder
+		for delta := range results {
+			sb.WriteString(delta)
+		}
+		Expect(sb.String()).To(Equal(blocking))
+	})
+
+	It("honors stop words", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "a b c d e f",
+			Tokens:      64,
+			Temperature: 0,
+			StopPrompts: []string{"g"},
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(ContainSubstring("g h"))
+	})
+
+	It("constrains generation with a GBNF grammar (tool-call path)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "Answer strictly yes or no: is water wet?",
+			Tokens:      4,
+			Temperature: 0,
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).To(Or(HavePrefix("yes"), HavePrefix("no")))
+	})
+
+	It("serves concurrent streams", func() {
+		const n = 4
+		type result struct {
+			text string
+			err  error
+		}
+		done := make(chan result, n)
+		for i := 0; i < n; i++ {
+			go func() {
+				results := make(chan string)
+				err := backend.PredictStream(&pb.PredictOptions{
+					Prompt: "Hello", Tokens: 8, Temperature: 0,
+				}, results)
+				var sb strings.Builder
+				for delta := range results {
+					sb.WriteString(delta)
+				}
+				done <- result{sb.String(), err}
+			}()
+		}
+		for i := 0; i < n; i++ {
+			r := <-done
+			Expect(r.err).NotTo(HaveOccurred())
+			Expect(r.text).NotTo(BeEmpty())
+		}
+	})
+})
+
+var _ = Describe("vllm-cpp chat e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	weatherTools := `[{"type":"function","function":{"name":"get_weather",` +
+		`"description":"Get the current weather for a city.",` +
+		`"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]`
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping chat e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	chatOpts := func() *pb.PredictOptions {
+		return &pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "Reply with one short sentence: what is the capital of France?"},
+			},
+			Tokens:      512,
+			Temperature: 0,
+		}
+	}
+
+	It("answers a plain chat turn through the engine-side template", func() {
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(BeEmpty())
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+	})
+
+	It("splits reasoning from content engine-side (auto-detected from the template)", func() {
+		// The Qwen3.5 chat template carries <think>, so the engine auto-selects
+		// the think_auto reasoning parser: a markerless answer stays pure
+		// content; if the model DOES think, the block arrives as
+		// ReasoningContent and never leaks into Message.
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(ContainSubstring("<think>"))
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+		for _, d := range reply.ChatDeltas {
+			Expect(d.ReasoningContent).NotTo(ContainSubstring("Paris"),
+				"the user-visible answer must not be swallowed into reasoning")
+		}
+	})
+
+	It("streams chat deltas that concatenate to the blocking answer", func() {
+		blocking, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan *pb.Reply, 64)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(chatOpts(), results)
+			close(results)
+		}()
+		var sb strings.Builder
+		for r := range results {
+			sb.WriteString(string(r.Message))
+		}
+		Expect(<-done).To(Succeed())
+		Expect(sb.String()).To(Equal(string(blocking.Message)))
+	})
+
+	It("emits a parsed tool call when tool_choice requires it", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "What is the weather in Rome right now?"},
+		}
+		opts.Tools = weatherTools
+		opts.ToolChoice = "required"
+		opts.Tokens = 256
+
+		reply, err := backend.PredictRich(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(reply.ChatDeltas).NotTo(BeEmpty())
+		var name, args string
+		for _, d := range reply.ChatDeltas {
+			for _, tc := range d.ToolCalls {
+				if tc.Name != "" {
+					name = tc.Name
+				}
+				args += tc.Arguments
+			}
+		}
+		Expect(name).To(Equal("get_weather"))
+		var parsed map[string]any
+		Expect(json.Unmarshal([]byte(args), &parsed)).To(Succeed(),
+			"tool arguments must be valid JSON: %q", args)
+		Expect(parsed).To(HaveKey("city"))
+	})
+
+	It("lets the engine decide on auto tool choice and streams tool deltas", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "Use the get_weather tool to check the weather in Rome."},
+		}
+		opts.Tools = weatherTools
+		opts.Tokens = 512
+
+		results := make(chan *pb.Reply, 128)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(opts, results)
+			close(results)
+		}()
+		sawToolDelta := false
+		for r := range results {
+			for _, d := range r.ChatDeltas {
+				if len(d.ToolCalls) > 0 {
+					sawToolDelta = true
+				}
+			}
+		}
+		Expect(<-done).To(Succeed())
+		// tool_choice auto is a LAZY constraint: the model may or may not call.
+		// With an explicit instruction the gate model reliably does; treat a
+		// no-call run as a soft signal rather than a hard failure only if the
+		// engine produced SOME output.
+		Expect(sawToolDelta).To(BeTrue(), "expected the engine to engage the tool")
+	})
+})

backend/go/vllm-cpp/govllmcpp.go

@@ -0,0 +1,182 @@
+package main
+
+// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v2).
+//
+// The structs below are hand-mirrored PODs of the C declarations, with
+// explicit padding so the Go layout matches the C layout on linux/darwin
+// amd64+arm64. Struct-by-value entry points (the *_default helpers) are NOT
+// bound - purego's struct-return support is platform-dependent - so the
+// defaults are replicated here and guarded by the vllm_abi_version check at
+// startup: a library whose ABI differs from what these mirrors were written
+// against is refused before any request runs.
+
+import (
+	"fmt"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+)
+
+// abiVersion is the VLLM_ABI_VERSION this file mirrors (vllm.h).
+const abiVersion = 5
+
+// vllm_status (vllm.h).
+const (
+	vllmOK = 0
+)
+
+// cModelParams mirrors vllm_model_params.
+type cModelParams struct {
+	ModelPath           uintptr // const char*
+	TokenizerConfigPath uintptr // const char*
+	BlockSize           int32
+	NumBlocks           int32
+	MaxModelLen         int32
+	MaxNumSeqs          int32
+	ToolParser          uintptr // const char*; NULL = auto-detect (ABI v4)
+	ReasoningParser     uintptr // const char*; NULL = auto-detect (ABI v5)
+}
+
+// cSamplingParams mirrors vllm_sampling_params (ABI v2, structured fields
+// included). Padding matches the C compiler's: the uint64 seed is 8-aligned,
+// and each pointer following an int32 is 8-aligned.
+type cSamplingParams struct {
+	Temperature          float32
+	TopP                 float32
+	TopK                 int32
+	MinP                 float32
+	MaxTokens            int32
+	_                    [4]byte
+	Seed                 uint64
+	HasSeed              int32
+	PresencePenalty      float32
+	FrequencyPenalty     float32
+	RepetitionPenalty    float32
+	MinTokens            int32
+	IgnoreEOS            int32
+	Stop                 uintptr // const char* const*
+	NStop                int32
+	_                    [4]byte
+	StructuredJSON       uintptr // const char*
+	StructuredRegex      uintptr // const char*
+	StructuredChoice     uintptr // const char* const*
+	NStructuredChoice    int32
+	_                    [4]byte
+	StructuredGrammar    uintptr // const char*
+	StructuredJSONObject int32
+	_                    [4]byte
+}
+
+// cCompletion mirrors vllm_completion.
+type cCompletion struct {
+	Text             uintptr // char*, caller-owned
+	FinishReason     uintptr // const char*, library-owned
+	PromptTokens     int32
+	CompletionTokens int32
+}
+
+// defaultSamplingParams mirrors vllm_sampling_params_default().
+func defaultSamplingParams() cSamplingParams {
+	return cSamplingParams{
+		Temperature:       1.0,
+		TopP:              1.0,
+		MaxTokens:         16,
+		RepetitionPenalty: 1.0,
+	}
+}
+
+// defaultModelParams mirrors vllm_model_params_default().
+func defaultModelParams() cModelParams {
+	return cModelParams{
+		BlockSize:  32,
+		NumBlocks:  256,
+		MaxNumSeqs: 8,
+	}
+}
+
+var (
+	vllmEngineLoad     func(params, out unsafe.Pointer) int32
+	vllmEngineFree     func(engine uintptr)
+	vllmComplete       func(engine uintptr, prompt string, params, out unsafe.Pointer) int32
+	vllmCompleteStream func(engine uintptr, prompt string, params unsafe.Pointer, cb uintptr, userData uintptr) int32
+	vllmChat           func(engine uintptr, requestJSON string, out unsafe.Pointer) int32
+	vllmChatStream     func(engine uintptr, requestJSON string, cb uintptr, userData uintptr) int32
+	vllmStringFree     func(s uintptr)
+	vllmCompletionFree func(out unsafe.Pointer)
+	vllmLastError      func() string
+	vllmVersion        func() string
+	vllmABIVersion     func() int32
+)
+
+type libFunc struct {
+	ptr  any
+	name string
+}
+
+// registerLib dlopens libvllm and binds the C ABI, refusing an ABI-version
+// mismatch (the struct mirrors above would be undefined behavior against a
+// different layout).
+func registerLib(libName string) error {
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: dlopen %s: %w", libName, err)
+	}
+	for _, lf := range []libFunc{
+		{&vllmEngineLoad, "vllm_engine_load"},
+		{&vllmEngineFree, "vllm_engine_free"},
+		{&vllmComplete, "vllm_complete"},
+		{&vllmCompleteStream, "vllm_complete_stream"},
+		{&vllmChat, "vllm_chat"},
+		{&vllmChatStream, "vllm_chat_stream"},
+		{&vllmStringFree, "vllm_string_free"},
+		{&vllmCompletionFree, "vllm_completion_free"},
+		{&vllmLastError, "vllm_last_error"},
+		{&vllmVersion, "vllm_version"},
+		{&vllmABIVersion, "vllm_abi_version"},
+	} {
+		purego.RegisterLibFunc(lf.ptr, lib, lf.name)
+	}
+	if v := vllmABIVersion(); v != abiVersion {
+		return fmt.Errorf("vllm-cpp: ABI mismatch: library reports v%d, backend built against v%d", v, abiVersion)
+	}
+	return nil
+}
+
+// cString returns a NUL-terminated byte slice for s. The backing array may be
+// passed to C for the duration of a call (the ABI borrows and copies); keep it
+// alive across the call with runtime.KeepAlive.
+func cString(s string) []byte {
+	b := make([]byte, len(s)+1)
+	copy(b, s)
+	return b
+}
+
+// cStringArray builds a NULL-free array of C-string pointers plus the backing
+// buffers that must stay alive for the duration of the C call.
+func cStringArray(ss []string) (ptrs []uintptr, backing [][]byte) {
+	backing = make([][]byte, 0, len(ss))
+	ptrs = make([]uintptr, 0, len(ss))
+	for _, s := range ss {
+		b := cString(s)
+		backing = append(backing, b)
+		ptrs = append(ptrs, uintptr(unsafe.Pointer(&b[0]))) // #nosec G103 -- borrowed by C for the call only
+	}
+	return ptrs, backing
+}
+
+// goString copies a NUL-terminated C string.
+func goString(p uintptr) string {
+	if p == 0 {
+		return ""
+	}
+	//nolint:govet // C-owned pointer handed over by purego, valid for this call
+	base := unsafe.Pointer(p) // #nosec G103 -- C-owned, copied out immediately
+	n := 0
+	for *(*byte)(unsafe.Add(base, n)) != 0 {
+		n++
+	}
+	if n == 0 {
+		return ""
+	}
+	return string(unsafe.Slice((*byte)(base), n))
+}

backend/go/vllm-cpp/main.go

@@ -0,0 +1,35 @@
+package main
+
+// Note: this is started internally by LocalAI and a server is allocated for each model
+import (
+	"flag"
+	"os"
+	"runtime"
+
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+func main() {
+	libName := os.Getenv("VLLM_CPP_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "./libvllm.dylib"
+		} else {
+			libName = "./libvllm.so"
+		}
+	}
+
+	if err := registerLib(libName); err != nil {
+		panic(err)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &VllmCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/vllm-cpp/options.go

@@ -0,0 +1,54 @@
+package main
+
+// Engine-sizing knobs carried through the model config's free-form
+// `options:` list ("key:value" entries), mirroring how the other in-house
+// backends pass engine-specific settings that have no proto field.
+
+import (
+	"strconv"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+type loadOptions struct {
+	blockSize  int32 // KV block size (tokens/block); engine default 32.
+	numBlocks  int32 // KV blocks to allocate; engine default 256.
+	maxNumSeqs int32 // max concurrent sequences; engine default 8.
+	// Engine-side parser selection (ABI v4/v5). Empty = the engine
+	// auto-detects from the chat template; "none" disables the reasoning
+	// split; unknown names fail the first chat call.
+	toolParser      string
+	reasoningParser string
+}
+
+func parseOptions(opts *pb.ModelOptions) loadOptions {
+	lo := loadOptions{}
+	for _, o := range opts.GetOptions() {
+		k, v, found := strings.Cut(o, ":")
+		if !found {
+			continue
+		}
+		switch strings.TrimSpace(k) {
+		case "block_size":
+			lo.blockSize = parseInt32(v, lo.blockSize)
+		case "num_blocks":
+			lo.numBlocks = parseInt32(v, lo.numBlocks)
+		case "max_num_seqs":
+			lo.maxNumSeqs = parseInt32(v, lo.maxNumSeqs)
+		case "tool_parser":
+			lo.toolParser = strings.TrimSpace(v)
+		case "reasoning_parser":
+			lo.reasoningParser = strings.TrimSpace(v)
+		}
+	}
+	return lo
+}
+
+func parseInt32(s string, fallback int32) int32 {
+	n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 32)
+	if err != nil || n <= 0 {
+		return fallback
+	}
+	return int32(n)
+}

backend/go/vllm-cpp/package.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/vllm-cpp $CURDIR/package/
+cp -fLv $CURDIR/libvllm.so $CURDIR/package/ 2>/dev/null || true
+cp -fLv $CURDIR/libvllm.dylib $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/vllm-cpp/run.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+# vllm.cpp ships ONE portable library per platform (SIMD tiers are per-file
+# with runtime dispatch), so there is no per-CPU variant probing here.
+if [ "$(uname)" = "Darwin" ]; then
+	LIBRARY="$CURDIR/libvllm.dylib"
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	LIBRARY="$CURDIR/libvllm.so"
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+fi
+
+export VLLM_CPP_LIBRARY=$LIBRARY
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LIBRARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/vllm-cpp "$@"
+fi
+
+echo "Using library: $LIBRARY"
+exec "$CURDIR"/vllm-cpp "$@"

backend/go/vllm-cpp/test.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+echo "Running vllm-cpp backend tests..."
+
+# Unit specs always run (struct-mirror layout, option/sampling mapping, load
+# validation). The e2e specs need a real model: set VLLM_CPP_MODEL to a .gguf
+# file or a safetensors model dir to enable them (see e2e_test.go).
+go test -v -timeout 1200s .
+
+echo "All vllm-cpp tests passed."

backend/go/vllm-cpp/vllmcpp_test.go

@@ -0,0 +1,162 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestVllmCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "vllm-cpp suite")
+}
+
+// The Go POD mirrors must match the C struct layout of vllm.h (ABI v2)
+// byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
+// amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
+var _ = Describe("C ABI struct mirrors", func() {
+	It("cModelParams matches vllm_model_params", func() {
+		var p cModelParams
+		Expect(unsafe.Offsetof(p.ModelPath)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TokenizerConfigPath)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.BlockSize)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.NumBlocks)).To(Equal(uintptr(20)))
+		Expect(unsafe.Offsetof(p.MaxModelLen)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.MaxNumSeqs)).To(Equal(uintptr(28)))
+		Expect(unsafe.Offsetof(p.ToolParser)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.ReasoningParser)).To(Equal(uintptr(40)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(48)))
+	})
+
+	It("cSamplingParams matches vllm_sampling_params (ABI v2)", func() {
+		var p cSamplingParams
+		Expect(unsafe.Offsetof(p.Temperature)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TopP)).To(Equal(uintptr(4)))
+		Expect(unsafe.Offsetof(p.TopK)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.MinP)).To(Equal(uintptr(12)))
+		Expect(unsafe.Offsetof(p.MaxTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.Seed)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.HasSeed)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.PresencePenalty)).To(Equal(uintptr(36)))
+		Expect(unsafe.Offsetof(p.FrequencyPenalty)).To(Equal(uintptr(40)))
+		Expect(unsafe.Offsetof(p.RepetitionPenalty)).To(Equal(uintptr(44)))
+		Expect(unsafe.Offsetof(p.MinTokens)).To(Equal(uintptr(48)))
+		Expect(unsafe.Offsetof(p.IgnoreEOS)).To(Equal(uintptr(52)))
+		Expect(unsafe.Offsetof(p.Stop)).To(Equal(uintptr(56)))
+		Expect(unsafe.Offsetof(p.NStop)).To(Equal(uintptr(64)))
+		Expect(unsafe.Offsetof(p.StructuredJSON)).To(Equal(uintptr(72)))
+		Expect(unsafe.Offsetof(p.StructuredRegex)).To(Equal(uintptr(80)))
+		Expect(unsafe.Offsetof(p.StructuredChoice)).To(Equal(uintptr(88)))
+		Expect(unsafe.Offsetof(p.NStructuredChoice)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.StructuredGrammar)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.StructuredJSONObject)).To(Equal(uintptr(112)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
+	})
+
+	It("cCompletion matches vllm_completion", func() {
+		var c cCompletion
+		Expect(unsafe.Offsetof(c.Text)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(c.FinishReason)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(c.PromptTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(c.CompletionTokens)).To(Equal(uintptr(20)))
+		Expect(unsafe.Sizeof(c)).To(Equal(uintptr(24)))
+	})
+})
+
+var _ = Describe("parseOptions", func() {
+	It("extracts the engine sizing knobs", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:64", "num_blocks:512", "max_num_seqs:32", "unknown:ignored",
+		}})
+		Expect(lo.blockSize).To(Equal(int32(64)))
+		Expect(lo.numBlocks).To(Equal(int32(512)))
+		Expect(lo.maxNumSeqs).To(Equal(int32(32)))
+	})
+	It("ignores malformed and non-positive values", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:abc", "num_blocks:-1", "max_num_seqs", "block_size:0",
+		}})
+		Expect(lo).To(Equal(loadOptions{}))
+	})
+})
+
+var _ = Describe("samplingFromPredict", func() {
+	It("maps the sampling fields onto the C POD", func() {
+		sp, _ := samplingFromPredict(&pb.PredictOptions{
+			Temperature:      0.7,
+			TopP:             0.9,
+			TopK:             40,
+			MinP:             0.05,
+			Tokens:           128,
+			Seed:             42,
+			Penalty:          1.1,
+			PresencePenalty:  0.5,
+			FrequencyPenalty: 0.25,
+			IgnoreEOS:        true,
+		})
+		Expect(sp.Temperature).To(BeNumerically("~", 0.7, 1e-6))
+		Expect(sp.TopP).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(sp.TopK).To(Equal(int32(40)))
+		Expect(sp.MinP).To(BeNumerically("~", 0.05, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(128)))
+		Expect(sp.HasSeed).To(Equal(int32(1)))
+		Expect(sp.Seed).To(Equal(uint64(42)))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.1, 1e-6))
+		Expect(sp.PresencePenalty).To(BeNumerically("~", 0.5, 1e-6))
+		Expect(sp.FrequencyPenalty).To(BeNumerically("~", 0.25, 1e-6))
+		Expect(sp.IgnoreEOS).To(Equal(int32(1)))
+	})
+
+	It("keeps the engine defaults for unset fields and stays unseeded", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{})
+		Expect(sp.TopP).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(0))) // unbounded, engine-capped.
+		Expect(sp.HasSeed).To(Equal(int32(0)))
+		Expect(sp.Stop).To(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).To(Equal(uintptr(0)))
+		Expect(keep).To(BeEmpty())
+	})
+
+	It("wires stop prompts and the grammar constraint", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{
+			StopPrompts: []string{"</s>", "\n\n"},
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(sp.NStop).To(Equal(int32(2)))
+		Expect(sp.Stop).NotTo(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).NotTo(Equal(uintptr(0)))
+		Expect(keep).NotTo(BeEmpty())
+	})
+})
+
+var _ = Describe("validModelPath", func() {
+	It("accepts a .gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "model.gguf")
+		Expect(os.WriteFile(p, []byte("GGUF"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).To(Succeed())
+	})
+	It("accepts a directory with config.json", func() {
+		dir := GinkgoT().TempDir()
+		Expect(os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o600)).To(Succeed())
+		Expect(validModelPath(dir)).To(Succeed())
+	})
+	It("refuses a directory without config.json (greedy-probe rule)", func() {
+		Expect(validModelPath(GinkgoT().TempDir())).NotTo(Succeed())
+	})
+	It("refuses a non-gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "weights.bin")
+		Expect(os.WriteFile(p, []byte("x"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).NotTo(Succeed())
+	})
+	It("refuses a missing path", func() {
+		Expect(validModelPath("/nonexistent/model.gguf")).NotTo(Succeed())
+	})
+})

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=080bbbe85230f624f0b52127f1ae1218247989f9
+WHISPER_CPP_VERSION?=97c56f1dc1d1100a9d859c865a20c82d22f823ed
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/index.yaml

@@ -156,6 +156,44 @@
     nvidia-cuda-12: "cuda12-whisper"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-whisper"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-whisper"
+- &vllm-cpp
+  name: "vllm-cpp"
+  alias: "vllm-cpp"
+  license: apache-2.0
+  description: |
+    vllm.cpp is a from-scratch C++20 port of vLLM created and maintained by the LocalAI team.
+    It mirrors vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching,
+    scheduler, sampler) on a portable tensor runtime with no Python, PyTorch or ggml at
+    inference time. It loads Hugging Face safetensors and GGUF checkpoints, supports
+    structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine,
+    and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan.
+  urls:
+    - https://github.com/mudler/vllm.cpp
+  tags:
+    - text-to-text
+    - LLM
+    - CPU
+    - GPU
+    - CUDA
+    - metal
+  capabilities:
+    default: "cpu-vllm-cpp"
+    nvidia: "cuda13-vllm-cpp"
+    metal: "metal-vllm-cpp"
+    vulkan: "vulkan-vllm-cpp"
+    nvidia-cuda-13: "cuda13-vllm-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp"
+- !!merge <<: *vllm-cpp
+  name: "vllm-cpp-development"
+  capabilities:
+    default: "cpu-vllm-cpp-development"
+    nvidia: "cuda13-vllm-cpp-development"
+    metal: "metal-vllm-cpp-development"
+    vulkan: "vulkan-vllm-cpp-development"
+    nvidia-cuda-13: "cuda13-vllm-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp-development"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp-development"
 - &crispasr
   name: "crispasr"
   alias: "crispasr"
@@ -1171,6 +1209,48 @@
     nvidia-l4t: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-moss-tts-cpp-development"
+- &magpiettscpp
+  name: "magpie-tts-cpp"
+  description: |
+    Magpie TTS C++ backend using GGML (magpie-tts.cpp). Native C++
+    text-to-speech for NVIDIA's Magpie TTS Multilingual 357M model (encoder +
+    autoregressive decoder over NanoCodec tokens), running from a single
+    self-contained GGUF (model, codec, tokenizer, G2P dictionaries) with no
+    Python at inference time. 22.05kHz mono output, 5 baked voices (Aria,
+    Jason, John, Leo, Sofia), 9+ languages.
+  urls:
+    - https://github.com/mudler/magpie-tts.cpp
+    - https://huggingface.co/mudler/magpie-tts.cpp-gguf
+  tags:
+    - text-to-speech
+    - tts
+  alias: "magpie-tts-cpp"
+  capabilities:
+    default: "cpu-magpie-tts-cpp"
+    nvidia: "cuda12-magpie-tts-cpp"
+    nvidia-cuda-13: "cuda13-magpie-tts-cpp"
+    nvidia-cuda-12: "cuda12-magpie-tts-cpp"
+    intel: "intel-sycl-f16-magpie-tts-cpp"
+    metal: "metal-magpie-tts-cpp"
+    amd: "rocm-magpie-tts-cpp"
+    vulkan: "vulkan-magpie-tts-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
+- !!merge <<: *magpiettscpp
+  name: "magpie-tts-cpp-development"
+  capabilities:
+    default: "cpu-magpie-tts-cpp-development"
+    nvidia: "cuda12-magpie-tts-cpp-development"
+    nvidia-cuda-13: "cuda13-magpie-tts-cpp-development"
+    nvidia-cuda-12: "cuda12-magpie-tts-cpp-development"
+    intel: "intel-sycl-f16-magpie-tts-cpp-development"
+    metal: "metal-magpie-tts-cpp-development"
+    amd: "rocm-magpie-tts-cpp-development"
+    vulkan: "vulkan-magpie-tts-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
 - &omnivoicecpp
   name: "omnivoice-cpp"
   description: |
@@ -1375,6 +1455,7 @@
   alias: "kokoro"
   name: "kokoro"
   capabilities:
+    default: "cpu-kokoro"
     nvidia: "cuda12-kokoro"
     intel: "intel-kokoro"
     amd: "rocm-kokoro"
@@ -4754,6 +4835,107 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp
+## magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "nvidia-l4t-arm64-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "nvidia-l4t-arm64-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cpu-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "metal-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "metal-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cpu-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda12-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "rocm-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f32-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f16-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "vulkan-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "vulkan-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda12-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "rocm-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f32-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "intel-sycl-f16-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-magpie-tts-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp
+- !!merge <<: *magpiettscpp
+  name: "cuda13-magpie-tts-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp
 ## omnivoice-cpp
 - !!merge <<: *omnivoicecpp
   name: "omnivoice-cpp-development"
@@ -5005,11 +5187,22 @@
 - !!merge <<: *kokoro
   name: "kokoro-development"
   capabilities:
+    default: "cpu-kokoro-development"
     nvidia: "cuda12-kokoro-development"
     intel: "intel-kokoro-development"
     amd: "rocm-kokoro-development"
     nvidia-l4t: "nvidia-l4t-kokoro-development"
     metal: "metal-kokoro-development"
+- !!merge <<: *kokoro
+  name: "cpu-kokoro"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-kokoro"
+  mirrors:
+    - localai/localai-backends:latest-cpu-kokoro
+- !!merge <<: *kokoro
+  name: "cpu-kokoro-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-kokoro"
+  mirrors:
+    - localai/localai-backends:master-cpu-kokoro
 - !!merge <<: *kokoro
   name: "cuda12-kokoro-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-kokoro"
@@ -6430,3 +6623,53 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-supertonic"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-supertonic
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-vllm-cpp

backend/python/coqui/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.82.1
+grpcio==1.83.0
 protobuf
 certifi
 packaging==26.2

backend/python/mlx-vlm/requirements-mps.txt

@@ -1 +1,3 @@
-git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
+torch
+torchvision

backend/python/nemo/install.sh

@@ -14,4 +14,28 @@ if [ "x${BUILD_PROFILE}" == "xintel" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
 fi
 
+# Darwin needs a newer interpreter than libbackend's 3.10 default. nemo_toolkit
+# pulls in text2num, a Rust extension built with maturin, and its macOS arm64
+# wheels start at cp311 (3.0.2 publishes cp311/cp312/cp313/cp314 and no cp310).
+# On 3.10 pip therefore falls back to the sdist and dies in the PEP 517 hook
+# with "No module named 'maturin'", since EXTRA_PIP_INSTALL_FLAGS carries
+# --no-build-isolation and nothing installs the build backend. Moving to 3.12
+# takes the prebuilt wheel and needs no Rust toolchain on the runner at all.
+#
+# Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel
+# for the same package and have no reason to move.
+if [ "x${BUILD_PROFILE}" == "xmps" ] || [ "x${BUILD_PROFILE}" == "xmetal" ]; then
+    PYTHON_VERSION="3.12"
+    # PYTHON_PATCH must move with it. libbackend builds the portable-Python URL
+    # as cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}-..., and
+    # the default patch is 18 for 3.10.18; leaving it alone asks for a 3.12.18
+    # that was never released and the download 404s.
+    #
+    # 11, not the 12 that sglang/install.sh uses for l4t13: at the 20250818 tag
+    # python-build-standalone published 3.12.12 for linux aarch64 but not for
+    # aarch64-apple-darwin, where 3.12.11 is the newest. Verified against the
+    # release assets rather than copied across.
+    PYTHON_PATCH="11"
+fi
+
 installRequirements

backend/python/sglang/requirements-cublas12-after.txt

@@ -2,3 +2,16 @@
 # (FunctionCallParser, ReasoningParser) move between releases.
 # 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
 sglang[all]>=0.5.11
+
+# Keep nvidia-modelopt on a stable release. sglang[all] pulls it in through its
+# `diffusion` extra with no version bound of its own, and install.sh passes a
+# GLOBAL --prerelease=allow (needed because flash-attn-4 only ships 4.0.0b*
+# wheels). Unbounded plus prereleases-allowed resolves to 0.46.0rc0, whose build
+# backend imports wheel_stub without declaring it as a build dependency; with
+# --no-build-isolation also in EXTRA_PIP_INSTALL_FLAGS nothing installs it, and
+# every cublas sglang image fails with "No module named 'wheel_stub'".
+#
+# Bounding this one package rather than dropping the global flag: the flag is
+# load-bearing for flash-attn-4, and this is the narrower change. Raise the
+# bound once 0.46.0 final ships.
+nvidia-modelopt<0.46

backend/python/sglang/requirements-cublas13-after.txt

@@ -2,3 +2,16 @@
 # (FunctionCallParser, ReasoningParser) move between releases.
 # 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
 sglang[all]>=0.5.11
+
+# Keep nvidia-modelopt on a stable release. sglang[all] pulls it in through its
+# `diffusion` extra with no version bound of its own, and install.sh passes a
+# GLOBAL --prerelease=allow (needed because flash-attn-4 only ships 4.0.0b*
+# wheels). Unbounded plus prereleases-allowed resolves to 0.46.0rc0, whose build
+# backend imports wheel_stub without declaring it as a build dependency; with
+# --no-build-isolation also in EXTRA_PIP_INSTALL_FLAGS nothing installs it, and
+# every cublas sglang image fails with "No module named 'wheel_stub'".
+#
+# Bounding this one package rather than dropping the global flag: the flag is
+# load-bearing for flash-attn-4, and this is the narrower change. Raise the
+# bound once 0.46.0 final ships.
+nvidia-modelopt<0.46

backend/python/transformers/requirements-cpu.txt

@@ -4,7 +4,7 @@ numba==0.60.0
 accelerate
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-cublas12.txt

@@ -4,7 +4,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-cublas13.txt

@@ -4,7 +4,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-hipblas.txt

@@ -5,7 +5,7 @@ transformers>=5.14.1
 llvmlite==0.43.0
 numba==0.60.0
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-intel.txt

@@ -5,7 +5,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-mps.txt

@@ -4,7 +4,7 @@ numba==0.60.0
 accelerate
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.0
+sentence-transformers==5.6.1
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.82.1
+grpcio==1.83.0
 protobuf==7.35.0
 certifi
 setuptools

backend/python/trl/reward_functions.py

@@ -5,12 +5,26 @@ All reward functions follow TRL's signature: (completions, **kwargs) -> list[flo
 """
 
 import json
+import os
 import re
 import math
 import string
 import functools
 
 
+# Opt-in flag for inline reward code. Compiling a caller-supplied Python body
+# is arbitrary code execution: the _SAFE_BUILTINS allowlist below is NOT a
+# security boundary — expressions like ().__class__.__bases__[0].__subclasses__()
+# escape it trivially to reach os.system. Since the fine-tuning endpoint is
+# unauthenticated by default, inline reward functions are disabled unless the
+# operator explicitly opts in by setting this env var to a truthy value.
+ALLOW_INLINE_ENV = "LOCALAI_TRL_ALLOW_INLINE_REWARD"
+
+
+def _inline_rewards_allowed():
+    return os.environ.get(ALLOW_INLINE_ENV, "").strip().lower() in ("1", "true", "yes", "on")
+
+
 # ---------------------------------------------------------------------------
 # Built-in reward functions
 # ---------------------------------------------------------------------------
@@ -224,6 +238,14 @@ def build_reward_functions(specs_json):
             reward_funcs.append(func)
 
         elif spec_type == "inline":
+            if not _inline_rewards_allowed():
+                raise ValueError(
+                    f"Inline reward function '{name}' rejected: inline reward code "
+                    f"executes arbitrary Python and is disabled by default. Set "
+                    f"{ALLOW_INLINE_ENV}=true on the backend to enable it (only on a "
+                    f"trusted, access-controlled instance), or use a builtin reward "
+                    f"function instead."
+                )
             code = spec.get("code", "")
             if not code.strip():
                 raise ValueError(f"Inline reward function '{name}' has no code")

backend/python/trl/test_reward_functions.py

@@ -0,0 +1,48 @@
+"""
+Tests for reward_functions.py — no gRPC / model deps, pure stdlib.
+
+Covers the inline-code opt-in gate (issue #11015): inline reward code is
+arbitrary code execution and must stay disabled unless the operator opts in.
+"""
+import os
+import unittest
+
+import reward_functions as rf
+
+INLINE_SPEC = [{"type": "inline", "name": "ok", "code": "return [1.0 for _ in completions]"}]
+
+
+class TestInlineRewardGate(unittest.TestCase):
+    def setUp(self):
+        self._saved = os.environ.pop(rf.ALLOW_INLINE_ENV, None)
+
+    def tearDown(self):
+        if self._saved is None:
+            os.environ.pop(rf.ALLOW_INLINE_ENV, None)
+        else:
+            os.environ[rf.ALLOW_INLINE_ENV] = self._saved
+
+    def test_inline_refused_by_default(self):
+        with self.assertRaises(ValueError) as cm:
+            rf.build_reward_functions(INLINE_SPEC)
+        self.assertIn("disabled by default", str(cm.exception))
+
+    def test_builtin_works_without_optin(self):
+        fns = rf.build_reward_functions([{"type": "builtin", "name": "format_reward"}])
+        self.assertEqual(len(fns), 1)
+        self.assertTrue(callable(fns[0]))
+
+    def test_inline_enabled_with_optin(self):
+        os.environ[rf.ALLOW_INLINE_ENV] = "true"
+        fns = rf.build_reward_functions(INLINE_SPEC)
+        self.assertEqual(fns[0](["x"]), [1.0])
+
+    def test_optin_falsey_values_stay_disabled(self):
+        for val in ("", "0", "false", "no", "off"):
+            os.environ[rf.ALLOW_INLINE_ENV] = val
+            with self.assertRaises(ValueError):
+                rf.build_reward_functions(INLINE_SPEC)
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/vllm/install.sh

@@ -119,7 +119,7 @@ if [ "$(uname -s)" = "Darwin" ]; then
     # can rewrite it. Darwin therefore follows vllm-metal and can lag the Linux
     # vllm pin (requirements-cublas13-after.txt, bumped independently against
     # vllm/vllm) until vllm-metal supports a newer vLLM.
-    VLLM_METAL_VERSION="v0.3.0.dev20260722081849"
+    VLLM_METAL_VERSION="v0.3.0.dev20260726174827"
 
     # The coupled vLLM source version is whatever this vllm-metal release builds
     # against -- it declares it in its own installer as `vllm_v=`. Derive it from

backend/python/vllm/requirements-cublas13-after.txt

@@ -3,8 +3,8 @@
 # on a cu130 host. Pull the cu130-flavoured wheel from vLLM's per-tag index
 # instead — the cublas13 case in install.sh adds --index-strategy=unsafe-best-match
 # so uv consults this index alongside PyPI.
---extra-index-url https://wheels.vllm.ai/0.25.1/cu130
+--extra-index-url https://wheels.vllm.ai/0.26.0/cu130
 # VERSION COUPLING: darwin/Apple-Silicon builds use vllm-metal (see install.sh),
 # which pins this exact vLLM version. Bumping vllm here means coordinating with a
 # vllm-metal release that supports the new version, or macOS/Metal builds break.
-vllm==0.25.1
+vllm==0.26.0

backend/python/vllm/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.82.1
+grpcio==1.83.0
 protobuf
 certifi
 setuptools

core/application/distributed.go

@@ -286,13 +286,14 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 		prefixProvider = prefixSync
 
 		// Invalidate the prefix-cache index whenever a replica row is removed.
-		// SetReplicaRemovedHook fires from the single chokepoint all removal paths
+		// AddReplicaRemovedHook fires from the single chokepoint all removal paths
 		// funnel through (RemoveNodeModel / RemoveAllNodeModelReplicas), so this
 		// one hook covers every path: reconciler scale-down, probe reaper,
 		// health-monitor reap, RemoteUnloaderAdapter, and the router. Registering
 		// it only inside this enabled block keeps the disabled path a true no-op
-		// (the registry stays hook-less).
-		registry.SetReplicaRemovedHook(func(model, node string, replica int) {
+		// for the prefix cache; other subsystems register their own hooks
+		// independently and are unaffected either way.
+		registry.AddReplicaRemovedHook(func(model, node string, replica int) {
 			if replica < 0 {
 				prefixSync.InvalidateNode(model, node)
 			} else {

core/application/router_factories.go

@@ -46,12 +46,12 @@ type lazyScorer struct {
 	modelName string
 }
 
-func (l *lazyScorer) Score(ctx context.Context, prompt string, candidates []string) ([]backend.CandidateScore, error) {
+func (l *lazyScorer) Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]backend.CandidateScore, error) {
 	cfg := l.app.adapterConfig(l.modelName)
 	if cfg == nil {
 		return nil, fmt.Errorf("scorer: model %q no longer available", l.modelName)
 	}
-	return backend.NewScorer(l.app.modelLoader, *cfg, l.app.applicationConfig).Score(ctx, prompt, candidates)
+	return backend.NewScorer(l.app.modelLoader, *cfg, l.app.applicationConfig).Score(ctx, prompt, stablePrefixLen, candidates)
 }
 
 // TokenCounter returns a func so the middleware's literal field type accepts

core/application/router_factories_test.go

@@ -109,7 +109,7 @@ var _ = Describe("router_factories lazy config resolution", func() {
 			Expect(lazy.modelName).To(Equal("score-test"))
 
 			removeCfg("score-test")
-			_, err := sc.Score(context.Background(), "prompt", []string{"a"})
+			_, err := sc.Score(context.Background(), "prompt", 0, []string{"a"})
 			Expect(err).To(HaveOccurred())
 			Expect(err.Error()).To(ContainSubstring("no longer available"))
 		})

core/application/startup.go

@@ -283,6 +283,14 @@ func New(opts ...config.AppOption) (*Application, error) {
 			distSvc.Registry,
 		)
 		application.modelLoader.SetModelStore(distStore)
+		// Drop the local stub when a model's last replica leaves the registry.
+		// The store reports local stubs UNION registry rows, and every removal
+		// path deletes the row only, so without this the frontend keeps
+		// reporting a model as loaded long after the replica is gone.
+		// Registered unconditionally: this is independent of the prefix cache.
+		distSvc.Registry.AddReplicaRemovedHook(
+			nodes.NewLocalStubInvalidator(distSvc.Registry, distStore),
+		)
 		// Start health monitor
 		distSvc.Health.Start(options.Context)
 		// Start replica reconciler for auto-scaling model replicas

core/backend/companion_options_test.go

@@ -113,34 +113,11 @@ var _ = Describe("companion artifact backend options", func() {
 		Expect(opts.Options).To(Equal([]string{"attention_backend:sdpa"}))
 	})
 
-	It("names the source repository when the companion is not resolved yet", func() {
-		// A companion that reaches load time WITHOUT a resolved snapshot must not
-		// vanish silently: emitting no option lets the backend fall back to its own
-		// hardcoded default, which is how a distributed longcat-video worker ended
-		// up trying to load the wrong base model and failing "base_model must point
-		// to a LongCat-Video checkpoint". Naming the DECLARED repository instead
-		// points the backend at the artifact the config actually asked for. The
-		// snapshot path (the staged, no-download fast path) is still preferred
-		// whenever the companion IS resolved.
+	It("skips a companion that has not been resolved yet", func() {
 		cfg := configWithCompanion()
 		cfg.Artifacts[1].Resolved = nil
 		opts := grpcModelOpts(cfg, "/models")
-
-		value, found := optionValue(opts.Options, "base_model")
-		Expect(found).To(BeTrue())
-		Expect(value).To(Equal("meituan-longcat/LongCat-Video"))
-		// The fallback is a repo reference, never a models-relative snapshot path.
-		Expect(value).ToNot(ContainSubstring(".artifacts"))
-	})
-
-	It("prefers the resolved snapshot path over the source repository", func() {
-		opts := grpcModelOpts(configWithCompanion(), "/models")
-		value, found := optionValue(opts.Options, "base_model")
-		Expect(found).To(BeTrue())
-		expected, err := modelartifacts.RelativeSnapshotPath(companionKey)
-		Expect(err).NotTo(HaveOccurred())
-		Expect(value).To(Equal(expected))
-		// The resolved fast path must never degrade to a bare repo id.
-		Expect(value).ToNot(Equal("meituan-longcat/LongCat-Video"))
+		_, found := optionValue(opts.Options, "base_model")
+		Expect(found).To(BeFalse())
 	})
 })

core/backend/options.go

@@ -166,6 +166,21 @@ func estimateModelSizeBytes(c config.ModelConfig, modelsPath string) int64 {
 	return int64(result.SizeBytes)
 }
 
+// effectiveThreads resolves the thread count a backend is asked to use.
+// Per-model threads wins: SetDefaults already fills an unset per-model value
+// from the app-level --threads, so overriding a set value with the app value
+// here would make the YAML `threads:` knob dead config (it did, for years —
+// e.g. a tiny VAD model could never opt down from the global pool size).
+func effectiveThreads(c config.ModelConfig, appThreads int) int {
+	if c.Threads != nil && *c.Threads > 0 {
+		return *c.Threads
+	}
+	if appThreads > 0 {
+		return appThreads
+	}
+	return 1
+}
+
 func ModelOptions(c config.ModelConfig, so *config.ApplicationConfig, opts ...model.Option) []model.Option {
 	defOpts := []model.Option{
 		model.WithBackendString(c.Backend),
@@ -178,16 +193,7 @@ func ModelOptions(c config.ModelConfig, so *config.ApplicationConfig, opts ...mo
 		defOpts = append(defOpts, model.WithModelFile(c.ModelFileName()))
 	}
 
-	threads := 1
-
-	if c.Threads != nil {
-		threads = *c.Threads
-	}
-
-	if so.Threads != 0 {
-		threads = so.Threads
-	}
-
+	threads := effectiveThreads(c, so.Threads)
 	c.Threads = &threads
 
 	grpcOpts := grpcModelOpts(c, so.SystemState.Model.ModelsPath)
@@ -294,13 +300,6 @@ func EffectiveBatchSize(c config.ModelConfig) int {
 //
 // An option the author set explicitly always wins: pinning a companion to a
 // local checkout has to beat the managed snapshot.
-//
-// A companion that is declared but NOT resolved falls back to its source
-// repository id rather than being dropped: a dropped companion is invisible to
-// the backend, which then loads its own hardcoded default and fails far away
-// from the cause. The repo-id fallback trades the staging fast path (the weights
-// are fetched on the worker) for correctness, and logs a warning so the missing
-// controller-side resolution is diagnosable.
 func withCompanionArtifactOptions(options []string, artifacts []modelartifacts.Spec) []string {
 	configured := make(map[string]struct{}, len(options))
 	for _, option := range options {
@@ -313,46 +312,19 @@ func withCompanionArtifactOptions(options []string, artifacts []modelartifacts.S
 	// reallocate away from) the config's own slice.
 	combined := slices.Clone(options)
 	for _, artifact := range artifacts {
-		if artifact.Target != modelartifacts.TargetCompanion {
+		if artifact.Target != modelartifacts.TargetCompanion || artifact.Resolved == nil {
 			continue
 		}
 		if _, exists := configured[artifact.Name]; exists {
 			xlog.Debug("keeping the configured companion option over the managed snapshot", "artifact", artifact.Name)
 			continue
 		}
-
-		// Preferred fast path: a resolved companion is surfaced as its staged,
-		// models-relative snapshot directory. Staging materializes exactly this
-		// path on a remote worker and the backend resolves it under its own
-		// ModelPath, so the weights are never fetched again at load time.
-		if artifact.Resolved != nil {
-			if snapshot, err := modelartifacts.RelativeSnapshotPath(artifact.Resolved.CacheKey); err == nil {
-				xlog.Debug("surfacing resolved companion snapshot to the backend", "artifact", artifact.Name, "path", snapshot)
-				combined = append(combined, artifact.Name+":"+snapshot)
-				continue
-			} else {
-				xlog.Warn("companion artifact has an unusable cache key; falling back to its source repository", "artifact", artifact.Name, "error", err)
-			}
-		}
-
-		// Fallback: the companion reached load time without a resolved snapshot
-		// (its resolved state never made it into the config the loader is serving
-		// from, e.g. after a controller restart or a peer-replica config reload).
-		// Emitting nothing here is what makes the failure so hard to see: the
-		// backend then falls back to its OWN hardcoded default companion, which on
-		// a distributed longcat-video worker meant fetching the wrong base model
-		// and failing "base_model must point to a LongCat-Video checkpoint". Name
-		// the DECLARED repository instead, so the backend at least fetches the
-		// artifact the config actually asked for. It is a warn because it means the
-		// no-download fast path was lost: the controller-side materialization or
-		// persistence for this companion needs investigating.
-		if repo := strings.TrimSpace(artifact.Source.Repo); repo != "" {
-			xlog.Warn("companion artifact is not resolved on the controller; the backend will fetch it by repository id (no staging fast path)",
-				"artifact", artifact.Name, "repo", repo)
-			combined = append(combined, artifact.Name+":"+repo)
+		snapshot, err := modelartifacts.RelativeSnapshotPath(artifact.Resolved.CacheKey)
+		if err != nil {
+			xlog.Warn("skipping companion artifact with an unusable cache key", "artifact", artifact.Name, "error", err)
 			continue
 		}
-		xlog.Warn("companion artifact is neither resolved nor has a source repository; the backend will get no option for it", "artifact", artifact.Name)
+		combined = append(combined, artifact.Name+":"+snapshot)
 	}
 	return combined
 }
@@ -450,6 +422,7 @@ func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
 		Options:              withCompanionArtifactOptions(c.Options, c.Artifacts),
 		Overrides:            c.Overrides,
 		EngineArgs:           engineArgsJSON,
+		EnableScore:          c.HasUsecases(config.FLAG_SCORE),
 		CLIPSkip:             int32(c.Diffusers.ClipSkip),
 		ControlNet:           c.Diffusers.ControlNet,
 		ContextSize:          int32(ctxSize),
@@ -509,6 +482,7 @@ func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
 			ApiKeyFile:            c.Proxy.APIKeyFile,
 			UpstreamModel:         c.Proxy.UpstreamModel,
 			RequestTimeoutSeconds: int32(c.Proxy.RequestTimeoutSeconds),
+			CachePrompt:           c.Proxy.CachePrompt,
 		}
 	}
 

core/backend/options_internal_test.go

@@ -120,6 +120,7 @@ var _ = Describe("grpcModelOpts NBatch", func() {
 		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}}
 		opts := grpcModelOpts(cfg, "/tmp/models")
 		Expect(opts.NBatch).To(BeEquivalentTo(512))
+		Expect(opts.EnableScore).To(BeFalse())
 	})
 
 	It("sizes the batch to the context window for score models", func() {
@@ -128,6 +129,14 @@ var _ = Describe("grpcModelOpts NBatch", func() {
 		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &scoreUsecase}
 		opts := grpcModelOpts(cfg, "/tmp/models")
 		Expect(opts.NBatch).To(BeEquivalentTo(4096))
+		Expect(opts.EnableScore).To(BeTrue())
+	})
+
+	It("enables score resources for a model with multiple usecases", func() {
+		usecases := config.FLAG_CHAT | config.FLAG_SCORE
+		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &usecases}
+		opts := grpcModelOpts(cfg, "/tmp/models")
+		Expect(opts.EnableScore).To(BeTrue())
 	})
 
 	It("keeps an explicit batch over the score default", func() {
@@ -355,3 +364,23 @@ var _ = Describe("gRPCPredictOpts model identity", func() {
 		Expect(opts.ModelIdentity).To(BeEmpty())
 	})
 })
+
+var _ = Describe("effectiveThreads", func() {
+	It("lets a per-model threads value override the app-level --threads", func() {
+		one := 1
+		cfg := config.ModelConfig{Threads: &one}
+		Expect(effectiveThreads(cfg, 10)).To(Equal(1),
+			"per-model threads is a real knob, not dead config under --threads")
+	})
+
+	It("falls back to the app-level threads when the model sets none", func() {
+		Expect(effectiveThreads(config.ModelConfig{}, 10)).To(Equal(10))
+		zero := 0
+		Expect(effectiveThreads(config.ModelConfig{Threads: &zero}, 10)).To(Equal(10),
+			"an explicit threads: 0 means unset, not zero threads")
+	})
+
+	It("never resolves to a non-positive thread count", func() {
+		Expect(effectiveThreads(config.ModelConfig{}, 0)).To(Equal(1))
+	})
+})

core/backend/preload.go

@@ -28,7 +28,7 @@ func PreloadModelByName(ctx context.Context, cl *config.ModelConfigLoader, ml *m
 		return nil, err
 	}
 
-	stages, err := pipelineStages(cl, &cfg.Pipeline, ml.ModelPath)
+	stages, err := pipelineStages(cl, &cfg.Pipeline, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 		return nil, err
 	}
@@ -59,7 +59,7 @@ var loadStage = PreloadModel
 // pipeline itself uses. A stage that fails to resolve is a misconfiguration,
 // so it fails fast rather than being deferred to load. A pipeline with no
 // stages set returns nil, which callers treat as "not a pipeline".
-func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath string) ([]PreloadStage, error) {
+func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath string, opts ...config.ConfigLoaderOption) ([]PreloadStage, error) {
 	voiceRec := ""
 	if p.VoiceRecognition != nil {
 		voiceRec = p.VoiceRecognition.Model
@@ -76,7 +76,7 @@ func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath
 		if s.name == "" {
 			continue
 		}
-		cfg, err := cl.LoadResolvedModelConfig(s.name, modelPath)
+		cfg, err := cl.LoadResolvedModelConfig(s.name, modelPath, opts...)
 		if err != nil {
 			return nil, fmt.Errorf("%s (%s): %w", s.role, s.name, err)
 		}
@@ -87,9 +87,11 @@ func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath
 
 // PreloadStages loads every present stage at once and waits for all of them, so
 // a pipeline warms in the time of its slowest stage rather than the sum. Absent
-// (nil-config) stages are skipped. A failed stage does not cancel the others —
-// they all run to completion so the joined error names every broken stage at
-// once, alongside the names that did load.
+// stages are skipped. Some callers represent an unset optional stage with a
+// nil config, while others materialize a default config with an empty name. A
+// failed stage does not cancel the others — they all run to completion so the
+// joined error names every broken stage at once, alongside the names that did
+// load.
 func PreloadStages(ctx context.Context, ml *model.ModelLoader, appConfig *config.ApplicationConfig, stages []PreloadStage) ([]string, error) {
 	var (
 		wg     sync.WaitGroup
@@ -98,7 +100,7 @@ func PreloadStages(ctx context.Context, ml *model.ModelLoader, appConfig *config
 		errs   []error
 	)
 	for _, s := range stages {
-		if s.Cfg == nil {
+		if s.Cfg == nil || s.Cfg.Name == "" {
 			continue
 		}
 		wg.Add(1)

core/backend/preload_internal_test.go

@@ -103,12 +103,13 @@ var _ = Describe("PreloadStages", func() {
 		return PreloadStage{Role: role, Cfg: &config.ModelConfig{Name: name}}
 	}
 
-	It("loads every present stage, skips absent (nil-config) ones, and returns the loaded names", func() {
+	It("loads every present stage, skips absent stages, and returns the loaded names", func() {
 		stubLoader(nil)
 
 		loaded, err := PreloadStages(context.Background(), nil, nil, []PreloadStage{
 			mkStage("vad", "vad-m"),
-			{Role: "transcription"}, // absent stage
+			{Role: "transcription"},
+			mkStage("tts", ""),
 			mkStage("llm", "llm-m"),
 		})
 

core/backend/score.go

@@ -23,6 +23,10 @@ type ScoreOptions struct {
 	// token count. Useful when comparing candidates of different
 	// lengths — without it, longer candidates score lower by default.
 	LengthNormalize bool
+	// StablePrefixLen is the byte length of the prompt prefix that stays
+	// identical across repeated scoring calls (0 = unknown); forwarded to
+	// the backend as a state-reuse boundary hint.
+	StablePrefixLen int
 }
 
 // CandidateScore is the per-candidate result. Mirrors pb.CandidateScore
@@ -42,9 +46,13 @@ type TokenLogProb struct {
 // Scorer evaluates a model's joint log-probability of each candidate
 // continuation given a shared prompt. Implemented by NewScorer over a
 // model-loaded backend; the router's score classifier consumes this
-// for multi-label policy selection.
+// for multi-label policy selection. stablePrefixLen is the byte length
+// of the prompt prefix that stays identical across calls (0 = unknown)
+// — backends use it to place a state-reuse point at the boundary, which
+// is what keeps repeat scoring fast on models that cannot rewind
+// (hybrid/recurrent architectures).
 type Scorer interface {
-	Score(ctx context.Context, prompt string, candidates []string) ([]CandidateScore, error)
+	Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]CandidateScore, error)
 }
 
 // NewScorer binds (loader, modelConfig, appConfig) into a Scorer. The
@@ -61,8 +69,8 @@ type modelScorer struct {
 	appConfig   *config.ApplicationConfig
 }
 
-func (m *modelScorer) Score(ctx context.Context, prompt string, candidates []string) ([]CandidateScore, error) {
-	fn, err := ModelScore(prompt, candidates, ScoreOptions{LengthNormalize: true}, m.loader, m.modelConfig, m.appConfig)
+func (m *modelScorer) Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]CandidateScore, error) {
+	fn, err := ModelScore(prompt, candidates, ScoreOptions{LengthNormalize: true, StablePrefixLen: stablePrefixLen}, m.loader, m.modelConfig, m.appConfig)
 	if err != nil {
 		return nil, err
 	}
@@ -103,6 +111,7 @@ func ModelScore(prompt string, candidates []string, opts ScoreOptions, loader *m
 			Candidates:           candidates,
 			IncludeTokenLogprobs: opts.IncludeTokenLogprobs,
 			LengthNormalize:      opts.LengthNormalize,
+			StablePrefixLen:      int32(opts.StablePrefixLen),
 		})
 		results := scoreResponseToCandidates(resp, opts.IncludeTokenLogprobs)
 		if appConfig.EnableTracing {

core/cli/run.go

@@ -243,6 +243,23 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		return nil
 	}
 
+	activatedListeners, err := systemdActivatedListeners()
+	if err != nil {
+		return fmt.Errorf("loading systemd socket activation listeners: %w", err)
+	}
+	activatedListener, err := selectSystemdListener(activatedListeners)
+	if err != nil {
+		for _, listener := range activatedListeners {
+			_ = listener.Close()
+		}
+		return err
+	}
+	if activatedListener != nil {
+		defer func() {
+			_ = activatedListener.Close()
+		}()
+	}
+
 	os.MkdirAll(r.BackendsPath, 0750)
 	os.MkdirAll(r.ModelsPath, 0750)
 
@@ -732,8 +749,13 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	// LAN, or VPN that's the historical "trusted network" deployment, but on
 	// a public IP it makes every model, gallery install, settings change, and
 	// admin endpoint reachable by anyone who can connect to the port.
+	listenAddress := r.Address
+	if activatedListener != nil {
+		listenAddress = activatedListener.Addr().String()
+	}
+
 	authConfigured := app.AuthDB() != nil || len(r.APIKeys) > 0
-	if err := requireAuthOrTrustedBind(r.Address, authConfigured, r.AllowInsecurePublicBind); err != nil {
+	if err := requireAuthOrTrustedBind(listenAddress, authConfigured, r.AllowInsecurePublicBind); err != nil {
 		return err
 	}
 
@@ -743,7 +765,11 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		return err
 	}
 
-	xlog.Info("LocalAI is started and running", "address", r.Address)
+	if activatedListener != nil {
+		appHTTP.Listener = activatedListener
+		xlog.Info("Using systemd socket activation listener", "address", listenAddress)
+	}
+	xlog.Info("LocalAI is started and running", "address", listenAddress)
 
 	// Start P2P if token was provided via CLI/env or loaded from runtime_settings.json
 	if token != "" || app.ApplicationConfig().P2PToken != "" {
@@ -762,11 +788,11 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	// backends like PostgreSQL need to call the embeddings API during
 	// collection initialization.
 	go func() {
-		waitForServerReady(r.Address, app.ApplicationConfig().Context)
+		waitForServerReady(listenAddress, app.ApplicationConfig().Context)
 		app.StartAgentPool()
 	}()
 
-	return appHTTP.Start(r.Address)
+	return appHTTP.Start(listenAddress)
 }
 
 // waitForServerReady polls the given address until the HTTP server is

core/cli/run_socket_activation.go

@@ -0,0 +1,17 @@
+package cli
+
+import (
+	"fmt"
+	"net"
+)
+
+func selectSystemdListener(listeners []net.Listener) (net.Listener, error) {
+	switch len(listeners) {
+	case 0:
+		return nil, nil
+	case 1:
+		return listeners[0], nil
+	default:
+		return nil, fmt.Errorf("systemd socket activation requires exactly one stream listener, got %d", len(listeners))
+	}
+}

core/cli/run_socket_activation_linux.go

@@ -0,0 +1,71 @@
+//go:build linux
+
+package cli
+
+import (
+	"fmt"
+	"net"
+	"os"
+	"strconv"
+)
+
+const systemdListenFDStart = 3
+
+func systemdActivatedListeners() ([]net.Listener, error) {
+	listenPID := os.Getenv("LISTEN_PID")
+	listenFDs := os.Getenv("LISTEN_FDS")
+	if listenPID == "" && listenFDs == "" {
+		return nil, nil
+	}
+
+	defer func() {
+		for _, key := range []string{"LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES"} {
+			_ = os.Unsetenv(key)
+		}
+	}()
+
+	pid, err := strconv.Atoi(listenPID)
+	if err != nil {
+		return nil, fmt.Errorf("invalid LISTEN_PID %q: %w", listenPID, err)
+	}
+	count, err := strconv.Atoi(listenFDs)
+	if err != nil || count < 0 {
+		return nil, fmt.Errorf("invalid LISTEN_FDS %q", listenFDs)
+	}
+	if pid != os.Getpid() || count == 0 {
+		return nil, nil
+	}
+
+	return listenersFromSystemdFDs(systemdListenFDStart, count)
+}
+
+func listenersFromSystemdFDs(start, count int) (_ []net.Listener, err error) {
+	listeners := make([]net.Listener, 0, count)
+	defer func() {
+		if err != nil {
+			for _, listener := range listeners {
+				_ = listener.Close()
+			}
+		}
+	}()
+
+	for offset := range count {
+		fd := uintptr(start + offset)
+		file := os.NewFile(fd, fmt.Sprintf("LISTEN_FD_%d", fd))
+		if file == nil {
+			return nil, fmt.Errorf("opening systemd listener file descriptor %d", fd)
+		}
+		listener, listenerErr := net.FileListener(file)
+		closeErr := file.Close()
+		if listenerErr != nil {
+			return nil, fmt.Errorf("using systemd file descriptor %d as a stream listener: %w", fd, listenerErr)
+		}
+		if closeErr != nil {
+			_ = listener.Close()
+			return nil, fmt.Errorf("closing inherited systemd file descriptor %d: %w", fd, closeErr)
+		}
+		listeners = append(listeners, listener)
+	}
+
+	return listeners, nil
+}

core/cli/run_socket_activation_other.go

@@ -0,0 +1,9 @@
+//go:build !linux
+
+package cli
+
+import "net"
+
+func systemdActivatedListeners() ([]net.Listener, error) {
+	return nil, nil
+}

core/cli/run_socket_activation_test.go

@@ -0,0 +1,101 @@
+//go:build linux
+
+package cli
+
+import (
+	"net"
+	"os"
+	"strconv"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("selectSystemdListener", func() {
+	It("keeps normal address binding when systemd passes no listener", func() {
+		listener, err := selectSystemdListener(nil)
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listener).To(BeNil())
+	})
+
+	It("uses the single stream listener passed by systemd", func() {
+		inherited, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(inherited.Close)
+
+		listener, err := selectSystemdListener([]net.Listener{inherited})
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listener).To(BeIdenticalTo(inherited))
+	})
+
+	It("rejects ambiguous activation with multiple stream listeners", func() {
+		first, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(first.Close)
+		second, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(second.Close)
+
+		listener, err := selectSystemdListener([]net.Listener{first, second})
+
+		Expect(err).To(MatchError(ContainSubstring("exactly one")))
+		Expect(listener).To(BeNil())
+	})
+})
+
+var _ = Describe("systemdActivatedListeners", func() {
+	It("turns an inherited TCP file descriptor into a working listener", func() {
+		original, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		file, err := original.(*net.TCPListener).File()
+		Expect(err).NotTo(HaveOccurred())
+		Expect(original.Close()).To(Succeed())
+
+		listeners, err := listenersFromSystemdFDs(int(file.Fd()), 1)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listeners).To(HaveLen(1))
+		DeferCleanup(listeners[0].Close)
+
+		client, err := net.Dial("tcp", listeners[0].Addr().String())
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(client.Close)
+		server, err := listeners[0].Accept()
+		Expect(err).NotTo(HaveOccurred())
+		Expect(server.Close()).To(Succeed())
+	})
+
+	It("ignores descriptors intended for another process and clears the activation environment", func() {
+		Expect(os.Setenv("LISTEN_PID", strconv.Itoa(os.Getpid()+1))).To(Succeed())
+		Expect(os.Setenv("LISTEN_FDS", "1")).To(Succeed())
+		Expect(os.Setenv("LISTEN_FDNAMES", "localai-http")).To(Succeed())
+		DeferCleanup(func() {
+			_ = os.Unsetenv("LISTEN_PID")
+			_ = os.Unsetenv("LISTEN_FDS")
+			_ = os.Unsetenv("LISTEN_FDNAMES")
+		})
+
+		listeners, err := systemdActivatedListeners()
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listeners).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_PID")).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_FDS")).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_FDNAMES")).To(BeEmpty())
+	})
+
+	It("reports malformed activation metadata instead of silently binding another socket", func() {
+		Expect(os.Setenv("LISTEN_PID", strconv.Itoa(os.Getpid()))).To(Succeed())
+		Expect(os.Setenv("LISTEN_FDS", "not-a-number")).To(Succeed())
+		DeferCleanup(func() {
+			_ = os.Unsetenv("LISTEN_PID")
+			_ = os.Unsetenv("LISTEN_FDS")
+		})
+
+		listeners, err := systemdActivatedListeners()
+
+		Expect(err).To(MatchError(ContainSubstring("LISTEN_FDS")))
+		Expect(listeners).To(BeNil())
+	})
+})

core/config/backend_capabilities.go

@@ -30,6 +30,7 @@ const (
 	UsecaseFaceRecognition     = "face_recognition"
 	UsecaseSpeakerRecognition  = "speaker_recognition"
 	UsecaseTokenClassify       = "token_classify"
+	UsecaseScore               = "score"
 )
 
 // GRPCMethod identifies a Backend service RPC from backend.proto.
@@ -60,6 +61,7 @@ const (
 	MethodVoiceEmbed         GRPCMethod = "VoiceEmbed"
 	MethodVoiceAnalyze       GRPCMethod = "VoiceAnalyze"
 	MethodTokenClassify      GRPCMethod = "TokenClassify"
+	MethodScore              GRPCMethod = "Score"
 )
 
 // UsecaseInfo describes a single known_usecase value and how it maps
@@ -192,6 +194,11 @@ var UsecaseInfoMap = map[string]UsecaseInfo{
 		GRPCMethod:  MethodTokenClassify,
 		Description: "Per-token classification (NER) via the TokenClassify RPC — the PII detector tier. Declared explicitly via known_usecases; never auto-guessed, since the token-classification head is not useful as general generation or embeddings.",
 	},
+	UsecaseScore: {
+		Flag:        FLAG_SCORE,
+		GRPCMethod:  MethodScore,
+		Description: "Joint log-probability scoring of candidate continuations via the Score RPC. Declared explicitly via known_usecases and usable alongside generation usecases.",
+	},
 }
 
 // BackendCapability describes which gRPC methods and usecases a backend supports.
@@ -241,8 +248,8 @@ func referenceVoiceCloning() *VoiceCloningCapability {
 var BackendCapabilities = map[string]BackendCapability{
 	// --- LLM / text generation backends ---
 	"llama-cpp": {
-		GRPCMethods:      []GRPCMethod{MethodPredict, MethodPredictStream, MethodEmbedding, MethodTokenizeString},
-		PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseEdit, UsecaseEmbeddings, UsecaseTokenize, UsecaseVision},
+		GRPCMethods:      []GRPCMethod{MethodPredict, MethodPredictStream, MethodEmbedding, MethodTokenizeString, MethodScore},
+		PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseEdit, UsecaseEmbeddings, UsecaseTokenize, UsecaseVision, UsecaseScore},
 		DefaultUsecases:  []string{UsecaseChat},
 		AcceptsImages:    true, // requires mmproj
 		Description:      "llama.cpp GGUF models — LLM inference with optional vision via mmproj",
@@ -466,6 +473,12 @@ var BackendCapabilities = map[string]BackendCapability{
 		VoiceCloning:     referenceVoiceCloning(),
 		Description:      "Qwen3 TTS C++ - text-to-speech with streaming, named speakers, voice design and cloning (qwentts.cpp / GGML)",
 	},
+	"magpie-tts-cpp": {
+		GRPCMethods:      []GRPCMethod{MethodTTS, MethodTTSStream},
+		PossibleUsecases: []string{UsecaseTTS},
+		DefaultUsecases:  []string{UsecaseTTS},
+		Description:      "Magpie TTS C++ - NVIDIA Magpie TTS Multilingual 357M with 5 baked voices and 9+ languages (magpie-tts.cpp / GGML)",
+	},
 	"faster-qwen3-tts": {
 		GRPCMethods:      []GRPCMethod{MethodTTS},
 		PossibleUsecases: []string{UsecaseTTS},

core/config/meta/registry.go

@@ -623,6 +623,13 @@ func DefaultRegistry() map[string]FieldMetaOverride {
 			Component:   "toggle",
 			Order:       89,
 		},
+		"pipeline.turn_detection.vad_window_sec": {
+			Section:     "pipeline",
+			Label:       "VAD Window (s)",
+			Description: "Widen the slice of recent audio the VAD rescans each turn-detection tick. Sized automatically from the commit silence threshold (server_vad silence window, or the semantic eagerness fallback) plus a warm-up margin — set only to widen it; values below the automatic floor are ignored.",
+			Component:   "number",
+			Order:       90,
+		},
 		"pipeline.disable_warmup": {
 			Section:     "pipeline",
 			Label:       "Disable Warmup",
@@ -630,6 +637,99 @@ func DefaultRegistry() map[string]FieldMetaOverride {
 			Component:   "toggle",
 			Order:       90,
 		},
+		"pipeline.classifier.enabled": {
+			Section:     "pipeline",
+			Label:       "Classifier Mode",
+			Description: "Replace autoregressive generation with prefill-only option selection: each user turn is scored against the option list via the Score primitive and the winning option's canned reply / tool call is emitted. Built for hardware that can afford prompt processing but not decode (e.g. a Raspberry Pi).",
+			Component:   "toggle",
+			Order:       91,
+		},
+		"pipeline.classifier.options": {
+			Section:     "pipeline",
+			Label:       "Classifier Options",
+			Description: "The intents the classifier scores each turn against. Each option has an id (also the scored route label — keep it short), a description of when it applies, an optional canned spoken reply, and an optional canned tool call {name, arguments}. A tool may also declare slots ([{name, type: number|enum|string, values, default, hint}]) whose \"{{name}}\" placeholders in arguments (and, optionally, the reply) are filled by a short grammar-constrained completion when the option wins — the hybrid between prefill-only classification and full generation (requires completion in the scoring model's known_usecases). Clients can replace the list per session via session.update localai_classifier.",
+			Component:   "json-editor",
+			Order:       92,
+		},
+		"pipeline.classifier.threshold": {
+			Section:     "pipeline",
+			Label:       "Classifier Threshold",
+			Description: "Softmax-probability floor the best option must clear; below it the fallback applies. 0 always picks the argmax.",
+			Component:   "slider",
+			Min:         f64(0),
+			Max:         f64(0.99),
+			Step:        f64(0.01),
+			Order:       93,
+		},
+		"pipeline.classifier.fallback.mode": {
+			Section:     "pipeline",
+			Label:       "Classifier Fallback",
+			Description: "What happens when no option clears the threshold: complete with no output, speak the canned fallback reply, or fall through to normal (slow) generation.",
+			Component:   "select",
+			Options: []FieldOption{
+				{Value: "none", Label: "none (empty response)"},
+				{Value: "reply", Label: "canned reply"},
+				{Value: "generate", Label: "generate"},
+			},
+			Order: 94,
+		},
+		"pipeline.classifier.fallback.reply": {
+			Section:     "pipeline",
+			Label:       "Classifier Fallback Reply",
+			Description: "The canned reply spoken when the fallback mode is 'reply' and no option clears the threshold.",
+			Component:   "text",
+			Order:       95,
+		},
+		"pipeline.classifier.normalization": {
+			Section:     "pipeline",
+			Label:       "Classifier Normalization",
+			Description: "How option scores feed the softmax: 'raw' compares joint log-probs (default); 'mean' divides by token count, which is fairer when option ids have very different lengths.",
+			Component:   "select",
+			Options: []FieldOption{
+				{Value: "raw", Label: "raw (joint log-prob)"},
+				{Value: "mean", Label: "mean (per-token)"},
+			},
+			Order: 96,
+		},
+		"pipeline.classifier.history_items": {
+			Section:     "pipeline",
+			Label:       "Classifier History Items",
+			Description: "What gets scored: 0 or -1 (default) score only the latest user message; a positive N includes the trailing N conversation messages, role-labeled. Prior turns echo option names and can dominate small scoring models — only opt in with a larger scorer.",
+			Component:   "number",
+			Order:       97,
+		},
+		"pipeline.classifier.model": {
+			Section:              "pipeline",
+			Label:                "Classifier Scoring Model",
+			Description:          "Optionally score on a different model config. Empty uses the pipeline LLM — scoring runs through the same llama.cpp slot as generation and shares its prompt cache, so a separate model is rarely needed.",
+			Component:            "model-select",
+			AutocompleteProvider: ProviderModels,
+			Order:                98,
+		},
+		"pipeline.classifier.address.names": {
+			Section:     "pipeline",
+			Label:       "Classifier Address Names",
+			Description: "Wake-word gate: only act on turns that mention one of these names as a whole word ('Drone go up', not just 'go up'). Matching is deterministic on the transcript; unaddressed turns skip scoring entirely.",
+			Component:   "string-list",
+			Order:       99,
+		},
+		"pipeline.classifier.address.mode": {
+			Section:     "pipeline",
+			Label:       "Classifier Address Mode",
+			Description: "What to do with unaddressed turns: 'ignore' completes silently (right for ambient conversation), 'reply' speaks the address reply.",
+			Component:   "select",
+			Options: []FieldOption{
+				{Value: "ignore", Label: "ignore (stay silent)"},
+				{Value: "reply", Label: "reply (speak the address reply)"},
+			},
+			Order: 100,
+		},
+		"pipeline.classifier.address.reply": {
+			Section:     "pipeline",
+			Label:       "Classifier Address Reply",
+			Description: "Spoken when an unaddressed turn arrives in 'reply' mode.",
+			Order:       101,
+		},
 
 		// --- Functions ---
 		"function.grammar.parallel_calls": {
@@ -822,6 +922,13 @@ func DefaultRegistry() map[string]FieldMetaOverride {
 			Min:         f64(0),
 			Order:       213,
 		},
+		"proxy.cache_prompt": {
+			Section:     "proxy",
+			Label:       "Proxy Anthropic Prompt Cache",
+			Description: "Inject Anthropic prompt-cache breakpoints (cache_control: ephemeral) on the stable prefix (system, tools, last message) when mode is translate and provider is anthropic. Serves the repeated prefix at the cache-read rate on multi-turn/agentic calls. No effect otherwise.",
+			Component:   "checkbox",
+			Order:       214,
+		},
 
 		// --- MITM intercept hosts ---
 		// Each host listed here is claimed by this model config; the

core/config/model_config.go

@@ -232,6 +232,15 @@ type ProxyConfig struct {
 	// means no per-request timeout (only the request context, which
 	// is bound to the client connection, applies).
 	RequestTimeoutSeconds int `yaml:"request_timeout_seconds,omitempty" json:"request_timeout_seconds,omitempty"`
+
+	// CachePrompt enables automatic Anthropic prompt-cache breakpoints
+	// (cache_control: ephemeral) on the stable prefix — system prompt,
+	// tools, and the last message block — when mode=translate and
+	// provider=anthropic. Anthropic then serves the repeated prefix at
+	// the cache-read rate (0.1x input), which sharply cuts cost on
+	// agentic/multi-turn workloads that re-send a large stable prefix.
+	// No effect for passthrough mode or non-Anthropic providers.
+	CachePrompt bool `yaml:"cache_prompt,omitempty" json:"cache_prompt,omitempty"`
 }
 
 // Proxy mode names. Validate() normalises an empty Mode to
@@ -669,6 +678,16 @@ type Pipeline struct {
 	// per session; retranscribe is server-side only. Unset keeps server_vad.
 	TurnDetection PipelineTurnDetection `yaml:"turn_detection,omitempty" json:"turn_detection,omitempty"`
 
+	// Classifier switches realtime responses to prefill-only option
+	// selection (LocalAI classifier mode): each user turn is scored
+	// against a fixed option list via the Score primitive and the winning
+	// option's canned reply / tool call is emitted, so weak hardware
+	// never pays for autoregressive decode. Nil means disabled; clients
+	// can still enable per session via session.update localai_classifier.
+	// Validated (and rejected loudly) at realtime session setup, like the
+	// pipeline model slots.
+	Classifier *PipelineClassifier `yaml:"classifier,omitempty" json:"classifier,omitempty"`
+
 	// DisableWarmup turns off eager pre-loading of the pipeline's sub-models at
 	// realtime session start. By default (false) LocalAI loads every configured
 	// sub-model backend (VAD, transcription, LLM, TTS, sound detection, voice
@@ -682,6 +701,65 @@ type Pipeline struct {
 	DisableWarmup bool `yaml:"disable_warmup,omitempty" json:"disable_warmup,omitempty"`
 }
 
+// PipelineClassifier is the YAML mirror of the realtime API's
+// localai_classifier extension (see
+// core/http/endpoints/openai/types/classifier.go, which documents the
+// field semantics and owns validation — the realtime session converts and
+// validates this block at setup).
+type PipelineClassifier struct {
+	Enabled bool `yaml:"enabled,omitempty" json:"enabled,omitempty"`
+	// Model optionally names a different config to score on. Empty uses
+	// the pipeline's llm — with slot-based Score the same process serves
+	// both scoring and generation and shares its prompt cache.
+	Model         string                      `yaml:"model,omitempty" json:"model,omitempty"`
+	Threshold     float64                     `yaml:"threshold,omitempty" json:"threshold,omitempty"`
+	Normalization string                      `yaml:"normalization,omitempty" json:"normalization,omitempty"`
+	HistoryItems  int                         `yaml:"history_items,omitempty" json:"history_items,omitempty"`
+	Fallback      *PipelineClassifierFallback `yaml:"fallback,omitempty" json:"fallback,omitempty"`
+	Options       []PipelineClassifierOption  `yaml:"options,omitempty" json:"options,omitempty"`
+	// Address gates every turn on the assistant being addressed by one of
+	// these names (wake-word behavior); see types.ClassifierAddress.
+	Address *PipelineClassifierAddress `yaml:"address,omitempty" json:"address,omitempty"`
+}
+
+// PipelineClassifierAddress mirrors types.ClassifierAddress for YAML.
+type PipelineClassifierAddress struct {
+	Names []string `yaml:"names,omitempty" json:"names,omitempty"`
+	Mode  string   `yaml:"mode,omitempty" json:"mode,omitempty"`
+	Reply string   `yaml:"reply,omitempty" json:"reply,omitempty"`
+}
+
+type PipelineClassifierOption struct {
+	ID          string                  `yaml:"id" json:"id"`
+	Description string                  `yaml:"description" json:"description"`
+	Reply       string                  `yaml:"reply,omitempty" json:"reply,omitempty"`
+	Tool        *PipelineClassifierTool `yaml:"tool,omitempty" json:"tool,omitempty"`
+}
+
+type PipelineClassifierTool struct {
+	Name string `yaml:"name" json:"name"`
+	// Arguments is a plain YAML map; the realtime session marshals it to
+	// the JSON arguments string of the emitted function call. With Slots
+	// it is a template: "{{name}}" values are filled by a constrained
+	// completion when the option wins.
+	Arguments map[string]any `yaml:"arguments,omitempty" json:"arguments,omitempty"`
+	// Slots declares the inferred arguments; see types.ClassifierSlot.
+	Slots []PipelineClassifierSlot `yaml:"slots,omitempty" json:"slots,omitempty"`
+}
+
+type PipelineClassifierSlot struct {
+	Name    string   `yaml:"name" json:"name"`
+	Type    string   `yaml:"type" json:"type"` // number | enum | string
+	Values  []string `yaml:"values,omitempty" json:"values,omitempty"`
+	Default string   `yaml:"default,omitempty" json:"default,omitempty"`
+	Hint    string   `yaml:"hint,omitempty" json:"hint,omitempty"`
+}
+
+type PipelineClassifierFallback struct {
+	Mode  string `yaml:"mode,omitempty" json:"mode,omitempty"`
+	Reply string `yaml:"reply,omitempty" json:"reply,omitempty"`
+}
+
 // PipelineCompaction configures summarize-then-drop for a realtime pipeline.
 type PipelineCompaction struct {
 	// Enabled turns summarize-then-drop on. Default false.
@@ -982,6 +1060,12 @@ type PipelineTurnDetection struct {
 	// are compared in the logs — a diagnostic for streaming/batch alignment
 	// at the cost of one extra decode per turn.
 	Retranscribe *bool `yaml:"retranscribe,omitempty" json:"retranscribe,omitempty"`
+	// VadWindowSec widens the slice of recent audio the VAD rescans each
+	// tick. The pipeline sizes it automatically from the commit silence
+	// threshold (server_vad silence window, or the semantic eagerness
+	// fallback) plus a warm-up margin; set this only to widen it further —
+	// values below the automatic floor are ignored.
+	VadWindowSec float64 `yaml:"vad_window_sec,omitempty" json:"vad_window_sec,omitempty"`
 }
 
 // TurnDetectionSemantic reports whether this pipeline defaults sessions to
@@ -1435,20 +1519,9 @@ func (c *ModelConfig) Validate() (bool, error) {
 			ProxyProviderOpenAI, ProxyProviderAnthropic)
 	}
 
-	// Score on llama-cpp bypasses the slot loop and races the
-	// llama_context against concurrent generation/embedding traffic
-	// (see backend/cpp/llama-cpp/grpc-server.cpp on Score). Reject the
-	// combination here so operators are forced to split the model.
-	// (token_classify is unaffected — it runs on the standalone
-	// privacy-filter backend, not llama-cpp.)
-	const scoreConflicts = FLAG_CHAT | FLAG_COMPLETION | FLAG_EMBEDDINGS
-	if (c.Backend == "llama-cpp" || c.Backend == "llama") &&
-		c.HasUsecases(FLAG_SCORE) && c.KnownUsecases != nil &&
-		*c.KnownUsecases&scoreConflicts != 0 {
-		return false, fmt.Errorf(
-			"known_usecases conflict on llama-cpp: score is incompatible " +
-				"with chat/completion/embeddings — split into separate model configs")
-	}
+	// Score on llama-cpp runs through the slot loop (SERVER_TASK_TYPE_SCORE,
+	// see backend/cpp/llama-cpp/patches/), so it is safe to combine with
+	// chat/completion/embeddings on one config — no conflict check needed.
 
 	// Pattern detector: validate built-in names and that each operator-defined
 	// pattern is a well-formed, anchored, bounded restricted-regex. Reject at
@@ -1576,9 +1649,10 @@ const (
 	// Marks a model as wired for the Score gRPC primitive (joint
 	// log-prob of candidate continuations under a shared prompt). Must
 	// be declared explicitly via `known_usecases: [score]` — there's
-	// no heuristic for it. On llama-cpp, Score bypasses the slot loop
-	// (direct llama_decode), so combining score with
-	// chat/completion/embeddings in one config is rejected at validation.
+	// no heuristic for it. On llama-cpp, Score runs through the slot
+	// loop (SERVER_TASK_TYPE_SCORE), so it may combine freely with
+	// chat/completion/embeddings on one config and shares the slot's
+	// prompt cache with generation.
 	FLAG_SCORE ModelConfigUsecase = 0b10000000000000000000
 
 	// Marks a model as wired for the Depth gRPC primitive (per-pixel
@@ -1691,9 +1765,9 @@ func GetUsecasesFromYAML(input []string) *ModelConfigUsecase {
 // either, they reserved the model for an internal direct-decode primitive
 // (the router classifier, or the PII NER tier). Letting GuessUsecases
 // paint chat/completion/embeddings on top would surface it in pickers it
-// was deliberately kept out of, and (on llama-cpp) reintroduce the slot
-// contention the conflict check exists to prevent. So a declared score or
-// token_classify list is authoritative.
+// was deliberately kept out of. So a declared score or token_classify
+// list is authoritative; declare the generation usecases explicitly
+// alongside score to serve both from one config.
 func (c *ModelConfig) HasUsecases(u ModelConfigUsecase) bool {
 	if c.KnownUsecases != nil {
 		if (u & *c.KnownUsecases) == u {
@@ -1883,8 +1957,8 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool {
 
 	if (u & FLAG_SCORE) == FLAG_SCORE {
 		// No heuristic: Score-intent is a deliberate operator choice
-		// (it reserves the model from generation traffic on llama-cpp),
-		// so HasUsecases(FLAG_SCORE) is true only when KnownUsecases
+		// (it keeps the model out of pickers it wasn't meant for), so
+		// HasUsecases(FLAG_SCORE) is true only when KnownUsecases
 		// declares it explicitly.
 		return false
 	}

core/config/model_config_loader.go

@@ -201,8 +201,8 @@ func (bcl *ModelConfigLoader) LoadModelConfigFileByNameDefaultOptions(modelName
 // survives unresolved into model loading and fails downstream — notably in
 // distributed mode with "backend name is empty". Mirrors the top-level alias
 // resolution in core/http/middleware/request.go.
-func (bcl *ModelConfigLoader) LoadResolvedModelConfig(modelName, modelPath string) (*ModelConfig, error) {
-	cfg, err := bcl.LoadModelConfigFileByName(modelName, modelPath)
+func (bcl *ModelConfigLoader) LoadResolvedModelConfig(modelName, modelPath string, opts ...ConfigLoaderOption) (*ModelConfig, error) {
+	cfg, err := bcl.LoadModelConfigFileByName(modelName, modelPath, opts...)
 	if err != nil {
 		return nil, err
 	}