harden(distributed): re-derive companion options when the reconciler replays a stored blob

Defensive follow-up to #11075; not the fix for the earlier longcat-video remote
failure (that was a stale worker backend, fixed by upgrading it).

The reconciler's scale-up path (ScheduleAndLoadModel) replays a ModelOptions
proto blob captured at first load and never runs grpcModelOpts. So a companion
resolved AFTER that blob was captured (e.g. the blob predates the companion's
resolution) would never reach a scaled-up replica, which would then load without
the companion option. applyCompanionOptions re-derives the model's managed
companion options from its CURRENT config (via the new
backend.CompanionArtifactOptions, wired through SmartRouterOptions.CompanionOptionsFor
in distributed startup) and merges in only those the blob is missing — an option
the blob already carries, including an author pin, always wins. nil-safe: single
node and any caller that does not wire the resolver are an unchanged no-op.

Also adds a regression guard for the controller->worker handoff:
FileStagingClient (the client wrapping every distributed LoadModel) must forward
ModelOptions.Options — the channel a managed companion option rides on — and
ModelPath to the backend unchanged. The test pins that so a future refactor of
the wrapper cannot silently drop options.

Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.agents/building-and-testing.md

@@ -28,6 +28,7 @@ The core Go suites (`./pkg`, `./core`, plus the in-process integration suite `./
   - **Build tags (`COVERAGE_TAGS`, passed via `GINKGO_TAGS`):** defaults to `debug auth`. The `auth` tag is required to compile the real (sqlite-backed) auth implementation and its ~150 `//go:build auth` tests — without it those files aren't built, the tests don't run, and the gate scores auth against a stub (~3.7% instead of ~38%). If you add new tag-gated tests, extend `COVERAGE_TAGS` or they won't count (and likely won't run in CI at all).
 - `make test-coverage-check` — runs `test-coverage`, then `scripts/coverage-check.sh` fails the build if total coverage is **below** the committed baseline in `coverage-baseline.txt`. The Linux job in `.github/workflows/test.yml` runs this instead of `make test`.
 - `make test-coverage-baseline` — regenerates and overwrites `coverage-baseline.txt` from the current run.
+- `make install-hooks` — sets `core.hooksPath` to the versioned `.githooks/`, whose `pre-commit` runs checks scoped to what's staged: Go changes → `make lint` + `make test-coverage-check`; `core/http/react-ui/` changes → `make test-ui-coverage-check` (Playwright e2e + UI coverage gate). A commit touching neither is skipped; bypass with `git commit --no-verify`. The hook resolves golangci-lint's new-from base to `upstream/master` → `origin/master` → `master`, so it works from a fork clone where `origin/master` is stale (passed to `make lint` via `LINT_NEW_FROM`).
 
 ### React UI coverage
 
@@ -37,11 +38,12 @@ The React UI (`core/http/react-ui/`) has **no component/unit tests** — its onl
 - **Browser:** the flake dev shell ships `chromium` and exports `PLAYWRIGHT_CHROMIUM_PATH`; `playwright.config.js` uses it via `launchOptions.executablePath`, and the Makefile skips `playwright install` when it's set. This avoids Playwright's downloaded browser, which can't resolve system libs (`libglib-2.0`, …) on NixOS. In CI (no `PLAYWRIGHT_CHROMIUM_PATH`) the Makefile falls back to `playwright install --with-deps chromium`.
 - The app is a React SPA, so coverage accumulates across in-app navigation within a test; a full `page.goto`/reload resets it.
 - `.nycrc.json` uses `all: true`, so **every `src/**` file is in the report**, including 0%-coverage ones — that's how you spot features with no test at all (sort the HTML report or `coverage-summary.json` by line% ascending). 
-- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`).
+- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`) and pre-commit on `core/http/react-ui/` changes.
 - **Why it has a tolerance (unlike the strict Go gate):** UI e2e coverage is *non-deterministic*. Specs that assert on state and end while async/lazy render work is still in flight collect those lines only when the render beats the coverage teardown — so the total drifts with machine speed/load (a fast local box reads higher than a slow CI runner), diffusely across many specs. The tolerance absorbs that drift, so set the baseline *below* the slow-CI floor, never to a fast-local `make test-ui-coverage-baseline` number, or CI flaps.
 - **Raising coverage is cheap:** a *render-smoke* spec (navigate to a route, assert its header renders) mounts a lazy page and runs its full render + initial effects, capturing most of its lines in a few lines of test — see `e2e/page-render-smoke.spec.js`. Auth is disabled in the test server (`isAdmin=true`), so `RequireAdmin`/`RequireFeature` routes render without a mock. The most *deterministic* win is removing a race: make a spec `await` a rendered element before ending (see `e2e/agents.spec.js` → AgentCreate) so its lines count every run.
 
 Rules (both gates):
-- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
+- **Install the hooks:** `make install-hooks` once per clone so lint + coverage run pre-commit. Don't lean on CI for what the hook catches.
+- **Don't work around the gate:** never `git commit --no-verify`, and never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
 - If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
 - The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.

.docker/llama-cpp-compile.sh

@@ -28,10 +28,6 @@ if [ -z "${BUILD_TYPE:-}" ]; then
   # variants with it (the host never *selects* SME unless it has it, but every variant must
   # still compile).
   if [ "${TARGETARCH}" = "arm64" ]; then
-    # The prebuilt base inherits default ports.ubuntu.com sources; honor the
-    # APT_*_MIRROR build args here like the from-source path does, so this
-    # apt step survives a mirror outage.
-    sh /LocalAI/.docker/apt-mirror.sh || true
     apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
     export CC=gcc-14 CXX=g++-14
   fi

.githooks/pre-commit

@@ -0,0 +1,72 @@
+#!/usr/bin/env sh
+#
+# LocalAI pre-commit hook. Install it (once per clone) with:
+#
+#     make install-hooks
+#
+# Runs only the checks relevant to what's staged:
+#   - Go files          -> make lint + make test-coverage-check
+#   - core/http/react-ui -> make test-ui-coverage-check (Playwright e2e + gate)
+#   - realtime state machines / specs -> make test-realtime-conformance
+#       (respcoord/**, turncoord/**, or formal-verification/** -- a pure .fizz
+#        spec edit must still re-verify the design, detected separately from Go)
+# A commit touching none of these is skipped entirely (other docs/YAML can't
+# change lint findings, Go coverage, the UI, or the realtime conformance gate).
+#
+# To bypass for a single commit (e.g. a WIP checkpoint): git commit --no-verify
+set -eu
+
+repo_root="$(git rev-parse --show-toplevel)"
+cd "$repo_root"
+
+staged="$(git diff --cached --name-only --diff-filter=ACMRD)"
+
+go_changed=0
+ui_changed=0
+rt_changed=0
+if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
+if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
+if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
+
+if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
+	echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
+	exit 0
+fi
+
+if [ "$go_changed" -eq 1 ]; then
+	# Resolve the ref golangci-lint's new-from-merge-base should compare
+	# against. .golangci.yml pins origin/master, which is correct in CI
+	# (origin == the canonical repo) but wrong from a fork clone, where
+	# origin/master lags behind and lint would report the whole upstream
+	# backlog. Prefer upstream/master, then origin/master, then master.
+	lint_base=""
+	for ref in upstream/master origin/master master; do
+		if git rev-parse --verify --quiet "${ref}^{commit}" >/dev/null 2>&1; then
+			lint_base="$ref"
+			break
+		fi
+	done
+
+	echo "pre-commit ▶ golangci-lint (make lint${lint_base:+, new-from $lint_base})"
+	make lint LINT_NEW_FROM="$lint_base"
+
+	echo "pre-commit ▶ coverage gate (make test-coverage-check) — builds and runs the"
+	echo "             pkg/core suites plus tests/e2e; can take a few minutes."
+	make test-coverage-check
+fi
+
+if [ "$ui_changed" -eq 1 ]; then
+	echo "pre-commit ▶ React UI e2e + coverage gate (make test-ui-coverage-check) —"
+	echo "             rebuilds the UI + ui-test-server, runs the Playwright specs, and"
+	echo "             fails if line coverage regressed; can take a couple of minutes."
+	make test-ui-coverage-check
+fi
+
+if [ "$rt_changed" -eq 1 ]; then
+	echo "pre-commit ▶ realtime state-machine conformance (make test-realtime-conformance) —"
+	echo "             Go transition/rapid tests under -race + FizzBee model check of the"
+	echo "             authoritative specs. Fail-closed: needs FizzBee (make install-fizzbee)."
+	make test-realtime-conformance
+fi
+
+echo "pre-commit ✓ all relevant checks passed"

.github/backend-matrix.yml

@@ -66,34 +66,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-kokoro'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'true'
-    backend: "kokoro"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-kokoro'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'true'
-    backend: "kokoro"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -899,19 +871,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-magpie-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1976,32 +1935,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-magpie-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-vllm-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "vllm-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2067,32 +2000,6 @@ include:
     backend: "moss-tts-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-vllm-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "vllm-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp'
-    base-image: "ubuntu:24.04"
-    ubuntu-version: '2404'
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -3056,19 +2963,6 @@ include:
     dockerfile: "./backend/Dockerfile.privacy-filter"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-vllm-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "vllm-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   # Vulkan: base-grpc-vulkan-amd64 carries the SDK. arm64 vulkan is a one-line
   # add once amd64 is proven in CI.
   - build-type: 'vulkan'
@@ -4689,20 +4583,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-magpie-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4717,50 +4597,7 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  # vllm-cpp
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-vllm-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "vllm-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-vllm-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "vllm-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   # omnivoice-cpp
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-magpie-tts-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4815,19 +4652,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f32'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f32-magpie-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4867,19 +4691,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f16'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f16-magpie-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4921,20 +4732,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4977,20 +4774,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-magpie-tts-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -5031,19 +4814,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2204'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-arm64-magpie-tts-cpp'
-    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2204'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -5083,19 +4853,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'hipblas'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-rocm-hipblas-magpie-tts-cpp'
-    base-image: "rocm/dev-ubuntu-24.04:6.4.4"
-    runs-on: 'ubuntu-latest'
-    skip-drivers: 'false'
-    backend: "magpie-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -6017,14 +5774,6 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-moss-tts-cpp"
     build-type: "metal"
     lang: "go"
-  - backend: "magpie-tts-cpp"
-    tag-suffix: "-metal-darwin-arm64-magpie-tts-cpp"
-    build-type: "metal"
-    lang: "go"
-  - backend: "vllm-cpp"
-    tag-suffix: "-metal-darwin-arm64-vllm-cpp"
-    build-type: "metal"
-    lang: "go"
   - backend: "omnivoice-cpp"
     tag-suffix: "-metal-darwin-arm64-omnivoice-cpp"
     build-type: "metal"

.github/workflows/bump_deps.yaml

@@ -50,10 +50,6 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
-          - repository: "mudler/vllm.cpp"
-            variable: "VLLM_CPP_VERSION"
-            branch: "main"
-            file: "backend/go/vllm-cpp/Makefile"
           - repository: "localai-org/moss-transcribe.cpp"
             variable: "MOSS_VERSION"
             branch: "master"
@@ -114,10 +110,6 @@ jobs:
             variable: "VIBEVOICE_CPP_VERSION"
             branch: "master"
             file: "backend/go/vibevoice-cpp/Makefile"
-          - repository: "mudler/magpie-tts.cpp"
-            variable: "MAGPIETTS_CPP_VERSION"
-            branch: "main"
-            file: "backend/go/magpie-tts-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7

.github/workflows/image-pr.yml

@@ -52,7 +52,7 @@
               tag-latest: 'false'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:24.04"
+              base-image: "ubuntu:22.04"
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'hipblas'

.github/workflows/image.yml

@@ -113,7 +113,7 @@
               tag-latest: 'auto'
               tag-suffix: '-gpu-nvidia-cuda-13'
               runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:24.04"
+              base-image: "ubuntu:22.04"
               skip-drivers: 'false'
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'

.github/workflows/lint.yml

@@ -61,7 +61,7 @@ jobs:
       # The backend matrix path filter fails silently: a miss emits an empty
       # matrix, every job goes green, and the change reaches no image (#10946).
       # Its tests need only node, so they ride along with this job.
-      - uses: actions/setup-node@v7
+      - uses: actions/setup-node@v4
         with:
           node-version: '20'
       - name: run CI script tests

.github/workflows/test-extra.yml

@@ -37,7 +37,6 @@ jobs:
       sglang: ${{ steps.detect.outputs.sglang }}
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
       qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
-      magpie-tts-cpp: ${{ steps.detect.outputs.magpie-tts-cpp }}
       rfdetr-cpp: ${{ steps.detect.outputs.rfdetr-cpp }}
       locate-anything-cpp: ${{ steps.detect.outputs.locate-anything-cpp }}
       vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
@@ -867,38 +866,6 @@ jobs:
       - name: Test qwen3-tts-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/qwen3-tts-cpp test
-  tests-magpie-tts-cpp:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.magpie-tts-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v7
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
-      - name: Setup Go
-        uses: actions/setup-go@v5
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Build magpie-tts-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp
-      - name: Test magpie-tts-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp test
   # Per-backend smoke for rfdetr-cpp: builds the .so + Go binary and runs
   # `make -C backend/go/rfdetr-cpp test`. test.sh fetches the small (~20 MB)
   # rfdetr-nano-q8_0 GGUF from the published mudler/rfdetr-cpp-nano HF repo

AGENTS.md

@@ -35,7 +35,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 
 ## Quick Reference
 
-- **Coverage gates**: Never lower a coverage baseline or widen a gate's tolerance to turn a red gate green — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
+- **Git hooks & coverage gates**: Run `make install-hooks` once per clone so the pre-commit lint + coverage gates run. **Never bypass them with `git commit --no-verify`, and never lower a coverage baseline or widen a gate's tolerance to turn a red gate green** — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
 - **Logging**: Use `github.com/mudler/xlog` (same API as slog)
 - **Go style**: Prefer `any` over `interface{}`
 - **Comments**: Explain *why*, not *what*

CONTRIBUTING.md

@@ -198,6 +198,7 @@ For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`C
 
 - Prefer modern Go idioms — for example, use `any` instead of `interface{}`.
 - Use [`golangci-lint`](https://golangci-lint.run) to catch common issues before submitting a PR.
+- Run `make install-hooks` once per clone to enable the pre-commit hook: Go changes run `make lint` + the coverage gate (`make test-coverage-check`); `core/http/react-ui/` changes run the Playwright e2e suite (`make test-ui`). Bypass a single commit with `git commit --no-verify`.
 - Use [`github.com/mudler/xlog`](https://github.com/mudler/xlog) for logging (same API as `slog`). Do not use `fmt.Println` or the standard `log` package for operational logging.
 - Use tab indentation for Go files (as defined in `.editorconfig`).
 
@@ -267,7 +268,7 @@ make test-e2e
 
 ### React UI tests and coverage
 
-The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
+The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI and pre-commit). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
 
 **If your change lowers UI coverage, raise it back by adding specs — do not widen the tolerance or hand-lower the baseline.** A *render-smoke* spec (navigate to a page, assert its header is visible) cheaply covers an entire lazy page. See `core/http/react-ui/e2e/page-render-smoke.spec.js` and the full policy in [.agents/building-and-testing.md](.agents/building-and-testing.md#react-ui-coverage).
 

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -103,7 +103,7 @@ COVERAGE_E2E_LABELS?=!real-models
 COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go
 
 
-.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
+.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check install-hooks build vendor lint lint-all
 
 all: help
 
@@ -269,7 +269,8 @@ LINT_EXCLUDE_DIRS_RE=/(backend/go/(piper|silero-vad|llm)|cmd/launcher)(/|$$)
 
 ## Set LINT_NEW_FROM to a git ref to override .golangci.yml's
 ## new-from-merge-base (origin/master). Useful from a fork clone where
-## origin/master is stale relative to the canonical repo.
+## origin/master is stale relative to the canonical repo — the pre-commit
+## hook passes the resolved upstream ref here so local lint matches CI.
 LINT_NEW_FROM?=
 lint:
 	@command -v golangci-lint >/dev/null 2>&1 || { \
@@ -288,6 +289,17 @@ lint-all:
 	}
 	golangci-lint run --new=false --new-from-merge-base= --new-from-rev= $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
 
+########################################################
+## Git hooks
+########################################################
+## Points git at the versioned .githooks/ directory so the pre-commit hook
+## (lint + coverage gate) runs locally. Run once per clone. Undo with:
+## `git config --unset core.hooksPath`. Skip a single commit with
+## `git commit --no-verify`.
+install-hooks:
+	git config core.hooksPath .githooks
+	@echo 'Installed git hooks: core.hooksPath -> .githooks (pre-commit runs lint + test-coverage-check on Go changes)'
+
 ########################################################
 ## E2E AIO tests (uses standard image with pre-configured models)
 ########################################################
@@ -625,7 +637,6 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/go/locate-anything-cpp test
 	$(MAKE) -C backend/go/depth-anything-cpp test
 	$(MAKE) -C backend/go/supertonic test
-	$(MAKE) -C backend/go/vllm-cpp test
 
 ##
 ## End-to-end gRPC tests that exercise a built backend container image.
@@ -1258,8 +1269,6 @@ BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
 BACKEND_MOSS_TTS_CPP = moss-tts-cpp|golang|.|false|true
-BACKEND_MAGPIE_TTS_CPP = magpie-tts-cpp|golang|.|false|true
-BACKEND_VLLM_CPP = vllm-cpp|golang|.|false|true
 BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
 BACKEND_LOCALVQE = localvqe|golang|.|false|true
@@ -1387,8 +1396,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TTS_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_MAGPIE_TTS_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
@@ -1408,7 +1415,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1443,7 +1450,7 @@ test-ui-e2e: build-ui-test-server
 UI_TEST_WORKERS ?=
 PLAYWRIGHT_WORKERS_FLAG = $(if $(UI_TEST_WORKERS),--workers=$(UI_TEST_WORKERS),)
 
-## Fast Playwright e2e run for local React UI validation.
+## Fast Playwright e2e run used by the pre-commit hook on React UI changes.
 ## Force-rebuilds the (non-instrumented) dist so the suite tests the working
 ## tree — not a stale dist the `react-ui` skip-guard would leave — re-embeds
 ## it into ui-test-server, and runs the specs. Uses the nix-provided browser

README.md

@@ -231,11 +231,9 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 
 | Backend | What it does |
 |---------|-------------|
-| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
 | [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
 | [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
-| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
 | [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
 | [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
 | [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |

backend/Dockerfile.golang

@@ -221,33 +221,6 @@ RUN if [ "${BACKEND}" = "crispasr" ]; then \
     apt-get clean && rm -rf /var/lib/apt/lists/*; \
 fi
 
-# sherpa-onnx links onnxruntime's CUDA execution provider, and
-# libonnxruntime_providers_cuda.so has cuDNN as a hard DT_NEEDED. The
-# onnxruntime GPU tarball does not ship cuDNN itself, so without this the
-# builder has none (the arm64 + CUDA 13 branch above is the only other place
-# that installs it) and package-gpu-libs.sh correctly refuses to produce a
-# package that references cuDNN with no cuDNN available to it.
-#
-# Installed per-backend rather than for every cublas build: the auto-detection
-# in package-gpu-libs.sh bundles only what a package actually references, so
-# the ggml backends would not grow either way, but they would all pay ~1.1 GB
-# of builder layer and registry cache for a library they never call.
-#
-# Runtime package only, no -dev: sherpa-onnx consumes onnxruntime's prebuilt
-# CUDA provider and never compiles against cuDNN headers. libcudnn9-cuda-N
-# carries the dispatcher plus all seven dlopen()ed sublibraries, which is what
-# complete_cudnn_family needs to assemble a whole bundle.
-RUN <<EOT bash
-    if [ "${BACKEND}" = "sherpa-onnx" ] && [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            libcudnn9-cuda-${CUDA_MAJOR_VERSION} && \
-        ldconfig && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
 COPY . /LocalAI
 
 RUN git config --global --add safe.directory /LocalAI

backend/Dockerfile.llama-cpp

@@ -111,10 +111,6 @@ RUN make -BC /LocalAI/backend/cpp/llama-cpp package
 # ============================================================================
 FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
 
-ARG APT_MIRROR
-ENV APT_MIRROR=${APT_MIRROR}
-ARG APT_PORTS_MIRROR
-ENV APT_PORTS_MIRROR=${APT_PORTS_MIRROR}
 ARG BUILD_TYPE
 ENV BUILD_TYPE=${BUILD_TYPE}
 ARG CUDA_DOCKER_ARCH

backend/backend.proto

@@ -181,13 +181,6 @@ message ScoreRequest {
   // PredictOptions.ModelIdentity for the full rationale. Empty means "no
   // identity supplied" and backends MUST skip the check.
   string ModelIdentity = 5;
-  // Byte length of the prompt prefix that stays identical across
-  // repeated scoring calls (e.g. a classifier's option-list system
-  // prompt — everything before the per-turn probe text). Backends that
-  // snapshot state (hybrid/recurrent models cannot rewind otherwise)
-  // use it to place a reuse point exactly at the boundary, so the next
-  // call re-processes only the tokens after it. 0 means unknown.
-  int32 stable_prefix_len = 6;
 }
 
 // CandidateScore is one row in the ScoreResponse, matching by index
@@ -500,11 +493,6 @@ message ModelOptions {
   // Proxy carries the cloud-proxy backend's per-model configuration.
   // Empty for non-proxy backends.
   ProxyOptions Proxy = 74;
-
-  // EnableScore reserves backend resources for the Score RPC. It is derived
-  // from the model's explicit `known_usecases: [score]` declaration so models
-  // that never score retain their ordinary serving footprint.
-  bool EnableScore = 75;
 }
 
 // ProxyOptions configures the cloud-proxy backend. UpstreamURL and
@@ -520,12 +508,6 @@ message ProxyOptions {
   string api_key_file = 5;
   string upstream_model = 6;
   int32 request_timeout_seconds = 7;
-  // cache_prompt enables automatic Anthropic prompt-cache breakpoints
-  // (cache_control: ephemeral) on the stable prefix — system, tools, and
-  // the last message block — when translating to the Anthropic provider.
-  // Cuts input cost on repeated/agentic calls (cache read = 0.1x). Only
-  // meaningful for mode=translate + provider=anthropic; ignored otherwise.
-  bool cache_prompt = 8;
 }
 
 message Result {

backend/cpp/bonsai/Makefile

@@ -41,7 +41,6 @@ define bonsai-build
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
-	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build llama.cpp
@@ -78,7 +77,6 @@ bonsai-cpu-all:
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
-	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build llama.cpp

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
+# Upstream pin lives below as DS4_VERSION?=efdadd41e20134af4f3381e1ed90e96fe4faef6f
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
+DS4_VERSION?=efdadd41e20134af4f3381e1ed90e96fe4faef6f
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=b054a8b983827c01aec59d4dc273a27c492c51c4
+IK_LLAMA_VERSION?=e5357286c0d433cd4384e82ed7e2b6d655f57087
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=1cbfd1988311775425d36c0ce066590f7d3049cf
+LLAMA_VERSION?=571d0d540df04f25298d0e159e520d9fc62ed121
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/disable-score-task.sh

@@ -1,43 +0,0 @@
-#!/bin/bash
-# Mark a copied gRPC server as targeting a llama.cpp fork that does not carry
-# LocalAI's slot-based Score patches. The RPC remains present in the shared
-# protobuf service, but responds with UNIMPLEMENTED instead of referencing
-# server task types and common_params fields absent from those forks.
-
-set -euo pipefail
-
-if [[ $# -ne 1 ]]; then
-    echo "usage: $0 <grpc-server.cpp>" >&2
-    exit 2
-fi
-
-SRC=$1
-
-if [[ ! -f "$SRC" ]]; then
-    echo "grpc-server.cpp not found at $SRC" >&2
-    exit 2
-fi
-
-if grep -q '^#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK' "$SRC"; then
-    echo "==> $SRC already disables the LocalAI score task, skipping"
-    exit 0
-fi
-
-awk '
-    !done && /^#include/ {
-        print "#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK 1"
-        print "// ^ injected by disable-score-task.sh for an unpatched llama.cpp fork"
-        print ""
-        done = 1
-    }
-    { print }
-    END {
-        if (!done) {
-            print "disable-score-task.sh: no #include anchor found" > "/dev/stderr"
-            exit 1
-        }
-    }
-' "$SRC" > "$SRC.tmp"
-mv "$SRC.tmp" "$SRC"
-
-echo "==> LocalAI score task disabled in $SRC"

backend/cpp/llama-cpp/grpc-server.cpp

@@ -52,7 +52,6 @@
 #include "common.h"
 #include "arg.h"
 #include "chat-auto-parser.h"
-#include "llama_compat.h" // fork-skew switches, generated by prepare.sh
 #include "message_content.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
@@ -152,6 +151,40 @@ static std::string base64_encode_bytes(const unsigned char* data, size_t len) {
 
 bool loaded_model; // TODO: add a mutex for this, but happens only once loading the model
 
+// Score bypasses the slot loop (see the comment on Score below) so it
+// must not run concurrently with any slot-loop RPC. These counters
+// are a defence-in-depth tripwire — ModelConfig.Validate already
+// rejects llama-cpp configs that mix score with chat/completion/
+// embeddings, so a healthy deployment never trips them. seq_cst is
+// load-bearing for the increment-then-check pattern below.
+static std::atomic<int> slot_loop_inflight{0};
+static std::atomic<int> score_inflight{0};
+
+// Increment-then-check, not check-then-increment: two simultaneous
+// racers both observe the other's increment and both abort cleanly.
+// Reversed, both could see zero and proceed.
+struct conflict_guard {
+    std::atomic<int>& self;
+    conflict_guard(const char* rpc, std::atomic<int>& self_, std::atomic<int>& other, const char* other_name)
+        : self(self_) {
+        self.fetch_add(1, std::memory_order_seq_cst);
+        int o = other.load(std::memory_order_seq_cst);
+        if (o > 0) {
+            fprintf(stderr,
+                "FATAL: %s called with %s=%d. The llama-cpp backend cannot "
+                "service Score and slot-loop RPCs concurrently — Score "
+                "bypasses the slot loop and races the llama_context. Bind "
+                "Score-using features to a model dedicated to scoring "
+                "(known_usecases: [score] with no chat/completion/embeddings).\n",
+                rpc, other_name, o);
+            std::abort();
+        }
+    }
+    ~conflict_guard() {
+        self.fetch_sub(1, std::memory_order_seq_cst);
+    }
+};
+
 static std::function<void(int)> shutdown_handler;
 static std::atomic_flag is_terminating = ATOMIC_FLAG_INIT;
 
@@ -580,13 +613,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
     // starts with '-'. Applied once after the loop via common_params_parse.
     std::vector<std::string> extra_argv;
 
-    // O_DIRECT intent from the `direct_io` option. Upstream folded
-    // use_mmap/use_mlock/use_direct_io into a single common_params::load_mode
-    // enum (ggml-org/llama.cpp#20834), so the three independent LocalAI settings
-    // can only be reduced to one value once all of them have been read, held
-    // here until the mmap/mlock fields arrive further down.
-    bool want_direct_io = false;
-
     auto add_device_options = [&](const std::string & devices) {
         const std::regex regex{ R"([,]+)" };
         std::sregex_token_iterator it{ devices.begin(), devices.end(), regex, -1 };
@@ -698,22 +724,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                     // If conversion fails, keep default value (0)
                 }
             }
-#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
-        } else if (!strcmp(optname, "n_rs_seq") || !strcmp(optname, "rs_seq")) {
-            // Recurrent-state rollback snapshots per sequence. Hybrid models
-            // (deltanet/conv layers) cannot rewind their state, so without
-            // snapshots any prompt-cache reuse that needs a rewind — e.g. a
-            // score task whose probe changed under a stable option-list
-            // prefix — falls back to a full re-prefill. Costs recurrent-state
-            // memory x (1 + N) per sequence; unsupported archs clamp to 0.
-            if (optval != NULL) {
-                try {
-                    params.n_rs_seq = std::stoi(optval_str);
-                } catch (const std::exception& e) {
-                    // If conversion fails, keep default value (0)
-                }
-            }
-#endif
         } else if (!strcmp(optname, "slot_prompt_similarity") || !strcmp(optname, "sps")) {
             if (optval != NULL) {
                 try {
@@ -858,9 +868,9 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         // --- O_DIRECT model loading (upstream --direct-io) ---
         } else if (!strcmp(optname, "direct_io") || !strcmp(optname, "use_direct_io")) {
             if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
-                want_direct_io = true;
+                params.use_direct_io = true;
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
-                want_direct_io = false;
+                params.use_direct_io = false;
             }
 
         // --- embedding normalization (upstream --embd-normalize) ---
@@ -1268,28 +1278,8 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
      lora_info.ptr = nullptr;
      params.lora_adapters.push_back(std::move(lora_info));
     }
-    // LocalAI keeps mmap, mlock and direct-I/O as three independent settings,
-    // while upstream now carries a single load mode. Fold them with the
-    // precedence the separate booleans used to give: direct I/O bypasses the
-    // page cache entirely, mlock implies mmap, and everything off is a plain
-    // buffered read. Forks that branched before ggml-org/llama.cpp#20834 still
-    // expose the booleans; prepare.sh probes the checkout and sets
-    // LOCALAI_LEGACY_LOAD_MODE in the generated llama_compat.h accordingly.
-#if LOCALAI_LEGACY_LOAD_MODE
-    params.use_mlock     = request->mlock();
-    params.use_mmap      = request->mmap();
-    params.use_direct_io = want_direct_io;
-#else
-    if (want_direct_io) {
-        params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO;
-    } else if (request->mlock()) {
-        params.load_mode = LLAMA_LOAD_MODE_MLOCK;
-    } else if (request->mmap()) {
-        params.load_mode = LLAMA_LOAD_MODE_MMAP;
-    } else {
-        params.load_mode = LLAMA_LOAD_MODE_NONE;
-    }
-#endif
+    params.use_mlock = request->mlock();
+    params.use_mmap = request->mmap();
 
     if (request->flashattention() == "on" || request->flashattention() == "enabled") {
         params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;
@@ -1374,17 +1364,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         }
     }
 
-#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
-    // Score-task suffix forking: reserve seq ids (and recurrent-state cells)
-    // beyond the slots so one scoring call decodes all candidate tails in a
-    // single batch (SERVER_TASK_TYPE_SCORE, patches/). Requires the unified
-    // KV cache — with per-sequence streams the extra ids would shrink every
-    // sequence's context to n_ctx / n_seq_max. Decided after both option
-    // passes so an explicit kv_unified:false wins and disables forking.
-    params.score_enabled = request->enablescore();
-    params.n_seq_score_forks = params.score_enabled && params.kv_unified ? SERVER_SCORE_FORK_SEQS : 0;
-#endif
-
     // Terminate/pad the override vectors only after BOTH the named-option loop
     // and the generic passthrough (common_params_parse above) have pushed their
     // real entries, so back() is the null sentinel the model loader asserts on.
@@ -1471,16 +1450,6 @@ public:
         common_params params;
         params_parse(ctx_server, request, params);
 
-#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
-        if (params.score_enabled && !params.kv_unified) {
-            const std::string error_msg =
-                "Score requires the unified KV cache; remove kv_unified:false or remove score from known_usecases";
-            result->set_message(error_msg);
-            result->set_success(false);
-            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, error_msg);
-        }
-#endif
-
         common_init();
         // Ensure debug logs are enabled after common_init() sets up logging
         common_log_set_verbosity_thold(params.verbosity);
@@ -1683,6 +1652,7 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
+        conflict_guard guard("PredictStream", slot_loop_inflight, score_inflight, "score_inflight");
         json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
 
@@ -2251,6 +2221,7 @@ public:
          if (params_base.model.path.empty()) {
              return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
          }
+         conflict_guard guard("Predict", slot_loop_inflight, score_inflight, "score_inflight");
          json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
         data["stream"] = false;
@@ -2784,6 +2755,7 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
+        conflict_guard guard("Embedding", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
 
         body["stream"] = false;
@@ -2893,6 +2865,7 @@ public:
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "\"documents\" must be a non-empty string array");
         }
 
+        conflict_guard guard("Rerank", slot_loop_inflight, score_inflight, "score_inflight");
 
         // Create and queue the task
         auto rd = ctx_server.get_response_reader();
@@ -2969,16 +2942,37 @@ public:
     // Score returns the model's joint log-probability of each candidate
     // continuation given a shared prompt.
     //
-    // Scoring runs as a single SERVER_TASK_TYPE_SCORE task through the
-    // slot loop (added by patches/ on top of upstream server-context), so
-    // it is safe to interleave with generation on the same process and it
-    // reuses any KV prefix the slot already holds across turns. The task
-    // decodes the shared prefix (prompt + longest common candidate token
-    // prefix) once on the slot's sequence; every candidate's unique tail
-    // then rides its own forked sequence and all tails are decoded
-    // together in one batch, so a warm scoring call costs roughly one
-    // forward pass over the new prompt tokens plus one batched pass over
-    // the candidate tails.
+    // WHY bypass the slot/task queue: upstream server_context exposes
+    // get_llama_context as "main thread only" and the slot loop's
+    // update_slots() owns the context whenever a task is in flight.
+    // No public synchronization primitive is available — so Score is
+    // unsafe to call concurrently with active generation through this
+    // backend. In practice routing-classifier calls happen before the
+    // request is routed to a generation backend, so the model used
+    // for Score is typically idle. Concurrent Score calls are
+    // serialised by a local mutex; KV-cache state is isolated behind
+    // a dedicated sequence ID cleared between candidates.
+    //
+    // A patch to server-context.cpp that adds SERVER_TASK_TYPE_SCORE
+    // and routes scoring through the slot loop would be the correct
+    // long-term fix; tracked as a follow-up.
+    //
+    // Perf TODO (measured: ~450 ms warm for 3 candidates on Arch-
+    // Router-1.5B Q4_K_M + Intel SYCL): the current loop re-decodes
+    // `prompt + candidate` from scratch for every candidate, throwing
+    // away the prompt's KV cache between iterations. A smarter
+    // version would:
+    //   1. Decode just the prompt once into score_seq_id.
+    //   2. Snapshot/cp that sequence (llama_memory_seq_cp) into a
+    //      per-candidate sequence id.
+    //   3. For each candidate, decode only its tokens onto the copy
+    //      (continuing from the saved prompt state), read logits.
+    //   4. llama_memory_seq_rm the copy.
+    // Estimated speedup: 3-candidate calls 450 ms -> ~150-200 ms,
+    // 6-candidate calls 630 ms -> ~220 ms. Single source-file change,
+    // no proto / Go-side changes needed. Worth doing once routing is
+    // wired into the middleware and Score is on the hot path of every
+    // chat request.
     grpc::Status Score(ServerContext* context, const backend::ScoreRequest* request, backend::ScoreResponse* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
@@ -2987,21 +2981,40 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-#ifdef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
-        (void) request;
-        (void) response;
-        return grpc::Status(grpc::StatusCode::UNIMPLEMENTED,
-            "Score is unavailable in this llama.cpp fork backend");
-#else
-        if (!params_base.score_enabled) {
-            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION,
-                "Score was not enabled when the model was loaded; add score to known_usecases");
-        }
         if (request->candidates_size() == 0) {
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "candidates must be non-empty");
         }
 
+        // Tripwire against the slot loop. Acquired before score_mutex
+        // so it fires even when this Score is queued behind another.
+        conflict_guard guard("Score", score_inflight, slot_loop_inflight, "slot_loop_inflight");
+
+        // Serialise concurrent Score calls. The slot loop is still
+        // free to race with us — see the class comment above.
+        static std::mutex score_mutex;
+        std::lock_guard<std::mutex> score_lock(score_mutex);
+
+        llama_context * lctx = ctx_server.get_llama_context();
+        if (lctx == nullptr) {
+            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "llama context unavailable (sleeping?)");
+        }
         const llama_vocab * vocab = ctx_server.impl->vocab;
+        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
+        const int32_t n_ctx = llama_n_ctx(lctx);
+        llama_memory_t mem = llama_get_memory(lctx);
+
+        // The KV-cache is sized to seq_to_stream.size() at load
+        // (typically equal to n_slots, often 1). Sequence IDs must
+        // be in [0, n_seq_max), so we can't pick a high-value
+        // "private" ID — we have to share with the slot. We clear
+        // the cache before AND after each candidate to keep
+        // scoring isolated from whatever state the slot held, and
+        // the static mutex above guarantees no other Score call is
+        // racing in the meantime. The slot loop is still free to
+        // race (see comment on this method) — Score must not run
+        // concurrently with generation through this backend.
+        const llama_seq_id score_seq_id = 0;
+        llama_memory_seq_rm(mem, score_seq_id, -1, -1);
 
         // Tokenize the shared prompt once with add_special=true so
         // BOS is prepended when the model requires it. parse_special
@@ -3010,15 +3023,6 @@ public:
         std::vector<llama_token> prompt_tokens = common_tokenize(vocab, prompt, /*add_special=*/true, /*parse_special=*/true);
         const int32_t prompt_len = (int32_t) prompt_tokens.size();
 
-        // Per candidate: full prompt+candidate token list and the
-        // divergence point, kept for piece rendering and empty-candidate
-        // handling after the task comes back.
-        std::vector<std::vector<llama_token>> cand_tokens(request->candidates_size());
-        std::vector<int32_t> cand_divergence(request->candidates_size(), 0);
-
-        // candidates that actually have tokens to score
-        std::vector<int32_t> included;
-
         for (int ci = 0; ci < request->candidates_size(); ci++) {
             const std::string & candidate_text = request->candidates(ci);
 
@@ -3035,135 +3039,9 @@ public:
                     break;
                 }
             }
-            divergence = std::min<int32_t>(divergence, (int32_t) full_tokens.size());
-
             const int32_t cand_len = (int32_t) full_tokens.size() - divergence;
-            if (cand_len > 0 && divergence < 1) {
-                // Need at least one prior token (typically BOS) to
-                // predict the first candidate token's logit. Tokeniser
-                // models without BOS + an empty prompt fall in here.
-                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
-                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
-            }
-            if (cand_len > SERVER_SCORE_MAX_CAND_TOKENS) {
-                // The context reserves logits outputs for at most this many
-                // candidate tokens per slot (server_n_outputs_max).
-                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
-                    "Score: candidate " + std::to_string(ci) + " is " + std::to_string(cand_len) +
-                    " tokens; the maximum is " + std::to_string(SERVER_SCORE_MAX_CAND_TOKENS));
-            }
-
-            cand_divergence[ci] = divergence;
-            cand_tokens[ci] = std::move(full_tokens);
-
-            if (cand_len > 0) {
-                included.push_back(ci);
-            }
-        }
-
-        auto rd = ctx_server.get_response_reader();
-        bool posted_task = false;
-
-        // Shared prefix bounds, needed again when stitching the results:
-        // n_shared is the longest common token prefix of the scored
-        // candidates, n_score_prompt the earliest divergence from the
-        // bare prompt (scored logprobs start there).
-        int32_t n_shared       = 0;
-        int32_t n_score_prompt = 0;
-
-        if (!included.empty()) {
-            const auto & first = cand_tokens[included[0]];
-
-            // the common prefix of a set is the shortest common prefix
-            // against any fixed member
-            n_shared = (int32_t) first.size();
-            for (int32_t ci : included) {
-                const auto & ft = cand_tokens[ci];
-                const int32_t lim = std::min<int32_t>(n_shared, (int32_t) ft.size());
-                int32_t match = 0;
-                while (match < lim && ft[match] == first[match]) {
-                    match++;
-                }
-                n_shared = match;
-            }
-
-            // below its divergence every candidate equals the prompt
-            // tokens, so n_score_prompt <= n_shared always holds
-            n_score_prompt = cand_divergence[included[0]];
-            for (int32_t ci : included) {
-                n_score_prompt = std::min(n_score_prompt, cand_divergence[ci]);
-            }
-
-            // Map the caller's stable-prefix byte length onto a token
-            // index: the last prompt token that ends at or before the
-            // boundary. A checkpoint forced there survives every future
-            // probe under the same option list, which is what keeps
-            // repeat scoring cheap on models that cannot rewind state.
-            int32_t n_stable_prompt = 0;
-            if (request->stable_prefix_len() > 0) {
-                size_t consumed = 0;
-                for (int32_t ti = 0; ti < n_score_prompt; ti++) {
-                    const size_t piece_len = common_token_to_piece(vocab, prompt_tokens[ti]).size();
-                    // BOS and other zero-length specials consume no prompt bytes
-                    if (consumed + piece_len > (size_t) request->stable_prefix_len()) {
-                        break;
-                    }
-                    consumed += piece_len;
-                    n_stable_prompt = ti + 1;
-                }
-            }
-
-            server_task task(SERVER_TASK_TYPE_SCORE);
-            task.id             = rd.queue_tasks.get_new_id();
-            task.index          = 0;
-            task.tokens         = server_tokens(llama_tokens(first.begin(), first.begin() + n_shared), false);
-            task.n_score_prompt = n_score_prompt;
-            task.n_stable_prompt = n_stable_prompt;
-            task.score_suffixes.reserve(included.size());
-            for (int32_t ci : included) {
-                task.score_suffixes.emplace_back(cand_tokens[ci].begin() + n_shared, cand_tokens[ci].end());
-            }
-
-            std::vector<server_task> tasks;
-            tasks.push_back(std::move(task));
-            rd.post_tasks(std::move(tasks));
-            posted_task = true;
-        }
-
-        // Wait for the shared-prefix and per-candidate logprob vectors.
-        // Context overflow and decode failures surface here as task errors.
-        std::vector<float> shared_logprobs;
-        std::vector<std::vector<float>> cand_logprobs;
-        if (posted_task) {
-            auto all_results = rd.wait_for_all([&context]() { return context->IsCancelled(); });
-            if (all_results.is_terminated) {
-                return grpc::Status(grpc::StatusCode::CANCELLED, "Request cancelled by client");
-            }
-            if (all_results.error) {
-                return grpc::Status(grpc::StatusCode::INTERNAL,
-                    all_results.error->to_json().value("message", "Error in receiving score results"));
-            }
-            if (all_results.results.size() != 1) {
-                return grpc::Status(grpc::StatusCode::INTERNAL, "expected a single score result");
-            }
-            auto * score_res = dynamic_cast<server_task_result_score*>(all_results.results[0].get());
-            if (score_res == nullptr) {
-                return grpc::Status(grpc::StatusCode::INTERNAL, "unexpected result type for score task");
-            }
-            shared_logprobs = std::move(score_res->shared_logprobs);
-            cand_logprobs   = std::move(score_res->cand_logprobs);
-            if (cand_logprobs.size() != included.size()) {
-                return grpc::Status(grpc::StatusCode::INTERNAL, "score result candidate count mismatch");
-            }
-        }
-
-        size_t inc = 0; // index into included / cand_logprobs
-        for (int ci = 0; ci < request->candidates_size(); ci++) {
-            const int32_t divergence = cand_divergence[ci];
-            const int32_t cand_len   = (int32_t) cand_tokens[ci].size() - divergence;
-
             backend::CandidateScore * cs = response->add_candidates();
-            cs->set_num_tokens(cand_len > 0 ? cand_len : 0);
+            cs->set_num_tokens(cand_len);
             if (cand_len <= 0) {
                 cs->set_log_prob(0.0);
                 if (request->length_normalize()) {
@@ -3171,57 +3049,101 @@ public:
                 }
                 continue;
             }
-
-            // Stitch the candidate's scored logprobs back together: the
-            // stretch inside the shared prefix (identical for every
-            // candidate) followed by its forked suffix. Suffix entries
-            // before the candidate's own divergence are prompt tokens
-            // decoded only as context — not scored.
-            std::vector<float> lp;
-            lp.reserve(cand_len);
-            for (int32_t t = divergence; t < n_shared; t++) {
-                const int32_t idx = t - n_score_prompt;
-                if (idx < 0 || idx >= (int32_t) shared_logprobs.size()) {
-                    return grpc::Status(grpc::StatusCode::INTERNAL,
-                        "Score: shared logprob index out of range for candidate " + std::to_string(ci));
-                }
-                lp.push_back(shared_logprobs[idx]);
+            if (divergence < 1) {
+                // Need at least one prior token (typically BOS) to
+                // predict the first candidate token's logit. Tokeniser
+                // models without BOS + an empty prompt fall in here.
+                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
             }
-            const auto & sfx_lp = cand_logprobs[inc++];
-            for (int32_t j = std::max(0, divergence - n_shared); j < (int32_t) sfx_lp.size(); j++) {
-                lp.push_back(sfx_lp[j]);
+            if ((int32_t) full_tokens.size() > n_ctx) {
+                return grpc::Status(grpc::StatusCode::OUT_OF_RANGE,
+                    "Score: prompt+candidate exceeds context size (got " +
+                    std::to_string(full_tokens.size()) + ", n_ctx=" + std::to_string(n_ctx) + ")");
             }
 
-            if ((int32_t) lp.size() != cand_len) {
+            // Build a batch covering the entire prompt+candidate. We
+            // need logits at (divergence-1) onward — those are the
+            // predictions for each candidate token.
+            llama_batch batch = llama_batch_init((int32_t) full_tokens.size(), 0, 1);
+            for (int32_t i = 0; i < (int32_t) full_tokens.size(); i++) {
+                batch.token[i]    = full_tokens[i];
+                batch.pos[i]      = i;
+                batch.n_seq_id[i] = 1;
+                batch.seq_id[i][0] = score_seq_id;
+                // logits[i] is "do we want the prediction *for the
+                // next token*, computed from this position?"
+                // We want predictions for candidate tokens at
+                // positions divergence .. full_tokens.size()-1, which
+                // come from logits at positions (divergence-1) ..
+                // (full_tokens.size()-2).
+                bool need_logit = (i >= divergence - 1) && (i < (int32_t) full_tokens.size() - 1);
+                batch.logits[i] = need_logit ? 1 : 0;
+            }
+            batch.n_tokens = (int32_t) full_tokens.size();
+
+            // Decode the batch. If decode fails (e.g. KV slot
+            // exhaustion), surface as INTERNAL — the caller will
+            // typically fall back to a sampling-based classifier.
+            int decode_err = llama_decode(lctx, batch);
+            if (decode_err != 0) {
+                llama_batch_free(batch);
+                llama_memory_seq_rm(mem, score_seq_id, -1, -1);
                 return grpc::Status(grpc::StatusCode::INTERNAL,
-                    "Score: result for candidate " + std::to_string(ci) + " is missing token logprobs");
+                    "llama_decode failed during Score: " + std::to_string(decode_err));
             }
 
+            // Sum log-probabilities of the actual candidate tokens.
             double total_log_prob = 0.0;
             for (int32_t k = 0; k < cand_len; k++) {
-                const float token_log_prob = lp[k];
-                if (std::isnan(token_log_prob)) {
+                // The k-th candidate token sits at full_tokens index
+                // (divergence + k). Its predicting logit is at batch
+                // position (divergence + k - 1).
+                int32_t logit_pos = divergence + k - 1;
+                const float * logits = llama_get_logits_ith(lctx, logit_pos);
+                if (logits == nullptr) {
+                    llama_batch_free(batch);
+                    llama_memory_seq_rm(mem, score_seq_id, -1, -1);
                     return grpc::Status(grpc::StatusCode::INTERNAL,
-                        "Score: incomplete result for candidate " + std::to_string(ci) +
-                        " at token " + std::to_string(k));
+                        "llama_get_logits_ith returned null at position " + std::to_string(logit_pos));
                 }
-                total_log_prob += (double) token_log_prob;
+                llama_token target_token = full_tokens[divergence + k];
+
+                // Compute log_softmax(logits)[target_token] with the
+                // max-subtraction stability trick.
+                float max_logit = logits[0];
+                for (int32_t v = 1; v < n_vocab; v++) {
+                    if (logits[v] > max_logit) max_logit = logits[v];
+                }
+                double sum_exp = 0.0;
+                for (int32_t v = 0; v < n_vocab; v++) {
+                    sum_exp += std::exp((double)(logits[v] - max_logit));
+                }
+                double token_log_prob = (double)(logits[target_token] - max_logit) - std::log(sum_exp);
+                total_log_prob += token_log_prob;
 
                 if (request->include_token_logprobs()) {
                     backend::TokenLogProb * tlp = cs->add_tokens();
-                    tlp->set_token(common_token_to_piece(vocab, cand_tokens[ci][divergence + k]));
+                    std::string piece = common_token_to_piece(lctx, target_token);
+                    tlp->set_token(piece);
                     tlp->set_log_prob(token_log_prob);
                 }
             }
 
             cs->set_log_prob(total_log_prob);
-            if (request->length_normalize()) {
+            if (request->length_normalize() && cand_len > 0) {
                 cs->set_length_normalized_log_prob(total_log_prob / (double) cand_len);
             }
+
+            llama_batch_free(batch);
+            // Drop this candidate's KV-cache contribution so the next
+            // candidate starts from a clean state. Without this, the
+            // next decode would conflict at positions 0..N-1 for our
+            // sequence ID.
+            llama_memory_seq_rm(mem, score_seq_id, -1, -1);
         }
 
         return grpc::Status::OK;
-#endif
     }
 
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
@@ -3232,6 +3154,7 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
+        conflict_guard guard("TokenizeString", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
         body["stream"] = false;
 
@@ -3253,6 +3176,7 @@ public:
 
     grpc::Status GetMetrics(ServerContext* /*context*/, const backend::MetricsRequest* /*request*/, backend::MetricsResponse* response) override {
 
+        conflict_guard guard("GetMetrics", slot_loop_inflight, score_inflight, "score_inflight");
 
 // request slots data using task queue
         auto rd = ctx_server.get_response_reader();

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-chat-parser.patch

@@ -1,225 +0,0 @@
-# MiniMax-M3 chat-template parser, vendored from upstream llama.cpp PR #24523.
-#
-# Upstream has since merged the *model* half of #24523 (LLM_ARCH_MINIMAX_M3,
-# src/models/minimax-m3.cpp, the gguf-py constants and conversion/minimax.py), so
-# only the chat half is carried here: M3's namespace token "]<]minimax[>[" collides
-# with the autoparser's markup delimiters, so common/chat.cpp needs a dedicated
-# template detection + PEG parser that upstream does not have yet.
-#
-# Rebased against LLAMA_VERSION 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3, which also
-# renamed common_chat_params::thinking_end_tag to thinking_end_tags (a vector).
-# LLAMA_VERSION is auto-bumped nightly; if a bump rejects this patch, re-vendor from
-# #24523 — or, once the chat half merges upstream, delete this file.
-# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
-diff --git a/common/chat.cpp b/common/chat.cpp
-index 7a6e7238c..2dd015a2e 100644
---- a/common/chat.cpp
-+++ b/common/chat.cpp
-@@ -2121,6 +2121,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
-     return data;
- }
- 
-+static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
-+                                                             const autoparser::generation_params & inputs) {
-+    common_chat_params data;
-+
-+    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
-+    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
-+    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
-+    data.supports_thinking  = true;
-+    data.thinking_start_tag = "<mm:think>";
-+    data.thinking_end_tags  = {"</mm:think>"};
-+
-+    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
-+    // params use the parameter name as the tag (<file_path>...</file_path>).
-+    const std::string NS           = "]<]minimax[>[";
-+    const std::string THINK_START  = "<mm:think>";
-+    const std::string THINK_END    = "</mm:think>";
-+    const std::string FC_START     = NS + "<tool_call>";
-+    const std::string FC_END       = NS + "</tool_call>";
-+    const std::string INVOKE_END   = NS + "</invoke>";
-+
-+    data.preserved_tokens   = {
-+        NS,
-+        "<tool_call>",
-+        "</tool_call>",
-+        THINK_START,
-+        THINK_END,
-+    };
-+
-+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
-+    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
-+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
-+
-+    const std::string GEN_PROMPT = data.generation_prompt;
-+
-+    if (inputs.has_continuation()) {
-+        const auto & msg = inputs.continue_msg;
-+
-+        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
-+        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
-+            data.generation_prompt += THINK_END + msg.render_content();
-+        }
-+
-+        data.prompt += data.generation_prompt;
-+    }
-+
-+    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
-+        auto generation_prompt = p.literal(GEN_PROMPT);
-+        auto end = p.end();
-+
-+        auto reasoning = p.eps();
-+        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
-+        if (extract_reasoning && inputs.enable_thinking) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
-+        } else if (extract_reasoning) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
-+        }
-+
-+        if (has_response_format) {
-+            auto response_format = p.rule("response-format",
-+                p.literal("```json") + p.space() +
-+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
-+                p.space() + p.literal("```"));
-+            return generation_prompt + reasoning + response_format + end;
-+        }
-+
-+        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
-+            return generation_prompt + reasoning + p.content(p.rest()) + end;
-+        }
-+
-+        auto tool_choice = p.choice();
-+        foreach_function(inputs.tools, [&](const json & tool) {
-+            const auto & function = tool.at("function");
-+            std::string  name     = function.at("name");
-+            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
-+            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
-+
-+            std::set<std::string> required;
-+            if (params.contains("required")) {
-+                params.at("required").get_to(required);
-+            }
-+
-+            auto schema_info = common_schema_info();
-+            schema_info.resolve_refs(params);
-+
-+            std::vector<common_peg_parser> required_parsers;
-+            std::vector<common_peg_parser> optional_parsers;
-+            for (const auto & [param_name, param_schema] : props.items()) {
-+                bool is_required = required.find(param_name) != required.end();
-+                bool is_string   = schema_info.resolves_to_string(param_schema);
-+
-+                const std::string p_close = NS + "</" + param_name + ">";
-+
-+                auto arg = p.tool_arg(
-+                    p.tool_arg_open(
-+                        p.literal(NS + "<") +
-+                        p.tool_arg_name(p.literal(param_name)) +
-+                        p.literal(">")) +
-+                    (is_string
-+                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
-+                                p.tool_arg_close(p.literal(p_close)), p_close)
-+                         : p.tool_arg_json_value(p.schema(p.json(),
-+                                                          "tool-" + name + "-arg-" + param_name + "-schema",
-+                                                          param_schema, false)) +
-+                           p.tool_arg_close(p.literal(p_close))));
-+
-+                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
-+                if (is_required) {
-+                    required_parsers.push_back(named_arg);
-+                } else {
-+                    optional_parsers.push_back(named_arg);
-+                }
-+            }
-+
-+            common_peg_parser args_seq = p.eps();
-+            for (size_t i = 0; i < required_parsers.size(); i++) {
-+                if (i > 0) {
-+                    args_seq = args_seq + p.space();
-+                }
-+                args_seq = args_seq + required_parsers[i];
-+            }
-+
-+            if (!optional_parsers.empty()) {
-+                common_peg_parser any_opt = p.choice();
-+                for (const auto & opt : optional_parsers) {
-+                    any_opt |= opt;
-+                }
-+                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
-+            }
-+
-+            common_peg_parser invoke_body = args_seq;
-+            auto func_parser = p.tool(
-+                p.tool_open(p.literal(NS + "<invoke name=\"") +
-+                            p.tool_name(p.literal(name)) + p.literal("\">")) +
-+                p.space() + invoke_body + p.space() +
-+                p.tool_close(p.literal(INVOKE_END)));
-+
-+            tool_choice |= p.rule("tool-" + name, func_parser);
-+        });
-+
-+        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
-+
-+        common_peg_parser tool_calls = p.eps();
-+        if (inputs.parallel_tool_calls) {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice +
-+                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
-+        } else {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
-+        }
-+
-+        if (!require_tools) {
-+            tool_calls = p.optional(tool_calls);
-+        }
-+
-+        auto content_before_tools = p.content(p.until(FC_START));
-+        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
-+    });
-+
-+    data.parser = parser.save();
-+
-+    if (include_grammar) {
-+        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
-+        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
-+            foreach_function(inputs.tools, [&](const json & tool) {
-+                const auto & function = tool.at("function");
-+                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
-+                builder.resolve_refs(schema);
-+            });
-+            if (has_response_format) {
-+                auto schema = inputs.json_schema;
-+                builder.resolve_refs(schema);
-+            }
-+            parser.build_grammar(builder, data.grammar_lazy);
-+        });
-+
-+        data.grammar_triggers = {
-+            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
-+        };
-+    }
-+
-+    return data;
-+}
-+
- // Cohere2 MoE (a.k.a. "North Code") parser.
- //
- // The assistant turn is fully marker-wrapped:
-@@ -2707,6 +2892,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
-         return common_chat_params_init_gigachat_v3(tmpl, params);
-     }
- 
-+    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
-+    // markup delimiters, so detect the template and use a dedicated parser.
-+    if (src.find("]<]minimax[>[") != std::string::npos &&
-+        src.find("<tool_call>") != std::string::npos &&
-+        src.find("<invoke name=") != std::string::npos) {
-+        LOG_DBG("Using specialized template: MiniMax-M3\n");
-+        return common_chat_params_init_minimax_m3(tmpl, params);
-+    }
-+
-     // DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
-     // The template source contains the token as a variable assignment, not as a literal in markup.
-     // V3.2 names the tool call block "function_calls", V4 names it "tool_calls".

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch

@@ -0,0 +1,814 @@
+# Vendored from upstream llama.cpp PR #24523 (Preliminary MiniMax-M3 support).
+# Rebased against LLAMA_VERSION 00fa7cb284cbf133fc426733bd64238a3588a33e (also applies cleanly
+# to the later pin 505b1ed15ca80e2a19f12ff4ac365e40fb374053). LLAMA_VERSION is auto-bumped
+# nightly; if a bump rejects this patch, re-vendor from #24523 — or, once #24523 merges
+# upstream, delete this file and bump LLAMA_VERSION normally.
+# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
+diff --git a/common/chat.cpp b/common/chat.cpp
+index 22d2ee4..440be9a 100644
+--- a/common/chat.cpp
++++ b/common/chat.cpp
+@@ -2035,6 +2035,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
+     return data;
+ }
+ 
++static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
++                                                             const autoparser::generation_params & inputs) {
++    common_chat_params data;
++
++    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
++    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
++    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
++    data.supports_thinking  = true;
++    data.thinking_start_tag = "<mm:think>";
++    data.thinking_end_tag   = "</mm:think>";
++
++    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
++    // params use the parameter name as the tag (<file_path>...</file_path>).
++    const std::string NS           = "]<]minimax[>[";
++    const std::string THINK_START  = "<mm:think>";
++    const std::string THINK_END    = "</mm:think>";
++    const std::string FC_START     = NS + "<tool_call>";
++    const std::string FC_END       = NS + "</tool_call>";
++    const std::string INVOKE_END   = NS + "</invoke>";
++
++    data.preserved_tokens   = {
++        NS,
++        "<tool_call>",
++        "</tool_call>",
++        THINK_START,
++        THINK_END,
++    };
++
++    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
++    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
++    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
++    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
++
++    const std::string GEN_PROMPT = data.generation_prompt;
++
++    if (inputs.has_continuation()) {
++        const auto & msg = inputs.continue_msg;
++
++        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
++        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
++            data.generation_prompt += THINK_END + msg.render_content();
++        }
++
++        data.prompt += data.generation_prompt;
++    }
++
++    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
++        auto generation_prompt = p.literal(GEN_PROMPT);
++        auto end = p.end();
++
++        auto reasoning = p.eps();
++        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
++        if (extract_reasoning && inputs.enable_thinking) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
++        } else if (extract_reasoning) {
++            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
++        }
++
++        if (has_response_format) {
++            auto response_format = p.rule("response-format",
++                p.literal("```json") + p.space() +
++                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
++                p.space() + p.literal("```"));
++            return generation_prompt + reasoning + response_format + end;
++        }
++
++        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
++            return generation_prompt + reasoning + p.content(p.rest()) + end;
++        }
++
++        auto tool_choice = p.choice();
++        foreach_function(inputs.tools, [&](const json & tool) {
++            const auto & function = tool.at("function");
++            std::string  name     = function.at("name");
++            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
++            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
++
++            std::set<std::string> required;
++            if (params.contains("required")) {
++                params.at("required").get_to(required);
++            }
++
++            auto schema_info = common_schema_info();
++            schema_info.resolve_refs(params);
++
++            std::vector<common_peg_parser> required_parsers;
++            std::vector<common_peg_parser> optional_parsers;
++            for (const auto & [param_name, param_schema] : props.items()) {
++                bool is_required = required.find(param_name) != required.end();
++                bool is_string   = schema_info.resolves_to_string(param_schema);
++
++                const std::string p_close = NS + "</" + param_name + ">";
++
++                auto arg = p.tool_arg(
++                    p.tool_arg_open(
++                        p.literal(NS + "<") +
++                        p.tool_arg_name(p.literal(param_name)) +
++                        p.literal(">")) +
++                    (is_string
++                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
++                                p.tool_arg_close(p.literal(p_close)), p_close)
++                         : p.tool_arg_json_value(p.schema(p.json(),
++                                                          "tool-" + name + "-arg-" + param_name + "-schema",
++                                                          param_schema, false)) +
++                           p.tool_arg_close(p.literal(p_close))));
++
++                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
++                if (is_required) {
++                    required_parsers.push_back(named_arg);
++                } else {
++                    optional_parsers.push_back(named_arg);
++                }
++            }
++
++            common_peg_parser args_seq = p.eps();
++            for (size_t i = 0; i < required_parsers.size(); i++) {
++                if (i > 0) {
++                    args_seq = args_seq + p.space();
++                }
++                args_seq = args_seq + required_parsers[i];
++            }
++
++            if (!optional_parsers.empty()) {
++                common_peg_parser any_opt = p.choice();
++                for (const auto & opt : optional_parsers) {
++                    any_opt |= opt;
++                }
++                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
++            }
++
++            common_peg_parser invoke_body = args_seq;
++            auto func_parser = p.tool(
++                p.tool_open(p.literal(NS + "<invoke name=\"") +
++                            p.tool_name(p.literal(name)) + p.literal("\">")) +
++                p.space() + invoke_body + p.space() +
++                p.tool_close(p.literal(INVOKE_END)));
++
++            tool_choice |= p.rule("tool-" + name, func_parser);
++        });
++
++        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
++
++        common_peg_parser tool_calls = p.eps();
++        if (inputs.parallel_tool_calls) {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice +
++                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
++        } else {
++            tool_calls = p.trigger_rule("tool-call",
++                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
++        }
++
++        if (!require_tools) {
++            tool_calls = p.optional(tool_calls);
++        }
++
++        auto content_before_tools = p.content(p.until(FC_START));
++        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
++    });
++
++    data.parser = parser.save();
++
++    if (include_grammar) {
++        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
++        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
++            foreach_function(inputs.tools, [&](const json & tool) {
++                const auto & function = tool.at("function");
++                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
++                builder.resolve_refs(schema);
++            });
++            if (has_response_format) {
++                auto schema = inputs.json_schema;
++                builder.resolve_refs(schema);
++            }
++            parser.build_grammar(builder, data.grammar_lazy);
++        });
++
++        data.grammar_triggers = {
++            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
++        };
++    }
++
++    return data;
++}
++
+ // Cohere2 MoE (a.k.a. "North Code") parser.
+ //
+ // The assistant turn is fully marker-wrapped:
+@@ -2612,6 +2797,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
+         return common_chat_params_init_gigachat_v3(tmpl, params);
+     }
+ 
++    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
++    // markup delimiters, so detect the template and use a dedicated parser.
++    if (src.find("]<]minimax[>[") != std::string::npos &&
++        src.find("<tool_call>") != std::string::npos &&
++        src.find("<invoke name=") != std::string::npos) {
++        LOG_DBG("Using specialized template: MiniMax-M3\n");
++        return common_chat_params_init_minimax_m3(tmpl, params);
++    }
++
+     // DeepSeek V3.2 format detection: template defines dsml_token and uses it for tool calls.
+     // The template source contains the token as a variable assignment, not as a literal in markup.
+     if (src.find("dsml_token") != std::string::npos &&
+diff --git a/conversion/__init__.py b/conversion/__init__.py
+index 02ea638..71de528 100644
+--- a/conversion/__init__.py
++++ b/conversion/__init__.py
+@@ -155,6 +155,8 @@ TEXT_MODEL_MAP: dict[str, str] = {
+     "MiniCPMForCausalLM": "minicpm",
+     "MiniCPMV4_6ForConditionalGeneration": "minicpm",
+     "MiniMaxM2ForCausalLM": "minimax",
++    "MiniMaxM3SparseForCausalLM": "minimax",
++    "MiniMaxM3SparseForConditionalGeneration": "minimax",
+     "Ministral3ForCausalLM": "mistral3",
+     "Mistral3ForConditionalGeneration": "mistral3",
+     "MistralForCausalLM": "llama",
+diff --git a/conversion/base.py b/conversion/base.py
+index 0421aa4..224481a 100644
+--- a/conversion/base.py
++++ b/conversion/base.py
+@@ -1154,7 +1154,8 @@ class TextModel(ModelBase):
+                 or "projector." in name or "pre_mm_projector_norm" in name \
+                 or "image_newline" in name or "view_seperator" in name \
+                 or "patch_embed" in name or "patch_embedding" in name \
+-                or "patch_merger." in name or "model.connector." in name:
++                or "patch_merger." in name or "patch_merge_mlp" in name \
++                or "model.connector." in name:
+             return None
+ 
+         return super().filter_tensors(item)
+@@ -1201,7 +1202,7 @@ class TextModel(ModelBase):
+             self.gguf_writer.add_embedding_length(n_embd)
+             logger.info(f"gguf: embedding length = {n_embd}")
+ 
+-        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
++        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
+             self.gguf_writer.add_feed_forward_length(n_ff)
+             logger.info(f"gguf: feed forward length = {n_ff}")
+ 
+diff --git a/conversion/minimax.py b/conversion/minimax.py
+index 4857775..4f637f5 100644
+--- a/conversion/minimax.py
++++ b/conversion/minimax.py
+@@ -52,3 +52,67 @@ class MiniMaxM2Model(TextModel):
+             return
+ 
+         yield from super().modify_tensors(data_torch, name, bid)
++
++
++@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
++class MiniMaxM3Model(TextModel):
++    # Text-only MiniMax-M3: MiniMax-M2 GQA + DeepSeek-V3 shared/leading-dense experts (swigluoai).
++    model_arch = gguf.MODEL_ARCH.MINIMAXM3
++    _experts_cache: dict[int, dict[str, Tensor]] = {}
++
++    def set_gguf_parameters(self):
++        # feed_forward_length comes from dense_intermediate_size (base); experts use intermediate_size.
++        super().set_gguf_parameters()
++
++        self.gguf_writer.add_expert_feed_forward_length(self.find_hparam(["intermediate_size"]))
++        self.gguf_writer.add_rope_dimension_count(self.find_hparam(["rotary_dim"]))
++        self.gguf_writer.add_expert_shared_count(self.find_hparam(["n_shared_experts"]))
++        self.gguf_writer.add_expert_weights_scale(self.find_hparam(["routed_scaling_factor"]))
++        self.gguf_writer.add_expert_weights_norm(True)
++
++        # leading dense layers: moe_layer_freq (ints) or mlp_layer_types (Transformers 5.12, strings)
++        moe_layer_freq = self.find_hparam(["moe_layer_freq", "mlp_layer_types"])
++        n_dense = 0
++        for v in moe_layer_freq:
++            if v == 0 or v == "dense":
++                n_dense += 1
++            else:
++                break
++        self.gguf_writer.add_leading_dense_block_count(n_dense)
++
++    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
++        # index_* (sparse-attn indexer) tensors are preserved but unused; the loader skips them
++        if name.startswith("language_model."):
++            name = name[len("language_model."):]
++
++        # Gemma-style (1+w) RMSNorm: bake +1 in so llama.cpp can use plain RMSNorm
++        if name.endswith("norm.weight"):
++            data_torch = data_torch + 1.0
++
++        # merge routed experts (w1/w2/w3); shared_experts.* passes through to *_shexp
++        if "block_sparse_moe.experts." in name:
++            n_experts = self.find_hparam(["num_local_experts", "num_experts"])
++            assert bid is not None
++
++            expert_cache = self._experts_cache.setdefault(bid, {})
++            expert_cache[name] = data_torch
++            expert_weights = ["w1", "w2", "w3"]
++
++            if len(expert_cache) < n_experts * len(expert_weights):
++                return
++
++            for w_name in expert_weights:
++                datas: list[Tensor] = []
++                for xid in range(n_experts):
++                    ename = f"model.layers.{bid}.block_sparse_moe.experts.{xid}.{w_name}.weight"
++                    datas.append(expert_cache[ename])
++                    del expert_cache[ename]
++
++                data_torch = torch.stack(datas, dim=0)
++                merged_name = f"model.layers.{bid}.block_sparse_moe.experts.{w_name}.weight"
++                yield from super().modify_tensors(data_torch, merged_name, bid)
++
++            del self._experts_cache[bid]
++            return
++
++        yield from super().modify_tensors(data_torch, name, bid)
+diff --git a/gguf-py/gguf/constants.py b/gguf-py/gguf/constants.py
+index 869e436..760e3dd 100644
+--- a/gguf-py/gguf/constants.py
++++ b/gguf-py/gguf/constants.py
+@@ -525,6 +525,7 @@ class MODEL_ARCH(IntEnum):
+     APERTUS          = auto()
+     COGVLM           = auto()
+     MINIMAXM2        = auto()
++    MINIMAXM3        = auto()
+     RND1             = auto()
+     PANGU_EMBED      = auto()
+     MISTRAL3         = auto()
+@@ -613,6 +614,10 @@ class MODEL_TENSOR(IntEnum):
+     MOE_LATENT_UP        = auto() # nemotron 3 super
+     ATTN_Q_NORM          = auto()
+     ATTN_K_NORM          = auto()
++    ATTN_INDEX_Q         = auto() # minimax-m3 sparse-attn indexer (unused)
++    ATTN_INDEX_K         = auto()
++    ATTN_INDEX_Q_NORM    = auto()
++    ATTN_INDEX_K_NORM    = auto()
+     LAYER_OUT_NORM       = auto()
+     LAYER_OUT_SCALE      = auto()
+     PER_LAYER_TOKEN_EMBD = auto() # gemma3n
+@@ -1105,6 +1110,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
+     MODEL_ARCH.GROVEMOE:         "grovemoe",
+     MODEL_ARCH.APERTUS:          "apertus",
+     MODEL_ARCH.MINIMAXM2:        "minimax-m2",
++    MODEL_ARCH.MINIMAXM3:        "minimax-m3",
+     MODEL_ARCH.COGVLM:           "cogvlm",
+     MODEL_ARCH.RND1:             "rnd1",
+     MODEL_ARCH.PANGU_EMBED:      "pangu-embedded",
+@@ -1163,6 +1169,10 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
+     MODEL_TENSOR.ATTN_GATE:                 "blk.{bid}.attn_gate",
+     MODEL_TENSOR.ATTN_Q_NORM:               "blk.{bid}.attn_q_norm",
+     MODEL_TENSOR.ATTN_K_NORM:               "blk.{bid}.attn_k_norm",
++    MODEL_TENSOR.ATTN_INDEX_Q:              "blk.{bid}.attn_index_q",
++    MODEL_TENSOR.ATTN_INDEX_K:              "blk.{bid}.attn_index_k",
++    MODEL_TENSOR.ATTN_INDEX_Q_NORM:         "blk.{bid}.attn_index_q_norm",
++    MODEL_TENSOR.ATTN_INDEX_K_NORM:         "blk.{bid}.attn_index_k_norm",
+     MODEL_TENSOR.ATTN_OUT_NORM:             "blk.{bid}.attn_output_norm",
+     MODEL_TENSOR.ATTN_POST_NORM:            "blk.{bid}.post_attention_norm",
+     MODEL_TENSOR.FFN_GATE_INP:              "blk.{bid}.ffn_gate_inp",
+@@ -4102,6 +4112,30 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
+         MODEL_TENSOR.FFN_UP_EXP,
+         MODEL_TENSOR.FFN_EXP_PROBS_B,
+     ],
++    MODEL_ARCH.MINIMAXM3: [
++        MODEL_TENSOR.TOKEN_EMBD,
++        MODEL_TENSOR.OUTPUT_NORM,
++        MODEL_TENSOR.OUTPUT,
++        MODEL_TENSOR.ATTN_NORM,
++        MODEL_TENSOR.ATTN_Q,
++        MODEL_TENSOR.ATTN_Q_NORM,
++        MODEL_TENSOR.ATTN_K,
++        MODEL_TENSOR.ATTN_K_NORM,
++        MODEL_TENSOR.ATTN_V,
++        MODEL_TENSOR.ATTN_OUT,
++        MODEL_TENSOR.FFN_NORM,
++        MODEL_TENSOR.FFN_GATE_INP,
++        MODEL_TENSOR.FFN_EXP_PROBS_B,
++        MODEL_TENSOR.FFN_GATE_EXP,
++        MODEL_TENSOR.FFN_DOWN_EXP,
++        MODEL_TENSOR.FFN_UP_EXP,
++        MODEL_TENSOR.FFN_GATE_SHEXP,
++        MODEL_TENSOR.FFN_DOWN_SHEXP,
++        MODEL_TENSOR.FFN_UP_SHEXP,
++        MODEL_TENSOR.FFN_GATE,
++        MODEL_TENSOR.FFN_DOWN,
++        MODEL_TENSOR.FFN_UP,
++    ],
+     MODEL_ARCH.COGVLM: [
+         MODEL_TENSOR.TOKEN_EMBD,
+         MODEL_TENSOR.OUTPUT_NORM,
+@@ -4128,6 +4162,10 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
+         MODEL_TENSOR.ATTN_Q_NORM,
+         MODEL_TENSOR.ATTN_K,
+         MODEL_TENSOR.ATTN_K_NORM,
++        MODEL_TENSOR.ATTN_INDEX_Q,
++        MODEL_TENSOR.ATTN_INDEX_K,
++        MODEL_TENSOR.ATTN_INDEX_Q_NORM,
++        MODEL_TENSOR.ATTN_INDEX_K_NORM,
+         MODEL_TENSOR.ATTN_V,
+         MODEL_TENSOR.ATTN_OUT,
+         MODEL_TENSOR.FFN_NORM,
+diff --git a/gguf-py/gguf/tensor_mapping.py b/gguf-py/gguf/tensor_mapping.py
+index 9efb36f..a62040b 100644
+--- a/gguf-py/gguf/tensor_mapping.py
++++ b/gguf-py/gguf/tensor_mapping.py
+@@ -717,6 +717,22 @@ class TensorNameMap:
+             "model.layers.{bid}.attention.key_layernorm",                     # apertus
+         ),
+ 
++        MODEL_TENSOR.ATTN_INDEX_Q: (
++            "model.layers.{bid}.self_attn.index_q_proj",                      # minimax-m3 (sparse-attn indexer)
++        ),
++
++        MODEL_TENSOR.ATTN_INDEX_K: (
++            "model.layers.{bid}.self_attn.index_k_proj",                      # minimax-m3
++        ),
++
++        MODEL_TENSOR.ATTN_INDEX_Q_NORM: (
++            "model.layers.{bid}.self_attn.index_q_norm",                      # minimax-m3
++        ),
++
++        MODEL_TENSOR.ATTN_INDEX_K_NORM: (
++            "model.layers.{bid}.self_attn.index_k_norm",                      # minimax-m3
++        ),
++
+         MODEL_TENSOR.ROPE_FREQS: (
+             "encoder.layers.{bid}.self_attention.rotary_emb.inv_freq",  # persimmon
+         ),
+diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
+index b890e66..cb8bfc8 100644
+--- a/src/llama-arch.cpp
++++ b/src/llama-arch.cpp
+@@ -125,6 +125,7 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
+     { LLM_ARCH_GROVEMOE,         "grovemoe"         },
+     { LLM_ARCH_APERTUS,          "apertus"          },
+     { LLM_ARCH_MINIMAX_M2,       "minimax-m2"       },
++    { LLM_ARCH_MINIMAX_M3,       "minimax-m3"       },
+     { LLM_ARCH_COGVLM,           "cogvlm"           },
+     { LLM_ARCH_RND1,             "rnd1"             },
+     { LLM_ARCH_PANGU_EMBED,      "pangu-embedded"   },
+@@ -395,6 +396,10 @@ static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
+     { LLM_TENSOR_ATTN_POST_NORM,                         "blk.%d.post_attention_norm" },
+     { LLM_TENSOR_ATTN_Q_NORM,                            "blk.%d.attn_q_norm" },
+     { LLM_TENSOR_ATTN_K_NORM,                            "blk.%d.attn_k_norm" },
++    { LLM_TENSOR_ATTN_INDEX_Q,                           "blk.%d.attn_index_q" },
++    { LLM_TENSOR_ATTN_INDEX_K,                           "blk.%d.attn_index_k" },
++    { LLM_TENSOR_ATTN_INDEX_Q_NORM,                      "blk.%d.attn_index_q_norm" },
++    { LLM_TENSOR_ATTN_INDEX_K_NORM,                      "blk.%d.attn_index_k_norm" },
+     { LLM_TENSOR_ATTN_GATE,                              "blk.%d.attn_gate" },
+     { LLM_TENSOR_FFN_POST_NORM,                          "blk.%d.post_ffw_norm" },
+     { LLM_TENSOR_FFN_POST_NORM_1,                        "blk.%d.post_ffw_norm_1" },
+@@ -761,6 +766,11 @@ static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
+     {LLM_TENSOR_FFN_NORM_EXPS,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
+     {LLM_TENSOR_ATTN_Q_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
+     {LLM_TENSOR_ATTN_K_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
++    // minimax-m3 sparse-attn indexer: unused (GGML_OP_NONE) so the loader skips it
++    {LLM_TENSOR_ATTN_INDEX_Q,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
++    {LLM_TENSOR_ATTN_INDEX_K,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
++    {LLM_TENSOR_ATTN_INDEX_Q_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
++    {LLM_TENSOR_ATTN_INDEX_K_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
+     {LLM_TENSOR_LAYER_OUT_NORM,             {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
+     {LLM_TENSOR_LAYER_OUT_SCALE,            {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
+     {LLM_TENSOR_ATTN_Q_A_NORM,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
+@@ -998,6 +1008,7 @@ bool llm_arch_supports_sm_tensor(const llm_arch & arch) {
+         case LLM_ARCH_LFM2:
+         case LLM_ARCH_LFM2MOE:
+         case LLM_ARCH_MINIMAX_M2:
++        case LLM_ARCH_MINIMAX_M3:
+         case LLM_ARCH_MISTRAL4:
+         case LLM_ARCH_KIMI_LINEAR:
+             return false;
+diff --git a/src/llama-arch.h b/src/llama-arch.h
+index a4f5091..2d50ead 100644
+--- a/src/llama-arch.h
++++ b/src/llama-arch.h
+@@ -144,6 +144,7 @@ enum llm_arch {
+     LLM_ARCH_TALKIE,
+     LLM_ARCH_MELLUM,
+     LLM_ARCH_EAGLE3,
++    LLM_ARCH_MINIMAX_M3,
+     LLM_ARCH_DFLASH,
+     LLM_ARCH_UNKNOWN,
+ };
+@@ -429,6 +430,10 @@ enum llm_tensor {
+     LLM_TENSOR_FFN_LATENT_UP,
+     LLM_TENSOR_ATTN_Q_NORM,
+     LLM_TENSOR_ATTN_K_NORM,
++    LLM_TENSOR_ATTN_INDEX_Q,      // minimax-m3 sparse-attn indexer (unused)
++    LLM_TENSOR_ATTN_INDEX_K,
++    LLM_TENSOR_ATTN_INDEX_Q_NORM,
++    LLM_TENSOR_ATTN_INDEX_K_NORM,
+     LLM_TENSOR_LAYER_OUT_NORM,
+     LLM_TENSOR_LAYER_OUT_SCALE,
+     LLM_TENSOR_POST_ATTN_NORM,
+diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
+index c8ecb0a..4c2c286 100644
+--- a/src/llama-graph.cpp
++++ b/src/llama-graph.cpp
+@@ -1719,6 +1719,16 @@ ggml_tensor * llm_graph_context::build_ffn(
+                 cur = ggml_reglu(ctx0, cur);
+                 cb(cur, "ffn_reglu", il);
+             } break;
++        case LLM_FFN_SWIGLU_OAI:
++            {
++                // clamped SwiGLU: parallel gate path (cur=gate, tmp=up)
++                GGML_ASSERT(gate && type_gate == LLM_FFN_PAR);
++                constexpr float alpha = 1.702f;
++                constexpr float limit = 7.0f;
++                cur = ggml_swiglu_oai(ctx0, cur, tmp, alpha, limit);
++                cb(cur, "ffn_swiglu_oai", il);
++                type_gate = LLM_FFN_SEQ; // gate*up already fused; skip the par multiply
++            } break;
+         default:
+             GGML_ABORT("fatal error");
+     }
+diff --git a/src/llama-graph.h b/src/llama-graph.h
+index c84cb6a..806ce7b 100644
+--- a/src/llama-graph.h
++++ b/src/llama-graph.h
+@@ -54,6 +54,7 @@ enum llm_ffn_op_type : int {
+     LLM_FFN_SWIGLU,
+     LLM_FFN_GEGLU,
+     LLM_FFN_REGLU,
++    LLM_FFN_SWIGLU_OAI,
+     LLM_FFN_SWIGLU_OAI_MOE,
+ };
+ 
+diff --git a/src/llama-model.cpp b/src/llama-model.cpp
+index d874813..7bb71c0 100644
+--- a/src/llama-model.cpp
++++ b/src/llama-model.cpp
+@@ -280,6 +280,8 @@ static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params
+             return new llama_model_apertus(params);
+         case LLM_ARCH_MINIMAX_M2:
+             return new llama_model_minimax_m2(params);
++        case LLM_ARCH_MINIMAX_M3:
++            return new llama_model_minimax_m3(params);
+         case LLM_ARCH_COGVLM:
+             return new llama_model_cogvlm(params);
+         case LLM_ARCH_PANGU_EMBED:
+@@ -807,6 +809,7 @@ const char * llm_type_name(llm_type type) {
+         case LLM_TYPE_310B_A15B:     return "310B.A15B";
+         case LLM_TYPE_355B_A32B:     return "355B.A32B";
+         case LLM_TYPE_397B_A17B:     return "397B.A17B";
++        case LLM_TYPE_428B_A23B:     return "428B.A23B";
+         case LLM_TYPE_685B_A37B:     return "685B.A37B";
+         case LLM_TYPE_744B_A40B:     return "744B.A40B";
+         case LLM_TYPE_E2B:           return "E2B";
+@@ -2532,6 +2535,7 @@ llama_rope_type llama_model_rope_type(const llama_model * model) {
+         case LLM_ARCH_GROVEMOE:
+         case LLM_ARCH_APERTUS:
+         case LLM_ARCH_MINIMAX_M2:
++        case LLM_ARCH_MINIMAX_M3:
+         case LLM_ARCH_COGVLM:
+         case LLM_ARCH_PANGU_EMBED:
+         case LLM_ARCH_AFMOE:
+diff --git a/src/llama-model.h b/src/llama-model.h
+index 45b054c..540e0d2 100644
+--- a/src/llama-model.h
++++ b/src/llama-model.h
+@@ -139,6 +139,7 @@ enum llm_type {
+     LLM_TYPE_310B_A15B, // /MiMo-V2-Flash
+     LLM_TYPE_355B_A32B, // GLM-4.5
+     LLM_TYPE_397B_A17B, // Qwen3.5
++    LLM_TYPE_428B_A23B, // MiniMax M3
+     LLM_TYPE_685B_A37B, // DeepSeek V3.2
+     LLM_TYPE_744B_A40B, // GLM-5
+     LLM_TYPE_E2B,
+diff --git a/src/models/minimax-m3.cpp b/src/models/minimax-m3.cpp
+new file mode 100644
+index 0000000..137852a
+--- /dev/null
++++ b/src/models/minimax-m3.cpp
+@@ -0,0 +1,197 @@
++#include "models.h"
++
++// MiniMax-M3, text-only: MiniMax-M2 GQA (per-head QK-norm, partial rotary) + DeepSeek-V3
++// leading-dense/routed/shared experts (swigluoai). Sparse attn -> dense; vision + MTP dropped.
++
++void llama_model_minimax_m3::load_arch_hparams(llama_model_loader & ml) {
++    ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
++    ml.get_key(LLM_KV_LEADING_DENSE_BLOCK_COUNT,   hparams.n_layer_dense_lead, false);
++    ml.get_key(LLM_KV_EXPERT_FEED_FORWARD_LENGTH,  hparams.n_ff_exp);
++    ml.get_key(LLM_KV_EXPERT_SHARED_COUNT,         hparams.n_expert_shared);
++    ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE,        hparams.expert_weights_scale, false);
++    ml.get_key(LLM_KV_EXPERT_WEIGHTS_NORM,         hparams.expert_weights_norm, false);
++    ml.get_key(LLM_KV_EXPERT_GATING_FUNC,          hparams.expert_gating_func);
++
++    switch (hparams.n_layer()) {
++        case 60: type = LLM_TYPE_428B_A23B; break;
++        default: type = LLM_TYPE_UNKNOWN;
++    }
++}
++
++void llama_model_minimax_m3::load_arch_tensors(llama_model_loader &) {
++    LLAMA_LOAD_LOCALS;
++    const int64_t n_expert_shared = hparams.n_expert_shared;
++    const int64_t n_ff_exp        = hparams.n_ff_exp;
++
++    tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
++
++    output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
++    output      = create_tensor(tn(LLM_TENSOR_OUTPUT,      "weight"), {n_embd, n_vocab}, 0);
++
++    for (int i = 0; i < n_layer; ++i) {
++        auto & layer = layers[i];
++
++        create_tensor_qkv(layer, i, n_embd, n_embd_head_k * n_head, n_embd_gqa, n_embd_gqa, 0);
++        layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), { n_embd_head_k * n_head, n_embd }, 0);
++
++        layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
++        // per-head QK-norm (one head_dim vector)
++        layer.attn_q_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_NORM, "weight", i), {n_embd_head_k}, 0);
++        layer.attn_k_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_NORM, "weight", i), {n_embd_head_k}, 0);
++
++        // sparse-attn indexer (unused): GGML_OP_NONE -> loader skips; NOT_REQUIRED -> older GGUFs still load;
++        // SKIP_IF_VIRTUAL -> no-file loader (test-llama-archs) skips them too
++        const int64_t n_index_head = 4;    // sparse_num_index_heads
++        const int64_t d_index      = 128;  // sparse_index_dim
++        const int idx_flags = TENSOR_NOT_REQUIRED | TENSOR_SKIP_IF_VIRTUAL;
++        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q,      "weight", i), {n_embd, n_index_head * d_index}, idx_flags);
++        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K,      "weight", i), {n_embd, d_index},                idx_flags);
++        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q_NORM, "weight", i), {d_index},                        idx_flags);
++        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K_NORM, "weight", i), {d_index},                        idx_flags);
++
++        layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
++
++        if (i < (int) hparams.n_layer_dense_lead) {
++            layer.ffn_gate = create_tensor(tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd,   n_ff}, 0);
++            layer.ffn_down = create_tensor(tn(LLM_TENSOR_FFN_DOWN, "weight", i), {  n_ff, n_embd}, 0);
++            layer.ffn_up   = create_tensor(tn(LLM_TENSOR_FFN_UP,   "weight", i), {n_embd,   n_ff}, 0);
++        } else {
++            layer.ffn_gate_inp    = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP,    "weight", i), {n_embd, n_expert}, 0);
++            layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias",   i), {n_expert}, 0);
++            layer.ffn_gate_exps   = create_tensor(tn(LLM_TENSOR_FFN_GATE_EXPS,   "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
++            layer.ffn_down_exps   = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS,   "weight", i), {n_ff_exp, n_embd, n_expert}, 0);
++            layer.ffn_up_exps     = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,     "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
++
++            layer.ffn_gate_shexp = create_tensor(tn(LLM_TENSOR_FFN_GATE_SHEXP, "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
++            layer.ffn_down_shexp = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "weight", i), {        n_ff_exp * n_expert_shared, n_embd}, 0);
++            layer.ffn_up_shexp   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
++        }
++    }
++}
++
++std::unique_ptr<llm_graph_context> llama_model_minimax_m3::build_arch_graph(const llm_graph_params & params) const {
++    return std::make_unique<graph>(*this, params);
++}
++
++llama_model_minimax_m3::graph::graph(const llama_model & model, const llm_graph_params & params) : llm_graph_context(params) {
++    const int64_t n_embd_head = hparams.n_embd_head_v();
++
++    GGML_ASSERT(n_embd_head == hparams.n_embd_head_k());
++    // partial rotary: head_dim != n_rot, so don't assert n_embd_head == n_rot
++
++    ggml_tensor * cur;
++    ggml_tensor * inpL;
++
++    inpL = build_inp_embd(model.tok_embd);
++
++    ggml_tensor * inp_pos = build_inp_pos();
++    auto inp_attn = build_attn_inp_kv();
++    ggml_tensor * inp_out_ids = build_inp_out_ids();
++
++    for (int il = 0; il < n_layer; ++il) {
++        ggml_tensor * inpSA = inpL;
++
++        // self-attention
++        {
++            cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);
++            cb(cur, "attn_norm", il);
++
++            auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur,
++                    n_embd_head, n_head, n_head_kv, il);
++
++            // per-head QK RMSNorm (weights include Gemma +1)
++            Qcur = build_norm(Qcur, model.layers[il].attn_q_norm, NULL, LLM_NORM_RMS, il);
++            cb(Qcur, "Qcur_normed", il);
++            Kcur = build_norm(Kcur, model.layers[il].attn_k_norm, NULL, LLM_NORM_RMS, il);
++            cb(Kcur, "Kcur_normed", il);
++
++            Qcur = ggml_rope_ext(
++                ctx0, Qcur, inp_pos, nullptr,
++                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
++                ext_factor, attn_factor, beta_fast, beta_slow
++                );
++            Kcur = ggml_rope_ext(
++                ctx0, Kcur, inp_pos, nullptr,
++                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
++                ext_factor, attn_factor, beta_fast, beta_slow
++                );
++
++            cb(Qcur, "Qcur", il);
++            cb(Kcur, "Kcur", il);
++            cb(Vcur, "Vcur", il);
++
++            cur = build_attn(inp_attn,
++                    model.layers[il].wo, NULL, model.layers[il].wo_s,
++                    Qcur, Kcur, Vcur, nullptr, nullptr, nullptr, 1.0f/sqrtf(float(n_embd_head)), il);
++        }
++
++        if (il == n_layer - 1 && inp_out_ids) {
++            cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);
++            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids);
++        }
++
++        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);
++        cb(ffn_inp, "ffn_inp", il);
++
++        cur = build_norm(ffn_inp, model.layers[il].ffn_norm, NULL, LLM_NORM_RMS, il);
++        cb(cur, "ffn_norm", il);
++
++        if ((uint32_t) il < hparams.n_layer_dense_lead) {
++            // leading dense
++            cur = build_ffn(cur,
++                    model.layers[il].ffn_up,   NULL, NULL,
++                    model.layers[il].ffn_gate, NULL, NULL,
++                    model.layers[il].ffn_down, NULL, NULL,
++                    NULL,
++                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
++            cb(cur, "ffn_out", il);
++        } else {
++            // routed experts
++            ggml_tensor * moe_out = build_moe_ffn(cur,
++                    model.layers[il].ffn_gate_inp,
++                    model.layers[il].ffn_up_exps,
++                    model.layers[il].ffn_gate_exps,
++                    model.layers[il].ffn_down_exps,
++                    model.layers[il].ffn_exp_probs_b,
++                    n_expert, n_expert_used,
++                    LLM_FFN_SWIGLU_OAI_MOE, hparams.expert_weights_norm,
++                    hparams.expert_weights_scale,
++                    (llama_expert_gating_func_type) hparams.expert_gating_func,
++                    il);
++            cb(moe_out, "ffn_moe_out", il);
++
++            // shared expert
++            ggml_tensor * ffn_shexp = build_ffn(cur,
++                    model.layers[il].ffn_up_shexp,   NULL, NULL,
++                    model.layers[il].ffn_gate_shexp, NULL, NULL,
++                    model.layers[il].ffn_down_shexp, NULL, NULL,
++                    NULL,
++                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
++            cb(ffn_shexp, "ffn_shexp", il);
++
++            cur = ggml_add(ctx0, moe_out, ffn_shexp);
++            cb(cur, "ffn_out", il);
++        }
++
++        cur = ggml_add(ctx0, cur, ffn_inp);
++
++        cur = build_cvec(cur, il);
++        cb(cur, "l_out", il);
++
++        // input for next layer
++        inpL = cur;
++    }
++
++    cur = inpL;
++
++    cur = build_norm(cur, model.output_norm, NULL, LLM_NORM_RMS, -1);
++    cb(cur, "result_norm", -1);
++    res->t_embd = cur;
++
++    // lm_head
++    cur = build_lora_mm(model.output, cur, model.output_s);
++    cb(cur, "result_output", -1);
++    res->t_logits = cur;
++
++    ggml_build_forward_expand(gf, cur);
++}
+diff --git a/src/models/models.h b/src/models/models.h
+index 7a52e7b..5e2a826 100644
+--- a/src/models/models.h
++++ b/src/models/models.h
+@@ -1870,6 +1870,17 @@ struct llama_model_minimax_m2 : public llama_model_base {
+     std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
+ };
+ 
++struct llama_model_minimax_m3 : public llama_model_base {
++    llama_model_minimax_m3(const struct llama_model_params & params) : llama_model_base(params) {}
++    void load_arch_hparams(llama_model_loader & ml) override;
++    void load_arch_tensors(llama_model_loader & ml) override;
++
++    struct graph : public llm_graph_context {
++        graph(const llama_model & model, const llm_graph_params & params);
++    };
++
++    std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
++};
+ 
+ struct llama_model_cogvlm : public llama_model_base {
+     llama_model_cogvlm(const struct llama_model_params & params) : llama_model_base(params) {}
+diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp
+index f39abe7..2085f43 100644
+--- a/tests/test-llama-archs.cpp
++++ b/tests/test-llama-archs.cpp
+@@ -352,6 +352,7 @@ static bool moe_mandatory(const llm_arch arch) {
+         case LLM_ARCH_LLADA_MOE:
+         case LLM_ARCH_GROVEMOE:
+         case LLM_ARCH_MINIMAX_M2:
++        case LLM_ARCH_MINIMAX_M3:
+         case LLM_ARCH_RND1:
+         case LLM_ARCH_PADDLEOCR:
+         case LLM_ARCH_MIMO2:

backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch

@@ -1,599 +0,0 @@
-diff --git a/common/common.cpp b/common/common.cpp
-index 8f13217..fc584e1 100644
---- a/common/common.cpp
-+++ b/common/common.cpp
-@@ -1591,8 +1591,10 @@ struct llama_context_params common_context_params_to_llama(const common_params &
-     auto cparams = llama_context_default_params();
-
-     cparams.n_ctx             = params.n_ctx;
--    cparams.n_seq_max         = params.n_parallel;
--    cparams.n_rs_seq          = params.speculative.need_n_rs_seq();
-+    // score-task forks need seq ids (and recurrent-state cells) of their
-+    // own beyond the parallel slots
-+    cparams.n_seq_max         = params.n_parallel + params.n_seq_score_forks;
-+    cparams.n_rs_seq          = std::max(params.speculative.need_n_rs_seq(), (uint32_t) std::max(0, params.n_rs_seq));
-     cparams.n_outputs_max     = std::max(params.n_outputs_max, 0);
-     cparams.n_batch           = params.n_batch;
-     cparams.n_ubatch          = params.n_ubatch;
-diff --git a/common/common.h b/common/common.h
-index bffc176..e313bd6 100644
---- a/common/common.h
-+++ b/common/common.h
-@@ -455,6 +455,9 @@ struct common_params {
-     int32_t n_keep                =     0; // number of tokens to keep from initial prompt
-     int32_t n_chunks              =    -1; // max number of chunks to process (-1 = unlimited)
-     int32_t n_parallel            =     1; // number of parallel sequences to decode
-+    int32_t n_seq_score_forks     =     0; // extra seq ids beyond n_parallel, reserved for server score-task forks
-+    int32_t n_rs_seq              =     0; // recurrent-state rollback snapshots per seq (hybrid models cannot rewind without them; lets score tasks reuse a cached prompt across probe changes)
-+    bool    score_enabled          = false; // reserve server resources for the Score task type
-     int32_t n_sequences           =     1; // number of sequences to decode
-     int32_t n_outputs_max         =     0; // max outputs in a batch (0 = n_batch)
-     int32_t grp_attn_n            =     1; // group-attention factor
-diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt
-index 780df32..1d2fe8f 100644
---- a/tools/CMakeLists.txt
-+++ b/tools/CMakeLists.txt
-@@ -41,3 +41,4 @@ else()
-     add_subdirectory(fit-params)
-     add_subdirectory(results)
- endif()
-+add_subdirectory(grpc-server)
-diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
-index 715477e..de5bed8 100644
---- a/tools/server/server-context.cpp
-+++ b/tools/server/server-context.cpp
-@@ -49,7 +49,16 @@ static uint32_t server_n_outputs_max(const common_params & params) {
-
-     const uint32_t n_outputs_per_seq = 1 + common_speculative_n_max(&params.speculative);
-
--    const uint64_t n_outputs = (uint64_t) params.n_parallel * n_outputs_per_seq;
-+    // score tasks (SERVER_TASK_TYPE_SCORE) output logits for every candidate
-+    // token, so reserve room for a bounded candidate tail per parallel slot
-+    if (!params.score_enabled) {
-+        return std::max<uint32_t>(1, std::min<uint64_t>(n_batch,
-+            (uint64_t) params.n_parallel * n_outputs_per_seq));
-+    }
-+
-+    const uint32_t n_outputs_score_seq = 1 + SERVER_SCORE_MAX_CAND_TOKENS;
-+
-+    const uint64_t n_outputs = (uint64_t) params.n_parallel * std::max(n_outputs_per_seq, n_outputs_score_seq);
-
-     return std::max<uint32_t>(1, std::min<uint64_t>(n_batch, n_outputs));
- }
-@@ -202,6 +211,26 @@ struct server_slot {
-
-     std::vector<completion_token_output> generated_token_probs;
-
-+    // SERVER_TASK_TYPE_SCORE: shared-prefix token logprobs harvested
-+    // incrementally across batch views (NaN = not yet produced)
-+    std::vector<float> score_logprobs;
-+
-+    // SERVER_TASK_TYPE_SCORE: per-candidate suffix token logprobs; entry
-+    // [c][0] comes from the last shared token's logits during prompt
-+    // processing, the rest from the forked suffix decode
-+    std::vector<std::vector<float>> score_cand_logprobs;
-+
-+    // SERVER_TASK_TYPE_SCORE: the prompt completed but some candidate has
-+    // suffix tokens beyond the first, so a forked decode is still needed
-+    bool score_suffix_pending = false;
-+
-+    // SERVER_TASK_TYPE_SCORE: where the current task's tokens diverged from
-+    // the slot's previous cache. When the memory cannot rewind there and a
-+    // re-prefill follows, a checkpoint at this position lets the next
-+    // scoring call over the same stable prefix (e.g. a classifier's option
-+    // list) resume from it instead of re-processing the whole prompt.
-+    int32_t score_divergence = -1;
-+
-     bool has_next_token = true;
-     bool has_new_line   = false;
-     bool truncated      = false;
-@@ -311,6 +340,10 @@ struct server_slot {
-         }
-         generated_tokens.clear();
-         generated_token_probs.clear();
-+        score_logprobs.clear();
-+        score_cand_logprobs.clear();
-+        score_suffix_pending = false;
-+        score_divergence = -1;
-         json_schema = json();
-
-         // clear speculative decoding stats
-@@ -2205,6 +2238,229 @@ private:
-         queue_results.send(std::move(res));
-     }
-
-+    // log(sum(exp(logits))) with max-subtraction for stability — the
-+    // log_softmax denominator shared by every token read from one output
-+    static double score_log_denom(const float * logits, int32_t n_vocab) {
-+        float max_logit = logits[0];
-+        for (int32_t v = 1; v < n_vocab; ++v) {
-+            max_logit = std::max(max_logit, logits[v]);
-+        }
-+        double sum_exp = 0.0;
-+        for (int32_t v = 0; v < n_vocab; ++v) {
-+            sum_exp += std::exp((double)(logits[v] - max_logit));
-+        }
-+        return (double) max_logit + std::log(sum_exp);
-+    }
-+
-+    // Harvest logprobs for SCORE tasks from the current batch view: the
-+    // shared-prefix scored tokens, and — from the last shared token's
-+    // logits — the first suffix token of every candidate. The scored
-+    // region can straddle ubatch boundaries for long prompts, so this
-+    // accumulates view by view instead of reading everything when the
-+    // prompt completes.
-+    void collect_score_logprobs(server_slot & slot, const llama_batch & batch) {
-+        const int32_t n_prompt = slot.task->n_score_prompt;
-+        const int32_t n_total  = slot.task->n_tokens();
-+        const auto &  suffixes = slot.task->score_suffixes;
-+
-+        const size_t n_shared_scored = (size_t) std::max(0, n_total - n_prompt);
-+
-+        if (slot.score_logprobs.size() != n_shared_scored) {
-+            slot.score_logprobs.assign(n_shared_scored, NAN);
-+        }
-+        if (slot.score_cand_logprobs.size() != suffixes.size()) {
-+            slot.score_cand_logprobs.resize(suffixes.size());
-+            for (size_t c = 0; c < suffixes.size(); ++c) {
-+                slot.score_cand_logprobs[c].assign(suffixes[c].size(), NAN);
-+            }
-+        }
-+
-+        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
-+
-+        for (int32_t i = 0; i < batch.n_tokens; ++i) {
-+            if (!batch.logits[i] || batch.seq_id[i][0] != slot.id) {
-+                continue;
-+            }
-+
-+            // the output at position p predicts the task token at index p + 1;
-+            // score tasks are text-only, so positions equal token indices
-+            const int32_t target = batch.pos[i] + 1;
-+            if (target < n_prompt || target > n_total) {
-+                continue;
-+            }
-+
-+            const float * logits = llama_get_logits_ith(slot.ctx_tgt, i);
-+            if (logits == nullptr) {
-+                SLT_ERR(slot, "failed to get logits for score target %d\n", target);
-+                continue;
-+            }
-+
-+            const double log_denom = score_log_denom(logits, n_vocab);
-+
-+            if (target < n_total) {
-+                const llama_token tok = slot.task->tokens[target];
-+                slot.score_logprobs[target - n_prompt] = (float) ((double) logits[tok] - log_denom);
-+            } else {
-+                // the last shared token predicts the first suffix token of
-+                // every candidate
-+                for (size_t c = 0; c < suffixes.size(); ++c) {
-+                    if (!suffixes[c].empty()) {
-+                        slot.score_cand_logprobs[c][0] = (float) ((double) logits[suffixes[c][0]] - log_denom);
-+                    }
-+                }
-+            }
-+        }
-+    }
-+
-+    void send_score(server_slot & slot) {
-+        auto res = std::make_unique<server_task_result_score>();
-+        res->id              = slot.task->id;
-+        res->index           = slot.task->index;
-+        res->shared_logprobs = std::move(slot.score_logprobs);
-+        res->cand_logprobs   = std::move(slot.score_cand_logprobs);
-+
-+        slot.score_logprobs.clear();
-+        slot.score_cand_logprobs.clear();
-+
-+        SLT_DBG(slot, "sending score result, n_shared = %zu, n_cand = %zu\n",
-+                res->shared_logprobs.size(), res->cand_logprobs.size());
-+
-+        queue_results.send(std::move(res));
-+    }
-+
-+    // Decode the candidate suffixes of a completed score prompt: fork one
-+    // sequence per candidate off the slot's shared prefix (metadata-only
-+    // for the unified KV cache, copy-on-write for recurrent state) and
-+    // decode all unique suffix tokens in as few llama_decode calls as the
-+    // fork/batch/output budgets allow, harvesting a logprob for every
-+    // suffix token that predicts a following one.
-+    bool decode_score_suffixes(server_slot & slot) {
-+        const auto & suffixes = slot.task->score_suffixes;
-+
-+        auto * mem = llama_get_memory(ctx_tgt);
-+
-+        // seq ids beyond the slots are reserved for score forks at context
-+        // creation (common_params::n_seq_score_forks)
-+        const int32_t seq_base    = (int32_t) slots.size();
-+        const int32_t n_forks_max = std::min<int32_t>(SERVER_SCORE_FORK_SEQS, (int32_t) llama_n_seq_max(ctx_tgt) - seq_base);
-+
-+        if (n_forks_max < 1) {
-+            SLT_ERR(slot, "no fork sequences reserved for score suffixes (n_seq_max = %d, n_slots = %d)\n",
-+                    (int32_t) llama_n_seq_max(ctx_tgt), seq_base);
-+            return false;
-+        }
-+
-+        const int32_t   n_batch_max = llama_n_batch(ctx_tgt);
-+        const int32_t   n_vocab     = llama_vocab_n_tokens(vocab);
-+        const llama_pos pos0        = slot.prompt.tokens.pos_next();
-+
-+        std::vector<size_t> pending;
-+        for (size_t c = 0; c < suffixes.size(); ++c) {
-+            // single-token suffixes were fully scored from the last shared
-+            // token's logits during prompt processing
-+            if (suffixes[c].size() > 1) {
-+                if ((int32_t) suffixes[c].size() > n_batch_max) {
-+                    SLT_ERR(slot, "score suffix of candidate %zu (%zu tokens) exceeds n_batch (%d)\n",
-+                            c, suffixes[c].size(), n_batch_max);
-+                    return false;
-+                }
-+                pending.push_back(c);
-+            }
-+        }
-+
-+        size_t next = 0;
-+        while (next < pending.size()) {
-+            // greedy-pack candidates into one decode within the fork,
-+            // batch and reserved-output budgets
-+            std::vector<size_t> chunk;
-+            int32_t n_tok = 0;
-+            int32_t n_out = 0;
-+            while (next < pending.size() && (int32_t) chunk.size() < n_forks_max) {
-+                const int32_t m = (int32_t) suffixes[pending[next]].size();
-+                if (!chunk.empty() && (n_tok + m > n_batch_max || n_out + m - 1 > SERVER_SCORE_MAX_CAND_TOKENS)) {
-+                    break;
-+                }
-+                chunk.push_back(pending[next]);
-+                n_tok += m;
-+                n_out += m - 1;
-+                next++;
-+            }
-+
-+            llama_batch fb = llama_batch_init(n_tok, 0, 1);
-+
-+            for (size_t k = 0; k < chunk.size(); ++k) {
-+                const llama_seq_id seq = seq_base + (llama_seq_id) k;
-+                const auto & sfx = suffixes[chunk[k]];
-+
-+                llama_memory_seq_rm(mem, seq, -1, -1);
-+                llama_memory_seq_cp(mem, slot.id, seq, -1, -1);
-+
-+                for (size_t j = 0; j < sfx.size(); ++j) {
-+                    common_batch_add(fb, sfx[j], pos0 + (llama_pos) j, { seq }, j + 1 < sfx.size());
-+                }
-+            }
-+
-+            const int ret = llama_decode(ctx_tgt, fb);
-+
-+            if (ret == 0) {
-+                int32_t i = 0;
-+                for (size_t k = 0; k < chunk.size(); ++k) {
-+                    const auto & sfx = suffixes[chunk[k]];
-+                    auto & out = slot.score_cand_logprobs[chunk[k]];
-+
-+                    for (size_t j = 0; j < sfx.size(); ++j, ++i) {
-+                        if (j + 1 >= sfx.size()) {
-+                            continue; // last suffix token predicts nothing
-+                        }
-+                        const float * logits = llama_get_logits_ith(ctx_tgt, i);
-+                        if (logits == nullptr) {
-+                            SLT_ERR(slot, "failed to get logits for suffix token %zu of score candidate %zu\n", j, chunk[k]);
-+                            continue;
-+                        }
-+                        const double log_denom = score_log_denom(logits, n_vocab);
-+                        out[j + 1] = (float) ((double) logits[sfx[j + 1]] - log_denom);
-+                    }
-+                }
-+            }
-+
-+            for (size_t k = 0; k < chunk.size(); ++k) {
-+                llama_memory_seq_rm(mem, seq_base + (llama_seq_id) k, -1, -1);
-+            }
-+
-+            llama_batch_free(fb);
-+
-+            if (ret != 0) {
-+                SLT_ERR(slot, "score suffix decode failed, ret = %d\n", ret);
-+                return false;
-+            }
-+        }
-+
-+        return true;
-+    }
-+
-+    // score slots whose prompt completed this iteration decode their
-+    // candidate suffixes here, after every batch view was consumed — a
-+    // mid-view llama_decode would clobber logits other slots still read
-+    void update_score_suffixes() {
-+        for (auto & slot : slots) {
-+            if (!slot.score_suffix_pending) {
-+                continue;
-+            }
-+            slot.score_suffix_pending = false;
-+
-+            if (!slot.is_processing() || !slot.task || slot.task->type != SERVER_TASK_TYPE_SCORE) {
-+                continue; // the task was aborted mid-iteration
-+            }
-+
-+            if (decode_score_suffixes(slot)) {
-+                send_score(slot);
-+            } else {
-+                send_error(slot, "failed to decode score candidate suffixes", ERROR_TYPE_SERVER);
-+            }
-+            slot.release();
-+        }
-+    }
-+
-     //
-     // Functions to process the task
-     //
-@@ -2341,6 +2597,7 @@ private:
-             case SERVER_TASK_TYPE_INFILL:
-             case SERVER_TASK_TYPE_EMBEDDING:
-             case SERVER_TASK_TYPE_RERANK:
-+            case SERVER_TASK_TYPE_SCORE:
-                 {
-                     // special case: if input is provided via CLI, tokenize it first
-                     // otherwise, no need to tokenize as it's already done inside the HTTP thread
-@@ -2832,6 +3089,13 @@ private:
-                 break; // stop any further processing
-             }
-         }
-+
-+        try {
-+            update_score_suffixes();
-+        } catch (const std::exception & e) {
-+            SRV_ERR("update_score_suffixes() failed: %s\n", e.what());
-+            abort_all_slots("update_score_suffixes() failed: " + std::string(e.what()));
-+        }
-     }
-
-     void pre_decode() {
-@@ -3154,6 +3418,16 @@ private:
-                                     n_past = std::min(n_past, slot.alora_invocation_start - 1);
-                                 }
-
-+                                // score tasks need the logits that predict the first candidate
-+                                // token, so the last shared-prompt token must be (re-)decoded
-+                                // even when the cache already covers it
-+                                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
-+                                    n_past = std::min(n_past, std::max(0, slot.task->n_score_prompt - 1));
-+                                    // remember the divergence point before the checkpoint
-+                                    // logic below possibly resets n_past to 0
-+                                    slot.score_divergence = n_past;
-+                                }
-+
-                                 const auto n_cache_reuse = slot.task->params.n_cache_reuse;
-
-                                 const bool can_cache_reuse =
-@@ -3395,8 +3669,12 @@ private:
-
-                     bool do_checkpoint = params_base.n_ctx_checkpoints > 0;
-
--                    // make checkpoints only for completion tasks
--                    do_checkpoint = do_checkpoint && slot.task->type == SERVER_TASK_TYPE_COMPLETION;
-+                    // make checkpoints for completion tasks, and for score tasks at the
-+                    // shared-prompt boundary: models whose memory cannot be partially
-+                    // rewound (SWA/hybrid/recurrent) would otherwise re-process the whole
-+                    // prompt for every candidate of a scoring call
-+                    do_checkpoint = do_checkpoint && (slot.task->type == SERVER_TASK_TYPE_COMPLETION ||
-+                                                      slot.task->type == SERVER_TASK_TYPE_SCORE);
-
-                     // make a checkpoint of the parts of the memory that cannot be rolled back.
-                     // checkpoints are created only if:
-@@ -3463,10 +3741,17 @@ private:
-                         // embedding requires all tokens in the batch to be output;
-                         // MTP also wants logits at every prompt position so the
-                         // streaming hook can mirror t_h_nextn into ctx_dft.
-+                        // score tasks need outputs at the positions that predict
-+                        // each candidate token (the token at index i predicts the
-+                        // task token at index i+1).
-+                        const bool need_score_logit =
-+                            slot.task->type == SERVER_TASK_TYPE_SCORE &&
-+                            slot.prompt.n_tokens() + 1 >= slot.task->n_score_prompt &&
-+                            slot.prompt.n_tokens() + 1 <  slot.task->n_tokens();
-                         add_ok &= batch.add(slot.id,
-                             cur_tok,
-                             slot.prompt.tokens.pos_next(),
--                            slot.need_embd());
-+                            slot.need_embd() || need_score_logit);
-                         slot.prompt.tokens.push_back(cur_tok);
-
-                         slot.n_prompt_tokens_processed++;
-@@ -3481,6 +3766,32 @@ private:
-                             }
-                         }
-
-+                        // score tasks: break at the shared-prompt boundary so the checkpoint
-+                        // below lands exactly there — the other candidates of the same
-+                        // scoring call re-process only their own tokens. Also break at the
-+                        // point where this task diverged from the previous cache: after a
-+                        // forced re-prefill a checkpoint there serves the next scoring call
-+                        // over the same stable prefix (e.g. a classifier's option list).
-+                        // The caller-declared stable-prefix boundary is the strongest of
-+                        // these: a checkpoint there is at or before every future task's
-+                        // divergence within the same option list, so it always survives
-+                        // and always restores.
-+                        if (do_checkpoint && slot.task->type == SERVER_TASK_TYPE_SCORE &&
-+                            (slot.prompt.n_tokens() == slot.task->n_score_prompt - 1 ||
-+                             (slot.task->n_stable_prompt > 0 &&
-+                              slot.prompt.n_tokens() == slot.task->n_stable_prompt &&
-+                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1) ||
-+                             (slot.prompt.n_tokens() == slot.score_divergence &&
-+                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1))) {
-+                            bool have_ckpt = false;
-+                            for (const auto & ckpt : slot.prompt.checkpoints) {
-+                                have_ckpt |= ckpt.n_tokens == slot.prompt.n_tokens();
-+                            }
-+                            if (!have_ckpt) {
-+                                break;
-+                            }
-+                        }
-+
-                         // process the last few tokens of the prompt separately in order to allow for a checkpoint to be created.
-                         // create checkpoints that many tokens before the end of the prompt:
-                         //  - 4 + n_ubatch
-@@ -3513,6 +3824,15 @@ private:
-                     const bool is_user_start = spans.is_user_start(n_tokens_start);
-                     const bool is_last_user_message = n_tokens_start == last_user_pos;
-
-+                    // a batch starting at the score boundary or divergence point must
-+                    // always checkpoint — min-step spacing would otherwise suppress it
-+                    // and every candidate / next scoring call would re-process the prompt
-+                    const bool is_score_boundary = slot.task->type == SERVER_TASK_TYPE_SCORE &&
-+                                                   (n_tokens_start == slot.task->n_score_prompt - 1 ||
-+                                                    (slot.task->n_stable_prompt > 0 &&
-+                                                     n_tokens_start == slot.task->n_stable_prompt) ||
-+                                                    n_tokens_start == slot.score_divergence);
-+
-                     // entire prompt has been processed
-                     if (slot.prompt.n_tokens() == slot.task->n_tokens()) {
-                         slot.state = SLOT_STATE_DONE_PROMPT;
-@@ -3528,8 +3848,8 @@ private:
-                         slot.init_sampler();
-                     } else {
-                         // skip ordinary mid-prompt checkpoints, unless the batch starts a user
--                        // message or we are near the end of the prompt
--                        if (!is_user_start && !near_prompt_end) {
-+                        // message, the score boundary, or we are near the end of the prompt
-+                        if (!is_user_start && !is_score_boundary && !near_prompt_end) {
-                             do_checkpoint = false;
-                         }
-                     }
-@@ -3546,10 +3866,10 @@ private:
-                     // do not checkpoint after mtmd chunks
-                     do_checkpoint = do_checkpoint && !has_mtmd;
-
--                    // no need to create checkpoints that are too close together, unless it's the last user message
-+                    // no need to create checkpoints that are too close together, unless it's the last user message or the score boundary
-                     do_checkpoint = do_checkpoint && (
-                             slot.prompt.checkpoints.empty() ||
--                            is_last_user_message || near_prompt_end ||
-+                            is_last_user_message || near_prompt_end || is_score_boundary ||
-                             n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
-                     SLT_DBG(slot, "main/do_checkpoint = %s, pos_min = %d, pos_max = %d\n", do_checkpoint ? "yes" : "no", pos_min, pos_max);
-
-@@ -3703,6 +4023,13 @@ private:
-                 }
-             }
-
-+            // score slots harvest logprobs from every view that contains
-+            // their outputs, not just the one holding the final token
-+            if (slot.task && slot.task->type == SERVER_TASK_TYPE_SCORE &&
-+                (slot.state == SLOT_STATE_PROCESSING_PROMPT || slot.state == SLOT_STATE_DONE_PROMPT)) {
-+                collect_score_logprobs(slot, batch_view);
-+            }
-+
-             if (!is_inside_view(slot.i_batch)) {
-                 // the required token not in this sub-batch, skip
-                 return;
-@@ -3724,6 +4051,25 @@ private:
-                     return;
-                 }
-
-+                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
-+                    // shared-prefix logprobs (and every candidate's first
-+                    // suffix logprob) were accumulated per view above;
-+                    // candidates with more suffix tokens still need the
-+                    // forked decode at the end of update_slots()
-+                    for (const auto & sfx : slot.task->score_suffixes) {
-+                        if (sfx.size() > 1) {
-+                            slot.score_suffix_pending = true;
-+                            break;
-+                        }
-+                    }
-+                    if (!slot.score_suffix_pending) {
-+                        send_score(slot);
-+                        slot.release();
-+                    }
-+                    slot.i_batch = -1;
-+                    return;
-+                }
-+
-                 GGML_ASSERT(slot.task->need_sampling());
-
-                 // prompt evaluated for next-token prediction
-diff --git a/tools/server/server-task.h b/tools/server/server-task.h
-index c3eea2e..fb3c178 100644
---- a/tools/server/server-task.h
-+++ b/tools/server/server-task.h
-@@ -13,10 +13,25 @@
-
- using json = nlohmann::ordered_json;
-
-+// SERVER_TASK_TYPE_SCORE emits one logits output per candidate token (plus
-+// the forced last-token output), and the context's output budget
-+// (n_outputs_max) is reserved up front — so candidate length must be
-+// bounded. Raising this raises the worst-case compute-buffer reservation
-+// by ~n_vocab * 4 bytes per extra output.
-+constexpr int32_t SERVER_SCORE_MAX_CAND_TOKENS = 64;
-+
-+// Maximum sequences forked off the shared prefix in one score suffix
-+// decode. The context is created with this many seq ids (and
-+// recurrent-state cells) beyond the parallel slots — see
-+// common_params::n_seq_score_forks; candidates in excess of the budget
-+// are decoded in successive chunks.
-+constexpr int32_t SERVER_SCORE_FORK_SEQS = 16;
-+
- enum server_task_type {
-     SERVER_TASK_TYPE_COMPLETION,
-     SERVER_TASK_TYPE_EMBEDDING,
-     SERVER_TASK_TYPE_RERANK,
-+    SERVER_TASK_TYPE_SCORE,
-     SERVER_TASK_TYPE_INFILL,
-     SERVER_TASK_TYPE_CANCEL,
-     SERVER_TASK_TYPE_CONTROL,
-@@ -153,6 +168,18 @@ struct server_task {
-     task_params   params;
-     server_tokens tokens;
-
-+    // used by SERVER_TASK_TYPE_SCORE: `tokens` holds the shared prefix
-+    // (prompt + longest common candidate token prefix) and logprobs are
-+    // returned for its tokens from n_score_prompt onward. Each candidate's
-+    // tokens beyond the shared prefix ride a forked sequence.
-+    int32_t n_score_prompt = 0;
-+    std::vector<llama_tokens> score_suffixes;
-+    // token index where the caller-declared stable prompt prefix ends
-+    // (0 = no hint): the option-list system prompt that repeats across
-+    // scoring calls. A context checkpoint is forced there so models that
-+    // cannot rewind state re-process only the per-call tail next time.
-+    int32_t n_stable_prompt = 0;
-+
-     // only used by CLI, this allow tokenizing CLI inputs on server side
-     // we need this because mtmd_context and vocab are not accessible outside of server_context
-     bool                    cli = false;
-@@ -197,6 +224,7 @@ struct server_task {
-         switch (type) {
-             case SERVER_TASK_TYPE_COMPLETION:
-             case SERVER_TASK_TYPE_INFILL:
-+            case SERVER_TASK_TYPE_SCORE:
-                 return true;
-             default:
-                 return false;
-@@ -494,6 +522,25 @@ struct server_task_result_rerank : server_task_result {
-     virtual json to_json() override;
- };
-
-+struct server_task_result_score : server_task_result {
-+    // log P(token | prefix) for the shared-prefix tokens after
-+    // n_score_prompt, in order; NaN marks positions the decode never
-+    // produced an output for
-+    std::vector<float> shared_logprobs;
-+
-+    // per candidate: logprobs of its suffix tokens, in task order (entry
-+    // 0 is the token right after the shared prefix, predicted by the last
-+    // shared token's logits)
-+    std::vector<std::vector<float>> cand_logprobs;
-+
-+    virtual json to_json() override {
-+        return json {
-+            {"shared_logprobs", shared_logprobs},
-+            {"cand_logprobs",   cand_logprobs},
-+        };
-+    }
-+};
-+
- struct server_task_result_error : server_task_result {
-     error_type err_type = ERROR_TYPE_SERVER;
-     std::string err_msg;

backend/cpp/llama-cpp/prepare.sh

@@ -31,26 +31,6 @@ cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 
-## Fork-skew probe. Upstream folded common_params::use_mmap / use_mlock /
-## use_direct_io into a single `load_mode` enum (ggml-org/llama.cpp#20834).
-## turboquant and bonsai compile this very same grpc-server.cpp against forks
-## that branched before that change, so the field set is decided from the
-## checkout in front of us rather than from a per-fork build flag: the flavor
-## targets disagree on whether they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and
-## probing heals itself the moment a fork rebases past the refactor.
-if grep -q "LLAMA_LOAD_MODE_MMAP" llama.cpp/include/llama.h; then
-    echo "==> llama.cpp carries the load-mode enum, using common_params::load_mode"
-    LEGACY_LOAD_MODE=0
-else
-    echo "==> llama.cpp predates the load-mode enum, using the legacy mmap/mlock/direct-io booleans"
-    LEGACY_LOAD_MODE=1
-fi
-cat > llama.cpp/tools/grpc-server/llama_compat.h <<EOF
-// Generated by backend/cpp/llama-cpp/prepare.sh. Do not edit.
-#pragma once
-#define LOCALAI_LEGACY_LOAD_MODE ${LEGACY_LOAD_MODE}
-EOF
-
 set +e
 if grep -q "grpc-server" llama.cpp/tools/CMakeLists.txt; then
     echo "grpc-server already added"

backend/cpp/turboquant/Makefile

@@ -47,7 +47,6 @@ define turboquant-build
 	# original under backend/cpp/llama-cpp/, so the stock llama-cpp build
 	# stays compiling against vanilla upstream.
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
-	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build llama.cpp
@@ -85,7 +84,6 @@ turboquant-cpu-all:
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build purge
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
-	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build llama.cpp

backend/go/cloud-proxy/provider_anthropic.go

@@ -32,9 +32,7 @@ import (
 type anthropicRequest struct {
 	Model         string               `json:"model"`
 	MaxTokens     int32                `json:"max_tokens"`
-	// System is `any`: a bare string normally, or []anthropicSystemBlock
-	// when cache_prompt is on (the block form carries cache_control).
-	System        any                  `json:"system,omitempty"`
+	System        string               `json:"system,omitempty"`
 	Messages      []anthropicMessage   `json:"messages"`
 	Stream        bool                 `json:"stream,omitempty"`
 	Temperature   *float64             `json:"temperature,omitempty"`
@@ -54,30 +52,9 @@ type anthropicMessage struct {
 }
 
 type anthropicTool struct {
-	Name         string                 `json:"name"`
-	Description  string                 `json:"description,omitempty"`
-	InputSchema  json.RawMessage        `json:"input_schema"`
-	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
-}
-
-// anthropicCacheControl marks a prompt-cache breakpoint. Anthropic caches
-// everything up to and including a block tagged {"type":"ephemeral"} (5-min
-// TTL) and serves that prefix at the cache-read rate (0.1x input) on later
-// calls that share it — the win on agentic/multi-turn workloads.
-type anthropicCacheControl struct {
-	Type string `json:"type"` // "ephemeral"
-}
-
-// ephemeralCacheControl is the single reused breakpoint marker.
-var ephemeralCacheControl = &anthropicCacheControl{Type: "ephemeral"}
-
-// anthropicSystemBlock is the block form of the top-level system field.
-// Anthropic accepts system as a bare string OR a list of text blocks; the
-// block form is required to attach cache_control to the system prompt.
-type anthropicSystemBlock struct {
-	Type         string                 `json:"type"` // "text"
-	Text         string                 `json:"text"`
-	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
+	Name        string          `json:"name"`
+	Description string          `json:"description,omitempty"`
+	InputSchema json.RawMessage `json:"input_schema"`
 }
 
 // anthropicToolChoice mirrors the four shapes Anthropic accepts:
@@ -104,9 +81,8 @@ type anthropicContentBlock struct {
 	// Tool-result block fields. tool_result uses `content` (not
 	// `text`) and pairs with `tool_use_id`; modelling them as
 	// distinct fields avoids ambiguity at marshal time.
-	ToolUseID     string                 `json:"tool_use_id,omitempty"`
-	ResultContent string                 `json:"content,omitempty"`
-	CacheControl  *anthropicCacheControl `json:"cache_control,omitempty"`
+	ToolUseID     string `json:"tool_use_id,omitempty"`
+	ResultContent string `json:"content,omitempty"`
 }
 
 type anthropicResponse struct {
@@ -180,11 +156,6 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 	if req.ToolChoice != nil && req.ToolChoice.Type == anthropicToolChoiceNone {
 		req.Tools, req.ToolChoice = nil, nil
 	}
-	// Prompt-cache breakpoint on the last tool: Anthropic caches the entire
-	// tool block up to the marked tool — usually a large, fully stable prefix.
-	if cfg.cachePrompt && len(req.Tools) > 0 {
-		req.Tools[len(req.Tools)-1].CacheControl = ephemeralCacheControl
-	}
 
 	var systemParts []string
 	for _, m := range opts.GetMessages() {
@@ -218,54 +189,15 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 			})
 		}
 	}
-	// System: block form (with cache_control) when caching is on, else the
-	// bare string. Only set when non-empty so `omitempty` still drops it.
-	if len(systemParts) > 0 {
-		joined := strings.Join(systemParts, "\n\n")
-		if cfg.cachePrompt {
-			req.System = []anthropicSystemBlock{{Type: "text", Text: joined, CacheControl: ephemeralCacheControl}}
-		} else {
-			req.System = joined
-		}
-	}
+	req.System = strings.Join(systemParts, "\n\n")
 
 	if len(req.Messages) == 0 && opts.GetPrompt() != "" {
 		req.Messages = []anthropicMessage{{Role: "user", Content: opts.GetPrompt()}}
 	}
 
-	// Prompt-cache breakpoint on the final message block caches the whole
-	// conversation prefix up to the newest turn. With the system + tools
-	// breakpoints above, Anthropic serves the entire stable head at the
-	// cache-read rate on the next agentic iteration (max 4 breakpoints; we
-	// use at most 3, so we never exceed the limit).
-	if cfg.cachePrompt {
-		markLastMessageCacheable(req.Messages)
-	}
-
 	return json.Marshal(req)
 }
 
-// markLastMessageCacheable tags the final block of the last message with a
-// cache_control breakpoint. String content is promoted to a single text
-// block so the marker has somewhere to attach; block content gets the marker
-// on its last element.
-func markLastMessageCacheable(msgs []anthropicMessage) {
-	if len(msgs) == 0 {
-		return
-	}
-	last := &msgs[len(msgs)-1]
-	switch c := last.Content.(type) {
-	case string:
-		if c != "" {
-			last.Content = []anthropicContentBlock{{Type: "text", Text: c, CacheControl: ephemeralCacheControl}}
-		}
-	case []anthropicContentBlock:
-		if len(c) > 0 {
-			c[len(c)-1].CacheControl = ephemeralCacheControl
-		}
-	}
-}
-
 // appendToolResult appends a tool_result block as a user message,
 // merging into a preceding user message that already carries blocks.
 // Anthropic concatenates consecutive same-role messages on its end,

backend/go/cloud-proxy/provider_anthropic_test.go

@@ -328,62 +328,3 @@ func TestBuildAnthropic_RoundTripsAssistantToolCalls(t *testing.T) {
 	g.Expect(r0["tool_use_id"]).To(Equal("call_abc"))
 	g.Expect(r0["content"]).To(Equal(`{"models":["a","b"]}`))
 }
-
-// TestPredict_Anthropic_PromptCache verifies that cache_prompt injects
-// exactly the intended cache_control breakpoints (system, last tool, last
-// message) when on, and none when off — asserting on the raw upstream body
-// because System becomes a block list that the typed struct hides.
-func TestPredict_Anthropic_PromptCache(t *testing.T) {
-	g := NewWithT(t)
-
-	// run issues one translate Predict and returns the raw body the fake
-	// Anthropic upstream received.
-	run := func(cachePrompt bool) string {
-		var rawBody string
-		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-			b, _ := io.ReadAll(r.Body)
-			rawBody = string(b)
-			w.Header().Set("Content-Type", "application/json")
-			_, _ = io.WriteString(w, `{"id":"m","type":"message","role":"assistant","content":[{"type":"text","text":"ok"}],"model":"claude-3-5-sonnet-20241022","usage":{"input_tokens":5,"output_tokens":2}}`)
-		}))
-		defer srv.Close()
-
-		t.Setenv("CLOUD_PROXY_ANTHROPIC_FAKE", "sk-ant-fake")
-		cp := NewCloudProxy()
-		err := cp.Load(&pb.ModelOptions{
-			Model: "claude-local",
-			Proxy: &pb.ProxyOptions{
-				UpstreamUrl:   srv.URL,
-				Mode:          modeTranslate,
-				Provider:      providerAnthropic,
-				ApiKeyEnv:     "CLOUD_PROXY_ANTHROPIC_FAKE",
-				UpstreamModel: "claude-3-5-sonnet-20241022",
-				CachePrompt:   cachePrompt,
-			},
-		})
-		g.Expect(err).NotTo(HaveOccurred())
-
-		_, err = cp.Predict(&pb.PredictOptions{
-			Messages: []*pb.Message{
-				{Role: "system", Content: "be brief"},
-				{Role: "user", Content: "hello"},
-			},
-			Tools:  `[{"type":"function","function":{"name":"t","parameters":{"type":"object"}}}]`,
-			Tokens: 32,
-		})
-		g.Expect(err).NotTo(HaveOccurred())
-		return rawBody
-	}
-
-	// cache_prompt ON: three ephemeral breakpoints (system + last tool +
-	// last message), and system is emitted in block form.
-	on := run(true)
-	g.Expect(strings.Count(on, `"cache_control":{"type":"ephemeral"}`)).To(Equal(3),
-		"expected 3 breakpoints (system, tool, last message); body=%s", on)
-	g.Expect(on).To(ContainSubstring(`"system":[{"type":"text","text":"be brief"`))
-
-	// cache_prompt OFF: no breakpoints, system stays a bare string.
-	off := run(false)
-	g.Expect(off).NotTo(ContainSubstring("cache_control"))
-	g.Expect(off).To(ContainSubstring(`"system":"be brief"`))
-}

backend/go/cloud-proxy/proxy.go

@@ -48,7 +48,6 @@ type proxyConfig struct {
 	upstreamModel string
 	localModel    string // ModelOptions.Model — fallback when upstream_model is unset
 	apiKey        string // resolved at Load time
-	cachePrompt   bool   // inject Anthropic prompt-cache breakpoints (translate+anthropic)
 }
 
 func NewCloudProxy() *CloudProxy {
@@ -107,7 +106,6 @@ func (c *CloudProxy) Load(opts *pb.ModelOptions) error {
 		upstreamModel: po.GetUpstreamModel(),
 		localModel:    opts.GetModel(),
 		apiKey:        key,
-		cachePrompt:   po.GetCachePrompt(),
 	})
 	xlog.Info("cloud-proxy: ready",
 		"upstream", po.GetUpstreamUrl(),

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=754b67289cf1137e3ed722885705f94132fc614f
+CRISPASR_VERSION?=3ab5f4ac13685966b47cc75dc7fd02f3c4a51beb
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/depth-anything-cpp/Makefile

@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=2028b47ac75a8659c6a9aa617baf09be193eb55f
+DEPTHANYTHING_VERSION?=f4e17dea695dd12ae76bea98ba58030996b98118
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/magpie-tts-cpp/.gitignore

@@ -1,6 +0,0 @@
-magpie-tts-cpp
-*.so
-*.dylib
-sources/
-package/
-magpie-tts-models/

backend/go/magpie-tts-cpp/CMakeLists.txt

@@ -1,35 +0,0 @@
-cmake_minimum_required(VERSION 3.16)
-project(gomagpiettscpp LANGUAGES C CXX)
-set(CMAKE_POSITION_INDEPENDENT_CODE ON)
-set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
-
-set(MAGPIE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/magpie-tts.cpp)
-
-# Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
-if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
-    set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
-endif()
-
-# The magpie-tts C-API is exported directly by the upstream shared library
-# (magpie_tts_capi_* in libmagpie-tts.so, ggml statically linked inside), so
-# unlike qwen3-tts-cpp / moss-tts-cpp no local C shim is needed -- this wrapper
-# only configures the upstream project as a purego-loadable shared library.
-set(MAGPIE_SHARED       ON  CACHE BOOL "" FORCE)
-set(MAGPIE_BUILD_CLI    OFF CACHE BOOL "" FORCE)
-set(MAGPIE_BUILD_TESTS  OFF CACHE BOOL "" FORCE)
-
-# Upstream FORCE-overwrites GGML_CUDA / GGML_METAL / GGML_VULKAN / GGML_HIP from
-# its own MAGPIE_GGML_* toggles, which would silently discard the -DGGML_*=ON
-# flags the LocalAI Makefile passes for GPU BUILD_TYPEs. Translate them into the
-# MAGPIE_GGML_* vocabulary before add_subdirectory. (GGML_SYCL / GGML_BLAS and
-# the CPU ISA flags are not touched by upstream and reach ggml unchanged.)
-foreach(_be CUDA METAL VULKAN HIP)
-    if(GGML_${_be})
-        set(MAGPIE_GGML_${_be} ON CACHE BOOL "" FORCE)
-    endif()
-endforeach()
-
-add_subdirectory(${MAGPIE_DIR} magpie EXCLUDE_FROM_ALL)
-
-# Place libmagpie-tts.so at the build root, where the Makefile picks it up.
-set_target_properties(magpie-tts PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/magpie-tts-cpp/Makefile

@@ -1,140 +0,0 @@
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-
-GOCMD?=go
-GO_TAGS?=
-JOBS?=$(shell nproc --ignore=1)
-
-# magpie-tts.cpp version
-MAGPIETTS_REPO?=https://github.com/mudler/magpie-tts.cpp
-MAGPIETTS_CPP_VERSION?=3008ff73fc2d2da9e4d743b09350aa7023e8980c
-SO_TARGET?=libgomagpiettscpp.so
-
-CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
-
-ifeq ($(NATIVE),false)
-	CMAKE_ARGS+=-DGGML_NATIVE=OFF
-endif
-
-ifeq ($(BUILD_TYPE),cublas)
-	CMAKE_ARGS+=-DGGML_CUDA=ON
-else ifeq ($(BUILD_TYPE),openblas)
-	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
-else ifeq ($(BUILD_TYPE),clblas)
-	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
-else ifeq ($(BUILD_TYPE),hipblas)
-	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
-	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
-	ROCM_HOME ?= /opt/rocm
-	ROCM_PATH ?= /opt/rocm
-	export CXX=$(ROCM_HOME)/llvm/bin/clang++
-	export CC=$(ROCM_HOME)/llvm/bin/clang
-	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
-	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
-else ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DGGML_VULKAN=ON
-else ifeq ($(OS),Darwin)
-	ifneq ($(BUILD_TYPE),metal)
-		CMAKE_ARGS+=-DGGML_METAL=OFF
-	else
-		CMAKE_ARGS+=-DGGML_METAL=ON
-		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
-	endif
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f16)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx \
-		-DGGML_SYCL_F16=ON
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f32)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx
-endif
-
-sources/magpie-tts.cpp:
-	mkdir -p sources/magpie-tts.cpp
-	cd sources/magpie-tts.cpp && \
-	git init && \
-	git remote add origin $(MAGPIETTS_REPO) && \
-	git fetch origin && \
-	git checkout $(MAGPIETTS_CPP_VERSION) && \
-	git submodule update --init --recursive --depth 1 --single-branch
-
-# Detect OS
-UNAME_S := $(shell uname -s)
-
-# Only build CPU variants on Linux
-ifeq ($(UNAME_S),Linux)
-	VARIANT_TARGETS = libgomagpiettscpp-avx.so libgomagpiettscpp-avx2.so libgomagpiettscpp-avx512.so libgomagpiettscpp-fallback.so
-else
-	# On non-Linux (e.g., Darwin), build only fallback variant (as a dylib)
-	VARIANT_TARGETS = libgomagpiettscpp-fallback.dylib
-endif
-
-magpie-tts-cpp: main.go gomagpiettscpp.go $(VARIANT_TARGETS)
-	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o magpie-tts-cpp ./
-
-package: magpie-tts-cpp
-	bash package.sh
-
-build: package
-
-clean: purge
-	rm -rf libgomagpiettscpp*.so libgomagpiettscpp*.dylib package sources/magpie-tts.cpp magpie-tts-cpp
-
-purge:
-	rm -rf build*
-
-# Variants must build sequentially
-.NOTPARALLEL:
-
-# Build all variants (Linux only)
-ifeq ($(UNAME_S),Linux)
-libgomagpiettscpp-avx.so: sources/magpie-tts.cpp
-	$(info ${GREEN}I magpie-tts-cpp build info:avx${RESET})
-	SO_TARGET=libgomagpiettscpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
-	rm -rf build-libgomagpiettscpp-avx.so
-
-libgomagpiettscpp-avx2.so: sources/magpie-tts.cpp
-	$(info ${GREEN}I magpie-tts-cpp build info:avx2${RESET})
-	SO_TARGET=libgomagpiettscpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
-	rm -rf build-libgomagpiettscpp-avx2.so
-
-libgomagpiettscpp-avx512.so: sources/magpie-tts.cpp
-	$(info ${GREEN}I magpie-tts-cpp build info:avx512${RESET})
-	SO_TARGET=libgomagpiettscpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomagpiettscpp-custom
-	rm -rf build-libgomagpiettscpp-avx512.so
-endif
-
-# Build fallback variant (all platforms)
-libgomagpiettscpp-fallback.so: sources/magpie-tts.cpp
-	$(info ${GREEN}I magpie-tts-cpp build info:fallback${RESET})
-	SO_TARGET=libgomagpiettscpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
-	rm -rf build-libgomagpiettscpp-fallback.so
-
-# Build fallback variant as a dylib (Darwin)
-libgomagpiettscpp-fallback.dylib: sources/magpie-tts.cpp
-	$(info ${GREEN}I magpie-tts-cpp build info:fallback (dylib)${RESET})
-	SO_TARGET=libgomagpiettscpp-fallback.dylib CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomagpiettscpp-custom
-	rm -rf build-libgomagpiettscpp-fallback.dylib
-
-libgomagpiettscpp-custom: CMakeLists.txt
-	mkdir -p build-$(SO_TARGET) && \
-	cd build-$(SO_TARGET) && \
-	cmake .. $(CMAKE_ARGS) && \
-	cmake --build . --config Release -j$(JOBS) --target magpie-tts && \
-	cd .. && \
-	(mv build-$(SO_TARGET)/libmagpie-tts.so ./$(SO_TARGET) 2>/dev/null || \
-	 mv build-$(SO_TARGET)/libmagpie-tts.dylib ./$(SO_TARGET) 2>/dev/null)
-
-test: magpie-tts-cpp
-	@echo "Running magpie-tts-cpp tests..."
-	bash test.sh
-	@echo "magpie-tts-cpp tests completed."
-
-all: magpie-tts-cpp package

backend/go/magpie-tts-cpp/README.md

@@ -1,69 +0,0 @@
-# Magpie TTS C++ backend
-
-This backend runs NVIDIA's **Magpie TTS Multilingual 357M** GGUF through
-[magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp), a from-scratch
-C++/ggml port (model + NanoCodec + tokenizer + G2P dictionaries in one
-self-contained GGUF, no Python at inference time). It generates **22.05 kHz
-mono** speech in 5 baked voices across 9+ languages.
-
-The library is loaded via purego (cgo-less `dlopen`) exactly like
-`qwen3-tts-cpp` / `moss-tts-cpp`; the flat C-API (`magpie_tts_capi_*`) is
-exported directly by the upstream shared library, so there is no local C shim.
-
-## Model configuration
-
-The model path points at the single GGUF:
-
-```yaml
-name: magpie-tts-cpp
-backend: magpie-tts-cpp
-parameters:
-  model: magpie-tts-multilingual-357m-q8_0.gguf
-known_usecases:
-  - tts
-options:
-  - "speaker:Aria"   # optional default voice (Aria, Jason, John, Leo, Sofia, or 0-4)
-  - "language:en"    # optional default language
-```
-
-GGUFs live at
-[mudler/magpie-tts.cpp-gguf](https://huggingface.co/mudler/magpie-tts.cpp-gguf)
-(q8_0 recommended: near-lossless, ~624 MB, fastest decode).
-
-## Voices and languages
-
-Magpie has 5 baked speakers - `Aria`, `Jason`, `John`, `Leo`, `Sofia` - and no
-voice cloning. The request `voice` accepts the names case-insensitively or the
-indices `0`-`4`; empty selects Aria. Languages: `en`, `es`, `de`, `fr`, `it`,
-`pt-BR`, `hi`, `vi`, `ko`, `ar-AE`, `ar-SA`, `ar-MSA` (case-insensitive;
-default `en`).
-
-## API example
-
-```bash
-curl http://localhost:8080/v1/audio/speech \
-  -H 'Content-Type: application/json' \
-  -d '{
-    "model": "magpie-tts-cpp",
-    "input": "Hello world, this is a test of the text to speech system.",
-    "voice": "sofia",
-    "language": "en"
-  }' \
-  --output speech.wav
-```
-
-## Native end-to-end test
-
-The labeled test loads a real GGUF, synthesizes WAVs (verifying rate, layout
-and non-silence), and exercises the streaming path:
-
-```bash
-make -C backend/go/magpie-tts-cpp magpie-tts-cpp
-
-MAGPIETTS_MODEL=/path/to/magpie-tts-multilingual-357m-q8_0.gguf \
-MAGPIETTS_LIBRARY=backend/go/magpie-tts-cpp/libgomagpiettscpp-fallback.so \
-  go test ./backend/go/magpie-tts-cpp -ginkgo.label-filter=e2e
-```
-
-`bash test.sh` does the same and auto-downloads the q8_0 GGUF when
-`MAGPIETTS_MODEL` is unset.

backend/go/magpie-tts-cpp/audio.go

@@ -1,93 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"encoding/binary"
-	"fmt"
-	"os"
-
-	"github.com/go-audio/audio"
-	"github.com/go-audio/wav"
-)
-
-// magpieSampleRate is the fixed Magpie TTS Multilingual (NanoCodec) output
-// rate: 22.05 kHz.
-const magpieSampleRate = 22050
-
-// magpieChannels is the fixed output layout: mono.
-const magpieChannels = 1
-
-// wavHeaderMono returns a 44-byte WAV header for a streaming 16-bit mono PCM
-// stream at 22050 Hz, with placeholder (0xFFFFFFFF) sizes since the total
-// length is unknown up front. Emitted as the first chunk of TTSStream so the
-// HTTP layer receives a self-describing WAV.
-func wavHeaderMono() []byte {
-	const blockAlign = magpieChannels * 2 // 16-bit mono
-	var buf bytes.Buffer
-	w := func(v any) { _ = binary.Write(&buf, binary.LittleEndian, v) }
-	buf.WriteString("RIFF")
-	w(uint32(0xFFFFFFFF))
-	buf.WriteString("WAVE")
-	buf.WriteString("fmt ")
-	w(uint32(16))                            // Subchunk1Size
-	w(uint16(1))                             // PCM
-	w(uint16(magpieChannels))                // mono
-	w(uint32(magpieSampleRate))              // sample rate
-	w(uint32(magpieSampleRate * blockAlign)) // byte rate = SR * blockAlign
-	w(uint16(blockAlign))                    // block align
-	w(uint16(16))                            // bits per sample
-	buf.WriteString("data")
-	w(uint32(0xFFFFFFFF))
-	return buf.Bytes()
-}
-
-// floatToPCM16LE clamps each sample to [-1,1] and encodes it as little-endian
-// signed 16-bit PCM.
-func floatToPCM16LE(samples []float32) []byte {
-	out := make([]byte, len(samples)*2)
-	for i, s := range samples {
-		if s > 1 {
-			s = 1
-		} else if s < -1 {
-			s = -1
-		}
-		v := int16(s * 32767)
-		out[i*2] = byte(v)        // #nosec G115 -- intentional little-endian split of a clamped int16
-		out[i*2+1] = byte(v >> 8) // #nosec G115 -- high byte of the same clamped int16
-	}
-	return out
-}
-
-// writeWAVMono writes float samples as a finalized 16-bit mono WAV at
-// 22050 Hz.
-func writeWAVMono(dst string, samples []float32) error {
-	f, err := os.Create(dst) // #nosec G304 -- dst is the server-chosen output path from the TTS request, not user-traversable
-	if err != nil {
-		return fmt.Errorf("magpie-tts: create %q: %w", dst, err)
-	}
-	enc := wav.NewEncoder(f, magpieSampleRate, 16, magpieChannels, 1)
-	ints := make([]int, len(samples))
-	for i, s := range samples {
-		if s > 1 {
-			s = 1
-		} else if s < -1 {
-			s = -1
-		}
-		ints[i] = int(s * 32767)
-	}
-	b := &audio.IntBuffer{
-		Format:         &audio.Format{NumChannels: magpieChannels, SampleRate: magpieSampleRate},
-		Data:           ints,
-		SourceBitDepth: 16,
-	}
-	if err := enc.Write(b); err != nil {
-		_ = enc.Close()
-		_ = f.Close()
-		return fmt.Errorf("magpie-tts: encode WAV: %w", err)
-	}
-	if err := enc.Close(); err != nil {
-		_ = f.Close()
-		return fmt.Errorf("magpie-tts: finalize WAV: %w", err)
-	}
-	return f.Close()
-}

backend/go/magpie-tts-cpp/e2e_test.go

@@ -1,127 +0,0 @@
-package main
-
-import (
-	"encoding/binary"
-	"math"
-	"os"
-
-	"github.com/ebitengine/purego"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func ttsReq(text, voice, lang, dst string) *pb.TTSRequest {
-	r := &pb.TTSRequest{Text: text, Voice: voice, Dst: dst}
-	if lang != "" {
-		r.Language = &lang
-	}
-	return r
-}
-
-// wavRMS parses a 16-bit PCM WAV file and returns (sampleRate, channels, RMS
-// of the normalized samples).
-func wavRMS(path string) (int, int, float64) {
-	data, err := os.ReadFile(path) // #nosec G304 -- test-owned temp file
-	Expect(err).ToNot(HaveOccurred())
-	Expect(len(data)).To(BeNumerically(">", 44))
-	Expect(string(data[0:4])).To(Equal("RIFF"))
-	Expect(string(data[8:12])).To(Equal("WAVE"))
-	channels := int(binary.LittleEndian.Uint16(data[22:24]))
-	rate := int(binary.LittleEndian.Uint32(data[24:28]))
-	// Find the data chunk (go-audio writes fmt first, data after).
-	off := 12
-	for off+8 <= len(data) {
-		id := string(data[off : off+4])
-		sz := int(binary.LittleEndian.Uint32(data[off+4 : off+8]))
-		if id == "data" {
-			pcm := data[off+8:]
-			if sz < len(pcm) {
-				pcm = pcm[:sz]
-			}
-			var sum float64
-			n := len(pcm) / 2
-			for i := 0; i < n; i++ {
-				s := float64(int16(binary.LittleEndian.Uint16(pcm[i*2:]))) / 32768.0
-				sum += s * s
-			}
-			Expect(n).To(BeNumerically(">", 0))
-			return rate, channels, math.Sqrt(sum / float64(n))
-		}
-		off += 8 + sz
-	}
-	Fail("no data chunk found in " + path)
-	return 0, 0, 0
-}
-
-var _ = Describe("magpie-tts-cpp e2e", Label("e2e"), func() {
-	var (
-		loaded  bool
-		backend *MagpieTtsCpp
-	)
-
-	BeforeEach(func() {
-		modelPath := os.Getenv("MAGPIETTS_MODEL")
-		if modelPath == "" {
-			Skip("MAGPIETTS_MODEL not set; skipping e2e")
-		}
-		if !loaded {
-			lib := os.Getenv("MAGPIETTS_LIBRARY")
-			if lib == "" {
-				lib = "./libgomagpiettscpp-fallback.so"
-			}
-			h, err := purego.Dlopen(lib, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-			Expect(err).ToNot(HaveOccurred())
-			purego.RegisterLibFunc(&CppAbiVersion, h, "magpie_tts_capi_abi_version")
-			purego.RegisterLibFunc(&CppLoad, h, "magpie_tts_capi_load")
-			purego.RegisterLibFunc(&CppFree, h, "magpie_tts_capi_free")
-			purego.RegisterLibFunc(&CppSynthesize, h, "magpie_tts_capi_synthesize")
-			purego.RegisterLibFunc(&CppFreeAudio, h, "magpie_tts_capi_free_audio")
-			purego.RegisterLibFunc(&CppLastError, h, "magpie_tts_capi_last_error")
-
-			backend = &MagpieTtsCpp{}
-			Expect(backend.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
-			loaded = true
-		}
-	})
-
-	It("synthesizes a non-silent 22.05 kHz mono WAV via TTS", func() {
-		dst := GinkgoT().TempDir() + "/out.wav"
-		Expect(backend.TTS(ttsReq("Hello world, this is a test.", "Aria", "en", dst))).To(Succeed())
-		rate, channels, rms := wavRMS(dst)
-		Expect(rate).To(Equal(22050))
-		Expect(channels).To(Equal(1))
-		Expect(rms).To(BeNumerically(">", 0.01), "audio should not be silent")
-	})
-
-	It("accepts a case-insensitive voice and a speaker index", func() {
-		dst := GinkgoT().TempDir() + "/out.wav"
-		Expect(backend.TTS(ttsReq("Short test.", "sofia", "en", dst))).To(Succeed())
-		Expect(backend.TTS(ttsReq("Short test.", "1", "en", dst))).To(Succeed())
-	})
-
-	It("rejects an unknown voice before reaching the engine", func() {
-		dst := GinkgoT().TempDir() + "/out.wav"
-		err := backend.TTS(ttsReq("Short test.", "not-a-speaker", "en", dst))
-		Expect(err).To(MatchError(ContainSubstring("unknown voice")))
-	})
-
-	It("streams a self-describing WAV via TTSStream", func() {
-		results := make(chan []byte, 4096)
-		done := make(chan error, 1)
-		go func() { done <- backend.TTSStream(ttsReq("Hello there, streaming test.", "", "", ""), results) }()
-
-		var chunks int
-		var first []byte
-		for c := range results {
-			if chunks == 0 {
-				first = c
-			}
-			chunks++
-		}
-		Expect(<-done).ToNot(HaveOccurred())
-		Expect(chunks).To(BeNumerically(">=", 2))
-		Expect(string(first[0:4])).To(Equal("RIFF"))
-		Expect(string(first[8:12])).To(Equal("WAVE"))
-	})
-})

backend/go/magpie-tts-cpp/gomagpiettscpp.go

@@ -1,147 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"path/filepath"
-	"unsafe"
-
-	"github.com/mudler/LocalAI/pkg/grpc/base"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	"github.com/mudler/xlog"
-)
-
-// capiABIVersion is the magpie_tts_capi.h surface this backend binds. Bumped
-// upstream on any breaking signature/semantics change; refuse to run on a
-// mismatch instead of crashing inside a miscompiled call.
-const capiABIVersion = 1
-
-var (
-	// magpie_tts_capi_abi_version() int
-	CppAbiVersion func() int
-	// magpie_tts_capi_load(gguf_path) -> ctx (NULL on failure)
-	CppLoad func(path string) uintptr
-	// magpie_tts_capi_free(ctx)
-	CppFree func(ctx uintptr)
-	// magpie_tts_capi_synthesize(ctx, text, language, speaker, out_n) -> float*
-	// 22050 Hz mono f32 PCM in [-1,1]; NULL on failure (see last_error).
-	CppSynthesize func(ctx uintptr, text, language, speaker string, outN unsafe.Pointer) uintptr
-	// magpie_tts_capi_free_audio(ptr)
-	CppFreeAudio func(ptr uintptr)
-	// magpie_tts_capi_last_error(ctx) -> const char* (ctx-owned, "" if none)
-	CppLastError func(ctx uintptr) string
-)
-
-// MagpieTtsCpp serves the Magpie TTS Multilingual 357M GGUF through the
-// magpie-tts.cpp C-API. The context is stateful (per-call error buffer, reused
-// graph allocator) and NOT safe for concurrent synthesize calls, so
-// base.SingleThread serializes everything behind the server-level lock.
-type MagpieTtsCpp struct {
-	base.SingleThread
-	ctx  uintptr
-	opts loadOptions
-}
-
-func (m *MagpieTtsCpp) Load(opts *pb.ModelOptions) error {
-	model := opts.ModelFile
-	if model == "" {
-		model = opts.ModelPath
-	}
-	if !filepath.IsAbs(model) && opts.ModelPath != "" {
-		model = filepath.Join(opts.ModelPath, model)
-	}
-
-	m.opts = parseOptions(opts.Options)
-
-	if abi := CppAbiVersion(); abi != capiABIVersion {
-		return fmt.Errorf("magpie-tts: C-API ABI mismatch: library reports v%d, backend built for v%d", abi, capiABIVersion)
-	}
-
-	xlog.Info("[magpie-tts-cpp] Load", "model", model)
-
-	ctx := CppLoad(model)
-	if ctx == 0 {
-		// Load failures have no context to query last_error on; the C side
-		// logs the reason to stderr.
-		return fmt.Errorf("magpie-tts: failed to load model %q", model)
-	}
-	m.ctx = ctx
-	return nil
-}
-
-// lastError surfaces the context's last error, falling back to a generic
-// message when the C side left it empty.
-func (m *MagpieTtsCpp) lastError() string {
-	if m.ctx == 0 {
-		return "no model loaded"
-	}
-	if e := CppLastError(m.ctx); e != "" {
-		return e
-	}
-	return "unknown error"
-}
-
-// synthesize runs one C-API synthesis and copies the PCM out of C memory.
-func (m *MagpieTtsCpp) synthesize(req *pb.TTSRequest) ([]float32, error) {
-	if m.ctx == 0 {
-		return nil, fmt.Errorf("magpie-tts: no model loaded")
-	}
-	if req.Text == "" {
-		return nil, fmt.Errorf("magpie-tts: TTS requires text")
-	}
-	speaker, err := resolveSpeaker(req.Voice, m.opts.speaker)
-	if err != nil {
-		return nil, err
-	}
-	lang := resolveLanguage(req.Language, m.opts.language)
-
-	var n int32
-	ptr := CppSynthesize(m.ctx, req.Text, lang, speaker, unsafe.Pointer(&n)) // #nosec G103 -- out-param for the purego-bound C-API
-	if ptr == 0 {
-		return nil, fmt.Errorf("magpie-tts: synthesis failed: %s", m.lastError())
-	}
-	// Register the free as soon as we own a non-null buffer, so the n<=0 guard
-	// below cannot leak it (defensive: the C contract returns NULL on failure).
-	defer CppFreeAudio(ptr)
-	if n <= 0 {
-		return nil, fmt.Errorf("magpie-tts: synthesis produced no samples")
-	}
-	//nolint:govet // C-allocated PCM, copied out before free
-	src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) // #nosec G103 -- C-allocated PCM, copied out before free
-	out := make([]float32, int(n))
-	copy(out, src)
-	return out, nil
-}
-
-func (m *MagpieTtsCpp) TTS(req *pb.TTSRequest) error {
-	if req.Dst == "" {
-		return fmt.Errorf("magpie-tts: TTS requires a destination path")
-	}
-	samples, err := m.synthesize(req)
-	if err != nil {
-		return err
-	}
-	return writeWAVMono(req.Dst, samples)
-}
-
-// TTSStream synthesizes one-shot (the magpie C-API has no streaming call) and
-// then emits a self-describing mono WAV: a header chunk followed by the PCM in
-// fixed-size slices, so the HTTP layer still receives a streamed WAV (the gRPC
-// TTSStream path never sets Message, so the backend owns the header - see
-// core/backend/tts.go:ModelTTSStream).
-func (m *MagpieTtsCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
-	defer close(results)
-	samples, err := m.synthesize(req)
-	if err != nil {
-		return err
-	}
-	results <- wavHeaderMono()
-	const sampleChunk = 4096 // mono samples per emitted chunk
-	for off := 0; off < len(samples); off += sampleChunk {
-		end := off + sampleChunk
-		if end > len(samples) {
-			end = len(samples)
-		}
-		results <- floatToPCM16LE(samples[off:end])
-	}
-	return nil
-}

backend/go/magpie-tts-cpp/magpiettscpp_test.go

@@ -1,115 +0,0 @@
-package main
-
-import (
-	"testing"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func TestMagpieTtsCpp(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "magpie-tts-cpp suite")
-}
-
-var _ = Describe("resolveSpeaker", func() {
-	It("canonicalizes case-insensitive names", func() {
-		for in, want := range map[string]string{
-			"aria": "Aria", "JASON": "Jason", "john": "John",
-			"Leo": "Leo", "sofia": "Sofia",
-		} {
-			got, err := resolveSpeaker(in, "")
-			Expect(err).ToNot(HaveOccurred())
-			Expect(got).To(Equal(want))
-		}
-	})
-	It("accepts indices 0-4", func() {
-		for in, want := range map[string]string{
-			"0": "Aria", "1": "Jason", "2": "John", "3": "Leo", "4": "Sofia",
-		} {
-			got, err := resolveSpeaker(in, "")
-			Expect(err).ToNot(HaveOccurred())
-			Expect(got).To(Equal(want))
-		}
-	})
-	It("selects the engine default on empty", func() {
-		got, err := resolveSpeaker("", "")
-		Expect(err).ToNot(HaveOccurred())
-		Expect(got).To(BeEmpty())
-	})
-	It("falls back to the model-level default speaker", func() {
-		got, err := resolveSpeaker("", "sofia")
-		Expect(err).ToNot(HaveOccurred())
-		Expect(got).To(Equal("Sofia"))
-	})
-	It("prefers the request voice over the fallback", func() {
-		got, err := resolveSpeaker("leo", "sofia")
-		Expect(err).ToNot(HaveOccurred())
-		Expect(got).To(Equal("Leo"))
-	})
-	It("rejects out-of-range indices", func() {
-		_, err := resolveSpeaker("5", "")
-		Expect(err).To(HaveOccurred())
-		_, err = resolveSpeaker("-1", "")
-		Expect(err).To(HaveOccurred())
-	})
-	It("rejects unknown names with the valid choices", func() {
-		_, err := resolveSpeaker("serena", "")
-		Expect(err).To(MatchError(ContainSubstring("Aria, Jason, John, Leo, Sofia")))
-	})
-})
-
-var _ = Describe("resolveLanguage", func() {
-	strp := func(s string) *string { return &s }
-
-	It("defaults to empty (engine picks en)", func() {
-		Expect(resolveLanguage(nil, "")).To(BeEmpty())
-	})
-	It("canonicalizes case for known codes", func() {
-		Expect(resolveLanguage(strp("EN"), "")).To(Equal("en"))
-		Expect(resolveLanguage(strp("pt-br"), "")).To(Equal("pt-BR"))
-		Expect(resolveLanguage(strp("AR-MSA"), "")).To(Equal("ar-MSA"))
-	})
-	It("falls back to the model-level default language", func() {
-		Expect(resolveLanguage(nil, "de")).To(Equal("de"))
-		Expect(resolveLanguage(strp(""), "PT-BR")).To(Equal("pt-BR"))
-	})
-	It("prefers the request language over the fallback", func() {
-		Expect(resolveLanguage(strp("it"), "de")).To(Equal("it"))
-	})
-	It("passes unknown codes through verbatim", func() {
-		Expect(resolveLanguage(strp("zh"), "")).To(Equal("zh"))
-	})
-})
-
-var _ = Describe("parseOptions", func() {
-	It("reads speaker and language defaults", func() {
-		o := parseOptions([]string{"speaker:Jason", "language:de"})
-		Expect(o.speaker).To(Equal("Jason"))
-		Expect(o.language).To(Equal("de"))
-	})
-	It("accepts the voice/lang aliases and ignores unknown keys", func() {
-		o := parseOptions([]string{"voice: sofia ", "lang: pt-BR", "bogus:1", "novalue"})
-		Expect(o.speaker).To(Equal("sofia"))
-		Expect(o.language).To(Equal("pt-BR"))
-	})
-})
-
-var _ = Describe("audio encoding", func() {
-	It("emits a well-formed streaming mono WAV header", func() {
-		h := wavHeaderMono()
-		Expect(h).To(HaveLen(44))
-		Expect(string(h[0:4])).To(Equal("RIFF"))
-		Expect(string(h[8:12])).To(Equal("WAVE"))
-		// channels (offset 22) == 1, sample rate (offset 24) == 22050
-		Expect(int(h[22]) | int(h[23])<<8).To(Equal(1))
-		Expect(int(h[24]) | int(h[25])<<8 | int(h[26])<<16 | int(h[27])<<24).To(Equal(22050))
-	})
-	It("clamps float PCM to int16", func() {
-		b := floatToPCM16LE([]float32{2, -2, 0})
-		Expect(b).To(HaveLen(6))
-		Expect(int16(uint16(b[0]) | uint16(b[1])<<8)).To(Equal(int16(32767)))
-		Expect(int16(uint16(b[2]) | uint16(b[3])<<8)).To(Equal(int16(-32767)))
-		Expect(int16(uint16(b[4]) | uint16(b[5])<<8)).To(Equal(int16(0)))
-	})
-})

backend/go/magpie-tts-cpp/main.go

@@ -1,54 +0,0 @@
-package main
-
-// Note: this is started internally by LocalAI and a server is allocated for each model
-import (
-	"flag"
-	"os"
-	"runtime"
-
-	"github.com/ebitengine/purego"
-	grpc "github.com/mudler/LocalAI/pkg/grpc"
-)
-
-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
-)
-
-type LibFuncs struct {
-	FuncPtr any
-	Name    string
-}
-
-func main() {
-	libName := os.Getenv("MAGPIETTS_LIBRARY")
-	if libName == "" {
-		if runtime.GOOS == "darwin" {
-			libName = "./libgomagpiettscpp-fallback.dylib"
-		} else {
-			libName = "./libgomagpiettscpp-fallback.so"
-		}
-	}
-
-	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-	if err != nil {
-		panic(err)
-	}
-
-	libFuncs := []LibFuncs{
-		{&CppAbiVersion, "magpie_tts_capi_abi_version"},
-		{&CppLoad, "magpie_tts_capi_load"},
-		{&CppFree, "magpie_tts_capi_free"},
-		{&CppSynthesize, "magpie_tts_capi_synthesize"},
-		{&CppFreeAudio, "magpie_tts_capi_free_audio"},
-		{&CppLastError, "magpie_tts_capi_last_error"},
-	}
-	for _, lf := range libFuncs {
-		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
-	}
-
-	flag.Parse()
-
-	if err := grpc.StartServer(*addr, &MagpieTtsCpp{}); err != nil {
-		panic(err)
-	}
-}

backend/go/magpie-tts-cpp/options.go

@@ -1,108 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"strconv"
-	"strings"
-)
-
-// loadOptions holds the parsed model-level options. Magpie is a single
-// self-contained GGUF (model + codec + tokenizer + G2P dictionaries), so the
-// options only cover synthesis defaults.
-type loadOptions struct {
-	// speaker is the default baked speaker when a request has no voice.
-	speaker string
-	// language is the default language when a request has none ("" = engine
-	// default, which is "en").
-	language string
-}
-
-func splitOption(o string) (key, value string, ok bool) {
-	i := strings.Index(o, ":")
-	if i < 0 {
-		return "", "", false
-	}
-	return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true
-}
-
-// parseOptions reads the backend "key:value" option slice. Unknown keys are
-// ignored.
-func parseOptions(opts []string) loadOptions {
-	var o loadOptions
-	for _, oo := range opts {
-		key, value, ok := splitOption(oo)
-		if !ok {
-			continue
-		}
-		switch key {
-		case "speaker", "voice":
-			o.speaker = value
-		case "language", "lang":
-			o.language = value
-		}
-	}
-	return o
-}
-
-// magpieSpeakers are the baked speakers of Magpie TTS Multilingual 357M, in
-// index order (the engine matches names exactly, so the Go side canonicalizes
-// case-insensitive names and 0-4 indices to these strings).
-var magpieSpeakers = []string{"Aria", "Jason", "John", "Leo", "Sofia"}
-
-// resolveSpeaker maps the request voice (falling back to the model-level
-// default) onto a canonical baked speaker name. Accepted forms:
-// case-insensitive names (aria, JASON, ...) and indices 0-4. Empty selects the
-// engine default (speaker 0, Aria). Anything else is rejected with the valid
-// choices, instead of surfacing the engine's late error.
-func resolveSpeaker(voice, fallback string) (string, error) {
-	v := strings.TrimSpace(voice)
-	if v == "" {
-		v = strings.TrimSpace(fallback)
-	}
-	if v == "" {
-		return "", nil
-	}
-	if idx, err := strconv.Atoi(v); err == nil {
-		if idx < 0 || idx >= len(magpieSpeakers) {
-			return "", fmt.Errorf("magpie-tts: speaker index %d out of range 0-%d", idx, len(magpieSpeakers)-1)
-		}
-		return magpieSpeakers[idx], nil
-	}
-	for _, s := range magpieSpeakers {
-		if strings.EqualFold(s, v) {
-			return s, nil
-		}
-	}
-	return "", fmt.Errorf("magpie-tts: unknown voice %q (valid: %s, or 0-%d)",
-		voice, strings.Join(magpieSpeakers, ", "), len(magpieSpeakers)-1)
-}
-
-// magpieLanguages are the canonical language codes the tokenizer's language
-// map knows (exact-match on the C side), keyed by their lowercase form so
-// requests can be case-insensitive.
-var magpieLanguages = map[string]string{
-	"en": "en", "es": "es", "de": "de", "fr": "fr", "it": "it",
-	"pt-br": "pt-BR", "hi": "hi", "vi": "vi", "ko": "ko",
-	"ar-ae": "ar-AE", "ar-sa": "ar-SA", "ar-msa": "ar-MSA",
-}
-
-// resolveLanguage picks the request language, else the model-level default,
-// else "" (the engine defaults to "en"), canonicalizing case for the known
-// codes. Unknown codes pass through verbatim so the engine reports them with
-// its own exact-vocabulary error.
-func resolveLanguage(reqLang *string, fallback string) string {
-	l := ""
-	if reqLang != nil {
-		l = strings.TrimSpace(*reqLang)
-	}
-	if l == "" {
-		l = strings.TrimSpace(fallback)
-	}
-	if l == "" {
-		return ""
-	}
-	if canon, ok := magpieLanguages[strings.ToLower(l)]; ok {
-		return canon
-	}
-	return l
-}

backend/go/magpie-tts-cpp/package.sh

@@ -1,65 +0,0 @@
-#!/bin/bash
-
-# Script to copy the appropriate libraries based on architecture
-# This script is used in the final stage of the Dockerfile
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-# Create lib directory
-mkdir -p $CURDIR/package/lib
-
-cp -avf $CURDIR/magpie-tts-cpp $CURDIR/package/
-cp -fv $CURDIR/libgomagpiettscpp-*.so $CURDIR/package/ 2>/dev/null || true
-cp -fv $CURDIR/libgomagpiettscpp-*.dylib $CURDIR/package/ 2>/dev/null || true
-cp -fv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    # x86_64 architecture
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    # ARM64 architecture
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ $(uname -s) = "Darwin" ]; then
-    echo "Detected Darwin"
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/go/magpie-tts-cpp/run.sh

@@ -1,57 +0,0 @@
-#!/bin/bash
-set -ex
-
-# Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath "$0")")
-
-cd /
-
-echo "CPU info:"
-if [ "$(uname)" != "Darwin" ]; then
-	grep -e "model\sname" /proc/cpuinfo | head -1
-	grep -e "flags" /proc/cpuinfo | head -1
-fi
-
-if [ "$(uname)" = "Darwin" ]; then
-	# macOS: single dylib variant (Metal or Accelerate)
-	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.dylib"
-	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
-else
-	LIBRARY="$CURDIR/libgomagpiettscpp-fallback.so"
-
-	if grep -q -e "\savx\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX    found OK"
-		if [ -e "$CURDIR"/libgomagpiettscpp-avx.so ]; then
-			LIBRARY="$CURDIR/libgomagpiettscpp-avx.so"
-		fi
-	fi
-
-	if grep -q -e "\savx2\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX2   found OK"
-		if [ -e "$CURDIR"/libgomagpiettscpp-avx2.so ]; then
-			LIBRARY="$CURDIR/libgomagpiettscpp-avx2.so"
-		fi
-	fi
-
-	# Check avx 512
-	if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX512F found OK"
-		if [ -e "$CURDIR"/libgomagpiettscpp-avx512.so ]; then
-			LIBRARY="$CURDIR/libgomagpiettscpp-avx512.so"
-		fi
-	fi
-
-	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
-fi
-
-export MAGPIETTS_LIBRARY=$LIBRARY
-
-# If there is a lib/ld.so, use it
-if [ -f "$CURDIR"/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using library: $LIBRARY"
-	exec "$CURDIR"/lib/ld.so "$CURDIR"/magpie-tts-cpp "$@"
-fi
-
-echo "Using library: $LIBRARY"
-exec "$CURDIR"/magpie-tts-cpp "$@"

backend/go/magpie-tts-cpp/test.sh

@@ -1,28 +0,0 @@
-#!/bin/bash
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-cd "$CURDIR"
-
-echo "Running magpie-tts-cpp backend tests..."
-
-# Auto-download the q8_0 GGUF only when MAGPIETTS_MODEL is not set.
-if [ -z "$MAGPIETTS_MODEL" ]; then
-    MODEL_DIR="./magpie-tts-models"
-    mkdir -p "$MODEL_DIR"
-    REPO_ID="mudler/magpie-tts.cpp-gguf"
-    BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
-    FILE="magpie-tts-multilingual-357m-q8_0.gguf"
-    dest="${MODEL_DIR}/${FILE}"
-    if [ -f "${dest}" ]; then
-        echo "  [skip] ${FILE}"
-    else
-        echo "  [download] ${FILE}..."
-        curl -L -o "${dest}" "${BASE_URL}/${FILE}" --progress-bar
-    fi
-    export MAGPIETTS_MODEL="${dest}"
-fi
-
-go test -v -timeout 1200s .
-
-echo "All magpie-tts-cpp tests passed."

backend/go/parakeet-cpp/Makefile

@@ -1,6 +1,6 @@
 # parakeet-cpp backend Makefile.
 #
-# Upstream pin lives below as PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
+# Upstream pin lives below as PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
 # (.github/bump_deps.sh) can find and update it - matches the
 # whisper.cpp / ds4 / vibevoice-cpp convention.
 #
@@ -15,7 +15,7 @@
 # That's what the L0 smoke test uses. The default target below does the
 # proper clone-at-pin + cmake build so CI doesn't need a side-checkout.
 
-PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
+PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
 PARAKEET_REPO?=https://github.com/mudler/parakeet.cpp
 
 GOCMD?=go

backend/go/qwen3-tts-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # qwentts.cpp version
 QWEN3TTS_REPO?=https://github.com/ServeurpersoCom/qwentts.cpp
-QWEN3TTS_CPP_VERSION?=35ebe5376b82a0a59d008586d55bbe623d449011
+QWEN3TTS_CPP_VERSION?=82cd05b9f3a175612dc89fd6943e610fab096ef5
 SO_TARGET?=libgoqwen3ttscpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=22516991cbdf725e69b0b4a87e52ca16cce07c2d
+STABLEDIFFUSION_GGML_VERSION?=8a51eb92848c1327a5aaeff5ad81a7a9a2435255
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/vllm-cpp/.gitignore

@@ -1,6 +0,0 @@
-sources/
-build/
-package/
-vllm-cpp
-libvllm.so
-libvllm.dylib

backend/go/vllm-cpp/Makefile

@@ -1,102 +0,0 @@
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-
-GOCMD?=go
-GO_TAGS?=
-# nproc doesn't exist on the macOS runners: an empty JOBS turns `-j$(JOBS)`
-# into bare `-j` (unlimited clang jobs), which swap-thrashes the 3-core Mac
-# until the 6h GHA timeout. Fall back to sysctl there, then to a constant.
-JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
-
-# vllm.cpp version
-VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
-VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845
-
-# The backend consumes only the stable C ABI (libvllm + include/vllm.h), so the
-# server, examples and tests of the engine are never built here.
-CMAKE_ARGS+=-DVLLM_CPP_SERVER=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_BUILD_EXAMPLES=OFF
-CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
-
-# vllm.cpp sets no global -march: SIMD tiers are per-file with runtime dispatch,
-# so ONE portable library serves every CPU of the target arch (unlike the
-# ggml-based backends and their avx/avx2/avx512 variant builds).
-UNAME_M := $(shell uname -m)
-
-ifeq ($(BUILD_TYPE),cublas)
-	# Blackwell-family targets only: other CUDA arches are build-supported
-	# upstream but have no runtime-proven fast path. amd64 gets the consumer
-	# (120a) + GB10 (121a) fat binary; arm64 CUDA (l4t-style images, DGX
-	# Spark) is GB10 only. Triton-AOT GDN cubins are vendored per-arch, no
-	# Python needed to consume them.
-	ifeq ($(UNAME_M),x86_64)
-		# NO -DVLLM_CPP_TRITON on fat builds: the vendored Triton-AOT cubin
-		# trees are per-arch and the engine refuses a multi-arch build unless
-		# pinned to one tree (unsound for the other arch). The non-AOT GDN
-		# path serves the fat binary; single-arch builds keep the cubins.
-		#
-		# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks
-		# compute_121a (GB10) and its ptxas rejects the sm_120a NVFP4 MMA
-		# kernels ("Vector type too large"), so no cuda-12 variant is shipped.
-		ifeq ($(CUDA_MAJOR_VERSION),12)
-      $(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
-		endif
-		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=120a;121a"
-	else
-		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
-	endif
-else ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF
-else ifeq ($(BUILD_TYPE),metal)
-	CMAKE_ARGS+=-DVLLM_CPP_METAL=ON
-else
-	CMAKE_ARGS+=-DVLLM_CPP_CUDA=OFF
-endif
-
-UNAME_S := $(shell uname -s)
-ifeq ($(UNAME_S),Darwin)
-	LIB=libvllm.dylib
-else
-	LIB=libvllm.so
-endif
-
-sources/vllm.cpp:
-	mkdir -p sources/vllm.cpp
-	cd sources/vllm.cpp && \
-	git init && \
-	git remote add origin $(VLLM_CPP_REPO) && \
-	git fetch --depth 1 origin $(VLLM_CPP_VERSION) && \
-	git checkout FETCH_HEAD
-
-$(LIB): sources/vllm.cpp
-	mkdir -p build && \
-	cd build && \
-	cmake ../sources/vllm.cpp $(CMAKE_ARGS) && \
-	cmake --build . --config Release -j$(JOBS) --target vllm_shared
-	cp -fL build/$(LIB) ./$(LIB)
-
-vllm-cpp: main.go govllmcpp.go backend.go options.go $(LIB)
-	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vllm-cpp ./
-
-package: vllm-cpp
-	bash package.sh
-
-build: package
-
-clean: purge
-	rm -rf libvllm.so libvllm.dylib package sources/vllm.cpp vllm-cpp
-
-purge:
-	rm -rf build
-
-.NOTPARALLEL:
-
-# The unit specs are pure Go (struct mirrors, option mapping, load
-# validation): no libvllm build is needed. The e2e specs skip unless
-# VLLM_CPP_MODEL points at a real model (then build the lib first).
-test:
-	@echo "Running vllm-cpp tests..."
-	bash test.sh
-	@echo "vllm-cpp tests completed."
-
-all: vllm-cpp package

backend/go/vllm-cpp/README.md

@@ -1,45 +0,0 @@
-# vllm-cpp backend
-
-LocalAI text-generation backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
-the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching,
-safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
-inference time.
-
-The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
-ABI v2) through purego:
-
-- `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
-  directory (`config.json` + safetensors). `context_size` maps to
-  `max_model_len`; `options: ["block_size:<n>", "num_blocks:<n>",
-  "max_num_seqs:<n>"]` size the KV cache and scheduler admission.
-- `Predict` -> `vllm_complete` (blocking).
-- `PredictStream` -> `vllm_complete_stream`; concurrent gRPC requests batch
-  continuously in the engine's shared AsyncLLM scheduler.
-- Chat / tool calling rides the SAME code path as the llama.cpp autoparser:
-  with `use_tokenizer_template: true` the backend implements
-  `PredictRich`/`PredictStreamRich` over the ABI v3 chat entry points
-  (`vllm_chat` / `vllm_chat_stream`). The ENGINE applies the model's chat
-  template (GGUF `tokenizer.chat_template` or `tokenizer_config.json`),
-  decides when a tool call engages (`tool_choice: auto` lowers to a LAZY
-  structural-tag decode constraint; `required`/named force one), parses tool
-  calls with its streaming Hermes-style parser, and the backend maps each
-  `chat.completion.chunk` onto `ChatDelta`/`ToolCallDelta` protos.
-- Without structured messages the plain path applies:
-  `PredictOptions.Grammar` -> the ABI's `structured_grammar` (GBNF) for
-  LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
-  choice constraints are also exposed by the ABI.
-
-Model config example:
-
-```yaml
-name: qwen3-vllm
-backend: vllm-cpp
-context_size: 8192
-parameters:
-  model: Qwen3-4B   # model dir (safetensors) or .gguf file
-options:
-- max_num_seqs:16
-```
-
-Testing: `make test` runs the unit specs; export `VLLM_CPP_MODEL=<model>` (and
-optionally `VLLM_CPP_LIBRARY=<libvllm path>`) to enable the e2e specs.

backend/go/vllm-cpp/backend.go

@@ -1,245 +0,0 @@
-package main
-
-// LocalAI gRPC backend over the vllm.cpp C ABI.
-//
-// Predict maps to the blocking vllm_complete; PredictStream maps to
-// vllm_complete_stream, whose per-delta C callback bridges into the gRPC
-// stream channel. Concurrent calls are intentional: every completion entry
-// point submits into the engine's shared AsyncLLM scheduler, so parallel
-// LocalAI requests batch continuously inside the engine (the reason this
-// backend embeds base.Base and not base.SingleThread).
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"runtime"
-	"strings"
-	"sync"
-	"unsafe"
-
-	"github.com/ebitengine/purego"
-	"github.com/mudler/LocalAI/pkg/grpc/base"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	"github.com/mudler/xlog"
-)
-
-type VllmCpp struct {
-	base.Base
-
-	engine uintptr
-	opts   loadOptions
-}
-
-// Stream registry: the per-request bridge between the C token callback and
-// the gRPC stream channel, keyed by an integer handle round-tripped through
-// the C user_data pointer (never a Go pointer across the ABI). The host gRPC
-// server drains the channel even after a client disconnect, so sends here
-// cannot wedge the engine's delivery loop.
-var (
-	streamsMu   sync.Mutex
-	streams     = map[uintptr]chan string{}
-	streamNext  uintptr
-	tokenCbOnce sync.Once
-	tokenCbPtr  uintptr
-)
-
-// tokenCallback is the single C-shared callback for every stream; it
-// dispatches on the user_data handle. Returning 0 aborts the in-flight
-// request (vllm_token_callback contract).
-func tokenCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
-	streamsMu.Lock()
-	results := streams[userData]
-	streamsMu.Unlock()
-	if results == nil {
-		return 0 // unknown request: stop generation.
-	}
-	if text := goString(delta); text != "" {
-		results <- text
-	}
-	return 1
-}
-
-func registerStream(results chan string) uintptr {
-	streamsMu.Lock()
-	defer streamsMu.Unlock()
-	streamNext++
-	streams[streamNext] = results
-	return streamNext
-}
-
-func unregisterStream(h uintptr) {
-	streamsMu.Lock()
-	defer streamsMu.Unlock()
-	delete(streams, h)
-}
-
-// validModelPath enforces the greedy-probe rule: when a model config has no
-// explicit backend, the loader probes every backend with the model name, so
-// Load must refuse anything vllm.cpp cannot serve (a GGUF file, or a HF-style
-// directory with config.json + safetensors).
-func validModelPath(model string) error {
-	info, err := os.Stat(model)
-	if err != nil {
-		return fmt.Errorf("vllm-cpp: model path %q not found: %w", model, err)
-	}
-	if info.IsDir() {
-		if _, err := os.Stat(filepath.Join(model, "config.json")); err != nil {
-			return fmt.Errorf("vllm-cpp: model dir %q has no config.json", model)
-		}
-		return nil
-	}
-	if strings.EqualFold(filepath.Ext(model), ".gguf") {
-		return nil
-	}
-	return fmt.Errorf("vllm-cpp: model %q is neither a .gguf file nor a config.json model dir", model)
-}
-
-func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
-	model := opts.ModelFile
-	if model == "" {
-		model = opts.ModelPath
-	}
-	if !filepath.IsAbs(model) && opts.ModelPath != "" {
-		model = filepath.Join(opts.ModelPath, model)
-	}
-	if err := validModelPath(model); err != nil {
-		return err
-	}
-
-	v.opts = parseOptions(opts)
-
-	mp := defaultModelParams()
-	if v.opts.blockSize > 0 {
-		mp.BlockSize = v.opts.blockSize
-	}
-	if v.opts.numBlocks > 0 {
-		mp.NumBlocks = v.opts.numBlocks
-	}
-	if opts.ContextSize > 0 {
-		mp.MaxModelLen = opts.ContextSize
-	}
-	if v.opts.maxNumSeqs > 0 {
-		mp.MaxNumSeqs = v.opts.maxNumSeqs
-	}
-
-	modelC := cString(model)
-	mp.ModelPath = uintptr(unsafe.Pointer(&modelC[0])) // #nosec G103 -- borrowed by C for the load call only
-	var toolParserC, reasoningParserC []byte
-	if v.opts.toolParser != "" {
-		toolParserC = cString(v.opts.toolParser)
-		mp.ToolParser = uintptr(unsafe.Pointer(&toolParserC[0])) // #nosec G103 -- borrowed by C for the load call only
-	}
-	if v.opts.reasoningParser != "" {
-		reasoningParserC = cString(v.opts.reasoningParser)
-		mp.ReasoningParser = uintptr(unsafe.Pointer(&reasoningParserC[0])) // #nosec G103 -- borrowed by C for the load call only
-	}
-
-	xlog.Info("[vllm-cpp] Load", "model", model, "engine", vllmVersion(),
-		"blockSize", mp.BlockSize, "numBlocks", mp.NumBlocks,
-		"maxModelLen", mp.MaxModelLen, "maxNumSeqs", mp.MaxNumSeqs)
-
-	var engine uintptr
-	rc := vllmEngineLoad(unsafe.Pointer(&mp), unsafe.Pointer(&engine)) // #nosec G103 -- POD out-params
-	runtime.KeepAlive(modelC)
-	runtime.KeepAlive(toolParserC)
-	runtime.KeepAlive(reasoningParserC)
-	if rc != vllmOK {
-		return fmt.Errorf("vllm-cpp: engine load failed: %s", vllmLastError())
-	}
-	v.engine = engine
-	return nil
-}
-
-func (v *VllmCpp) Free() error {
-	if v.engine != 0 {
-		vllmEngineFree(v.engine)
-		v.engine = 0
-	}
-	return nil
-}
-
-// samplingFromPredict lowers PredictOptions into the C sampling POD plus the
-// backing buffers that must stay alive for the duration of the C call.
-func samplingFromPredict(opts *pb.PredictOptions) (sp cSamplingParams, keep []any) {
-	sp = defaultSamplingParams()
-	sp.Temperature = opts.Temperature
-	if opts.TopP > 0 {
-		sp.TopP = opts.TopP
-	}
-	if opts.TopK > 0 {
-		sp.TopK = opts.TopK
-	}
-	if opts.MinP > 0 {
-		sp.MinP = opts.MinP
-	}
-	if opts.Tokens > 0 {
-		sp.MaxTokens = opts.Tokens
-	} else {
-		sp.MaxTokens = 0 // unbounded; the engine caps at max_model_len.
-	}
-	if opts.Seed > 0 {
-		sp.Seed = uint64(opts.Seed)
-		sp.HasSeed = 1
-	}
-	sp.PresencePenalty = opts.PresencePenalty
-	sp.FrequencyPenalty = opts.FrequencyPenalty
-	if opts.Penalty > 0 {
-		sp.RepetitionPenalty = opts.Penalty
-	}
-	if opts.IgnoreEOS {
-		sp.IgnoreEOS = 1
-	}
-	if len(opts.StopPrompts) > 0 {
-		ptrs, backing := cStringArray(opts.StopPrompts)
-		sp.Stop = uintptr(unsafe.Pointer(&ptrs[0])) // #nosec G103 -- borrowed by C for the call only
-		sp.NStop = int32(len(ptrs))
-		keep = append(keep, ptrs, backing)
-	}
-	if opts.Grammar != "" {
-		g := cString(opts.Grammar)
-		sp.StructuredGrammar = uintptr(unsafe.Pointer(&g[0])) // #nosec G103 -- borrowed by C for the call only
-		keep = append(keep, g)
-	}
-	return sp, keep
-}
-
-func (v *VllmCpp) Predict(opts *pb.PredictOptions) (string, error) {
-	if v.engine == 0 {
-		return "", fmt.Errorf("vllm-cpp: model not loaded")
-	}
-	sp, keep := samplingFromPredict(opts)
-	var out cCompletion
-	rc := vllmComplete(v.engine, opts.Prompt, unsafe.Pointer(&sp), unsafe.Pointer(&out)) // #nosec G103 -- POD in/out params
-	runtime.KeepAlive(keep)
-	if rc != vllmOK {
-		return "", fmt.Errorf("vllm-cpp: completion failed: %s", vllmLastError())
-	}
-	text := goString(out.Text)
-	vllmCompletionFree(unsafe.Pointer(&out)) // #nosec G103 -- frees out.Text
-	return text, nil
-}
-
-func (v *VllmCpp) PredictStream(opts *pb.PredictOptions, results chan string) error {
-	if v.engine == 0 {
-		close(results)
-		return fmt.Errorf("vllm-cpp: model not loaded")
-	}
-	tokenCbOnce.Do(func() {
-		tokenCbPtr = purego.NewCallback(tokenCallback)
-	})
-
-	sp, keep := samplingFromPredict(opts)
-	handle := registerStream(results)
-
-	go func() {
-		defer close(results)
-		defer unregisterStream(handle)
-		rc := vllmCompleteStream(v.engine, opts.Prompt, unsafe.Pointer(&sp), tokenCbPtr, handle) // #nosec G103 -- POD in-params
-		runtime.KeepAlive(keep)
-		if rc != vllmOK {
-			xlog.Error("[vllm-cpp] stream failed", "error", vllmLastError())
-		}
-	}()
-	return nil
-}

backend/go/vllm-cpp/chat.go

@@ -1,289 +0,0 @@
-package main
-
-// The rich chat path (AIModelRich): rides the ENGINE's serving pipeline via
-// the ABI v3 chat entry points, exactly like the llama-cpp autoparser flow.
-// The engine applies the model's chat template, decides when a tool call
-// engages (tool_choice auto lowers to a LAZY structural-tag decode
-// constraint), parses tool calls with its streaming-stateful Hermes-style
-// parser, and hands back chat.completion.chunk JSON that this file maps 1:1
-// onto pb.Reply ChatDelta / ToolCallDelta.
-
-import (
-	"encoding/json"
-	"fmt"
-	"sync"
-	"unsafe"
-
-	"github.com/ebitengine/purego"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	"github.com/mudler/xlog"
-)
-
-// useChatPath reports whether the request should go through the engine-side
-// chat pipeline: the model config asked for backend-side templating and the
-// host handed us structured messages.
-func useChatPath(opts *pb.PredictOptions) bool {
-	return opts.UseTokenizerTemplate && len(opts.Messages) > 0
-}
-
-// chatRequestJSON lowers PredictOptions into one OpenAI chat-completions
-// request object for the ABI (the engine ignores `model`/`stream`).
-func chatRequestJSON(opts *pb.PredictOptions, stream bool) (string, error) {
-	messages := make([]map[string]any, 0, len(opts.Messages))
-	for _, m := range opts.Messages {
-		msg := map[string]any{"role": m.Role, "content": m.Content}
-		if m.ToolCalls != "" {
-			var toolCalls any
-			if err := json.Unmarshal([]byte(m.ToolCalls), &toolCalls); err == nil {
-				msg["tool_calls"] = toolCalls
-			}
-		}
-		// Multi-turn tool identity + prior reasoning: a role="tool" reply
-		// carries the id (and optionally the name) of the assistant call it
-		// answers, and assistant history may carry its reasoning span. The
-		// engine's template context needs all three or a second turn after
-		// tool execution is malformed.
-		if m.ToolCallId != "" {
-			msg["tool_call_id"] = m.ToolCallId
-		}
-		if m.Name != "" {
-			msg["name"] = m.Name
-		}
-		if m.ReasoningContent != "" {
-			msg["reasoning"] = m.ReasoningContent
-		}
-		messages = append(messages, msg)
-	}
-	req := map[string]any{"messages": messages}
-
-	if opts.Tools != "" {
-		var tools any
-		if err := json.Unmarshal([]byte(opts.Tools), &tools); err != nil {
-			return "", fmt.Errorf("vllm-cpp: tools is not valid JSON: %w", err)
-		}
-		req["tools"] = tools
-	}
-	if opts.ToolChoice != "" {
-		var choice any
-		// ToolChoice arrives either as a bare string ("auto"/"required"/"none")
-		// or as the OpenAI named-function JSON object.
-		if err := json.Unmarshal([]byte(opts.ToolChoice), &choice); err == nil {
-			req["tool_choice"] = choice
-		} else {
-			req["tool_choice"] = opts.ToolChoice
-		}
-	}
-
-	req["temperature"] = opts.Temperature
-	if opts.TopP > 0 {
-		req["top_p"] = opts.TopP
-	}
-	if opts.TopK > 0 {
-		req["top_k"] = opts.TopK
-	}
-	if opts.Tokens > 0 {
-		req["max_tokens"] = opts.Tokens
-	}
-	if opts.Seed > 0 {
-		req["seed"] = opts.Seed
-	}
-	if len(opts.StopPrompts) > 0 {
-		req["stop"] = opts.StopPrompts
-	}
-	if opts.PresencePenalty != 0 {
-		req["presence_penalty"] = opts.PresencePenalty
-	}
-	if opts.FrequencyPenalty != 0 {
-		req["frequency_penalty"] = opts.FrequencyPenalty
-	}
-	if stream {
-		// The engine's request parser validates stream_options against the
-		// stream flag at parse time (before the ABI entry point forces it),
-		// so state the intent explicitly.
-		req["stream"] = true
-		req["stream_options"] = map[string]any{"include_usage": true}
-	}
-
-	b, err := json.Marshal(req)
-	if err != nil {
-		return "", err
-	}
-	return string(b), nil
-}
-
-// chatChunk is the subset of an OpenAI chat.completion(.chunk) object the
-// backend consumes.
-type chatChunk struct {
-	Object  string `json:"object"`
-	Choices []struct {
-		Delta        *chatDelta `json:"delta"`   // streaming chunks
-		Message      *chatDelta `json:"message"` // non-stream response
-		FinishReason string     `json:"finish_reason"`
-	} `json:"choices"`
-	Usage *struct {
-		PromptTokens     int32 `json:"prompt_tokens"`
-		CompletionTokens int32 `json:"completion_tokens"`
-	} `json:"usage"`
-}
-
-type chatDelta struct {
-	Content          string `json:"content"`
-	ReasoningContent string `json:"reasoning"`
-	ToolCalls        []struct {
-		Index    int32  `json:"index"`
-		ID       string `json:"id"`
-		Function struct {
-			Name      string `json:"name"`
-			Arguments string `json:"arguments"`
-		} `json:"function"`
-	} `json:"tool_calls"`
-}
-
-// toReply maps one parsed chunk onto a pb.Reply carrying the content bytes
-// plus the structured ChatDelta (the host prefers ChatDeltas when present).
-func (c *chatChunk) toReply() *pb.Reply {
-	reply := &pb.Reply{}
-	if c.Usage != nil {
-		reply.PromptTokens = c.Usage.PromptTokens
-		reply.Tokens = c.Usage.CompletionTokens
-	}
-	if len(c.Choices) == 0 {
-		return reply
-	}
-	d := c.Choices[0].Delta
-	if d == nil {
-		d = c.Choices[0].Message
-	}
-	if d == nil {
-		return reply
-	}
-	delta := &pb.ChatDelta{
-		Content:          d.Content,
-		ReasoningContent: d.ReasoningContent,
-	}
-	for _, tc := range d.ToolCalls {
-		delta.ToolCalls = append(delta.ToolCalls, &pb.ToolCallDelta{
-			Index:     tc.Index,
-			Id:        tc.ID,
-			Name:      tc.Function.Name,
-			Arguments: tc.Function.Arguments,
-		})
-	}
-	reply.Message = []byte(d.Content)
-	if delta.Content != "" || delta.ReasoningContent != "" ||
-		len(delta.ToolCalls) > 0 {
-		reply.ChatDeltas = []*pb.ChatDelta{delta}
-	}
-	return reply
-}
-
-// Chat-stream registry: chunk JSON arrives on the engine's delivery thread
-// through one shared C callback; the integer handle in user_data selects the
-// destination channel (never a Go pointer across the ABI).
-var (
-	chatStreamsMu  sync.Mutex
-	chatStreams    = map[uintptr]chan<- *pb.Reply{}
-	chatStreamNext uintptr
-	chatCbOnce     sync.Once
-	chatCbPtr      uintptr
-)
-
-func chatCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
-	chatStreamsMu.Lock()
-	results := chatStreams[userData]
-	chatStreamsMu.Unlock()
-	if results == nil {
-		return 0
-	}
-	_ = finished // the terminal call carries an empty delta; nothing to emit.
-	payload := goString(delta)
-	if payload == "" {
-		return 1
-	}
-	var chunk chatChunk
-	if err := json.Unmarshal([]byte(payload), &chunk); err != nil {
-		xlog.Error("[vllm-cpp] unparseable chat chunk", "error", err)
-		return 1
-	}
-	results <- chunk.toReply()
-	return 1
-}
-
-func registerChatStream(results chan<- *pb.Reply) uintptr {
-	chatStreamsMu.Lock()
-	defer chatStreamsMu.Unlock()
-	chatStreamNext++
-	chatStreams[chatStreamNext] = results
-	return chatStreamNext
-}
-
-func unregisterChatStream(h uintptr) {
-	chatStreamsMu.Lock()
-	defer chatStreamsMu.Unlock()
-	delete(chatStreams, h)
-}
-
-// PredictRich implements the non-streaming rich path. Without structured
-// messages it falls back to the plain Predict flow (LocalAI-side templating,
-// optional grammar constraint).
-func (v *VllmCpp) PredictRich(opts *pb.PredictOptions) (*pb.Reply, error) {
-	if !useChatPath(opts) {
-		text, err := v.Predict(opts)
-		if err != nil {
-			return nil, err
-		}
-		return &pb.Reply{Message: []byte(text)}, nil
-	}
-	if v.engine == 0 {
-		return nil, fmt.Errorf("vllm-cpp: model not loaded")
-	}
-	request, err := chatRequestJSON(opts, false)
-	if err != nil {
-		return nil, err
-	}
-	var out uintptr
-	rc := vllmChat(v.engine, request, unsafe.Pointer(&out)) // #nosec G103 -- char** out-param
-	if rc != vllmOK {
-		return nil, fmt.Errorf("vllm-cpp: chat failed: %s", vllmLastError())
-	}
-	payload := goString(out)
-	vllmStringFree(out)
-	var response chatChunk
-	if err := json.Unmarshal([]byte(payload), &response); err != nil {
-		return nil, fmt.Errorf("vllm-cpp: unparseable chat response: %w", err)
-	}
-	return response.toReply(), nil
-}
-
-// PredictStreamRich implements the streaming rich path. Contract: send into
-// the channel and return when finished; the host closes the channel.
-func (v *VllmCpp) PredictStreamRich(opts *pb.PredictOptions, results chan<- *pb.Reply) error {
-	if !useChatPath(opts) {
-		// Legacy bridge: run the plain stream and wrap deltas.
-		plain := make(chan string)
-		if err := v.PredictStream(opts, plain); err != nil {
-			return err
-		}
-		for delta := range plain {
-			results <- &pb.Reply{Message: []byte(delta)}
-		}
-		return nil
-	}
-	if v.engine == 0 {
-		return fmt.Errorf("vllm-cpp: model not loaded")
-	}
-	request, err := chatRequestJSON(opts, true)
-	if err != nil {
-		return err
-	}
-	chatCbOnce.Do(func() {
-		chatCbPtr = purego.NewCallback(chatCallback)
-	})
-	handle := registerChatStream(results)
-	defer unregisterChatStream(handle)
-	rc := vllmChatStream(v.engine, request, chatCbPtr, handle)
-	if rc != vllmOK {
-		return fmt.Errorf("vllm-cpp: chat stream failed: %s", vllmLastError())
-	}
-	return nil
-}

backend/go/vllm-cpp/chat_test.go

@@ -1,158 +0,0 @@
-package main
-
-import (
-	"encoding/json"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("useChatPath", func() {
-	It("requires tokenizer templating AND structured messages", func() {
-		Expect(useChatPath(&pb.PredictOptions{})).To(BeFalse())
-		Expect(useChatPath(&pb.PredictOptions{UseTokenizerTemplate: true})).To(BeFalse())
-		Expect(useChatPath(&pb.PredictOptions{
-			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
-		})).To(BeFalse())
-		Expect(useChatPath(&pb.PredictOptions{
-			UseTokenizerTemplate: true,
-			Messages:             []*pb.Message{{Role: "user", Content: "hi"}},
-		})).To(BeTrue())
-	})
-})
-
-var _ = Describe("chatRequestJSON", func() {
-	It("lowers messages, tools, tool_choice and sampling onto one request", func() {
-		out, err := chatRequestJSON(&pb.PredictOptions{
-			UseTokenizerTemplate: true,
-			Messages: []*pb.Message{
-				{Role: "system", Content: "be brief"},
-				{Role: "user", Content: "weather in Rome?"},
-			},
-			Tools:       `[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object"}}}]`,
-			ToolChoice:  "required",
-			Temperature: 0.2,
-			TopP:        0.9,
-			Tokens:      64,
-			StopPrompts: []string{"<|im_end|>"},
-		}, false)
-		Expect(err).NotTo(HaveOccurred())
-
-		var req map[string]any
-		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
-		Expect(req["messages"]).To(HaveLen(2))
-		Expect(req["tools"]).To(HaveLen(1))
-		Expect(req["tool_choice"]).To(Equal("required"))
-		Expect(req["max_tokens"]).To(BeNumerically("==", 64))
-		Expect(req["top_p"]).To(BeNumerically("~", 0.9, 1e-6))
-		Expect(req["stop"]).To(ConsistOf("<|im_end|>"))
-		Expect(req).NotTo(HaveKey("stream_options"))
-	})
-
-	It("parses a named-function tool_choice object and asks for stream usage", func() {
-		out, err := chatRequestJSON(&pb.PredictOptions{
-			Messages:   []*pb.Message{{Role: "user", Content: "hi"}},
-			ToolChoice: `{"type":"function","function":{"name":"get_weather"}}`,
-		}, true)
-		Expect(err).NotTo(HaveOccurred())
-
-		var req map[string]any
-		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
-		choice, ok := req["tool_choice"].(map[string]any)
-		Expect(ok).To(BeTrue())
-		Expect(choice["type"]).To(Equal("function"))
-		Expect(req["stream_options"]).To(HaveKeyWithValue("include_usage", true))
-	})
-
-	It("rejects malformed tools JSON", func() {
-		_, err := chatRequestJSON(&pb.PredictOptions{
-			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
-			Tools:    "{not json",
-		}, false)
-		Expect(err).To(HaveOccurred())
-	})
-})
-
-var _ = Describe("chatChunk.toReply", func() {
-	It("maps a streaming tool-call delta onto ChatDelta/ToolCallDelta", func() {
-		var chunk chatChunk
-		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{
-			"tool_calls":[{"index":0,"id":"call_1","function":{"name":"get_weather","arguments":"{\"city\":"}}]
-		},"finish_reason":null}]}`
-		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
-
-		reply := chunk.toReply()
-		Expect(reply.ChatDeltas).To(HaveLen(1))
-		Expect(reply.ChatDeltas[0].ToolCalls).To(HaveLen(1))
-		tc := reply.ChatDeltas[0].ToolCalls[0]
-		Expect(tc.Name).To(Equal("get_weather"))
-		Expect(tc.Id).To(Equal("call_1"))
-		Expect(tc.Arguments).To(Equal(`{"city":`))
-	})
-
-	It("maps a non-stream response message and usage", func() {
-		var chunk chatChunk
-		payload := `{"object":"chat.completion","choices":[{"message":{
-			"role":"assistant","content":"Sunny."},"finish_reason":"stop"}],
-			"usage":{"prompt_tokens":12,"completion_tokens":3}}`
-		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
-
-		reply := chunk.toReply()
-		Expect(string(reply.Message)).To(Equal("Sunny."))
-		Expect(reply.ChatDeltas).To(HaveLen(1))
-		Expect(reply.ChatDeltas[0].Content).To(Equal("Sunny."))
-		Expect(reply.PromptTokens).To(BeNumerically("==", 12))
-		Expect(reply.Tokens).To(BeNumerically("==", 3))
-	})
-
-	It("emits no ChatDelta for an empty role-only chunk", func() {
-		var chunk chatChunk
-		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":""}}]}`
-		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
-		Expect(chunk.toReply().ChatDeltas).To(BeEmpty())
-	})
-})
-
-var _ = Describe("chatRequestJSON multi-turn tool round trip", func() {
-	It("forwards tool_call_id, name, reasoning and assistant tool_calls", func() {
-		out, err := chatRequestJSON(&pb.PredictOptions{
-			UseTokenizerTemplate: true,
-			Messages: []*pb.Message{
-				{Role: "user", Content: "What is the weather in Rome?"},
-				{
-					Role:             "assistant",
-					ReasoningContent: "need the weather tool",
-					ToolCalls:        `[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Rome\"}"}}]`,
-				},
-				{Role: "tool", ToolCallId: "call_1", Name: "get_weather", Content: `{"temp": 21}`},
-			},
-		}, false)
-		Expect(err).NotTo(HaveOccurred())
-
-		var req struct {
-			Messages []map[string]any `json:"messages"`
-		}
-		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
-		Expect(req.Messages).To(HaveLen(3))
-
-		assistant := req.Messages[1]
-		Expect(assistant["reasoning"]).To(Equal("need the weather tool"))
-		calls, ok := assistant["tool_calls"].([]any)
-		Expect(ok).To(BeTrue())
-		Expect(calls).To(HaveLen(1))
-		call := calls[0].(map[string]any)
-		Expect(call["id"]).To(Equal("call_1"))
-
-		tool := req.Messages[2]
-		Expect(tool["role"]).To(Equal("tool"))
-		Expect(tool["tool_call_id"]).To(Equal("call_1"))
-		Expect(tool["name"]).To(Equal("get_weather"))
-		Expect(tool["content"]).To(Equal(`{"temp": 21}`))
-
-		user := req.Messages[0]
-		Expect(user).NotTo(HaveKey("tool_call_id"))
-		Expect(user).NotTo(HaveKey("name"))
-		Expect(user).NotTo(HaveKey("reasoning"))
-	})
-})

backend/go/vllm-cpp/e2e_test.go

@@ -1,281 +0,0 @@
-package main
-
-// E2E over a real model + the built libvllm. Gated on VLLM_CPP_MODEL (a .gguf
-// file or a safetensors model dir): without it the suite skips, so CI runs
-// only the unit specs. test.sh auto-downloads a small GGUF when the gate is
-// unset and the download is allowed.
-
-import (
-	"encoding/json"
-	"os"
-	"runtime"
-	"strings"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("vllm-cpp e2e", Label("e2e"), Ordered, func() {
-	var backend *VllmCpp
-
-	BeforeAll(func() {
-		modelPath := os.Getenv("VLLM_CPP_MODEL")
-		if modelPath == "" {
-			Skip("VLLM_CPP_MODEL not set; skipping e2e")
-		}
-		lib := os.Getenv("VLLM_CPP_LIBRARY")
-		if lib == "" {
-			if runtime.GOOS == "darwin" {
-				lib = "./libvllm.dylib"
-			} else {
-				lib = "./libvllm.so"
-			}
-		}
-		Expect(registerLib(lib)).To(Succeed())
-
-		backend = &VllmCpp{}
-		Expect(backend.Load(&pb.ModelOptions{
-			ModelFile:   modelPath,
-			ContextSize: 2048,
-		})).To(Succeed())
-	})
-
-	AfterAll(func() {
-		if backend != nil {
-			Expect(backend.Free()).To(Succeed())
-		}
-	})
-
-	It("refuses a foreign model artefact", func() {
-		other := &VllmCpp{}
-		Expect(other.Load(&pb.ModelOptions{ModelFile: "/nonexistent/foreign.bin"})).NotTo(Succeed())
-	})
-
-	It("completes a prompt (greedy)", func() {
-		text, err := backend.Predict(&pb.PredictOptions{
-			Prompt:      "The capital of France is",
-			Tokens:      16,
-			Temperature: 0,
-		})
-		Expect(err).NotTo(HaveOccurred())
-		Expect(text).NotTo(BeEmpty())
-	})
-
-	It("is deterministic under greedy decoding", func() {
-		opts := &pb.PredictOptions{Prompt: "1 2 3 4", Tokens: 8, Temperature: 0}
-		a, err := backend.Predict(opts)
-		Expect(err).NotTo(HaveOccurred())
-		b, err := backend.Predict(opts)
-		Expect(err).NotTo(HaveOccurred())
-		Expect(a).To(Equal(b))
-	})
-
-	It("streams deltas that concatenate to the blocking result", func() {
-		opts := &pb.PredictOptions{Prompt: "Count: one two", Tokens: 12, Temperature: 0}
-		blocking, err := backend.Predict(opts)
-		Expect(err).NotTo(HaveOccurred())
-
-		results := make(chan string)
-		Expect(backend.PredictStream(opts, results)).To(Succeed())
-		var sb strings.Builder
-		for delta := range results {
-			sb.WriteString(delta)
-		}
-		Expect(sb.String()).To(Equal(blocking))
-	})
-
-	It("honors stop words", func() {
-		text, err := backend.Predict(&pb.PredictOptions{
-			Prompt:      "a b c d e f",
-			Tokens:      64,
-			Temperature: 0,
-			StopPrompts: []string{"g"},
-		})
-		Expect(err).NotTo(HaveOccurred())
-		Expect(text).NotTo(ContainSubstring("g h"))
-	})
-
-	It("constrains generation with a GBNF grammar (tool-call path)", func() {
-		text, err := backend.Predict(&pb.PredictOptions{
-			Prompt:      "Answer strictly yes or no: is water wet?",
-			Tokens:      4,
-			Temperature: 0,
-			Grammar:     "root ::= \"yes\" | \"no\"",
-		})
-		Expect(err).NotTo(HaveOccurred())
-		Expect(text).To(Or(HavePrefix("yes"), HavePrefix("no")))
-	})
-
-	It("serves concurrent streams", func() {
-		const n = 4
-		type result struct {
-			text string
-			err  error
-		}
-		done := make(chan result, n)
-		for i := 0; i < n; i++ {
-			go func() {
-				results := make(chan string)
-				err := backend.PredictStream(&pb.PredictOptions{
-					Prompt: "Hello", Tokens: 8, Temperature: 0,
-				}, results)
-				var sb strings.Builder
-				for delta := range results {
-					sb.WriteString(delta)
-				}
-				done <- result{sb.String(), err}
-			}()
-		}
-		for i := 0; i < n; i++ {
-			r := <-done
-			Expect(r.err).NotTo(HaveOccurred())
-			Expect(r.text).NotTo(BeEmpty())
-		}
-	})
-})
-
-var _ = Describe("vllm-cpp chat e2e", Label("e2e"), Ordered, func() {
-	var backend *VllmCpp
-
-	weatherTools := `[{"type":"function","function":{"name":"get_weather",` +
-		`"description":"Get the current weather for a city.",` +
-		`"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]`
-
-	BeforeAll(func() {
-		modelPath := os.Getenv("VLLM_CPP_MODEL")
-		if modelPath == "" {
-			Skip("VLLM_CPP_MODEL not set; skipping chat e2e")
-		}
-		lib := os.Getenv("VLLM_CPP_LIBRARY")
-		if lib == "" {
-			if runtime.GOOS == "darwin" {
-				lib = "./libvllm.dylib"
-			} else {
-				lib = "./libvllm.so"
-			}
-		}
-		Expect(registerLib(lib)).To(Succeed())
-
-		backend = &VllmCpp{}
-		Expect(backend.Load(&pb.ModelOptions{
-			ModelFile:   modelPath,
-			ContextSize: 2048,
-		})).To(Succeed())
-	})
-
-	AfterAll(func() {
-		if backend != nil {
-			Expect(backend.Free()).To(Succeed())
-		}
-	})
-
-	chatOpts := func() *pb.PredictOptions {
-		return &pb.PredictOptions{
-			UseTokenizerTemplate: true,
-			Messages: []*pb.Message{
-				{Role: "user", Content: "Reply with one short sentence: what is the capital of France?"},
-			},
-			Tokens:      512,
-			Temperature: 0,
-		}
-	}
-
-	It("answers a plain chat turn through the engine-side template", func() {
-		reply, err := backend.PredictRich(chatOpts())
-		Expect(err).NotTo(HaveOccurred())
-		Expect(string(reply.Message)).NotTo(BeEmpty())
-		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
-	})
-
-	It("splits reasoning from content engine-side (auto-detected from the template)", func() {
-		// The Qwen3.5 chat template carries <think>, so the engine auto-selects
-		// the think_auto reasoning parser: a markerless answer stays pure
-		// content; if the model DOES think, the block arrives as
-		// ReasoningContent and never leaks into Message.
-		reply, err := backend.PredictRich(chatOpts())
-		Expect(err).NotTo(HaveOccurred())
-		Expect(string(reply.Message)).NotTo(ContainSubstring("<think>"))
-		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
-		for _, d := range reply.ChatDeltas {
-			Expect(d.ReasoningContent).NotTo(ContainSubstring("Paris"),
-				"the user-visible answer must not be swallowed into reasoning")
-		}
-	})
-
-	It("streams chat deltas that concatenate to the blocking answer", func() {
-		blocking, err := backend.PredictRich(chatOpts())
-		Expect(err).NotTo(HaveOccurred())
-
-		results := make(chan *pb.Reply, 64)
-		done := make(chan error, 1)
-		go func() {
-			done <- backend.PredictStreamRich(chatOpts(), results)
-			close(results)
-		}()
-		var sb strings.Builder
-		for r := range results {
-			sb.WriteString(string(r.Message))
-		}
-		Expect(<-done).To(Succeed())
-		Expect(sb.String()).To(Equal(string(blocking.Message)))
-	})
-
-	It("emits a parsed tool call when tool_choice requires it", func() {
-		opts := chatOpts()
-		opts.Messages = []*pb.Message{
-			{Role: "user", Content: "What is the weather in Rome right now?"},
-		}
-		opts.Tools = weatherTools
-		opts.ToolChoice = "required"
-		opts.Tokens = 256
-
-		reply, err := backend.PredictRich(opts)
-		Expect(err).NotTo(HaveOccurred())
-		Expect(reply.ChatDeltas).NotTo(BeEmpty())
-		var name, args string
-		for _, d := range reply.ChatDeltas {
-			for _, tc := range d.ToolCalls {
-				if tc.Name != "" {
-					name = tc.Name
-				}
-				args += tc.Arguments
-			}
-		}
-		Expect(name).To(Equal("get_weather"))
-		var parsed map[string]any
-		Expect(json.Unmarshal([]byte(args), &parsed)).To(Succeed(),
-			"tool arguments must be valid JSON: %q", args)
-		Expect(parsed).To(HaveKey("city"))
-	})
-
-	It("lets the engine decide on auto tool choice and streams tool deltas", func() {
-		opts := chatOpts()
-		opts.Messages = []*pb.Message{
-			{Role: "user", Content: "Use the get_weather tool to check the weather in Rome."},
-		}
-		opts.Tools = weatherTools
-		opts.Tokens = 512
-
-		results := make(chan *pb.Reply, 128)
-		done := make(chan error, 1)
-		go func() {
-			done <- backend.PredictStreamRich(opts, results)
-			close(results)
-		}()
-		sawToolDelta := false
-		for r := range results {
-			for _, d := range r.ChatDeltas {
-				if len(d.ToolCalls) > 0 {
-					sawToolDelta = true
-				}
-			}
-		}
-		Expect(<-done).To(Succeed())
-		// tool_choice auto is a LAZY constraint: the model may or may not call.
-		// With an explicit instruction the gate model reliably does; treat a
-		// no-call run as a soft signal rather than a hard failure only if the
-		// engine produced SOME output.
-		Expect(sawToolDelta).To(BeTrue(), "expected the engine to engage the tool")
-	})
-})

backend/go/vllm-cpp/govllmcpp.go

@@ -1,182 +0,0 @@
-package main
-
-// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v2).
-//
-// The structs below are hand-mirrored PODs of the C declarations, with
-// explicit padding so the Go layout matches the C layout on linux/darwin
-// amd64+arm64. Struct-by-value entry points (the *_default helpers) are NOT
-// bound - purego's struct-return support is platform-dependent - so the
-// defaults are replicated here and guarded by the vllm_abi_version check at
-// startup: a library whose ABI differs from what these mirrors were written
-// against is refused before any request runs.
-
-import (
-	"fmt"
-	"unsafe"
-
-	"github.com/ebitengine/purego"
-)
-
-// abiVersion is the VLLM_ABI_VERSION this file mirrors (vllm.h).
-const abiVersion = 5
-
-// vllm_status (vllm.h).
-const (
-	vllmOK = 0
-)
-
-// cModelParams mirrors vllm_model_params.
-type cModelParams struct {
-	ModelPath           uintptr // const char*
-	TokenizerConfigPath uintptr // const char*
-	BlockSize           int32
-	NumBlocks           int32
-	MaxModelLen         int32
-	MaxNumSeqs          int32
-	ToolParser          uintptr // const char*; NULL = auto-detect (ABI v4)
-	ReasoningParser     uintptr // const char*; NULL = auto-detect (ABI v5)
-}
-
-// cSamplingParams mirrors vllm_sampling_params (ABI v2, structured fields
-// included). Padding matches the C compiler's: the uint64 seed is 8-aligned,
-// and each pointer following an int32 is 8-aligned.
-type cSamplingParams struct {
-	Temperature          float32
-	TopP                 float32
-	TopK                 int32
-	MinP                 float32
-	MaxTokens            int32
-	_                    [4]byte
-	Seed                 uint64
-	HasSeed              int32
-	PresencePenalty      float32
-	FrequencyPenalty     float32
-	RepetitionPenalty    float32
-	MinTokens            int32
-	IgnoreEOS            int32
-	Stop                 uintptr // const char* const*
-	NStop                int32
-	_                    [4]byte
-	StructuredJSON       uintptr // const char*
-	StructuredRegex      uintptr // const char*
-	StructuredChoice     uintptr // const char* const*
-	NStructuredChoice    int32
-	_                    [4]byte
-	StructuredGrammar    uintptr // const char*
-	StructuredJSONObject int32
-	_                    [4]byte
-}
-
-// cCompletion mirrors vllm_completion.
-type cCompletion struct {
-	Text             uintptr // char*, caller-owned
-	FinishReason     uintptr // const char*, library-owned
-	PromptTokens     int32
-	CompletionTokens int32
-}
-
-// defaultSamplingParams mirrors vllm_sampling_params_default().
-func defaultSamplingParams() cSamplingParams {
-	return cSamplingParams{
-		Temperature:       1.0,
-		TopP:              1.0,
-		MaxTokens:         16,
-		RepetitionPenalty: 1.0,
-	}
-}
-
-// defaultModelParams mirrors vllm_model_params_default().
-func defaultModelParams() cModelParams {
-	return cModelParams{
-		BlockSize:  32,
-		NumBlocks:  256,
-		MaxNumSeqs: 8,
-	}
-}
-
-var (
-	vllmEngineLoad     func(params, out unsafe.Pointer) int32
-	vllmEngineFree     func(engine uintptr)
-	vllmComplete       func(engine uintptr, prompt string, params, out unsafe.Pointer) int32
-	vllmCompleteStream func(engine uintptr, prompt string, params unsafe.Pointer, cb uintptr, userData uintptr) int32
-	vllmChat           func(engine uintptr, requestJSON string, out unsafe.Pointer) int32
-	vllmChatStream     func(engine uintptr, requestJSON string, cb uintptr, userData uintptr) int32
-	vllmStringFree     func(s uintptr)
-	vllmCompletionFree func(out unsafe.Pointer)
-	vllmLastError      func() string
-	vllmVersion        func() string
-	vllmABIVersion     func() int32
-)
-
-type libFunc struct {
-	ptr  any
-	name string
-}
-
-// registerLib dlopens libvllm and binds the C ABI, refusing an ABI-version
-// mismatch (the struct mirrors above would be undefined behavior against a
-// different layout).
-func registerLib(libName string) error {
-	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-	if err != nil {
-		return fmt.Errorf("vllm-cpp: dlopen %s: %w", libName, err)
-	}
-	for _, lf := range []libFunc{
-		{&vllmEngineLoad, "vllm_engine_load"},
-		{&vllmEngineFree, "vllm_engine_free"},
-		{&vllmComplete, "vllm_complete"},
-		{&vllmCompleteStream, "vllm_complete_stream"},
-		{&vllmChat, "vllm_chat"},
-		{&vllmChatStream, "vllm_chat_stream"},
-		{&vllmStringFree, "vllm_string_free"},
-		{&vllmCompletionFree, "vllm_completion_free"},
-		{&vllmLastError, "vllm_last_error"},
-		{&vllmVersion, "vllm_version"},
-		{&vllmABIVersion, "vllm_abi_version"},
-	} {
-		purego.RegisterLibFunc(lf.ptr, lib, lf.name)
-	}
-	if v := vllmABIVersion(); v != abiVersion {
-		return fmt.Errorf("vllm-cpp: ABI mismatch: library reports v%d, backend built against v%d", v, abiVersion)
-	}
-	return nil
-}
-
-// cString returns a NUL-terminated byte slice for s. The backing array may be
-// passed to C for the duration of a call (the ABI borrows and copies); keep it
-// alive across the call with runtime.KeepAlive.
-func cString(s string) []byte {
-	b := make([]byte, len(s)+1)
-	copy(b, s)
-	return b
-}
-
-// cStringArray builds a NULL-free array of C-string pointers plus the backing
-// buffers that must stay alive for the duration of the C call.
-func cStringArray(ss []string) (ptrs []uintptr, backing [][]byte) {
-	backing = make([][]byte, 0, len(ss))
-	ptrs = make([]uintptr, 0, len(ss))
-	for _, s := range ss {
-		b := cString(s)
-		backing = append(backing, b)
-		ptrs = append(ptrs, uintptr(unsafe.Pointer(&b[0]))) // #nosec G103 -- borrowed by C for the call only
-	}
-	return ptrs, backing
-}
-
-// goString copies a NUL-terminated C string.
-func goString(p uintptr) string {
-	if p == 0 {
-		return ""
-	}
-	//nolint:govet // C-owned pointer handed over by purego, valid for this call
-	base := unsafe.Pointer(p) // #nosec G103 -- C-owned, copied out immediately
-	n := 0
-	for *(*byte)(unsafe.Add(base, n)) != 0 {
-		n++
-	}
-	if n == 0 {
-		return ""
-	}
-	return string(unsafe.Slice((*byte)(base), n))
-}

backend/go/vllm-cpp/main.go

@@ -1,35 +0,0 @@
-package main
-
-// Note: this is started internally by LocalAI and a server is allocated for each model
-import (
-	"flag"
-	"os"
-	"runtime"
-
-	grpc "github.com/mudler/LocalAI/pkg/grpc"
-)
-
-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
-)
-
-func main() {
-	libName := os.Getenv("VLLM_CPP_LIBRARY")
-	if libName == "" {
-		if runtime.GOOS == "darwin" {
-			libName = "./libvllm.dylib"
-		} else {
-			libName = "./libvllm.so"
-		}
-	}
-
-	if err := registerLib(libName); err != nil {
-		panic(err)
-	}
-
-	flag.Parse()
-
-	if err := grpc.StartServer(*addr, &VllmCpp{}); err != nil {
-		panic(err)
-	}
-}

backend/go/vllm-cpp/options.go

@@ -1,54 +0,0 @@
-package main
-
-// Engine-sizing knobs carried through the model config's free-form
-// `options:` list ("key:value" entries), mirroring how the other in-house
-// backends pass engine-specific settings that have no proto field.
-
-import (
-	"strconv"
-	"strings"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-)
-
-type loadOptions struct {
-	blockSize  int32 // KV block size (tokens/block); engine default 32.
-	numBlocks  int32 // KV blocks to allocate; engine default 256.
-	maxNumSeqs int32 // max concurrent sequences; engine default 8.
-	// Engine-side parser selection (ABI v4/v5). Empty = the engine
-	// auto-detects from the chat template; "none" disables the reasoning
-	// split; unknown names fail the first chat call.
-	toolParser      string
-	reasoningParser string
-}
-
-func parseOptions(opts *pb.ModelOptions) loadOptions {
-	lo := loadOptions{}
-	for _, o := range opts.GetOptions() {
-		k, v, found := strings.Cut(o, ":")
-		if !found {
-			continue
-		}
-		switch strings.TrimSpace(k) {
-		case "block_size":
-			lo.blockSize = parseInt32(v, lo.blockSize)
-		case "num_blocks":
-			lo.numBlocks = parseInt32(v, lo.numBlocks)
-		case "max_num_seqs":
-			lo.maxNumSeqs = parseInt32(v, lo.maxNumSeqs)
-		case "tool_parser":
-			lo.toolParser = strings.TrimSpace(v)
-		case "reasoning_parser":
-			lo.reasoningParser = strings.TrimSpace(v)
-		}
-	}
-	return lo
-}
-
-func parseInt32(s string, fallback int32) int32 {
-	n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 32)
-	if err != nil || n <= 0 {
-		return fallback
-	}
-	return int32(n)
-}

backend/go/vllm-cpp/package.sh

@@ -1,61 +0,0 @@
-#!/bin/bash
-
-# Script to copy the appropriate libraries based on architecture
-# This script is used in the final stage of the Dockerfile
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-# Create lib directory
-mkdir -p $CURDIR/package/lib
-
-cp -avf $CURDIR/vllm-cpp $CURDIR/package/
-cp -fLv $CURDIR/libvllm.so $CURDIR/package/ 2>/dev/null || true
-cp -fLv $CURDIR/libvllm.dylib $CURDIR/package/ 2>/dev/null || true
-cp -fv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    # x86_64 architecture
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    # ARM64 architecture
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ $(uname -s) = "Darwin" ]; then
-    echo "Detected Darwin"
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/go/vllm-cpp/run.sh

@@ -1,29 +0,0 @@
-#!/bin/bash
-set -ex
-
-# Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath "$0")")
-
-cd /
-
-# vllm.cpp ships ONE portable library per platform (SIMD tiers are per-file
-# with runtime dispatch), so there is no per-CPU variant probing here.
-if [ "$(uname)" = "Darwin" ]; then
-	LIBRARY="$CURDIR/libvllm.dylib"
-	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
-else
-	LIBRARY="$CURDIR/libvllm.so"
-	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
-fi
-
-export VLLM_CPP_LIBRARY=$LIBRARY
-
-# If there is a lib/ld.so, use it
-if [ -f "$CURDIR"/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using library: $LIBRARY"
-	exec "$CURDIR"/lib/ld.so "$CURDIR"/vllm-cpp "$@"
-fi
-
-echo "Using library: $LIBRARY"
-exec "$CURDIR"/vllm-cpp "$@"

backend/go/vllm-cpp/test.sh

@@ -1,14 +0,0 @@
-#!/bin/bash
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-cd "$CURDIR"
-
-echo "Running vllm-cpp backend tests..."
-
-# Unit specs always run (struct-mirror layout, option/sampling mapping, load
-# validation). The e2e specs need a real model: set VLLM_CPP_MODEL to a .gguf
-# file or a safetensors model dir to enable them (see e2e_test.go).
-go test -v -timeout 1200s .
-
-echo "All vllm-cpp tests passed."

backend/go/vllm-cpp/vllmcpp_test.go

@@ -1,162 +0,0 @@
-package main
-
-import (
-	"os"
-	"path/filepath"
-	"testing"
-	"unsafe"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func TestVllmCpp(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "vllm-cpp suite")
-}
-
-// The Go POD mirrors must match the C struct layout of vllm.h (ABI v2)
-// byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
-// amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
-var _ = Describe("C ABI struct mirrors", func() {
-	It("cModelParams matches vllm_model_params", func() {
-		var p cModelParams
-		Expect(unsafe.Offsetof(p.ModelPath)).To(Equal(uintptr(0)))
-		Expect(unsafe.Offsetof(p.TokenizerConfigPath)).To(Equal(uintptr(8)))
-		Expect(unsafe.Offsetof(p.BlockSize)).To(Equal(uintptr(16)))
-		Expect(unsafe.Offsetof(p.NumBlocks)).To(Equal(uintptr(20)))
-		Expect(unsafe.Offsetof(p.MaxModelLen)).To(Equal(uintptr(24)))
-		Expect(unsafe.Offsetof(p.MaxNumSeqs)).To(Equal(uintptr(28)))
-		Expect(unsafe.Offsetof(p.ToolParser)).To(Equal(uintptr(32)))
-		Expect(unsafe.Offsetof(p.ReasoningParser)).To(Equal(uintptr(40)))
-		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(48)))
-	})
-
-	It("cSamplingParams matches vllm_sampling_params (ABI v2)", func() {
-		var p cSamplingParams
-		Expect(unsafe.Offsetof(p.Temperature)).To(Equal(uintptr(0)))
-		Expect(unsafe.Offsetof(p.TopP)).To(Equal(uintptr(4)))
-		Expect(unsafe.Offsetof(p.TopK)).To(Equal(uintptr(8)))
-		Expect(unsafe.Offsetof(p.MinP)).To(Equal(uintptr(12)))
-		Expect(unsafe.Offsetof(p.MaxTokens)).To(Equal(uintptr(16)))
-		Expect(unsafe.Offsetof(p.Seed)).To(Equal(uintptr(24)))
-		Expect(unsafe.Offsetof(p.HasSeed)).To(Equal(uintptr(32)))
-		Expect(unsafe.Offsetof(p.PresencePenalty)).To(Equal(uintptr(36)))
-		Expect(unsafe.Offsetof(p.FrequencyPenalty)).To(Equal(uintptr(40)))
-		Expect(unsafe.Offsetof(p.RepetitionPenalty)).To(Equal(uintptr(44)))
-		Expect(unsafe.Offsetof(p.MinTokens)).To(Equal(uintptr(48)))
-		Expect(unsafe.Offsetof(p.IgnoreEOS)).To(Equal(uintptr(52)))
-		Expect(unsafe.Offsetof(p.Stop)).To(Equal(uintptr(56)))
-		Expect(unsafe.Offsetof(p.NStop)).To(Equal(uintptr(64)))
-		Expect(unsafe.Offsetof(p.StructuredJSON)).To(Equal(uintptr(72)))
-		Expect(unsafe.Offsetof(p.StructuredRegex)).To(Equal(uintptr(80)))
-		Expect(unsafe.Offsetof(p.StructuredChoice)).To(Equal(uintptr(88)))
-		Expect(unsafe.Offsetof(p.NStructuredChoice)).To(Equal(uintptr(96)))
-		Expect(unsafe.Offsetof(p.StructuredGrammar)).To(Equal(uintptr(104)))
-		Expect(unsafe.Offsetof(p.StructuredJSONObject)).To(Equal(uintptr(112)))
-		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
-	})
-
-	It("cCompletion matches vllm_completion", func() {
-		var c cCompletion
-		Expect(unsafe.Offsetof(c.Text)).To(Equal(uintptr(0)))
-		Expect(unsafe.Offsetof(c.FinishReason)).To(Equal(uintptr(8)))
-		Expect(unsafe.Offsetof(c.PromptTokens)).To(Equal(uintptr(16)))
-		Expect(unsafe.Offsetof(c.CompletionTokens)).To(Equal(uintptr(20)))
-		Expect(unsafe.Sizeof(c)).To(Equal(uintptr(24)))
-	})
-})
-
-var _ = Describe("parseOptions", func() {
-	It("extracts the engine sizing knobs", func() {
-		lo := parseOptions(&pb.ModelOptions{Options: []string{
-			"block_size:64", "num_blocks:512", "max_num_seqs:32", "unknown:ignored",
-		}})
-		Expect(lo.blockSize).To(Equal(int32(64)))
-		Expect(lo.numBlocks).To(Equal(int32(512)))
-		Expect(lo.maxNumSeqs).To(Equal(int32(32)))
-	})
-	It("ignores malformed and non-positive values", func() {
-		lo := parseOptions(&pb.ModelOptions{Options: []string{
-			"block_size:abc", "num_blocks:-1", "max_num_seqs", "block_size:0",
-		}})
-		Expect(lo).To(Equal(loadOptions{}))
-	})
-})
-
-var _ = Describe("samplingFromPredict", func() {
-	It("maps the sampling fields onto the C POD", func() {
-		sp, _ := samplingFromPredict(&pb.PredictOptions{
-			Temperature:      0.7,
-			TopP:             0.9,
-			TopK:             40,
-			MinP:             0.05,
-			Tokens:           128,
-			Seed:             42,
-			Penalty:          1.1,
-			PresencePenalty:  0.5,
-			FrequencyPenalty: 0.25,
-			IgnoreEOS:        true,
-		})
-		Expect(sp.Temperature).To(BeNumerically("~", 0.7, 1e-6))
-		Expect(sp.TopP).To(BeNumerically("~", 0.9, 1e-6))
-		Expect(sp.TopK).To(Equal(int32(40)))
-		Expect(sp.MinP).To(BeNumerically("~", 0.05, 1e-6))
-		Expect(sp.MaxTokens).To(Equal(int32(128)))
-		Expect(sp.HasSeed).To(Equal(int32(1)))
-		Expect(sp.Seed).To(Equal(uint64(42)))
-		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.1, 1e-6))
-		Expect(sp.PresencePenalty).To(BeNumerically("~", 0.5, 1e-6))
-		Expect(sp.FrequencyPenalty).To(BeNumerically("~", 0.25, 1e-6))
-		Expect(sp.IgnoreEOS).To(Equal(int32(1)))
-	})
-
-	It("keeps the engine defaults for unset fields and stays unseeded", func() {
-		sp, keep := samplingFromPredict(&pb.PredictOptions{})
-		Expect(sp.TopP).To(BeNumerically("~", 1.0, 1e-6))
-		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.0, 1e-6))
-		Expect(sp.MaxTokens).To(Equal(int32(0))) // unbounded, engine-capped.
-		Expect(sp.HasSeed).To(Equal(int32(0)))
-		Expect(sp.Stop).To(Equal(uintptr(0)))
-		Expect(sp.StructuredGrammar).To(Equal(uintptr(0)))
-		Expect(keep).To(BeEmpty())
-	})
-
-	It("wires stop prompts and the grammar constraint", func() {
-		sp, keep := samplingFromPredict(&pb.PredictOptions{
-			StopPrompts: []string{"</s>", "\n\n"},
-			Grammar:     "root ::= \"yes\" | \"no\"",
-		})
-		Expect(sp.NStop).To(Equal(int32(2)))
-		Expect(sp.Stop).NotTo(Equal(uintptr(0)))
-		Expect(sp.StructuredGrammar).NotTo(Equal(uintptr(0)))
-		Expect(keep).NotTo(BeEmpty())
-	})
-})
-
-var _ = Describe("validModelPath", func() {
-	It("accepts a .gguf file", func() {
-		dir := GinkgoT().TempDir()
-		p := filepath.Join(dir, "model.gguf")
-		Expect(os.WriteFile(p, []byte("GGUF"), 0o600)).To(Succeed())
-		Expect(validModelPath(p)).To(Succeed())
-	})
-	It("accepts a directory with config.json", func() {
-		dir := GinkgoT().TempDir()
-		Expect(os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o600)).To(Succeed())
-		Expect(validModelPath(dir)).To(Succeed())
-	})
-	It("refuses a directory without config.json (greedy-probe rule)", func() {
-		Expect(validModelPath(GinkgoT().TempDir())).NotTo(Succeed())
-	})
-	It("refuses a non-gguf file", func() {
-		dir := GinkgoT().TempDir()
-		p := filepath.Join(dir, "weights.bin")
-		Expect(os.WriteFile(p, []byte("x"), 0o600)).To(Succeed())
-		Expect(validModelPath(p)).NotTo(Succeed())
-	})
-	It("refuses a missing path", func() {
-		Expect(validModelPath("/nonexistent/model.gguf")).NotTo(Succeed())
-	})
-})

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=97c56f1dc1d1100a9d859c865a20c82d22f823ed
+WHISPER_CPP_VERSION?=080bbbe85230f624f0b52127f1ae1218247989f9
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/index.yaml

@@ -156,44 +156,6 @@
     nvidia-cuda-12: "cuda12-whisper"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-whisper"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-whisper"
-- &vllm-cpp
-  name: "vllm-cpp"
-  alias: "vllm-cpp"
-  license: apache-2.0
-  description: |
-    vllm.cpp is a from-scratch C++20 port of vLLM created and maintained by the LocalAI team.
-    It mirrors vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching,
-    scheduler, sampler) on a portable tensor runtime with no Python, PyTorch or ggml at
-    inference time. It loads Hugging Face safetensors and GGUF checkpoints, supports
-    structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine,
-    and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan.
-  urls:
-    - https://github.com/mudler/vllm.cpp
-  tags:
-    - text-to-text
-    - LLM
-    - CPU
-    - GPU
-    - CUDA
-    - metal
-  capabilities:
-    default: "cpu-vllm-cpp"
-    nvidia: "cuda13-vllm-cpp"
-    metal: "metal-vllm-cpp"
-    vulkan: "vulkan-vllm-cpp"
-    nvidia-cuda-13: "cuda13-vllm-cpp"
-    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp"
-    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp"
-- !!merge <<: *vllm-cpp
-  name: "vllm-cpp-development"
-  capabilities:
-    default: "cpu-vllm-cpp-development"
-    nvidia: "cuda13-vllm-cpp-development"
-    metal: "metal-vllm-cpp-development"
-    vulkan: "vulkan-vllm-cpp-development"
-    nvidia-cuda-13: "cuda13-vllm-cpp-development"
-    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp-development"
-    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp-development"
 - &crispasr
   name: "crispasr"
   alias: "crispasr"
@@ -1209,48 +1171,6 @@
     nvidia-l4t: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-moss-tts-cpp-development"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-moss-tts-cpp-development"
-- &magpiettscpp
-  name: "magpie-tts-cpp"
-  description: |
-    Magpie TTS C++ backend using GGML (magpie-tts.cpp). Native C++
-    text-to-speech for NVIDIA's Magpie TTS Multilingual 357M model (encoder +
-    autoregressive decoder over NanoCodec tokens), running from a single
-    self-contained GGUF (model, codec, tokenizer, G2P dictionaries) with no
-    Python at inference time. 22.05kHz mono output, 5 baked voices (Aria,
-    Jason, John, Leo, Sofia), 9+ languages.
-  urls:
-    - https://github.com/mudler/magpie-tts.cpp
-    - https://huggingface.co/mudler/magpie-tts.cpp-gguf
-  tags:
-    - text-to-speech
-    - tts
-  alias: "magpie-tts-cpp"
-  capabilities:
-    default: "cpu-magpie-tts-cpp"
-    nvidia: "cuda12-magpie-tts-cpp"
-    nvidia-cuda-13: "cuda13-magpie-tts-cpp"
-    nvidia-cuda-12: "cuda12-magpie-tts-cpp"
-    intel: "intel-sycl-f16-magpie-tts-cpp"
-    metal: "metal-magpie-tts-cpp"
-    amd: "rocm-magpie-tts-cpp"
-    vulkan: "vulkan-magpie-tts-cpp"
-    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp"
-    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
-- !!merge <<: *magpiettscpp
-  name: "magpie-tts-cpp-development"
-  capabilities:
-    default: "cpu-magpie-tts-cpp-development"
-    nvidia: "cuda12-magpie-tts-cpp-development"
-    nvidia-cuda-13: "cuda13-magpie-tts-cpp-development"
-    nvidia-cuda-12: "cuda12-magpie-tts-cpp-development"
-    intel: "intel-sycl-f16-magpie-tts-cpp-development"
-    metal: "metal-magpie-tts-cpp-development"
-    amd: "rocm-magpie-tts-cpp-development"
-    vulkan: "vulkan-magpie-tts-cpp-development"
-    nvidia-l4t: "nvidia-l4t-arm64-magpie-tts-cpp-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-magpie-tts-cpp-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
 - &omnivoicecpp
   name: "omnivoice-cpp"
   description: |
@@ -1455,7 +1375,6 @@
   alias: "kokoro"
   name: "kokoro"
   capabilities:
-    default: "cpu-kokoro"
     nvidia: "cuda12-kokoro"
     intel: "intel-kokoro"
     amd: "rocm-kokoro"
@@ -4835,107 +4754,6 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-moss-tts-cpp
-## magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "nvidia-l4t-arm64-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-arm64-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "nvidia-l4t-arm64-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-arm64-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cuda13-nvidia-l4t-arm64-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cpu-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-cpu-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "metal-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "metal-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-metal-darwin-arm64-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cpu-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-cpu-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cuda12-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-12-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "rocm-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-rocm-hipblas-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "intel-sycl-f32-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-intel-sycl-f32-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "intel-sycl-f16-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-intel-sycl-f16-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "vulkan-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-vulkan-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "vulkan-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-vulkan-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cuda12-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-12-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "rocm-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-rocm-hipblas-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "intel-sycl-f32-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-intel-sycl-f32-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "intel-sycl-f16-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-intel-sycl-f16-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cuda13-magpie-tts-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-13-magpie-tts-cpp
-- !!merge <<: *magpiettscpp
-  name: "cuda13-magpie-tts-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-13-magpie-tts-cpp
 ## omnivoice-cpp
 - !!merge <<: *omnivoicecpp
   name: "omnivoice-cpp-development"
@@ -5187,22 +5005,11 @@
 - !!merge <<: *kokoro
   name: "kokoro-development"
   capabilities:
-    default: "cpu-kokoro-development"
     nvidia: "cuda12-kokoro-development"
     intel: "intel-kokoro-development"
     amd: "rocm-kokoro-development"
     nvidia-l4t: "nvidia-l4t-kokoro-development"
     metal: "metal-kokoro-development"
-- !!merge <<: *kokoro
-  name: "cpu-kokoro"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-kokoro"
-  mirrors:
-    - localai/localai-backends:latest-cpu-kokoro
-- !!merge <<: *kokoro
-  name: "cpu-kokoro-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-kokoro"
-  mirrors:
-    - localai/localai-backends:master-cpu-kokoro
 - !!merge <<: *kokoro
   name: "cuda12-kokoro-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-kokoro"
@@ -6623,53 +6430,3 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-supertonic"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-supertonic
-- !!merge <<: *vllm-cpp
-  name: "cpu-vllm-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:latest-cpu-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "cpu-vllm-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:master-cpu-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "cuda13-vllm-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "cuda13-vllm-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-13-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "nvidia-l4t-arm64-vllm-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "nvidia-l4t-arm64-vllm-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "vulkan-vllm-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-vulkan-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "vulkan-vllm-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-vulkan-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "metal-vllm-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-vllm-cpp
-- !!merge <<: *vllm-cpp
-  name: "metal-vllm-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-vllm-cpp"
-  mirrors:
-    - localai/localai-backends:master-metal-darwin-arm64-vllm-cpp

backend/python/coqui/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.83.0
+grpcio==1.82.1
 protobuf
 certifi
 packaging==26.2

backend/python/mlx-vlm/requirements-mps.txt

@@ -1,3 +1 @@
-git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
-torch
-torchvision
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4

backend/python/nemo/install.sh

@@ -14,28 +14,4 @@ if [ "x${BUILD_PROFILE}" == "xintel" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
 fi
 
-# Darwin needs a newer interpreter than libbackend's 3.10 default. nemo_toolkit
-# pulls in text2num, a Rust extension built with maturin, and its macOS arm64
-# wheels start at cp311 (3.0.2 publishes cp311/cp312/cp313/cp314 and no cp310).
-# On 3.10 pip therefore falls back to the sdist and dies in the PEP 517 hook
-# with "No module named 'maturin'", since EXTRA_PIP_INSTALL_FLAGS carries
-# --no-build-isolation and nothing installs the build backend. Moving to 3.12
-# takes the prebuilt wheel and needs no Rust toolchain on the runner at all.
-#
-# Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel
-# for the same package and have no reason to move.
-if [ "x${BUILD_PROFILE}" == "xmps" ] || [ "x${BUILD_PROFILE}" == "xmetal" ]; then
-    PYTHON_VERSION="3.12"
-    # PYTHON_PATCH must move with it. libbackend builds the portable-Python URL
-    # as cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}-..., and
-    # the default patch is 18 for 3.10.18; leaving it alone asks for a 3.12.18
-    # that was never released and the download 404s.
-    #
-    # 11, not the 12 that sglang/install.sh uses for l4t13: at the 20250818 tag
-    # python-build-standalone published 3.12.12 for linux aarch64 but not for
-    # aarch64-apple-darwin, where 3.12.11 is the newest. Verified against the
-    # release assets rather than copied across.
-    PYTHON_PATCH="11"
-fi
-
 installRequirements

backend/python/sglang/requirements-cublas12-after.txt

@@ -2,16 +2,3 @@
 # (FunctionCallParser, ReasoningParser) move between releases.
 # 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
 sglang[all]>=0.5.11
-
-# Keep nvidia-modelopt on a stable release. sglang[all] pulls it in through its
-# `diffusion` extra with no version bound of its own, and install.sh passes a
-# GLOBAL --prerelease=allow (needed because flash-attn-4 only ships 4.0.0b*
-# wheels). Unbounded plus prereleases-allowed resolves to 0.46.0rc0, whose build
-# backend imports wheel_stub without declaring it as a build dependency; with
-# --no-build-isolation also in EXTRA_PIP_INSTALL_FLAGS nothing installs it, and
-# every cublas sglang image fails with "No module named 'wheel_stub'".
-#
-# Bounding this one package rather than dropping the global flag: the flag is
-# load-bearing for flash-attn-4, and this is the narrower change. Raise the
-# bound once 0.46.0 final ships.
-nvidia-modelopt<0.46

backend/python/sglang/requirements-cublas13-after.txt

@@ -2,16 +2,3 @@
 # (FunctionCallParser, ReasoningParser) move between releases.
 # 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
 sglang[all]>=0.5.11
-
-# Keep nvidia-modelopt on a stable release. sglang[all] pulls it in through its
-# `diffusion` extra with no version bound of its own, and install.sh passes a
-# GLOBAL --prerelease=allow (needed because flash-attn-4 only ships 4.0.0b*
-# wheels). Unbounded plus prereleases-allowed resolves to 0.46.0rc0, whose build
-# backend imports wheel_stub without declaring it as a build dependency; with
-# --no-build-isolation also in EXTRA_PIP_INSTALL_FLAGS nothing installs it, and
-# every cublas sglang image fails with "No module named 'wheel_stub'".
-#
-# Bounding this one package rather than dropping the global flag: the flag is
-# load-bearing for flash-attn-4, and this is the narrower change. Raise the
-# bound once 0.46.0 final ships.
-nvidia-modelopt<0.46

backend/python/transformers/requirements-cpu.txt

@@ -4,7 +4,7 @@ numba==0.60.0
 accelerate
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.1
+sentence-transformers==5.6.0
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-cublas12.txt

@@ -4,7 +4,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.1
+sentence-transformers==5.6.0
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-cublas13.txt

@@ -4,7 +4,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.1
+sentence-transformers==5.6.0
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-hipblas.txt

@@ -5,7 +5,7 @@ transformers>=5.14.1
 llvmlite==0.43.0
 numba==0.60.0
 bitsandbytes
-sentence-transformers==5.6.1
+sentence-transformers==5.6.0
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-intel.txt

@@ -5,7 +5,7 @@ llvmlite==0.43.0
 numba==0.60.0
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.1
+sentence-transformers==5.6.0
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements-mps.txt

@@ -4,7 +4,7 @@ numba==0.60.0
 accelerate
 transformers>=5.14.1
 bitsandbytes
-sentence-transformers==5.6.1
+sentence-transformers==5.6.0
 diffusers
 soundfile
 protobuf==7.35.0

backend/python/transformers/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.83.0
+grpcio==1.82.1
 protobuf==7.35.0
 certifi
 setuptools

backend/python/vllm/install.sh

@@ -119,7 +119,7 @@ if [ "$(uname -s)" = "Darwin" ]; then
     # can rewrite it. Darwin therefore follows vllm-metal and can lag the Linux
     # vllm pin (requirements-cublas13-after.txt, bumped independently against
     # vllm/vllm) until vllm-metal supports a newer vLLM.
-    VLLM_METAL_VERSION="v0.3.0.dev20260726174827"
+    VLLM_METAL_VERSION="v0.3.0.dev20260722081849"
 
     # The coupled vLLM source version is whatever this vllm-metal release builds
     # against -- it declares it in its own installer as `vllm_v=`. Derive it from

backend/python/vllm/requirements-cublas13-after.txt

@@ -3,8 +3,8 @@
 # on a cu130 host. Pull the cu130-flavoured wheel from vLLM's per-tag index
 # instead — the cublas13 case in install.sh adds --index-strategy=unsafe-best-match
 # so uv consults this index alongside PyPI.
---extra-index-url https://wheels.vllm.ai/0.26.0/cu130
+--extra-index-url https://wheels.vllm.ai/0.25.1/cu130
 # VERSION COUPLING: darwin/Apple-Silicon builds use vllm-metal (see install.sh),
 # which pins this exact vLLM version. Bumping vllm here means coordinating with a
 # vllm-metal release that supports the new version, or macOS/Metal builds break.
-vllm==0.26.0
+vllm==0.25.1

backend/python/vllm/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.83.0
+grpcio==1.82.1
 protobuf
 certifi
 setuptools

core/application/distributed.go

@@ -10,6 +10,7 @@ import (
 	"time"
 
 	"github.com/google/uuid"
+	"github.com/mudler/LocalAI/core/backend"
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/core/services/agents"
 	"github.com/mudler/LocalAI/core/services/distributed"
@@ -286,14 +287,13 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 		prefixProvider = prefixSync
 
 		// Invalidate the prefix-cache index whenever a replica row is removed.
-		// AddReplicaRemovedHook fires from the single chokepoint all removal paths
+		// SetReplicaRemovedHook fires from the single chokepoint all removal paths
 		// funnel through (RemoveNodeModel / RemoveAllNodeModelReplicas), so this
 		// one hook covers every path: reconciler scale-down, probe reaper,
 		// health-monitor reap, RemoteUnloaderAdapter, and the router. Registering
 		// it only inside this enabled block keeps the disabled path a true no-op
-		// for the prefix cache; other subsystems register their own hooks
-		// independently and are unaffected either way.
-		registry.AddReplicaRemovedHook(func(model, node string, replica int) {
+		// (the registry stays hook-less).
+		registry.SetReplicaRemovedHook(func(model, node string, replica int) {
 			if replica < 0 {
 				prefixSync.InvalidateNode(model, node)
 			} else {
@@ -378,6 +378,22 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 			cfg.Distributed.BackendInstallTimeoutOrDefault(),
 			cfg.Distributed.ModelLoadTimeoutOrDefault(),
 		),
+		// Re-derive managed companion options from the model's current config when
+		// the reconciler replays a stored ModelOptions blob (which never runs
+		// grpcModelOpts). Without this, a replica scaled up from a blob captured
+		// before the companion resolved loads without the companion option and the
+		// backend fetches its own wrong default. nil-safe: an unknown model or a
+		// config with no companions yields no options.
+		CompanionOptionsFor: func(modelName string) []string {
+			if configLoader == nil {
+				return nil
+			}
+			modelCfg, ok := configLoader.GetModelConfig(modelName)
+			if !ok {
+				return nil
+			}
+			return backend.CompanionArtifactOptions(modelCfg)
+		},
 	})
 
 	// Wire staging-progress broadcasting so file-staging shows up on every

core/application/router_factories.go

@@ -46,12 +46,12 @@ type lazyScorer struct {
 	modelName string
 }
 
-func (l *lazyScorer) Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]backend.CandidateScore, error) {
+func (l *lazyScorer) Score(ctx context.Context, prompt string, candidates []string) ([]backend.CandidateScore, error) {
 	cfg := l.app.adapterConfig(l.modelName)
 	if cfg == nil {
 		return nil, fmt.Errorf("scorer: model %q no longer available", l.modelName)
 	}
-	return backend.NewScorer(l.app.modelLoader, *cfg, l.app.applicationConfig).Score(ctx, prompt, stablePrefixLen, candidates)
+	return backend.NewScorer(l.app.modelLoader, *cfg, l.app.applicationConfig).Score(ctx, prompt, candidates)
 }
 
 // TokenCounter returns a func so the middleware's literal field type accepts

core/application/router_factories_test.go

@@ -109,7 +109,7 @@ var _ = Describe("router_factories lazy config resolution", func() {
 			Expect(lazy.modelName).To(Equal("score-test"))
 
 			removeCfg("score-test")
-			_, err := sc.Score(context.Background(), "prompt", 0, []string{"a"})
+			_, err := sc.Score(context.Background(), "prompt", []string{"a"})
 			Expect(err).To(HaveOccurred())
 			Expect(err.Error()).To(ContainSubstring("no longer available"))
 		})

core/application/startup.go

@@ -283,14 +283,6 @@ func New(opts ...config.AppOption) (*Application, error) {
 			distSvc.Registry,
 		)
 		application.modelLoader.SetModelStore(distStore)
-		// Drop the local stub when a model's last replica leaves the registry.
-		// The store reports local stubs UNION registry rows, and every removal
-		// path deletes the row only, so without this the frontend keeps
-		// reporting a model as loaded long after the replica is gone.
-		// Registered unconditionally: this is independent of the prefix cache.
-		distSvc.Registry.AddReplicaRemovedHook(
-			nodes.NewLocalStubInvalidator(distSvc.Registry, distStore),
-		)
 		// Start health monitor
 		distSvc.Health.Start(options.Context)
 		// Start replica reconciler for auto-scaling model replicas

core/backend/companion_options_test.go

@@ -113,11 +113,34 @@ var _ = Describe("companion artifact backend options", func() {
 		Expect(opts.Options).To(Equal([]string{"attention_backend:sdpa"}))
 	})
 
-	It("skips a companion that has not been resolved yet", func() {
+	It("names the source repository when the companion is not resolved yet", func() {
+		// A companion that reaches load time WITHOUT a resolved snapshot must not
+		// vanish silently: emitting no option lets the backend fall back to its own
+		// hardcoded default, which is how a distributed longcat-video worker ended
+		// up trying to load the wrong base model and failing "base_model must point
+		// to a LongCat-Video checkpoint". Naming the DECLARED repository instead
+		// points the backend at the artifact the config actually asked for. The
+		// snapshot path (the staged, no-download fast path) is still preferred
+		// whenever the companion IS resolved.
 		cfg := configWithCompanion()
 		cfg.Artifacts[1].Resolved = nil
 		opts := grpcModelOpts(cfg, "/models")
-		_, found := optionValue(opts.Options, "base_model")
-		Expect(found).To(BeFalse())
+
+		value, found := optionValue(opts.Options, "base_model")
+		Expect(found).To(BeTrue())
+		Expect(value).To(Equal("meituan-longcat/LongCat-Video"))
+		// The fallback is a repo reference, never a models-relative snapshot path.
+		Expect(value).ToNot(ContainSubstring(".artifacts"))
+	})
+
+	It("prefers the resolved snapshot path over the source repository", func() {
+		opts := grpcModelOpts(configWithCompanion(), "/models")
+		value, found := optionValue(opts.Options, "base_model")
+		Expect(found).To(BeTrue())
+		expected, err := modelartifacts.RelativeSnapshotPath(companionKey)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(value).To(Equal(expected))
+		// The resolved fast path must never degrade to a bare repo id.
+		Expect(value).ToNot(Equal("meituan-longcat/LongCat-Video"))
 	})
 })

core/backend/ctx_propagation_test.go

@@ -157,33 +157,6 @@ var _ = Describe("X-LocalAI-Node ctx propagation contract", func() {
 		stampViaRouterCtx()
 	})
 
-	// Regression for #10636: a canceled request context must NOT cancel the
-	// model LOAD. The heavy image/audio backends bind the load to the request
-	// context so the routing holder reaches the SmartRouter; but a large
-	// diffusers/LLM model on a slow (e.g. shared-memory iGPU) host can take
-	// far longer to load than the client stays connected. If the request's
-	// cancellation propagates to the load, the LoadModel RPC is aborted, the
-	// backend process is torn down, and every retry restarts from scratch and
-	// never converges. The load must instead run to completion and cache while
-	// still carrying the request's routing holder value.
-	It("ImageGeneration does not propagate request cancellation to the model load", func() {
-		canceledCtx, cancel := context.WithCancel(reqCtx)
-		cancel() // client disconnected while the (slow) load was still running
-
-		_, err := backend.ImageGeneration(canceledCtx, 64, 64, 1, 0, "p", "", "", "/tmp/out.png", loader, modelCfg, appCfg, nil)
-		// The load reached the router (short-circuit sentinel), i.e. it was
-		// NOT aborted early by the already-canceled request context.
-		Expect(err).To(HaveOccurred())
-		Expect(err.Error()).To(ContainSubstring("router short-circuit (test)"))
-
-		routerCtx := routerCtxOf()
-		Expect(routerCtx).ToNot(BeNil(), "router callback must have been invoked")
-		Expect(routerCtx.Err()).To(BeNil(),
-			"a canceled request must not cancel the model load")
-		// The routing holder value still propagates despite the decoupling.
-		stampViaRouterCtx()
-	})
-
 	It("does NOT leak the holder when the app context is used instead", func() {
 		// Sanity: the bug being fixed manifests as the router getting
 		// appCfg.Context (no holder) instead of reqCtx (holder). A direct

core/backend/embeddings.go

@@ -40,14 +40,10 @@ func (e *modelEmbedder) Embed(ctx context.Context, text string) ([]float32, erro
 
 func ModelEmbedding(ctx context.Context, s string, tokens []int, loader *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig) (func() ([]float32, error), error) {
 
-	// model.WithContext carries the request context into the load so distributed
-	// routing decisions reach the request's X-LocalAI-Node holder via
-	// distributedhdr.Stamp. context.WithoutCancel keeps those values but drops
-	// the request's cancellation, so a slow first load still completes and
-	// caches if the client disconnects instead of aborting the LoadModel RPC and
-	// tearing down the backend process (issue #10636). Inference below keeps the
-	// cancellable ctx, so a disconnect still stops generation.
-	opts := ModelOptions(modelConfig, appConfig, model.WithContext(context.WithoutCancel(ctx)))
+	// model.WithContext(ctx) overrides the app-context default set in
+	// ModelOptions so distributed routing decisions reach the request's
+	// X-LocalAI-Node holder via distributedhdr.Stamp.
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
 
 	inferenceModel, err := loader.Load(opts...)
 	if err != nil {

core/backend/image.go

@@ -13,14 +13,10 @@ import (
 
 func ImageGeneration(ctx context.Context, height, width, step, seed int, positive_prompt, negative_prompt, src, dst string, loader *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig, refImages []string) (func() error, error) {
 
-	// model.WithContext carries the request context into the load so distributed
-	// routing decisions reach the request's X-LocalAI-Node holder via
-	// distributedhdr.Stamp. context.WithoutCancel keeps those values but drops
-	// the request's cancellation, so a slow first load still completes and
-	// caches if the client disconnects instead of aborting the LoadModel RPC and
-	// tearing down the backend process (issue #10636). Inference below keeps the
-	// cancellable ctx, so a disconnect still stops generation.
-	opts := ModelOptions(modelConfig, appConfig, model.WithContext(context.WithoutCancel(ctx)))
+	// model.WithContext(ctx) overrides the app-context default set in
+	// ModelOptions so distributed routing decisions reach the request's
+	// X-LocalAI-Node holder via distributedhdr.Stamp.
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
 	inferenceModel, err := loader.Load(
 		opts...,
 	)

core/backend/llm.go

@@ -133,12 +133,7 @@ func ModelInference(ctx context.Context, s string, messages schema.Messages, ima
 	}
 	ctx = distributedhdr.MaybeWithPrefixChain(ctx, c.ModelID(), chainSource)
 
-	// context.WithoutCancel decouples the model load from the request's
-	// cancellation while preserving its routing values, so a slow load still
-	// completes and caches if the client disconnects instead of aborting the
-	// LoadModel RPC mid-load (issue #10636). Inference below keeps the
-	// cancellable ctx, so a disconnect still stops generation.
-	opts := ModelOptions(*c, o, model.WithContext(context.WithoutCancel(ctx)))
+	opts := ModelOptions(*c, o, model.WithContext(ctx))
 	inferenceModel, err := loader.Load(opts...)
 	if err != nil {
 		recordModelLoadFailure(o, c.Name, c.Backend, err, map[string]any{"model_file": modelFile})

core/backend/options.go

@@ -166,21 +166,6 @@ func estimateModelSizeBytes(c config.ModelConfig, modelsPath string) int64 {
 	return int64(result.SizeBytes)
 }
 
-// effectiveThreads resolves the thread count a backend is asked to use.
-// Per-model threads wins: SetDefaults already fills an unset per-model value
-// from the app-level --threads, so overriding a set value with the app value
-// here would make the YAML `threads:` knob dead config (it did, for years —
-// e.g. a tiny VAD model could never opt down from the global pool size).
-func effectiveThreads(c config.ModelConfig, appThreads int) int {
-	if c.Threads != nil && *c.Threads > 0 {
-		return *c.Threads
-	}
-	if appThreads > 0 {
-		return appThreads
-	}
-	return 1
-}
-
 func ModelOptions(c config.ModelConfig, so *config.ApplicationConfig, opts ...model.Option) []model.Option {
 	defOpts := []model.Option{
 		model.WithBackendString(c.Backend),
@@ -193,7 +178,16 @@ func ModelOptions(c config.ModelConfig, so *config.ApplicationConfig, opts ...mo
 		defOpts = append(defOpts, model.WithModelFile(c.ModelFileName()))
 	}
 
-	threads := effectiveThreads(c, so.Threads)
+	threads := 1
+
+	if c.Threads != nil {
+		threads = *c.Threads
+	}
+
+	if so.Threads != 0 {
+		threads = so.Threads
+	}
+
 	c.Threads = &threads
 
 	grpcOpts := grpcModelOpts(c, so.SystemState.Model.ModelsPath)
@@ -300,6 +294,13 @@ func EffectiveBatchSize(c config.ModelConfig) int {
 //
 // An option the author set explicitly always wins: pinning a companion to a
 // local checkout has to beat the managed snapshot.
+//
+// A companion that is declared but NOT resolved falls back to its source
+// repository id rather than being dropped: a dropped companion is invisible to
+// the backend, which then loads its own hardcoded default and fails far away
+// from the cause. The repo-id fallback trades the staging fast path (the weights
+// are fetched on the worker) for correctness, and logs a warning so the missing
+// controller-side resolution is diagnosable.
 func withCompanionArtifactOptions(options []string, artifacts []modelartifacts.Spec) []string {
 	configured := make(map[string]struct{}, len(options))
 	for _, option := range options {
@@ -307,26 +308,71 @@ func withCompanionArtifactOptions(options []string, artifacts []modelartifacts.S
 			configured[name] = struct{}{}
 		}
 	}
-
 	// Copy before appending: opts.Options would otherwise share (and could
 	// reallocate away from) the config's own slice.
-	combined := slices.Clone(options)
+	return append(slices.Clone(options), ManagedCompanionOptions(artifacts, configured)...)
+}
+
+// ManagedCompanionOptions synthesizes the "<artifact name>:<value>" option for
+// every companion artifact whose name is not already in `configured`, so callers
+// can append them to a ModelOptions.Options slice. It is exported because the
+// distributed reconciler replays a stored proto blob that never runs
+// grpcModelOpts, and it must re-derive the same companion options from the
+// current config so a scaled-up replica gets them too.
+//
+// A resolved companion is surfaced as its staged, models-relative snapshot
+// directory (the no-download fast path a remote worker resolves under its own
+// ModelPath). A companion that reached load time WITHOUT a resolved snapshot
+// falls back to its source repository id rather than being dropped: dropping it
+// is invisible to the backend, which then loads its OWN hardcoded default and
+// fails far from the cause (a distributed longcat-video worker fetched the wrong
+// base model and failed "base_model must point to a LongCat-Video checkpoint").
+func ManagedCompanionOptions(artifacts []modelartifacts.Spec, configured map[string]struct{}) []string {
+	var out []string
 	for _, artifact := range artifacts {
-		if artifact.Target != modelartifacts.TargetCompanion || artifact.Resolved == nil {
+		if artifact.Target != modelartifacts.TargetCompanion {
 			continue
 		}
-		if _, exists := configured[artifact.Name]; exists {
-			xlog.Debug("keeping the configured companion option over the managed snapshot", "artifact", artifact.Name)
+		if configured != nil {
+			if _, exists := configured[artifact.Name]; exists {
+				xlog.Debug("keeping the configured companion option over the managed snapshot", "artifact", artifact.Name)
+				continue
+			}
+		}
+
+		if artifact.Resolved != nil {
+			if snapshot, err := modelartifacts.RelativeSnapshotPath(artifact.Resolved.CacheKey); err == nil {
+				out = append(out, artifact.Name+":"+snapshot)
+				continue
+			} else {
+				xlog.Warn("companion artifact has an unusable cache key; falling back to its source repository", "artifact", artifact.Name, "error", err)
+			}
+		}
+
+		if repo := strings.TrimSpace(artifact.Source.Repo); repo != "" {
+			xlog.Warn("companion artifact is not resolved on the controller; the backend will fetch it by repository id (no staging fast path)",
+				"artifact", artifact.Name, "repo", repo)
+			out = append(out, artifact.Name+":"+repo)
 			continue
 		}
-		snapshot, err := modelartifacts.RelativeSnapshotPath(artifact.Resolved.CacheKey)
-		if err != nil {
-			xlog.Warn("skipping companion artifact with an unusable cache key", "artifact", artifact.Name, "error", err)
-			continue
-		}
-		combined = append(combined, artifact.Name+":"+snapshot)
+		xlog.Warn("companion artifact is neither resolved nor has a source repository; the backend will get no option for it", "artifact", artifact.Name)
 	}
-	return combined
+	return out
+}
+
+// CompanionArtifactOptions returns the managed companion options a model's
+// current config would contribute, skipping any companion name the config
+// already pins in Options. It is the reconciler's entry point for re-deriving
+// companion options when it replays a stored ModelOptions blob (which predates,
+// and so cannot carry, a companion resolved after that blob was captured).
+func CompanionArtifactOptions(c config.ModelConfig) []string {
+	configured := make(map[string]struct{}, len(c.Options))
+	for _, option := range c.Options {
+		if name, _, found := strings.Cut(option, ":"); found {
+			configured[name] = struct{}{}
+		}
+	}
+	return ManagedCompanionOptions(c.Artifacts, configured)
 }
 
 func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
@@ -422,7 +468,6 @@ func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
 		Options:              withCompanionArtifactOptions(c.Options, c.Artifacts),
 		Overrides:            c.Overrides,
 		EngineArgs:           engineArgsJSON,
-		EnableScore:          c.HasUsecases(config.FLAG_SCORE),
 		CLIPSkip:             int32(c.Diffusers.ClipSkip),
 		ControlNet:           c.Diffusers.ControlNet,
 		ContextSize:          int32(ctxSize),
@@ -482,7 +527,6 @@ func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
 			ApiKeyFile:            c.Proxy.APIKeyFile,
 			UpstreamModel:         c.Proxy.UpstreamModel,
 			RequestTimeoutSeconds: int32(c.Proxy.RequestTimeoutSeconds),
-			CachePrompt:           c.Proxy.CachePrompt,
 		}
 	}
 

core/backend/options_internal_test.go

@@ -120,7 +120,6 @@ var _ = Describe("grpcModelOpts NBatch", func() {
 		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}}
 		opts := grpcModelOpts(cfg, "/tmp/models")
 		Expect(opts.NBatch).To(BeEquivalentTo(512))
-		Expect(opts.EnableScore).To(BeFalse())
 	})
 
 	It("sizes the batch to the context window for score models", func() {
@@ -129,14 +128,6 @@ var _ = Describe("grpcModelOpts NBatch", func() {
 		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &scoreUsecase}
 		opts := grpcModelOpts(cfg, "/tmp/models")
 		Expect(opts.NBatch).To(BeEquivalentTo(4096))
-		Expect(opts.EnableScore).To(BeTrue())
-	})
-
-	It("enables score resources for a model with multiple usecases", func() {
-		usecases := config.FLAG_CHAT | config.FLAG_SCORE
-		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &usecases}
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.EnableScore).To(BeTrue())
 	})
 
 	It("keeps an explicit batch over the score default", func() {
@@ -364,23 +355,3 @@ var _ = Describe("gRPCPredictOpts model identity", func() {
 		Expect(opts.ModelIdentity).To(BeEmpty())
 	})
 })
-
-var _ = Describe("effectiveThreads", func() {
-	It("lets a per-model threads value override the app-level --threads", func() {
-		one := 1
-		cfg := config.ModelConfig{Threads: &one}
-		Expect(effectiveThreads(cfg, 10)).To(Equal(1),
-			"per-model threads is a real knob, not dead config under --threads")
-	})
-
-	It("falls back to the app-level threads when the model sets none", func() {
-		Expect(effectiveThreads(config.ModelConfig{}, 10)).To(Equal(10))
-		zero := 0
-		Expect(effectiveThreads(config.ModelConfig{Threads: &zero}, 10)).To(Equal(10),
-			"an explicit threads: 0 means unset, not zero threads")
-	})
-
-	It("never resolves to a non-positive thread count", func() {
-		Expect(effectiveThreads(config.ModelConfig{}, 0)).To(Equal(1))
-	})
-})

core/backend/preload.go

@@ -28,7 +28,7 @@ func PreloadModelByName(ctx context.Context, cl *config.ModelConfigLoader, ml *m
 		return nil, err
 	}
 
-	stages, err := pipelineStages(cl, &cfg.Pipeline, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
+	stages, err := pipelineStages(cl, &cfg.Pipeline, ml.ModelPath)
 	if err != nil {
 		return nil, err
 	}
@@ -59,7 +59,7 @@ var loadStage = PreloadModel
 // pipeline itself uses. A stage that fails to resolve is a misconfiguration,
 // so it fails fast rather than being deferred to load. A pipeline with no
 // stages set returns nil, which callers treat as "not a pipeline".
-func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath string, opts ...config.ConfigLoaderOption) ([]PreloadStage, error) {
+func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath string) ([]PreloadStage, error) {
 	voiceRec := ""
 	if p.VoiceRecognition != nil {
 		voiceRec = p.VoiceRecognition.Model
@@ -76,7 +76,7 @@ func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath
 		if s.name == "" {
 			continue
 		}
-		cfg, err := cl.LoadResolvedModelConfig(s.name, modelPath, opts...)
+		cfg, err := cl.LoadResolvedModelConfig(s.name, modelPath)
 		if err != nil {
 			return nil, fmt.Errorf("%s (%s): %w", s.role, s.name, err)
 		}
@@ -87,11 +87,9 @@ func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath
 
 // PreloadStages loads every present stage at once and waits for all of them, so
 // a pipeline warms in the time of its slowest stage rather than the sum. Absent
-// stages are skipped. Some callers represent an unset optional stage with a
-// nil config, while others materialize a default config with an empty name. A
-// failed stage does not cancel the others — they all run to completion so the
-// joined error names every broken stage at once, alongside the names that did
-// load.
+// (nil-config) stages are skipped. A failed stage does not cancel the others —
+// they all run to completion so the joined error names every broken stage at
+// once, alongside the names that did load.
 func PreloadStages(ctx context.Context, ml *model.ModelLoader, appConfig *config.ApplicationConfig, stages []PreloadStage) ([]string, error) {
 	var (
 		wg     sync.WaitGroup
@@ -100,7 +98,7 @@ func PreloadStages(ctx context.Context, ml *model.ModelLoader, appConfig *config
 		errs   []error
 	)
 	for _, s := range stages {
-		if s.Cfg == nil || s.Cfg.Name == "" {
+		if s.Cfg == nil {
 			continue
 		}
 		wg.Add(1)

core/backend/preload_internal_test.go

@@ -103,13 +103,12 @@ var _ = Describe("PreloadStages", func() {
 		return PreloadStage{Role: role, Cfg: &config.ModelConfig{Name: name}}
 	}
 
-	It("loads every present stage, skips absent stages, and returns the loaded names", func() {
+	It("loads every present stage, skips absent (nil-config) ones, and returns the loaded names", func() {
 		stubLoader(nil)
 
 		loaded, err := PreloadStages(context.Background(), nil, nil, []PreloadStage{
 			mkStage("vad", "vad-m"),
-			{Role: "transcription"},
-			mkStage("tts", ""),
+			{Role: "transcription"}, // absent stage
 			mkStage("llm", "llm-m"),
 		})
 

core/backend/rerank.go

@@ -57,14 +57,10 @@ func (r *modelReranker) Rerank(ctx context.Context, query string, documents []st
 }
 
 func Rerank(ctx context.Context, request *proto.RerankRequest, loader *model.ModelLoader, appConfig *config.ApplicationConfig, modelConfig config.ModelConfig) (*proto.RerankResult, error) {
-	// model.WithContext carries the request context into the load so distributed
-	// routing decisions reach the request's X-LocalAI-Node holder via
-	// distributedhdr.Stamp. context.WithoutCancel keeps those values but drops
-	// the request's cancellation, so a slow first load still completes and
-	// caches if the client disconnects instead of aborting the LoadModel RPC and
-	// tearing down the backend process (issue #10636). Inference below keeps the
-	// cancellable ctx, so a disconnect still stops generation.
-	opts := ModelOptions(modelConfig, appConfig, model.WithContext(context.WithoutCancel(ctx)))
+	// model.WithContext(ctx) overrides the app-context default set in
+	// ModelOptions so distributed routing decisions reach the request's
+	// X-LocalAI-Node holder via distributedhdr.Stamp.
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
 	rerankModel, err := loader.Load(opts...)
 	if err != nil {
 		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)

core/backend/score.go

@@ -23,10 +23,6 @@ type ScoreOptions struct {
 	// token count. Useful when comparing candidates of different
 	// lengths — without it, longer candidates score lower by default.
 	LengthNormalize bool
-	// StablePrefixLen is the byte length of the prompt prefix that stays
-	// identical across repeated scoring calls (0 = unknown); forwarded to
-	// the backend as a state-reuse boundary hint.
-	StablePrefixLen int
 }
 
 // CandidateScore is the per-candidate result. Mirrors pb.CandidateScore
@@ -46,13 +42,9 @@ type TokenLogProb struct {
 // Scorer evaluates a model's joint log-probability of each candidate
 // continuation given a shared prompt. Implemented by NewScorer over a
 // model-loaded backend; the router's score classifier consumes this
-// for multi-label policy selection. stablePrefixLen is the byte length
-// of the prompt prefix that stays identical across calls (0 = unknown)
-// — backends use it to place a state-reuse point at the boundary, which
-// is what keeps repeat scoring fast on models that cannot rewind
-// (hybrid/recurrent architectures).
+// for multi-label policy selection.
 type Scorer interface {
-	Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]CandidateScore, error)
+	Score(ctx context.Context, prompt string, candidates []string) ([]CandidateScore, error)
 }
 
 // NewScorer binds (loader, modelConfig, appConfig) into a Scorer. The
@@ -69,8 +61,8 @@ type modelScorer struct {
 	appConfig   *config.ApplicationConfig
 }
 
-func (m *modelScorer) Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]CandidateScore, error) {
-	fn, err := ModelScore(prompt, candidates, ScoreOptions{LengthNormalize: true, StablePrefixLen: stablePrefixLen}, m.loader, m.modelConfig, m.appConfig)
+func (m *modelScorer) Score(ctx context.Context, prompt string, candidates []string) ([]CandidateScore, error) {
+	fn, err := ModelScore(prompt, candidates, ScoreOptions{LengthNormalize: true}, m.loader, m.modelConfig, m.appConfig)
 	if err != nil {
 		return nil, err
 	}
@@ -111,7 +103,6 @@ func ModelScore(prompt string, candidates []string, opts ScoreOptions, loader *m
 			Candidates:           candidates,
 			IncludeTokenLogprobs: opts.IncludeTokenLogprobs,
 			LengthNormalize:      opts.LengthNormalize,
-			StablePrefixLen:      int32(opts.StablePrefixLen),
 		})
 		results := scoreResponseToCandidates(resp, opts.IncludeTokenLogprobs)
 		if appConfig.EnableTracing {

core/backend/transcript.go

@@ -51,14 +51,10 @@ func loadTranscriptionModel(ctx context.Context, ml *model.ModelLoader, modelCon
 	if modelConfig.Backend == "" {
 		modelConfig.Backend = model.WhisperBackend
 	}
-	// model.WithContext carries the request context into the load so distributed
-	// routing decisions reach the request's X-LocalAI-Node holder via
-	// distributedhdr.Stamp. context.WithoutCancel keeps those values but drops
-	// the request's cancellation, so a slow first load still completes and
-	// caches if the client disconnects instead of aborting the LoadModel RPC and
-	// tearing down the backend process (issue #10636). Inference below keeps the
-	// cancellable ctx, so a disconnect still stops generation.
-	opts := ModelOptions(modelConfig, appConfig, model.WithContext(context.WithoutCancel(ctx)))
+	// model.WithContext(ctx) overrides the app-context default set in
+	// ModelOptions so distributed routing decisions reach the request's
+	// X-LocalAI-Node holder via distributedhdr.Stamp.
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
 	transcriptionModel, err := ml.Load(opts...)
 	if err != nil {
 		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)

core/backend/tts.go

@@ -57,14 +57,10 @@ func ModelTTS(
 	appConfig *config.ApplicationConfig,
 	modelConfig config.ModelConfig,
 ) (string, *proto.Result, error) {
-	// model.WithContext carries the request context into the load so distributed
-	// routing decisions reach the request's X-LocalAI-Node holder via
-	// distributedhdr.Stamp. context.WithoutCancel keeps those values but drops
-	// the request's cancellation, so a slow first load still completes and
-	// caches if the client disconnects instead of aborting the LoadModel RPC and
-	// tearing down the backend process (issue #10636). Inference below keeps the
-	// cancellable ctx, so a disconnect still stops generation.
-	opts := ModelOptions(modelConfig, appConfig, model.WithContext(context.WithoutCancel(ctx)))
+	// model.WithContext(ctx) overrides the app-context default set in
+	// ModelOptions so distributed routing decisions reach the request's
+	// X-LocalAI-Node holder via distributedhdr.Stamp.
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
 	ttsModel, err := loader.Load(opts...)
 	if err != nil {
 		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)
@@ -164,9 +160,7 @@ func ModelTTSStream(
 	modelConfig config.ModelConfig,
 	audioCallback func([]byte) error,
 ) error {
-	// See ModelTTS above: WithoutCancel decouples the load from request
-	// cancellation while preserving routing values (issue #10636).
-	opts := ModelOptions(modelConfig, appConfig, model.WithContext(context.WithoutCancel(ctx)))
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
 	ttsModel, err := loader.Load(opts...)
 	if err != nil {
 		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)

core/backend/vad.go

@@ -14,14 +14,10 @@ func VAD(request *schema.VADRequest,
 	ml *model.ModelLoader,
 	appConfig *config.ApplicationConfig,
 	modelConfig config.ModelConfig) (*schema.VADResponse, error) {
-	// model.WithContext carries the request context into the load so distributed
-	// routing decisions reach the request's X-LocalAI-Node holder via
-	// distributedhdr.Stamp. context.WithoutCancel keeps those values but drops
-	// the request's cancellation, so a slow first load still completes and
-	// caches if the client disconnects instead of aborting the LoadModel RPC and
-	// tearing down the backend process (issue #10636). Inference below keeps the
-	// cancellable ctx, so a disconnect still stops generation.
-	opts := ModelOptions(modelConfig, appConfig, model.WithContext(context.WithoutCancel(ctx)))
+	// model.WithContext(ctx) overrides the app-context default set in
+	// ModelOptions so distributed routing decisions reach the request's
+	// X-LocalAI-Node holder via distributedhdr.Stamp.
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
 	vadModel, err := ml.Load(opts...)
 	if err != nil {
 		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)

core/cli/run.go

@@ -243,23 +243,6 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		return nil
 	}
 
-	activatedListeners, err := systemdActivatedListeners()
-	if err != nil {
-		return fmt.Errorf("loading systemd socket activation listeners: %w", err)
-	}
-	activatedListener, err := selectSystemdListener(activatedListeners)
-	if err != nil {
-		for _, listener := range activatedListeners {
-			_ = listener.Close()
-		}
-		return err
-	}
-	if activatedListener != nil {
-		defer func() {
-			_ = activatedListener.Close()
-		}()
-	}
-
 	os.MkdirAll(r.BackendsPath, 0750)
 	os.MkdirAll(r.ModelsPath, 0750)
 
@@ -749,13 +732,8 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	// LAN, or VPN that's the historical "trusted network" deployment, but on
 	// a public IP it makes every model, gallery install, settings change, and
 	// admin endpoint reachable by anyone who can connect to the port.
-	listenAddress := r.Address
-	if activatedListener != nil {
-		listenAddress = activatedListener.Addr().String()
-	}
-
 	authConfigured := app.AuthDB() != nil || len(r.APIKeys) > 0
-	if err := requireAuthOrTrustedBind(listenAddress, authConfigured, r.AllowInsecurePublicBind); err != nil {
+	if err := requireAuthOrTrustedBind(r.Address, authConfigured, r.AllowInsecurePublicBind); err != nil {
 		return err
 	}
 
@@ -765,11 +743,7 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		return err
 	}
 
-	if activatedListener != nil {
-		appHTTP.Listener = activatedListener
-		xlog.Info("Using systemd socket activation listener", "address", listenAddress)
-	}
-	xlog.Info("LocalAI is started and running", "address", listenAddress)
+	xlog.Info("LocalAI is started and running", "address", r.Address)
 
 	// Start P2P if token was provided via CLI/env or loaded from runtime_settings.json
 	if token != "" || app.ApplicationConfig().P2PToken != "" {
@@ -788,11 +762,11 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	// backends like PostgreSQL need to call the embeddings API during
 	// collection initialization.
 	go func() {
-		waitForServerReady(listenAddress, app.ApplicationConfig().Context)
+		waitForServerReady(r.Address, app.ApplicationConfig().Context)
 		app.StartAgentPool()
 	}()
 
-	return appHTTP.Start(listenAddress)
+	return appHTTP.Start(r.Address)
 }
 
 // waitForServerReady polls the given address until the HTTP server is

core/cli/run_socket_activation.go

@@ -1,17 +0,0 @@
-package cli
-
-import (
-	"fmt"
-	"net"
-)
-
-func selectSystemdListener(listeners []net.Listener) (net.Listener, error) {
-	switch len(listeners) {
-	case 0:
-		return nil, nil
-	case 1:
-		return listeners[0], nil
-	default:
-		return nil, fmt.Errorf("systemd socket activation requires exactly one stream listener, got %d", len(listeners))
-	}
-}

core/cli/run_socket_activation_linux.go

@@ -1,71 +0,0 @@
-//go:build linux
-
-package cli
-
-import (
-	"fmt"
-	"net"
-	"os"
-	"strconv"
-)
-
-const systemdListenFDStart = 3
-
-func systemdActivatedListeners() ([]net.Listener, error) {
-	listenPID := os.Getenv("LISTEN_PID")
-	listenFDs := os.Getenv("LISTEN_FDS")
-	if listenPID == "" && listenFDs == "" {
-		return nil, nil
-	}
-
-	defer func() {
-		for _, key := range []string{"LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES"} {
-			_ = os.Unsetenv(key)
-		}
-	}()
-
-	pid, err := strconv.Atoi(listenPID)
-	if err != nil {
-		return nil, fmt.Errorf("invalid LISTEN_PID %q: %w", listenPID, err)
-	}
-	count, err := strconv.Atoi(listenFDs)
-	if err != nil || count < 0 {
-		return nil, fmt.Errorf("invalid LISTEN_FDS %q", listenFDs)
-	}
-	if pid != os.Getpid() || count == 0 {
-		return nil, nil
-	}
-
-	return listenersFromSystemdFDs(systemdListenFDStart, count)
-}
-
-func listenersFromSystemdFDs(start, count int) (_ []net.Listener, err error) {
-	listeners := make([]net.Listener, 0, count)
-	defer func() {
-		if err != nil {
-			for _, listener := range listeners {
-				_ = listener.Close()
-			}
-		}
-	}()
-
-	for offset := range count {
-		fd := uintptr(start + offset)
-		file := os.NewFile(fd, fmt.Sprintf("LISTEN_FD_%d", fd))
-		if file == nil {
-			return nil, fmt.Errorf("opening systemd listener file descriptor %d", fd)
-		}
-		listener, listenerErr := net.FileListener(file)
-		closeErr := file.Close()
-		if listenerErr != nil {
-			return nil, fmt.Errorf("using systemd file descriptor %d as a stream listener: %w", fd, listenerErr)
-		}
-		if closeErr != nil {
-			_ = listener.Close()
-			return nil, fmt.Errorf("closing inherited systemd file descriptor %d: %w", fd, closeErr)
-		}
-		listeners = append(listeners, listener)
-	}
-
-	return listeners, nil
-}