fix(audio-cpp): forward voice reference transcripts

Saved voices send ref_text, but Fish Audio requires reference_text. Derive the canonical parameter while preserving explicit overrides. Both TTS modes use the shared builder. Add regression cases and document the parameter alias. Assisted-by: Codex:gpt-6
fix(agents): pass pool limits to the LocalAGI agent pool
2026-09-12 14:22:11 -04:00 · 2026-09-12 16:06:11 +00:00 · 2026-09-12 08:15:23 +00:00 · 2026-09-11 23:34:59 +00:00 · 2026-09-11 23:19:28 +00:00 · 2026-09-11 23:13:32 +00:00
797 changed files with 57594 additions and 5612 deletions
@@ -49,6 +49,40 @@ AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
 A human reviewer owns the contribution; the AI's involvement is recorded
 via `Assisted-by` (see below).

+### Exception: automation operated by a maintainer
+
+The rule above addresses the common case, an AI assistant helping a human
+contributor who then signs off. It does not fit automation that a
+maintainer runs themselves, which opens pull requests with no human
+submitter to sign. Applied literally there, nothing ever signs and the
+DCO check blocks the pull request permanently.
+
+A maintainer-operated bot MUST therefore add a `Signed-off-by` trailer
+naming **the maintainer who operates it**, not the bot and not the model:
+
+```
+Assisted-by: Codex:gpt-5
+Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
+```
+
+This is not the AI certifying the DCO. The maintainer is, exactly as they
+do for a commit they typed by hand: they configured the automation, they
+own its output, and they take responsibility for it when they merge it.
+The `Assisted-by` trailer still records that a model produced the code, so
+the provenance trail is unchanged.
+
+The exception is narrow and does not widen the rule for anyone else:
+
+- It applies only to automation a LocalAI maintainer operates and whose
+  output that maintainer reviews before merge.
+- The sign-off names a real person who accepts DCO responsibility.
+- An AI assistant helping an outside contributor still MUST NOT sign off.
+  That contributor adds their own trailer.
+- A bot MUST NOT sign off on behalf of anyone other than its operator, and
+  MUST NOT add a trailer for a contributor whose branch it pushes to. If
+  automation contributes to someone else's branch, it leaves the sign-off
+  to that contributor.
+
 ## Attribution

 When AI tools contribute to LocalAI development, proper attribution helps
@@ -236,6 +236,58 @@ Use these HTTP status codes:

 If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.

+## Control-plane database health metrics
+
+In distributed mode the frontend registers three OpenTelemetry gauges over the
+PostgreSQL control-plane database (`core/services/monitoring/control_plane_db.go`,
+wired in `core/application/distributed.go`). They reach `/metrics` through the
+same Prometheus exporter as the rest of the API metrics.
+
+| Metric | Meaning | Page when |
+|--------|---------|-----------|
+| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising |
+| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 |
+| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table |
+
+A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on.
+While it grows, autovacuum can reclaim nothing anywhere in the database no
+matter how often it runs, so the dead tuple ratio keeps climbing and a six-row
+registry table can reach hundreds of megabytes. Tuning autovacuum does not help.
+The fix is to find the transaction holding the horizon open and clear it:
+
+```sql
+SELECT pid, state, age(backend_xmin) AS xmin_age, now() - xact_start AS xact_age, query
+FROM pg_stat_activity
+WHERE backend_xmin IS NOT NULL
+ORDER BY age(backend_xmin) DESC;
+```
+
+Then `pg_terminate_backend(pid)` on the offenders, and `VACUUM (VERBOSE)` the
+bloated tables once the horizon has moved.
+
+**A healthy-looking xmin age does not on its own prove the horizon is free.**
+The gauge reads `pg_stat_activity`, which only sees live backends. Two other
+things pin the very same horizon and are invisible there, so either one can hold
+vacuum back while the gauge reads 0:
+
+```sql
+SELECT gid, prepared, database, transaction FROM pg_prepared_xacts;
+SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;
+```
+
+An orphaned prepared transaction is cleared with `ROLLBACK PREPARED '<gid>'`,
+and a stale slot with `pg_drop_replication_slot('<slot_name>')`. Check both
+before concluding that a bloated table has some other cause.
+
+Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not
+translate into database load. Failed and timed-out samples cost the same interval
+as successful ones, so a database that is already struggling is not retried on
+every scrape. A failed sample reports the last good values rather than failing the
+scrape, because these gauges matter most when the database is struggling. Before
+the first successful sample the gauges are absent rather than zero, since a zero
+xmin age would read as a healthy horizon: alert on `absent()` too if you need to
+distinguish "healthy" from "never sampled".
+
 ## Advertising surfaces — where to register a new capability

 Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.
@@ -77,6 +77,56 @@ spectrum. **Metal (Darwin) only** - it is a no-op on CUDA/CPU. Enable with
 budget). Gallery entries built on this: `deepseek-v4-flash-q4-ssd` (153 GB Flash
 on a 128 GB Mac) and `deepseek-v4-pro-q2-ssd` (433 GB Pro, experimental).

+## CUDA architecture (do not build without one)
+
+`backend/cpp/ds4/Makefile` drives upstream's **object targets** directly
+(`$(MAKE) -C ds4 ds4.o ds4_cuda.o ...`), which bypasses upstream's own guard:
+its `cuda` target refuses to build unless `CUDA_ARCH` is set, and offers
+`cuda-spark` (sm_121, DGX Spark / GB10) and `cuda-generic` (native) instead.
+Built with no `-arch`, nvcc targets its default architecture and the kernels run
+as JIT'd PTX. On GB10 that silently corrupted every prefill batch of >=128
+tokens - the model emitted text unrelated to the prompt and never closed its
+thinking block, so `content` came back empty - and cost close to two orders of
+magnitude of prefill throughput (4.21 t/s vs 325.70 t/s, same box, same model).
+Short prompts stayed correct, which is why it went unnoticed.
+
+The Makefile therefore picks a gencode list from `CUDA_MAJOR_VERSION` (a build
+arg the backend matrix already declares, forwarded by `Dockerfile.ds4`) and
+`uname -m`, and passes it as `NVCC_ARCH_FLAGS` to the sub-make. Upstream's
+`CUDA_ARCH` accepts a single value, so it cannot express the fat binary the
+shipped images need; a command-line assignment beats its `:=`. An empty
+`CUDA_MAJOR_VERSION` falls back to upstream's `native` for local developer
+builds, and an unrecognised one is a hard error - no CI runner has a GPU, so a
+silent `native` there is exactly the failure mode this guards against.
+
+`DS4_CUDA_HAVE_MXF4` is deliberately unset: upstream defines it only for
+single-arch sm_120/sm_121 builds and guards it with a plain `#ifdef` rather than
+`__CUDA_ARCH__`, so it cannot be combined with older archs. It gates an optional
+MXFP4 indexer fast path whose `#ifndef` branch returns 0, so omitting it costs
+speed, not correctness.
+
+### Verifying a build
+
+Check which flags a configuration resolves to, without compiling anything:
+
+```
+make -C backend/cpp/ds4 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 NATIVE=false \
+  --eval='show: ; @echo [$(DS4_ARCH_MAKEVARS)]' show
+```
+
+Do not use `make -n` for this: the recipe is `+$(MAKE) ...`, and the `+` prefix
+makes it run even under `-n`.
+
+Then exercise the failure mode itself against a built backend. It only appears
+above one prefill batch, so the ordinary `predict` spec cannot catch it:
+
+```
+BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \
+BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \
+BACKEND_TEST_CAPS=health,load,predict,long_prefill \
+go test -count=1 -timeout=30m -v ./tests/e2e-backends/...
+```
+
 ## Build matrix

 | Build | Where | Notes |
@@ -59,7 +59,9 @@ backend/rust/*/target
 backend-images
 local-backends
 local-ai
+.claude
 .crush
+.tools
 protoc
 tests

@@ -5,7 +5,7 @@ This PR fixes #
 **Notes for Reviewers**


-**[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
+**[Signed commits](../CONTRIBUTING.md#commit-messages)**
 - [ ] Yes, I signed my commits.
 - [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable
 
@@ -166,6 +166,19 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-whisper-medusa'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'true'
+    backend: "whisper-medusa"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: ''
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -354,6 +367,19 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-funasr'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'cublas'
    cuda-major-version: "12"
    cuda-minor-version: "8"
@@ -626,6 +652,19 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "1"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-whisper-medusa'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "whisper-medusa"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'cublas'
    cuda-major-version: "12"
    cuda-minor-version: "8"
@@ -1069,6 +1108,19 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-funasr'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'cublas'
    cuda-major-version: "13"
    cuda-minor-version: "0"
@@ -1404,6 +1456,19 @@ include:
    backend: "qwen-asr"
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
+  - build-type: 'l4t'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-funasr'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    ubuntu-version: '2404'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
  - build-type: 'l4t'
    cuda-major-version: "13"
    cuda-minor-version: "0"
@@ -2388,6 +2453,19 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-funasr'
+    runs-on: 'ubuntu-latest'
+    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+    skip-drivers: 'false'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'hipblas'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2668,6 +2746,19 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2204'
+  - build-type: 'l4t'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-funasr'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    skip-drivers: 'true'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2204'
  - build-type: 'l4t'
    cuda-major-version: "12"
    cuda-minor-version: "0"
@@ -2877,6 +2968,19 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'intel'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-funasr'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'intel'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -3208,9 +3312,10 @@ include:
  # consumed. Same reason CUDA needs its toolkit in base-image rather than in a
  # builder image: this is the ds4 shape, not the llama-cpp one.
  #
-  # No ROCm entry: upstream has no HIP configuration. No CUDA arm64 or L4T
-  # entry: upstream documents and validates CUDA on x86 only. Darwin/Metal is in
-  # the includeDarwin matrix below, built by scripts/build/audio-cpp-darwin.sh.
+  # ROCm uses upstream's HIP backend and the project-wide ROCm 7.2.1 base. No
+  # CUDA arm64 or L4T entry: upstream documents and validates CUDA on x86 only.
+  # Darwin/Metal is in the includeDarwin matrix below, built by
+  # scripts/build/audio-cpp-darwin.sh.
  #
  # No vulkan entry either, though Dockerfile.audio-cpp and the backend Makefile
  # both handle BUILD_TYPE=vulkan for local builds. Every other vulkan backend
@@ -3278,6 +3383,19 @@ include:
    dockerfile: "./backend/Dockerfile.audio-cpp"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-audio-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+    skip-drivers: 'false'
+    backend: "audio-cpp"
+    dockerfile: "./backend/Dockerfile.audio-cpp"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: ''
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -3754,6 +3872,19 @@ include:
    dockerfile: "./backend/Dockerfile.golang"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-stablediffusion-ggml'
+    runs-on: 'ubuntu-latest'
+    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+    skip-drivers: 'false'
+    backend: "stablediffusion-ggml"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'sycl_f16'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -6058,6 +6189,20 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-funasr'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: ''
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -6072,6 +6217,20 @@ include:
    dockerfile: "./backend/Dockerfile.python"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-funasr'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "funasr"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: ''
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -6311,6 +6470,9 @@ includeDarwin:
  - backend: "mlx"
    tag-suffix: "-metal-darwin-arm64-mlx"
    build-type: "mps"
+  - backend: "mlx-video"
+    tag-suffix: "-metal-darwin-arm64-mlx-video"
+    build-type: "mps"
  - backend: "chatterbox"
    tag-suffix: "-metal-darwin-arm64-chatterbox"
    build-type: "mps"
@@ -6447,6 +6609,9 @@ includeDarwin:
  - backend: "qwen-asr"
    tag-suffix: "-metal-darwin-arm64-qwen-asr"
    build-type: "mps"
+  - backend: "funasr"
+    tag-suffix: "-metal-darwin-arm64-funasr"
+    build-type: "mps"
  - backend: "nemo"
    tag-suffix: "-metal-darwin-arm64-nemo"
    build-type: "mps"
@@ -0,0 +1,49 @@
+#!/bin/bash
+# Bump the CTranslate2 ROCm Python wheel release pin used by faster-whisper.
+set -xe
+
+REPO=$1   # OpenNMT/CTranslate2
+FILE=$2   # backend/python/faster-whisper/install.sh
+VAR=$3    # CTRANSLATE2_VERSION (used for output file names so the workflow can read them)
+
+if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
+    echo "usage: $0 <repo> <install-script> <var-name>" >&2
+    exit 1
+fi
+
+LATEST_RELEASE=$(curl -sS -H "Accept: application/vnd.github+json" \
+    "https://api.github.com/repos/$REPO/releases/latest")
+LATEST_TAG=$(python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])" <<< "$LATEST_RELEASE")
+
+set +e
+CURRENT_VERSION=$(grep -m1 "^${VAR}=" "$FILE" | cut -d= -f2 | sed -E 's/^\$\{[^:]+:-([^}]+)\}$/\1/')
+CTRANSLATE2_ROCM_WHEEL_OS=$(grep -m1 '^CTRANSLATE2_ROCM_WHEEL_OS=' "$FILE" | cut -d= -f2 | sed -E 's/^\$\{[^:]+:-([^}]+)\}$/\1/')
+set -e
+
+if [ -z "$CURRENT_VERSION" ]; then
+    echo "Could not find $VAR in $FILE."
+    exit 0
+fi
+
+if [ -z "$CTRANSLATE2_ROCM_WHEEL_OS" ]; then
+    echo "Could not find CTRANSLATE2_ROCM_WHEEL_OS in $FILE."
+    exit 0
+fi
+
+ASSET_NAME="rocm-python-wheels-${CTRANSLATE2_ROCM_WHEEL_OS}.zip"
+LATEST_RELEASE="$LATEST_RELEASE" python3 - "$ASSET_NAME" <<'PY'
+import json
+import os
+import sys
+
+asset_name = sys.argv[1]
+release = json.loads(os.environ["LATEST_RELEASE"])
+assets = {asset.get("name") for asset in release.get("assets", [])}
+if asset_name not in assets:
+    raise SystemExit(f"Could not find release asset {asset_name!r}")
+PY
+
+sed -i "$FILE" -e "s/^${VAR}=.*/${VAR}=\${${VAR}:-${LATEST_TAG}}/"
+
+echo "Changes: https://github.com/$REPO/compare/${CURRENT_VERSION}...${LATEST_TAG}" >> "${VAR}_message.txt"
+echo "${LATEST_TAG}" >> "${VAR}_commit.txt"
@@ -3,9 +3,9 @@
 # darwin (Apple Silicon) install path. The macOS/Metal build
 # (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is
 # version-locked to a specific vLLM source release. install.sh derives that vLLM
-# version at build time from vllm-metal's own installer at the pinned
-# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh,
-# which bumps the Linux cu130 wheel pin.
+# version, and the wheel asset name, at build time from the pinned tag, so there
+# is only ONE value to bump here -- mirroring bump_vllm_wheel.sh, which bumps the
+# Linux cu130 wheel pin.
 #
 # This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the
 # darwin build can only use the exact vLLM version vllm-metal supports, so it may
@@ -23,15 +23,20 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
    exit 1
 fi

-# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
-# /releases/latest returns the newest one (with its cp312 wheel asset).
+# vllm-metal ships frequent .dev releases, flagged as prereleases, alongside the
+# stable ones. /releases/latest skips the prereleases and returns the newest
+# stable tag, which is what darwin should pin: upstream deletes and re-cuts .dev
+# tags, and a pin to a deleted tag 404s the whole build.
 LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
    "https://api.github.com/repos/$REPO/releases/latest" \
    | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")

-# The coupled vLLM source version lives in vllm-metal's installer at that tag.
-NEW_VLLM_VERSION=$(gh_curl \
-    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
+# The coupled vLLM release lives in .github/vllm-release-tag.commit at that tag
+# (since vllm-metal 0.28); releases predating that file pinned it inline in their
+# own install.sh. The extractor reads both forms.
+NEW_VLLM_VERSION=$( { gh_curl \
+    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/.github/vllm-release-tag.commit" \
+    || gh_curl "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh"; } \
    | "$(dirname "${BASH_SOURCE[0]}")/../scripts/lib/extract-vllm-metal-version.sh")

 if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+python3 - <<'PY'
+from pathlib import Path
+
+home = Path("website/layouts/index.html").read_text()
+css = Path("website/static/css/site.css").read_text()
+install = Path("docs/content/getting-started/install.md").read_text()
+containers = Path("docs/content/getting-started/containers.md").read_text()
+
+def require(condition, message):
+    if not condition:
+        raise SystemExit(f"FAIL: {message}")
+
+require("Drop-in replacement for most upstream APIs." in home,
+        "homepage must use the requested drop-in API heading")
+require("Everything else plugs into LocalAI." not in home,
+        "old runtime heading must be removed")
+require("When the engine we need" not in home,
+        "hero must describe user outcomes instead of team implementation")
+require('href="mailto:contact@localai.io"' in home and "business" in home.lower(),
+        "homepage must provide a direct business contact action")
+require(home.index('id="localai"') < home.index('id="proof-quotes"') < home.index('id="mission"'),
+        "headline testimonials must directly follow the runtime section")
+require(home.count('id="proof-quotes"') == 1,
+        "headline testimonials must appear exactly once")
+require('id="engines"' not in home and "Engines we build" not in home,
+        "homepage engine showcase must be removed")
+require('href="/docs/installation/index.html"' in home,
+        "installation guide action must use the direct installation URL")
+require('<iframe' in install and "youtube.com/embed/cMVNnlqwfw4" in install,
+        "installation page must embed the walkthrough video")
+require("## Quick Start" not in install,
+        "installation landing page must not duplicate Quick Start")
+for text in ("CUDA 12", "CUDA 13", "ROCm", "Intel", "Jetson", "Vulkan", "fallback"):
+    require(text.lower() in containers.lower(), f"GPU chooser must explain {text}")
+require('class="sn__e"><a href="https://github.com/mudler/parakeet.cpp">parakeet.cpp</a>' in home,
+        "capability engine names must link to their repositories")
+require(".pane{min-height:" in css.replace(" ", ""),
+        "all installation panes must have a fixed minimum height")
+
+print("website review 143 source checks passed")
+PY
@@ -29,10 +29,6 @@ updates:
    schedule:
      # Check for updates to GitHub Actions every weekday
      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/bark"
-    schedule:
-      interval: "weekly"
  - package-ecosystem: "pip"
    directory: "/backend/python/common/template"
    schedule:
@@ -55,30 +51,10 @@ updates:
    ignore:
    - dependency-name: "torch"
    - dependency-name: "transformers"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/exllama"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/exllama2"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/mamba"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/openvoice"
-    schedule:
-      interval: "weekly"
  - package-ecosystem: "pip"
    directory: "/backend/python/rerankers"
    schedule:
      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/sentencetransformers"
-    schedule:
-      interval: "weekly"
  - package-ecosystem: "pip"
    directory: "/backend/python/transformers"
    schedule:
@@ -86,44 +62,4 @@ updates:
  - package-ecosystem: "pip"
    directory: "/backend/python/vllm"
    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/chainlit"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/functions"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/langchain/langchainpy-localai-example"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/langchain-chroma"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/streamlit-bot"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/k8sgpt"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/kubernetes"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/langchain"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "gomod"
-    directory: "/examples/semantic-todo"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/telegram-bot"
-    schedule:
-      interval: "weekly"
+      interval: "weekly"
@@ -166,7 +166,40 @@ jobs:
          push-to-fork: ci-forks/LocalAI
          commit-message: ':arrow_up: Update ${{ matrix.repository }}'
          title: 'chore: :arrow_up: Update ${{ matrix.repository }} to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/${{ matrix.variable }}"
+          branch: "bump/${{ matrix.variable }}"
+          body: ${{ steps.bump.outputs.message }}
+          signoff: true
+
+  bump-ctranslate2-rocm-wheel:
+    # CTranslate2's ROCm wheels are published as release assets, so the
+    # faster-whisper hipblas install path pins the release tag used in the URL.
+    if: github.repository == 'mudler/LocalAI'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - name: Bump CTranslate2 ROCm wheel pin 🔧
+        id: bump
+        run: |
+          bash .github/bump_ctranslate2_rocm_wheel.sh OpenNMT/CTranslate2 backend/python/faster-whisper/install.sh CTRANSLATE2_VERSION
+          {
+            echo 'message<<EOF'
+            cat "CTRANSLATE2_VERSION_message.txt"
+            echo EOF
+          } >> "$GITHUB_OUTPUT"
+          {
+            echo 'commit<<EOF'
+            cat "CTRANSLATE2_VERSION_commit.txt"
+            echo EOF
+          } >> "$GITHUB_OUTPUT"
+          rm -rfv CTRANSLATE2_VERSION_message.txt CTRANSLATE2_VERSION_commit.txt
+      - name: Create Pull Request
+        uses: peter-evans/create-pull-request@v8
+        with:
+          token: ${{ secrets.UPDATE_BOT_TOKEN }}
+          push-to-fork: ci-forks/LocalAI
+          commit-message: ':arrow_up: Update OpenNMT/CTranslate2 ROCm wheel'
+          title: 'chore: :arrow_up: Update OpenNMT/CTranslate2 ROCm wheel to `${{ steps.bump.outputs.commit }}`'
+          branch: "update/CTRANSLATE2_VERSION"
          body: ${{ steps.bump.outputs.message }}
          signoff: true

@@ -203,7 +236,7 @@ jobs:
          push-to-fork: ci-forks/LocalAI
          commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel'
          title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_VERSION"
+          branch: "bump/VLLM_VERSION"
          body: ${{ steps.bump.outputs.message }}
          signoff: true

@@ -241,6 +274,6 @@ jobs:
          push-to-fork: ci-forks/LocalAI
          commit-message: ':arrow_up: Update vllm-project/vllm-metal (darwin)'
          title: 'chore: :arrow_up: Update vllm-metal (darwin) to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_METAL_VERSION"
+          branch: "bump/VLLM_METAL_VERSION"
          body: ${{ steps.bump.outputs.message }}
          signoff: true
@@ -31,13 +31,14 @@ jobs:
            messages: [
                {
                role: "system",
-                content: "Write a discord message with a bullet point summary of the release notes."
+                content: "Write a Discord message with a bullet point summary of the release notes. Keep the complete message under 1800 characters."
                },
                {
                role: "user",
                content: $input
                }
-            ]
+            ],
+            max_tokens: 450
            }')

            # Send the request to LocalAI API
@@ -46,7 +47,7 @@ jobs:
            -d "$json_payload")

            # Extract the summary from the response
-            summary=$(echo $response | jq -r '.choices[0].message.content')
+            summary=$(printf '%s' "$response" | jq -er '.choices[0].message.content | strings | .[0:1800]')

            # Print the summary
            #  -H "Authorization: Bearer $API_KEY" \
@@ -14,6 +14,7 @@ on:

 permissions:
  contents: write
+  pull-requests: write

 concurrency:
  group: refresh-site-counters
@@ -30,15 +31,25 @@ jobs:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: ./.github/ci/refresh-site-counters.sh

-      - name: Commit only if something moved
+      - name: Show changes
        run: |
          if git diff --quiet -- website/data/stats.yaml; then
-            echo "counters unchanged, nothing to commit"
-            exit 0
+            echo "counters unchanged"
+          else
+            git diff --unified=0 -- website/data/stats.yaml
          fi
-          git diff --unified=0 -- website/data/stats.yaml
-          git config user.name  "github-actions[bot]"
-          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-          git add website/data/stats.yaml
-          git commit -m "chore(website): refresh the counters"
-          git push
+
+      - name: Create pull request when counters moved
+        uses: peter-evans/create-pull-request@v8
+        with:
+          token: ${{ secrets.UPDATE_BOT_TOKEN }}
+          push-to-fork: ci-forks/LocalAI
+          commit-message: "chore(website): refresh the counters"
+          title: "chore(website): refresh the counters"
+          body: |
+            Weekly refresh of the landing-page counters from the GitHub API.
+
+            This PR was created automatically by the `refresh-site-counters` workflow.
+          branch: update/site-counters
+          delete-branch: true
+          labels: automated
@@ -31,7 +31,7 @@ jobs:
        if: ${{ !github.repository.fork && github.actor != 'dependabot[bot]' }}
      - name: Run Gosec Security Scanner
        if: ${{ !github.repository.fork && github.actor != 'dependabot[bot]' }}
-        uses: securego/gosec@v2.27.1
+        uses: securego/gosec@v2.29.0
        with:
          # we let the report trigger content trigger a failure using the GitHub Security features.
          # backend/go/supertonic is excluded: it vendors upstream supertone-inc/supertonic
@@ -526,6 +526,7 @@ jobs:
      - name: Build llama-cpp backend image and run gRPC e2e tests
        run: |
          make test-extra-backend-llama-cpp
+          make test-extra-backend-llama-cpp-embeddings
  tests-llama-cpp-grpc-transcription:
    needs: detect-changes
    if: needs.detect-changes.outputs.llama-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
@@ -65,6 +65,12 @@ jobs:
      - name: Test (with coverage gate)
        run: |
          PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test-coverage-check
+      # tests/integration is outside the coverage roots because its store specs
+      # need a live backend. test-stores builds and installs local-store before
+      # running the complete suite, so new local-store specs are collected
+      # automatically without adding another workflow entry.
+      - name: Test local-store integration
+        run: PATH="$PATH:$HOME/go/bin" make test-stores
      - name: Upload coverage report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
@@ -74,8 +80,13 @@ jobs:
            coverage/coverage.out
            coverage/coverage.html
          if-no-files-found: ignore
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
        uses: mxschmitt/action-tmate@v3.23
        with:
          detached: true
@@ -119,8 +130,13 @@ jobs:
          export PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
          PATH="$PATH:$HOME/go/bin" make protogen-go
          PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target test
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
        uses: mxschmitt/action-tmate@v3.23
        with:
          detached: true
@@ -77,8 +77,13 @@ jobs:
      - name: Test
        run: |
            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
        uses: mxschmitt/action-tmate@v3.23
        with:
          detached: true
@@ -63,8 +63,13 @@ jobs:
      - name: Test Backend E2E
        run: |
          PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
        uses: mxschmitt/action-tmate@v3.23
        with:
          detached: true
@@ -88,8 +88,13 @@ jobs:
      # CPU and runs the token_classify capability spec (byte-offset contract).
      - name: Run live PII NER backend E2E
        run: PATH="$PATH:$HOME/go/bin" make test-extra-backend-privacy-filter
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
        uses: mxschmitt/action-tmate@v3.23
        with:
          detached: true
@@ -52,6 +52,8 @@ jobs:
        run: |
          sudo apt-get update
          sudo apt-get install -y build-essential libopus-dev
+      - name: Run stale chunk recovery tests
+        run: PATH="$PATH:$HOME/go/bin" make test-ui-stale-chunk
      # Builds an instrumented UI bundle, runs the Playwright specs, and fails
      # if line coverage regressed beyond the jitter tolerance (the gate is
      # in `make test-ui-coverage-check`). PLAYWRIGHT_CHROMIUM_PATH is unset
@@ -73,8 +75,13 @@ jobs:
          path: core/http/react-ui/coverage/
          if-no-files-found: ignore
          retention-days: 7
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
        uses: mxschmitt/action-tmate@v3.23
        with:
          detached: true
@@ -0,0 +1,21 @@
+## Design Context
+
+### Users
+
+LocalAI serves both single-host users who want to install and try models quickly and experienced developers, ML engineers, system administrators, and DevOps operators who manage production hosts or distributed clusters. The interface must support first-time discovery without hiding the runtime state, configuration, and control that returning operators need.
+
+### Brand Personality
+
+Capable, easy to use, and trustworthy. The interface should make sophisticated local-AI infrastructure feel understandable and under control. It should be direct and calm rather than playful, ornamental, or intimidating.
+
+### Aesthetic Direction
+
+Use LocalAI's established technical, editorial design language: Geist typography, compact information density, sharp geometry, deep blue-black surfaces, action blue, mint for healthy/local/live state, and amber only for decisions requiring attention. Support both dark and light themes. Avoid generic card dashboards, decorative gradients, glass effects, and visual noise.
+
+### Design Principles
+
+1. Use progressive disclosure to serve newcomers and operators in the same workflow: make the common path obvious, then reveal operational depth in context.
+2. Organize navigation around user intent and lifecycle state, not implementation concepts or nested containers.
+3. Give each resource one canonical home; expose discovery, installed state, and runtime state as clear views of that resource instead of duplicating management surfaces.
+4. Keep operational status visible and trustworthy through precise labels, explicit scope, and actionable state—not decoration.
+5. Preserve information density for expert use while flattening navigation and reducing repeated summaries, tabs, rails, and panels.
@@ -27,6 +27,7 @@ To be removed, open a pull request deleting your row, or email

 | Organisation | What they use it for | Status |
 |---|---|---|
+| [walcz.de](https://walcz.de) | Self-hosted appliance for a German B2B consultancy: local-only inference on AMD Strix Halo (gfx1151/ROCm), agents with MCP tools, RAG over an internal knowledge base, and a document/bookkeeping pipeline. | Production |
 | _Your organisation here_ | | |

 ## What this list is not
@@ -8,7 +8,7 @@ Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development w

 LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:

- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
+- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. One exception: automation a maintainer operates signs off with *that maintainer's* identity, since no other human submitter exists to certify it. See [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md).
 - **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
 - **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
 - **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.
@@ -33,6 +33,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 | [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) | LocalAI Assistant chat modality — adding admin tools to the in-process MCP server, editing skill prompts, keeping REST + MCP + skills in sync |
 | [.agents/backend-signing.md](.agents/backend-signing.md) | Backend OCI image signing (keyless cosign + sigstore-go) — producer-side CI setup, consumer-side gallery `verification:` block, strict mode (`LOCALAI_REQUIRE_BACKEND_INTEGRITY`), revocation via `not_before` |
 | [.agents/preparing-a-release.md](.agents/preparing-a-release.md) | Cutting a release: PR labels, `RELEASE_NOTES_vX.Y.Z.md`, the blog post under `website/content/blog/`, and the demo clips under `website/static/media/` |
+| [.impeccable.md](.impeccable.md) | Design context for UI/UX work — users, brand personality, aesthetic direction, and design principles |

 ## Quick Reference

@@ -218,7 +218,7 @@ LocalAI follows the **same guidelines as the Linux kernel project** for AI-assis

 The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:

- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
+- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. Automation operated by a maintainer is the one exception: it signs off with that maintainer's identity, because there is no other human submitter to certify it.
 - **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
 - **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:

@@ -1,5 +1,7 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/mlx-video backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
+.NOTPARALLEL: backends/whisper-medusa
+.NOTPARALLEL: backends/funasr

 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -34,6 +36,11 @@ TEST_FLAKES?=5
 RANDOM := $(shell bash -c 'echo $$RANDOM')

 VERSION?=$(shell git describe --always --tags || echo "dev" )
+# fyne package only accepts numeric x[.y[.z]] app versions, so reduce git
+# describe output (v4.9.0, v4.9.0-14-gabc1234, or a bare sha on untagged
+# checkouts) to its numeric core; anything non-numeric falls back to 0.0.0.
+# Without this the packaged launcher reports itself as version 0.0.0 (#11673).
+LAUNCHER_APP_VERSION?=$(shell v=$$(echo "$(VERSION)" | sed -E 's/^v//; s/[+-].*$$//'); echo "$$v" | grep -qE '^[0-9]+(\.[0-9]+){0,2}$$' && echo "$$v" || echo "0.0.0")
 # go tool nm ./local-ai | grep Commit
 LD_FLAGS?=-s -w
 override LD_FLAGS += -X "github.com/mudler/LocalAI/internal.Version=$(VERSION)"
@@ -103,7 +110,7 @@ COVERAGE_E2E_LABELS?=!real-models
 COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go


-.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
+.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-stale-chunk test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all

 all: help

@@ -235,7 +242,7 @@ test-ci-scripts:
 ## pure stdlib on purpose so they run without any backend venv; the list is
 ## explicit because their siblings (model_identity_test) import grpc and the
 ## generated protobufs, which only exist inside a built backend.
-PYTHON_HELPER_TESTS?=python_utils_test vllm_utils_test model_utils_test mlx_utils_test parent_watch_test
+PYTHON_HELPER_TESTS?=python_utils_test vllm_utils_test model_utils_test mlx_utils_test parent_watch_test temp_utils_test
 test-python-helpers:
 	cd backend/python/common && python3 -m unittest $(PYTHON_HELPER_TESTS)

@@ -388,9 +395,17 @@ test-e2e: build-mock-backend build-cloud-proxy-backend prepare-e2e run-e2e-image
 	$(MAKE) teardown-e2e
 	docker rmi localai-tests

+# `docker stop` returns as soon as the container exits, but Docker reaps a
+# `--rm` container asynchronously after that. The `docker rmi localai-tests` in
+# test-e2e then loses the race against the reaper and fails on a still
+# referenced image, turning a green suite red. Removing the container ourselves
+# is synchronous, so the image reference is gone before we return. It also
+# covers the case where nothing is running, which `docker stop` could not
+# because it rejects an empty argument list.
 teardown-e2e:
 	rm -rf $(TEST_DIR) || true
-	docker stop $$(docker ps -q --filter ancestor=localai-tests)
+	@CONTAINERS=$$(docker ps -aq --filter ancestor=localai-tests 2>/dev/null); \
+	if [ -n "$$CONTAINERS" ]; then docker rm -f $$CONTAINERS || true; fi

 ########################################################
 ## Integration and unit tests
@@ -599,6 +614,7 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/vllm
 	$(MAKE) -C backend/python/vllm-omni
 	$(MAKE) -C backend/python/longcat-video
+	$(MAKE) -C backend/python/mlx-video
 	$(MAKE) -C backend/python/sglang
 	$(MAKE) -C backend/python/vibevoice
 	$(MAKE) -C backend/python/liquid-audio
@@ -608,9 +624,11 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/fish-speech
 	$(MAKE) -C backend/python/faster-qwen3-tts
 	$(MAKE) -C backend/python/qwen-asr
+	$(MAKE) -C backend/python/funasr
 	$(MAKE) -C backend/python/nemo
 	$(MAKE) -C backend/python/voxcpm
 	$(MAKE) -C backend/python/faster-whisper
+	$(MAKE) -C backend/python/whisper-medusa
 	$(MAKE) -C backend/python/whisperx
 	$(MAKE) -C backend/python/ace-step
 	$(MAKE) -C backend/python/trl
@@ -631,6 +649,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/vllm test
 	$(MAKE) -C backend/python/vllm-omni test
 	$(MAKE) -C backend/python/longcat-video test
+	$(MAKE) -C backend/python/mlx-video test
 	$(MAKE) -C backend/python/vibevoice test
 	$(MAKE) -C backend/python/liquid-audio test
 	$(MAKE) -C backend/python/moonshine test
@@ -639,9 +658,11 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/fish-speech test
 	$(MAKE) -C backend/python/faster-qwen3-tts test
 	$(MAKE) -C backend/python/qwen-asr test
+	$(MAKE) -C backend/python/funasr test
 	$(MAKE) -C backend/python/nemo test
 	$(MAKE) -C backend/python/voxcpm test
 	$(MAKE) -C backend/python/faster-whisper test
+	$(MAKE) -C backend/python/whisper-medusa test
 	$(MAKE) -C backend/python/whisperx test
 	$(MAKE) -C backend/python/ace-step test
 	$(MAKE) -C backend/python/trl test
@@ -676,6 +697,7 @@ test-extra: prepare-test-extra
 ##   BACKEND_TEST_PROMPT      Override the prompt used in predict/stream specs.
 ##   BACKEND_TEST_OPTIONS     Comma-separated Options[] entries forwarded to LoadModel,
 ##                            e.g. "tool_parser:hermes,reasoning_parser:qwen3".
+##   BACKEND_TEST_EMBEDDING_LAYOUT Expected EmbeddingResult layout: "final" or "per_token".
 ##
 ## Direct usage (image already built, no docker-build-* dependency):
 ##
@@ -705,6 +727,7 @@ test-extra-backend: protogen-go
 	BACKEND_TEST_CAPS="$$BACKEND_TEST_CAPS" \
 	BACKEND_TEST_PROMPT="$$BACKEND_TEST_PROMPT" \
 	BACKEND_TEST_OPTIONS="$$BACKEND_TEST_OPTIONS" \
+	BACKEND_TEST_EMBEDDING_LAYOUT="$$BACKEND_TEST_EMBEDDING_LAYOUT" \
 	BACKEND_TEST_TOOL_PROMPT="$$BACKEND_TEST_TOOL_PROMPT" \
 	BACKEND_TEST_TOOL_NAME="$$BACKEND_TEST_TOOL_NAME" \
 	BACKEND_TEST_CACHE_TYPE_K="$$BACKEND_TEST_CACHE_TYPE_K" \
@@ -724,6 +747,15 @@ test-extra-backend-llama-cpp: docker-build-llama-cpp
 	BACKEND_TEST_CAPS=health,load,predict,stream,logprobs,logit_bias \
 	$(MAKE) test-extra-backend

+## Raw llama.cpp embeddings are required by Go-side pooling. This exercises the
+## real C++ backend and verifies that it marks the flattened matrix per-token.
+test-extra-backend-llama-cpp-embeddings: docker-build-llama-cpp
+	BACKEND_IMAGE=local-ai-backend:llama-cpp \
+	BACKEND_TEST_CAPS=health,load,embeddings \
+	BACKEND_TEST_OPTIONS=pooling:none \
+	BACKEND_TEST_EMBEDDING_LAYOUT=per_token \
+	$(MAKE) test-extra-backend
+
 test-extra-backend-ik-llama-cpp: docker-build-ik-llama-cpp
 	BACKEND_IMAGE=local-ai-backend:ik-llama-cpp $(MAKE) test-extra-backend

@@ -813,6 +845,7 @@ test-extra-backend-tinygrad-embeddings: docker-build-tinygrad
 	BACKEND_IMAGE=local-ai-backend:tinygrad \
 	BACKEND_TEST_MODEL_NAME=Qwen/Qwen3-0.6B \
 	BACKEND_TEST_CAPS=health,load,embeddings \
+	BACKEND_TEST_EMBEDDING_LAYOUT=final \
 	$(MAKE) test-extra-backend

 ## tinygrad — Stable Diffusion 1.5. The original CompVis/runwayml repos have
@@ -1233,6 +1266,10 @@ backends/mlx:
 	BACKEND=mlx $(MAKE) build-darwin-python-backend
 	./local-ai backends install "ocifile://$(abspath ./backend-images/mlx.tar)"

+backends/mlx-video:
+	BACKEND=mlx-video $(MAKE) build-darwin-python-backend
+	./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-video.tar)"
+
 backends/diffuser-darwin:
 	BACKEND=diffusers $(MAKE) build-darwin-python-backend
 	./local-ai backends install "ocifile://$(abspath ./backend-images/diffusers.tar)"
@@ -1319,6 +1356,7 @@ BACKEND_RERANKERS = rerankers|python|.|false|true
 BACKEND_TRANSFORMERS = transformers|python|.|false|true
 BACKEND_OUTETTS = outetts|python|.|false|true
 BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
+BACKEND_WHISPER_MEDUSA = whisper-medusa|python|.|false|true
 BACKEND_COQUI = coqui|python|.|false|true
 BACKEND_RFDETR = rfdetr|python|.|false|true
 BACKEND_INSIGHTFACE = insightface|python|.|false|true
@@ -1340,6 +1378,7 @@ BACKEND_QWEN_TTS = qwen-tts|python|.|false|true
 BACKEND_FISH_SPEECH = fish-speech|python|.|false|true
 BACKEND_FASTER_QWEN3_TTS = faster-qwen3-tts|python|.|false|true
 BACKEND_QWEN_ASR = qwen-asr|python|.|false|true
+BACKEND_FUNASR = funasr|python|.|false|true
 BACKEND_NEMO = nemo|python|.|false|true
 BACKEND_VOXCPM = voxcpm|python|.|false|true
 BACKEND_WHISPERX = whisperx|python|.|false|true
@@ -1410,6 +1449,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_RERANKERS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_TRANSFORMERS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OUTETTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
+$(eval $(call generate-docker-build-target,$(BACKEND_WHISPER_MEDUSA)))
 $(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_INSIGHTFACE)))
@@ -1431,6 +1471,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_QWEN_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FISH_SPEECH)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_QWEN3_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN_ASR)))
+$(eval $(call generate-docker-build-target,$(BACKEND_FUNASR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_NEMO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VOXCPM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
@@ -1460,6 +1501,8 @@ docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar

 docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-audio-cpp
+docker-build-backends: docker-build-whisper-medusa
+docker-build-backends: docker-build-funasr

 ########################################################
 ### Mock Backend for E2E Tests
@@ -1505,6 +1548,13 @@ test-ui: build-mock-backend protogen-go
 	$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui
 	cd core/http/react-ui && sh $(CURDIR)/scripts/ensure-playwright-browser.sh && bunx playwright test $(PLAYWRIGHT_WORKERS_FLAG)

+## The stale-chunk specs need the production code-split bundle. The V8 coverage
+## bundle below inlines dynamic imports to keep every page in its denominator.
+test-ui-stale-chunk: build-mock-backend protogen-go
+	cd core/http/react-ui && bun install && bun run build
+	$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui
+	cd core/http/react-ui && sh $(CURDIR)/scripts/ensure-playwright-browser.sh && bunx playwright test --grep @production-chunks --workers=1
+
 ## React UI code coverage from the Playwright e2e suite. Builds a
 ## NON-instrumented bundle with source maps (COVERAGE_V8=true), re-embeds it
 ## into the ui-test-server (the dist is //go:embed'ed at compile time), runs the
@@ -1520,7 +1570,7 @@ test-ui-coverage: build-mock-backend protogen-go
 	$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui && \
 	( cd core/http/react-ui && rm -rf .nyc_output coverage && \
 	    sh $(CURDIR)/scripts/ensure-playwright-browser.sh && \
-	    PW_V8_COVERAGE=1 bunx playwright test $(PLAYWRIGHT_WORKERS_FLAG) && bun run coverage:report )
+	    PW_V8_COVERAGE=1 bunx playwright test --grep-invert @production-chunks $(PLAYWRIGHT_WORKERS_FLAG) && bun run coverage:report )

 ## UI coverage baseline (committed) and the strict gate that compares against
 ## it — the React mirror of test-coverage-baseline / test-coverage-check.
@@ -1603,7 +1653,7 @@ site-serve: site
 build-launcher-darwin:
 	rm -rf dist/LocalAI.app cmd/launcher/LocalAI.app
 	mkdir -p dist
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION)
 	mv cmd/launcher/LocalAI.app dist/LocalAI.app
 	bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app

@@ -1630,4 +1680,4 @@ release-launcher-darwin: notarize-launcher-darwin
 	@echo "dist/LocalAI.dmg is ready"

 build-launcher-linux:
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION) && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
@@ -315,10 +315,6 @@ Past sponsors

 A special thanks to individual sponsors, a full list is on [GitHub](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler). Special shout out to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!

-## Star history
-
-[![LocalAI Star history Chart](https://api.star-history.com/svg?repos=go-skynet/LocalAI&type=Date)](https://star-history.com/#go-skynet/LocalAI&Date)
-
 ## License

 LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/) and maintained by the [LocalAI team](#team).
@@ -6,13 +6,14 @@ ARG APT_PORTS_MIRROR=""
 # ASR, VAD, diarization, source separation and music generation, wrapped as a
 # LocalAI gRPC backend.
 #
-# BASE_IMAGE is ubuntu:24.04 for cpu and vulkan builds, or
-# nvidia/cuda:<ver>-devel-ubuntu24.04 for cublas builds; both ship apt and
-# Ubuntu Noble packages, and the CUDA base additionally provides
-# /usr/local/cuda. BUILD_TYPE selects the engine backend in the Makefile:
-# "" = portable CPU with all ggml CPU variants, "cublas" ->
-# -DENGINE_ENABLE_CUDA=ON, "vulkan" -> -DENGINE_ENABLE_VULKAN=ON. Darwin
-# (Metal) builds bypass this Dockerfile entirely.
+# BASE_IMAGE is ubuntu:24.04 for cpu and vulkan builds,
+# nvidia/cuda:<ver>-devel-ubuntu24.04 for cublas builds, or
+# rocm/dev-ubuntu-24.04:<ver> for hipblas builds. All ship apt and Ubuntu Noble
+# packages; the GPU bases also provide their toolkits. BUILD_TYPE selects the
+# engine backend in the Makefile: "" = portable CPU with all ggml CPU variants,
+# "cublas" -> -DENGINE_ENABLE_CUDA=ON, "hipblas" -> -DENGINE_ENABLE_HIP=ON,
+# and "vulkan" -> -DENGINE_ENABLE_VULKAN=ON. Darwin (Metal) builds bypass this
+# Dockerfile entirely.
 #
 # Upstream needs GCC 13 or newer, which ubuntu:24.04 and the CUDA 12/13
 # devel-ubuntu24.04 images all provide.
@@ -62,7 +63,7 @@ ENV BUILD_TYPE=${BUILD_TYPE} \
    APT_MIRROR=${APT_MIRROR} \
    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
    DEBIAN_FRONTEND=noninteractive \
-    PATH=/usr/local/cuda/bin:${PATH}
+    PATH=/opt/rocm/bin:/usr/local/cuda/bin:${PATH}

 WORKDIR /build

@@ -73,7 +74,7 @@ WORKDIR /build
 # fallback of its own.
 #
 # BUILD_TYPE=vulkan additionally needs the loader headers and glslc; both are in
-# Noble. The CUDA toolkit for BUILD_TYPE=cublas comes from BASE_IMAGE.
+# Noble. The CUDA and ROCm toolkits come from their matching BASE_IMAGE.
 RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
    sh /usr/local/sbin/apt-mirror && \
    apt-get update && \
@@ -83,6 +84,9 @@ RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mi
    if [ "${BUILD_TYPE}" = "vulkan" ]; then \
        apt-get install -y --no-install-recommends libvulkan-dev glslc; \
    fi && \
+    if [ "${BUILD_TYPE}" = "hipblas" ]; then \
+        apt-get install -y --no-install-recommends hipblas-dev rocblas-dev; \
+    fi && \
    if [ "${TARGETARCH}" = "arm64" ]; then \
        apt-get install -y --no-install-recommends gcc-14 g++-14; \
    fi && \
@@ -10,6 +10,7 @@ FROM ${BASE_IMAGE} AS builder
 ARG BUILD_TYPE
 ARG TARGETARCH
 ARG TARGETVARIANT
+ARG CUDA_MAJOR_VERSION

 ENV BUILD_TYPE=${BUILD_TYPE} \
    DEBIAN_FRONTEND=noninteractive \
@@ -35,7 +36,8 @@ RUN apt-get update && \
 COPY . /LocalAI

 RUN --mount=type=cache,target=/root/.ccache,id=ds4-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package
+    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} \
+        CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} NATIVE=false grpc-server package

 FROM scratch
 COPY --from=builder /LocalAI/backend/cpp/ds4/package/. ./
@@ -47,7 +47,7 @@ The backend system provides language-specific Dockerfiles that handle the build
 - **mlx**: Apple Silicon optimization
 - **diffusers**: Stable Diffusion models
 - **longcat-video**: CUDA text/image-to-video and speech-driven avatar generation
- **Audio**: coqui, faster-whisper, kitten-tts
+- **Audio**: coqui, faster-whisper, funasr, kitten-tts
 - **Vision**: mlx-vlm, rfdetr
 - **Specialized**: rerankers, chatterbox, kokoro

@@ -499,6 +499,10 @@ message ModelOptions {
  // applied verbatim to the backend's engine constructor (e.g. vLLM AsyncEngineArgs).
  // Unknown keys produce an error at LoadModel time.
  string EngineArgs = 73;
+  string OriginalConfigFile = 77;
+
+  // EnvVars carries environment variables to be passed to the backend process.
+  map<string, string> EnvVars = 76;

  // Proxy carries the cloud-proxy backend's per-model configuration.
  // Empty for non-proxy backends.
@@ -536,8 +540,28 @@ message Result {
  bool success = 2;
 }

+// EmbeddingLayout describes whether embeddings contains one final vector or
+// a matrix of per-token vectors. Go-side pooling must never infer this from
+// tokens/dim alone: a one-token raw matrix and a final vector have the same
+// shape.
+enum EmbeddingLayout {
+  EMBEDDING_LAYOUT_UNSPECIFIED = 0;
+  EMBEDDING_LAYOUT_FINAL = 1;
+  EMBEDDING_LAYOUT_PER_TOKEN = 2;
+}
+
 message EmbeddingResult {
  repeated float embeddings = 1;
+  // Shape of the payload above: dim is the embedding width, tokens is the
+  // number of vectors packed into `embeddings` (1 when the backend pooled
+  // server-side, N with pooling:none; total across prompts if a request
+  // carried several). tokens=0/dim=0 means the backend predates shape
+  // reporting. prompt_tokens is the number of prompt tokens evaluated, for
+  // usage accounting.
+  int32 tokens = 2;
+  int32 dim = 3;
+  int32 prompt_tokens = 4;
+  EmbeddingLayout layout = 5;
 }

 message TranscriptRequest {
@@ -9,7 +9,7 @@
 # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean
 # rebuild and so the bump bot can see the pin.

-AUDIO_CPP_VERSION?=9d6e7b39236e0151ad28a70fab0d538b84ce8718
+AUDIO_CPP_VERSION?=efb04233dab73aeee4b2912042a90e7b36329061
 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp

 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -77,6 +77,16 @@ endif

 ifeq ($(BUILD_TYPE),cublas)
    CMAKE_ARGS += -DENGINE_ENABLE_CUDA=ON "-DCMAKE_CUDA_ARCHITECTURES=$(CUDA_ARCHITECTURES)"
+else ifeq ($(BUILD_TYPE),hipblas)
+    ROCM_HOME ?= /opt/rocm
+    ROCM_PATH ?= /opt/rocm
+    export CXX=$(ROCM_HOME)/llvm/bin/clang++
+    export CC=$(ROCM_HOME)/llvm/bin/clang
+    AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+    # audio.cpp forwards GPU_TARGETS to CMake's semicolon-delimited HIP list.
+    comma := ,
+    HIP_TARGETS := $(subst $(comma),;,$(AMDGPU_TARGETS))
+    CMAKE_ARGS += -DENGINE_ENABLE_HIP=ON "-DGPU_TARGETS=$(HIP_TARGETS)"
 else ifeq ($(BUILD_TYPE),vulkan)
    CMAKE_ARGS += -DENGINE_ENABLE_VULKAN=ON
 else ifeq ($(UNAME_S),Darwin)
@@ -29,6 +29,7 @@ const NamedTask kTaskNames[] = {
    {Task::VoiceDesign, "vdes"},
    {Task::SpeakerRecognition, "spk"},
    {Task::Svc, "svc"},
+    {Task::Midi, "midi"},
 };

 // Accepted on input but never emitted. "spkrec" was this backend's own earlier
@@ -25,6 +25,7 @@ enum class Task {
    VoiceDesign,
    SpeakerRecognition,
    Svc,
+    Midi,
 };

 // Mirrors engine::runtime::RunMode.
@@ -361,7 +361,7 @@ static void test_names_round_trip() {
                        Task::SourceSeparation, Task::AudioGeneration, Task::Tts,
                        Task::VoiceCloning, Task::VoiceConversion,
                        Task::SpeechToSpeech, Task::Alignment, Task::VoiceDesign,
-                        Task::SpeakerRecognition, Task::Svc};
+                        Task::SpeakerRecognition, Task::Svc, Task::Midi};
    for (const Task t : all) {
        Task parsed = Task::Vad;
        const bool ok = parse_task_name(task_name(t), parsed);
@@ -22,7 +22,8 @@ bool voice_is_reference_file(const std::string &voice) {

 RequestShape build_tts_shape(const backend::TTSRequest &request) {
    RequestShape shape;
-    shape.has_voice_reference = voice_is_reference_file(request.voice());
+    shape.has_voice_reference = voice_is_reference_file(request.voice()) ||
+                                request.params().find("multi_reference_cond") != request.params().end();
    // !empty() as well as has_instructions(), and it must match the guard in
    // build_tts_request: a request whose instructions are an empty string
    // carries no style condition, so telling routing to prefer VoiceDesign for
@@ -134,6 +135,13 @@ build_tts_request(const backend::TTSRequest &request,
        task.options["language"] = request.language();
    }

+    // Saved voice profiles send ref_text; Fish Audio reads reference_text.
+    // Derive the alias before copying params so an explicit canonical key wins.
+    const auto reference_text = request.params().find("ref_text");
+    if (reference_text != request.params().end()) {
+        task.options["reference_text"] = reference_text->second;
+    }
+
    // LAST, so an explicit params entry wins over anything derived above. That
    // matters for "caption": a caller who sets params[caption] has named the
    // exact string they want, and it must not be overwritten by `instructions`.
@@ -128,6 +128,14 @@ static void test_tts_shape() {
        check(!shape.has_voice_reference,
              "shape: a preset name is not a voice reference");
    }
+    {
+        backend::TTSRequest request;
+        (*request.mutable_params())["multi_reference_cond"] =
+            R"([{"audio":"one.wav","text":"one"}])";
+        const auto shape = build_tts_shape(request);
+        check(shape.has_voice_reference,
+              "shape: multi-reference conditioning is a voice reference");
+    }
    {
        backend::TTSRequest request;
        request.set_voice(dir.string());
@@ -367,6 +375,40 @@ static void test_tts_language_and_params() {
          "tts params: an explicit param overrides the derived caption");
 }

+static void test_tts_reference_transcript() {
+    backend::TTSRequest request;
+    request.set_text("New speech to generate.");
+    request.set_voice("reference.wav");
+    (*request.mutable_params())["ref_text"] = "The saved voice transcript.";
+
+    const auto task = build_tts_request(request, clip(24000, 1));
+    check(option_or(task.options, "reference_text", "") ==
+              "The saved voice transcript.",
+          "tts reference: saved transcript reaches Fish Audio's option");
+    check(option_or(task.options, "ref_text", "") ==
+              "The saved voice transcript.",
+          "tts reference: original transcript parameter is preserved");
+    check(task.text_input->text == "New speech to generate.",
+          "tts reference: transcript does not replace synthesis text");
+
+    (*request.mutable_params())["reference_text"] = "Explicit transcript.";
+    const auto explicit_task = build_tts_request(request, clip(24000, 1));
+    check(option_or(explicit_task.options, "reference_text", "") ==
+              "Explicit transcript.",
+          "tts reference: explicit canonical parameter wins over alias");
+
+    (*request.mutable_params())["reference_text"] = "";
+    const auto empty_task = build_tts_request(request, clip(24000, 1));
+    check(has_key(empty_task.options, "reference_text") &&
+              empty_task.options.at("reference_text").empty(),
+          "tts reference: explicit empty canonical parameter is preserved");
+
+    request.mutable_params()->clear();
+    const auto missing_task = build_tts_request(request, clip(24000, 1));
+    check(!has_key(missing_task.options, "reference_text"),
+          "tts reference: no transcript is invented when none was supplied");
+}
+
 static void test_sound_generation_minimal() {
    backend::SoundGenerationRequest request;
    request.set_text("a distant thunderstorm");
@@ -554,6 +596,7 @@ int main() {
    test_tts_empty_language_is_not_a_language();
    test_tts_clip_and_instructions();
    test_tts_language_and_params();
+    test_tts_reference_transcript();
    test_sound_generation_minimal();
    test_sound_generation_full();
    test_transform_text_absent();
@@ -69,7 +69,8 @@ static_assert(kEngine(engine::runtime::VoiceTaskKind::VoiceDesign) == 10, "Voice
 static_assert(kEngine(engine::runtime::VoiceTaskKind::SpeakerRecognition) == 11, "VoiceTaskKind drifted");
 // The last member. Pinning it pins the member count too, as long as the
 // enumerators stay contiguous and unassigned, which upstream's declaration is.
-static_assert(kEngine(engine::runtime::VoiceTaskKind::Svc) == 12,
+static_assert(kEngine(engine::runtime::VoiceTaskKind::Svc) == 12, "VoiceTaskKind drifted");
+static_assert(kEngine(engine::runtime::VoiceTaskKind::Midi) == 13,
              "engine::runtime::VoiceTaskKind gained, lost or reordered a member. "
              "audiocpp_backend::Task mirrors it positionally: update capability_routing.h, "
              "to_engine_task and from_engine_task together, then move this pin.");
@@ -87,6 +88,7 @@ static_assert(kMirror(Task::Alignment) == 9, "Task drifted from VoiceTaskKind");
 static_assert(kMirror(Task::VoiceDesign) == 10, "Task drifted from VoiceTaskKind");
 static_assert(kMirror(Task::SpeakerRecognition) == 11, "Task drifted from VoiceTaskKind");
 static_assert(kMirror(Task::Svc) == 12, "Task drifted from VoiceTaskKind");
+static_assert(kMirror(Task::Midi) == 13, "Task drifted from VoiceTaskKind");

 static_assert(static_cast<int>(engine::runtime::RunMode::Offline) == 0, "RunMode drifted");
 static_assert(static_cast<int>(engine::runtime::RunMode::Streaming) == 1,
@@ -101,6 +103,9 @@ engine::core::BackendType parse_backend_type(const std::string &value) {
    if (value == "cuda") {
        return engine::core::BackendType::Cuda;
    }
+    if (value == "hip" || value == "rocm") {
+        return engine::core::BackendType::Hip;
+    }
    if (value == "vulkan") {
        return engine::core::BackendType::Vulkan;
    }
@@ -114,7 +119,7 @@ engine::core::BackendType parse_backend_type(const std::string &value) {
        return engine::core::BackendType::Cpu;
    }
    throw ConfigError("audio-cpp: unknown backend option '" + value +
-                      "'. Known backends: cpu, cuda, vulkan, metal, best");
+                      "'. Known backends: cpu, cuda, hip, rocm, vulkan, metal, best");
 }

 std::filesystem::path executable_directory() {
@@ -241,6 +246,7 @@ engine::runtime::VoiceTaskKind to_engine_task(Task task) {
    case Task::VoiceDesign:        return K::VoiceDesign;
    case Task::SpeakerRecognition: return K::SpeakerRecognition;
    case Task::Svc:                return K::Svc;
+    case Task::Midi:               return K::Midi;
    }
    // Unreachable for any valid enumerator. No `default:` label, so -Wswitch
    // still reports a member this switch stops covering.
@@ -263,6 +269,7 @@ Task from_engine_task(engine::runtime::VoiceTaskKind kind) {
    case K::VoiceDesign:        return Task::VoiceDesign;
    case K::SpeakerRecognition: return Task::SpeakerRecognition;
    case K::Svc:                return Task::Svc;
+    case K::Midi:               return Task::Midi;
    }
    return Task::Vad;
 }
@@ -59,6 +59,12 @@ bool starts_with(const std::string &value, const std::string &prefix) {
           value.compare(0, prefix.size(), prefix) == 0;
 }

+bool is_known_backend(const std::string &value) {
+    return value == "cpu" || value == "cuda" || value == "hip" ||
+           value == "rocm" || value == "vulkan" || value == "metal" ||
+           value == "best";
+}
+
 } // namespace

 ParsedOptions parse_model_options(const std::vector<std::string> &entries) {
@@ -107,6 +113,12 @@ ParsedOptions parse_model_options(const std::vector<std::string> &entries) {
        } else if (key == "task") {
            parsed.options.task = value;
        } else if (key == "backend") {
+            if (!is_known_backend(value)) {
+                parsed.error = "audio-cpp: unknown backend option '" + value +
+                               "'. Known backends: cpu, cuda, hip, rocm, "
+                               "vulkan, metal, best";
+                return parsed;
+            }
            parsed.options.backend = value;
        } else if (key == "model_spec_override") {
            parsed.options.model_spec_override = value;
@@ -17,7 +17,7 @@ struct ModelOptions {
    std::string family;
    // Pins the audio.cpp task, overriding RPC-based routing. Empty means route.
    std::string task;
-    // ggml backend: cpu, cuda, vulkan, metal, best.
+    // ggml backend: cpu, cuda, hip (or rocm), vulkan, metal, best.
    std::string backend = "cpu";
    int device = 0;
    // True once a `device:` entry has been seen. 0 is both the default and a
@@ -75,6 +75,11 @@ static void test_scalar_options() {
    check(parse_model_options({"live_idle_timeout_ms:0"}).options.live_idle_timeout_ms == 0,
          "an explicit 0 turns the live idle limit off rather than reverting to "
          "the default");
+
+    check(parse_model_options({"backend:hip"}).error.empty(),
+          "HIP backend option is accepted");
+    check(parse_model_options({"backend:rocm"}).error.empty(),
+          "ROCm backend alias is accepted");
 }

 // Values containing colons must survive: split on the FIRST colon only.
@@ -124,6 +129,8 @@ static void test_errors() {
          "negative device is rejected");
    check(!parse_model_options({"threads:x"}).error.empty(),
          "non-numeric threads is rejected");
+    check(!parse_model_options({"backend:unknown"}).error.empty(),
+          "unknown compute backend is rejected before model loading");

    // Values too large for int must be rejected, not silently wrapped into a
    // negative device index that then reaches the ggml backend selector.
@@ -1,7 +1,7 @@

 # Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-BONSAI_VERSION?=9ca265a57f85f2117942490f421f64a226dd9847
+BONSAI_VERSION?=312bb2a93ea2bf798333fa859614fbf913ecb9e2
 LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp

 CMAKE_ARGS?=
@@ -41,6 +41,7 @@ define bonsai-build
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
+	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
@@ -79,6 +80,7 @@ bonsai-cpu-all:
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
+	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
@@ -0,0 +1,24 @@
+#!/bin/bash
+# Adapt the shared llama.cpp gRPC source to the older JSON API in Bonsai.
+
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+    echo "usage: $0 <grpc-server.cpp>" >&2
+    exit 2
+fi
+
+SRC=$1
+if [[ ! -f "$SRC" ]]; then
+    echo "grpc-server.cpp not found at $SRC" >&2
+    exit 2
+fi
+
+if grep -q 'common_json_error' "$SRC"; then
+    echo "==> patching $SRC to use the Bonsai JSON exception type"
+    awk '{ gsub(/common_json_error/, "json::parse_error"); print }' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+    echo "==> Bonsai JSON exception patch OK"
+else
+    echo "==> $SRC already uses a Bonsai-compatible JSON exception type, skipping"
+fi
@@ -84,9 +84,10 @@ elseif(DS4_GPU STREQUAL "cpu")
    set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
 endif()

-# Upstream splits distributed inference, tensor-parallel transport, the SSD
-# expert cache, and layer placement into GPU-agnostic translation units. Link
-# them regardless of DS4_GPU.
+# Upstream splits image preprocessing, distributed inference, tensor-parallel
+# transport, the SSD expert cache, and layer placement into GPU-agnostic
+# translation units. Link them regardless of DS4_GPU.
+list(APPEND DS4_OBJS "${DS4_DIR}/ds4_image.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_distributed.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_tp.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_ssd.o")
@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=84cc882352757baf628a1776badf7cc54d584e28
+# Upstream pin lives below as DS4_VERSION?=6289c516273979173abbc062209a81dd3706b804
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.

-DS4_VERSION?=84cc882352757baf628a1776badf7cc54d584e28
+DS4_VERSION?=6289c516273979173abbc062209a81dd3706b804
 DS4_REPO?=https://github.com/antirez/ds4

 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -18,21 +18,83 @@ UNAME_S := $(shell uname -s)

 CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release

-# Upstream splits distributed inference, tensor-parallel transport, the SSD
-# expert cache, and layer placement into GPU-agnostic translation units. They
-# are shared by every GPU mode, so append them unconditionally below.
+# nvcc must be told the target architecture explicitly for a cublas build, and
+# this is not a tuning knob. Upstream's Makefile leaves CUDA_ARCH empty and its
+# `cuda` target REFUSES to build without one, offering `cuda-spark`
+# (CUDA_ARCH=sm_121) and `cuda-generic` (CUDA_ARCH=native) instead. We drive its
+# object targets directly, which bypasses that guard: nvcc then compiles with no
+# -arch at all, and the kernels run as JIT'd PTX for its default architecture.
+# On GB10 (sm_121) that silently produced corrupt inference output above a
+# ~128-token prefill batch and ~77x slower prefill (4.21 t/s vs 325.70 t/s,
+# measured on the same box with the same model). No CI runner has a GPU, so
+# `native` has nothing to enumerate there.
+#
+# Upstream's CUDA_ARCH takes a SINGLE value (see its sm_120/sm_121 special cases
+# and the `-arch=$(CUDA_ARCH)` fallback), so it cannot express the fat binary
+# these images need. NVCC_ARCH_FLAGS is overridden instead: a command-line
+# assignment wins over the `:=` in upstream's Makefile, and its NVCCFLAGS
+# expands whatever we pass.
+#
+# The architecture lists are copied from backend/go/vllm-cpp/Makefile rather
+# than invented, so the two CUDA images cover the same GPUs: amd64 datacenter +
+# consumer, and l4t/arm64 covering Orin (87), Thor (110) and GB10 (121a).
+#
+# -DDS4_CUDA_HAVE_MXF4=1 is deliberately NOT set. Upstream only defines it for
+# single-arch sm_120/sm_121 builds and guards the code with a plain #ifdef
+# rather than __CUDA_ARCH__, so it cannot be combined with older archs in one
+# fat binary. It gates an optional MXFP4 indexer fast path whose #ifndef branch
+# returns 0 and falls back to the generic path, so omitting it costs some speed
+# on GB10, not correctness. Revisit if upstream adds __CUDA_ARCH__ guards.
+#
+# An EMPTY CUDA_MAJOR_VERSION means a local developer build, not CI: fall back
+# to upstream's own `native` handling, which needs a GPU present but is what a
+# developer building on their own machine wants. Both variables are `?=` so an
+# explicit value on the command line always wins.
+UNAME_M := $(shell uname -m)
+CUDA_MAJOR_VERSION ?=
+ifeq ($(BUILD_TYPE),cublas)
+ifeq ($(CUDA_MAJOR_VERSION),13)
+ifeq ($(UNAME_M),aarch64)
+    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_87,code=sm_87 \
+                           -gencode arch=compute_90a,code=sm_90a \
+                           -gencode arch=compute_100a,code=sm_100a \
+                           -gencode arch=compute_110,code=sm_110 \
+                           -gencode arch=compute_121a,code=sm_121a
+else
+    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_80,code=sm_80 \
+                           -gencode arch=compute_86,code=sm_86 \
+                           -gencode arch=compute_89,code=sm_89 \
+                           -gencode arch=compute_90a,code=sm_90a \
+                           -gencode arch=compute_100a,code=sm_100a \
+                           -gencode arch=compute_103a,code=sm_103a \
+                           -gencode arch=compute_120a,code=sm_120a \
+                           -gencode arch=compute_121a,code=sm_121a
+endif
+    DS4_ARCH_MAKEVARS := NVCC_ARCH_FLAGS="$(DS4_NVCC_ARCH_FLAGS)"
+else ifeq ($(CUDA_MAJOR_VERSION),)
+    # Local build: let upstream resolve the host GPU.
+    DS4_ARCH_MAKEVARS := CUDA_ARCH=native
+else
+    $(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (13 does). Leave it empty for a native build, or pass DS4_NVCC_ARCH_FLAGS explicitly.)
+endif
+endif
+
+# Upstream splits image preprocessing, distributed inference, tensor-parallel
+# transport, the SSD expert cache, and layer placement into GPU-agnostic
+# translation units. They are shared by every GPU mode, so append them
+# unconditionally below.
 ifeq ($(BUILD_TYPE),cublas)
    CMAKE_ARGS += -DDS4_GPU=cuda
-    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
+    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
        cuda/mmq/ds4_ggml_stubs.o cuda/mmq/ds4_mmq.o cuda/mmq/ds4_mmq_d2r.o \
        cuda/mmq/quantize.o cuda/mmq/mmid.o cuda/mmq/mmvq.o cuda/mmq/ds4_repack.o
 else ifeq ($(UNAME_S),Darwin)
    CMAKE_ARGS += -DDS4_GPU=metal
-    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
    # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
    CMAKE_ARGS += -DDS4_GPU=cpu
-    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif

 ifneq ($(NATIVE),true)
@@ -57,11 +119,11 @@ ds4:
 # the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
 ds4/ds4.o: ds4
 ifeq ($(BUILD_TYPE),cublas)
-	+$(MAKE) -C ds4 $(DS4_OBJ_TARGET)
+	+$(MAKE) -C ds4 $(DS4_ARCH_MAKEVARS) $(DS4_OBJ_TARGET)
 else ifeq ($(UNAME_S),Darwin)
-	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
-	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif

 grpc-server: ds4/ds4.o
@@ -92,7 +92,8 @@ std::string json_escape(const std::string &in) {

 } // namespace

-DsmlParser::DsmlParser() = default;
+DsmlParser::DsmlParser(bool starts_in_thinking)
+    : state_(starts_in_thinking ? State::THINK : State::TEXT) {}

 bool DsmlParser::IsInDsmlStructural() const {
    switch (state_) {
@@ -17,7 +17,9 @@ struct ParserEvent {
 // Streaming parser. Stateless across instances; one per Predict call.
 class DsmlParser {
 public:
-    DsmlParser();
+    // The chat prompt may already contain the opening thinking marker, so the
+    // generated text can begin directly with reasoning bytes.
+    explicit DsmlParser(bool starts_in_thinking = false);

    // Feed a chunk of raw model-emitted text. Appends classified events to
    // `out`. May buffer the tail of `chunk` internally if it looks like a
@@ -43,7 +45,7 @@ public:

 private:
    enum class State { TEXT, THINK, TOOL_CALLS, INVOKE, PARAM_VALUE };
-    State state_ = State::TEXT;
+    State state_;
    std::string buf_;
    std::string current_tool_name_;
    int tool_index_ = -1;
@@ -0,0 +1,133 @@
+// SPDX-License-Identifier: MIT
+// Standalone regression tests for the DSML streaming parser.
+//
+// The repository's backend/cpp/run-unit-tests.sh harness compiles each
+// *_test.cpp as a single translation unit, so include the implementation here.
+
+#include "dsml_parser.cpp"
+
+#include <cstdio>
+#include <string>
+#include <type_traits>
+#include <vector>
+
+namespace {
+
+struct ParsedText {
+    std::string content;
+    std::string reasoning;
+};
+
+int failures = 0;
+
+void check_equal(const std::string &got, const std::string &want,
+                 const char *name) {
+    if (got == want) return;
+    std::fprintf(stderr, "FAIL %s: got \"%s\", want \"%s\"\n",
+                 name, got.c_str(), want.c_str());
+    failures++;
+}
+
+void collect_text(const std::vector<ds4cpp::ParserEvent> &events,
+                  ParsedText *parsed) {
+    for (const auto &event : events) {
+        if (event.type == ds4cpp::ParserEvent::CONTENT) {
+            parsed->content += event.text;
+        } else if (event.type == ds4cpp::ParserEvent::REASONING) {
+            parsed->reasoning += event.text;
+        }
+    }
+}
+
+ParsedText parse_chunks(ds4cpp::DsmlParser *parser,
+                        const std::vector<std::string> &chunks) {
+    ParsedText parsed;
+    for (const auto &chunk : chunks) {
+        std::vector<ds4cpp::ParserEvent> events;
+        parser->Feed(chunk, events);
+        collect_text(events, &parsed);
+    }
+    std::vector<ds4cpp::ParserEvent> events;
+    parser->Flush(events);
+    collect_text(events, &parsed);
+    return parsed;
+}
+
+template <typename Parser>
+void test_reasoning_opened_by_prompt() {
+    if constexpr (!std::is_constructible_v<Parser, bool>) {
+        std::fprintf(stderr,
+                     "FAIL reasoning_opened_by_prompt: parser cannot start in thinking state\n");
+        failures++;
+    } else {
+        Parser parser(true);
+        ParsedText parsed = parse_chunks(
+            &parser,
+            {"We need to calculate factorial recursively.</think>Here is the answer."});
+        check_equal(parsed.reasoning,
+                    "We need to calculate factorial recursively.",
+                    "reasoning_opened_by_prompt:reasoning");
+        check_equal(parsed.content, "Here is the answer.",
+                    "reasoning_opened_by_prompt:content");
+    }
+}
+
+template <typename Parser>
+Parser text_parser() {
+    if constexpr (std::is_constructible_v<Parser, bool>) {
+        return Parser(false);
+    } else {
+        return Parser();
+    }
+}
+
+void test_reasoning_disabled() {
+    auto parser = text_parser<ds4cpp::DsmlParser>();
+    ParsedText parsed = parse_chunks(&parser, {"Here is the answer."});
+    check_equal(parsed.reasoning, "", "reasoning_disabled:reasoning");
+    check_equal(parsed.content, "Here is the answer.",
+                "reasoning_disabled:content");
+}
+
+void test_explicit_think_tag() {
+    auto parser = text_parser<ds4cpp::DsmlParser>();
+    ParsedText parsed = parse_chunks(
+        &parser, {"<think>reasoning</think>answer"});
+    check_equal(parsed.reasoning, "reasoning", "explicit_think_tag:reasoning");
+    check_equal(parsed.content, "answer", "explicit_think_tag:content");
+}
+
+template <typename Parser>
+void test_split_think_close_marker() {
+    if constexpr (!std::is_constructible_v<Parser, bool>) {
+        std::fprintf(stderr,
+                     "FAIL split_think_close_marker: parser cannot start in thinking state\n");
+        failures++;
+    } else {
+        Parser parser(true);
+        ParsedText parsed = parse_chunks(
+            &parser,
+            {"We need ", "to calculate ", "factorial", "</thi", "nk>",
+             "Here is ", "the answer."});
+        check_equal(parsed.reasoning, "We need to calculate factorial",
+                    "split_think_close_marker:reasoning");
+        check_equal(parsed.content, "Here is the answer.",
+                    "split_think_close_marker:content");
+    }
+}
+
+} // namespace
+
+int main() {
+    test_reasoning_opened_by_prompt<ds4cpp::DsmlParser>();
+    test_reasoning_disabled();
+    test_explicit_think_tag();
+    test_split_think_close_marker<ds4cpp::DsmlParser>();
+
+    if (failures == 0) {
+        std::fprintf(stderr, "all dsml_parser checks passed\n");
+        return 0;
+    }
+    std::fprintf(stderr, "%d check(s) failed\n", failures);
+    return 1;
+}
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+#include <algorithm>
+
+namespace ds4cpp {
+
+inline int EffectiveGenerationLimit(int requested, int context_size,
+                                    int session_position) {
+  const int limit = requested > 0 ? requested : 256;
+  const int room = context_size - session_position;
+  if (room <= 1) return 0;
+  return std::min(limit, room - 1);
+}
+
+inline int RemainingGenerationBudget(int effective_limit, int produced) {
+  if (effective_limit <= produced) return 0;
+  return effective_limit - produced;
+}
+
+inline int SpeculativeAcceptedCapacity(int remaining, int draft_allowance,
+                                       int buffer_capacity) {
+  if (remaining <= 0 || draft_allowance < 0 || buffer_capacity <= 0) return 0;
+  return std::min({remaining, draft_allowance + 1, buffer_capacity});
+}
+
+} // namespace ds4cpp
@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: MIT
+
+#include "generation_limits.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+void check_equal(int got, int want, const char *name) {
+  if (got == want) return;
+  std::fprintf(stderr, "FAIL %s: got %d, want %d\n", name, got, want);
+  failures++;
+}
+
+// Mutation caught: treating omitted or negative max_tokens as unlimited instead
+// of preserving DS4's legacy 256-token default.
+void test_nonpositive_uses_legacy_default_when_space_permits() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 4096, 100), 256,
+              "zero max_tokens uses legacy default");
+  check_equal(ds4cpp::EffectiveGenerationLimit(-1, 4096, 100), 256,
+              "negative max_tokens uses legacy default");
+}
+
+// Mutation caught: applying the legacy default without clamping it to the
+// post-prefill context room and reserved slot.
+void test_legacy_default_is_clamped_by_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 300, 100), 199,
+              "legacy default is context-clamped");
+}
+
+// Mutation caught: allowing an explicitly large request to overrun the
+// post-prefill context boundary.
+void test_large_positive_limit_is_clamped_to_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32768, 32768, 100), 32667,
+              "large positive is context-clamped");
+}
+
+// Mutation caught: replacing every positive request with the legacy default
+// rather than preserving a smaller configured limit.
+void test_smaller_positive_limit_is_preserved() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(64, 4096, 100), 64,
+              "smaller positive is preserved");
+}
+
+// Mutation caught: consuming the final context slot instead of reserving it as
+// required by DS4's generation loop.
+void test_no_usable_room_returns_zero() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32, 100, 99), 0,
+              "one remaining context slot is not usable");
+}
+
+// Mutation caught: sending the original generation limit to a later
+// speculative cycle instead of subtracting tokens already produced.
+void test_remaining_budget_accounts_for_produced_tokens() {
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 4), 6,
+              "remaining budget subtracts produced tokens");
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 12), 0,
+              "remaining budget never becomes negative");
+}
+
+// Mutation caught: giving speculative evaluation capacity beyond either the
+// output budget, the draft allowance plus its first target token, or the fixed
+// accepted-token buffer.
+void test_speculative_capacity_obeys_all_bounds() {
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(3, 8, 8), 3,
+              "capacity respects remaining output budget");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 4, 8), 5,
+              "capacity includes one target token beyond draft allowance");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 8, 6), 6,
+              "capacity respects fixed buffer");
+}
+
+} // namespace
+
+int main() {
+  test_nonpositive_uses_legacy_default_when_space_permits();
+  test_legacy_default_is_clamped_by_context();
+  test_large_positive_limit_is_clamped_to_context();
+  test_smaller_positive_limit_is_preserved();
+  test_no_usable_room_returns_zero();
+  test_remaining_budget_accounts_for_produced_tokens();
+  test_speculative_capacity_obeys_all_bounds();
+
+  if (failures == 0) {
+    std::fprintf(stderr, "all generation limit checks passed\n");
+    return 0;
+  }
+  std::fprintf(stderr, "%d check(s) failed\n", failures);
+  return 1;
+}
@@ -10,7 +10,9 @@

 #include "dsml_parser.h"   // populated in Task 12
 #include "dsml_renderer.h" // populated in Task 16
+#include "generation_limits.h"
 #include "kv_cache.h"      // populated in Task 17
+#include "request_lifecycle.h"

 extern "C" {
 #include "ds4.h"
@@ -35,6 +37,7 @@ extern "C" {
 #include <mutex>
 #include <string>
 #include <thread>
+#include <utility>
 #include <vector>

 using grpc::Server;
@@ -69,6 +72,21 @@ int g_route_timeout_sec = 60;

 std::atomic<Server *> g_server{nullptr};

+static bool server_context_cancelled(void *ud) {
+    return static_cast<ServerContext *>(ud)->IsCancelled();
+}
+
+static void set_session_cancel(void *target, ds4cpp::CancelCallback callback,
+                               void *userdata) noexcept {
+    ds4_session_set_cancel(static_cast<ds4_session *>(target), callback, userdata);
+}
+
+static bool request_should_continue(ds4cpp::RequestLifecycle *request,
+                                    ServerContext *context) {
+    request->ObserveContextCancellation(context->IsCancelled());
+    return request->ShouldContinue();
+}
+
 // Parse a "key:value" option string. Returns empty when no colon.
 static std::pair<std::string, std::string> split_option(const std::string &opt) {
    auto colon = opt.find(':');
@@ -238,37 +256,58 @@ static bool apply_engine_option(ds4_engine_options *opt, const std::string &key,

 // When acting as a distributed coordinator, block until the worker route
 // covers all layers (ds4_session_distributed_route_ready == 1) or the timeout
-// elapses. Returns an empty string on success, or an error message to return
-// to the client. No-op when not distributed.
+// elapses. No-op when not distributed.
 //
 // Takes the g_engine_mu lock by reference and RELEASES it during each poll
 // sleep. The wait can span up to g_route_timeout_sec seconds while workers
 // connect; holding g_engine_mu the whole time would block the Status/Health
 // readiness probes (they also lock g_engine_mu), making LocalAI's loader treat
 // a still-starting worker as hung.
-static std::string wait_route_ready(std::unique_lock<std::mutex> &lock) {
-    if (!g_distributed) return "";
+struct RouteWaitResult {
+    ds4cpp::RouteWaitDecision decision;
+    std::string error;
+};
+
+static RouteWaitResult wait_route_ready(std::unique_lock<std::mutex> &lock,
+                                        ServerContext *context) {
+    if (!g_distributed) return {ds4cpp::RouteWaitDecision::Ready, ""};
    char err[256] = {0};
    const int deadline_polls = g_route_timeout_sec * 10; // 100ms per poll
    for (int i = 0; i <= deadline_polls; ++i) {
        int ready = ds4_session_distributed_route_ready(g_session, err, sizeof(err));
-        if (ready == 1) return "";
-        if (ready < 0) {
-            return std::string("ds4 distributed route error: ") +
-                   (err[0] ? err : "unknown");
+        switch (ds4cpp::DecideRouteWait(ready, context->IsCancelled())) {
+        case ds4cpp::RouteWaitDecision::Ready:
+            return {ds4cpp::RouteWaitDecision::Ready, ""};
+        case ds4cpp::RouteWaitDecision::Error:
+            return {ds4cpp::RouteWaitDecision::Error,
+                    std::string("ds4 distributed route error: ") +
+                        (err[0] ? err : "unknown")};
+        case ds4cpp::RouteWaitDecision::Cancelled:
+            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+        case ds4cpp::RouteWaitDecision::Pending:
+            break;
        }
+        if (i == deadline_polls) break;
        // Release the lock while sleeping so Status/Health and other RPCs can
        // interleave during worker startup.
        lock.unlock();
        struct timespec ts = {0, 100L * 1000L * 1000L}; // 100ms
        nanosleep(&ts, nullptr);
        lock.lock();
+        if (context->IsCancelled()) {
+            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+        }
        // A concurrent Free() may have torn down the engine while we slept.
        if (!g_engine || !g_session) {
-            return "ds4: model unloaded while waiting for distributed route";
+            return {ds4cpp::RouteWaitDecision::Error,
+                    "ds4: model unloaded while waiting for distributed route"};
        }
    }
-    return "ds4 distributed route incomplete: workers not connected (layers uncovered)";
+    if (context->IsCancelled()) {
+        return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+    }
+    return {ds4cpp::RouteWaitDecision::Error,
+            "ds4 distributed route incomplete: workers not connected (layers uncovered)"};
 }

 static void append_token_text(ds4_engine *engine, int token, std::string &out) {
@@ -341,9 +380,9 @@ static void collect_done(void *) {}
 struct StreamCtx {
    ds4_engine *engine;
    ServerWriter<backend::Reply> *writer;
+    ds4cpp::RequestLifecycle *request;
    ds4cpp::DsmlParser parser;
    int tokens;
-    bool aborted;
    // Track which tool indices we've seen TOOL_START for, so subsequent
    // ARGS deltas can elide the redundant id/name fields.
    std::vector<bool> tool_started;
@@ -351,7 +390,7 @@ struct StreamCtx {

 static void stream_emit(void *ud, int token) {
    auto *s = static_cast<StreamCtx *>(ud);
-    if (s->aborted) return;
+    if (!s->request->ShouldContinue()) return;
    if (token == ds4_token_eos(s->engine)) return;
    size_t len = 0;
    const char *text = ds4_token_text(s->engine, token, &len);
@@ -401,7 +440,7 @@ static void stream_emit(void *ud, int token) {
    reply.set_message(chunk);
    reply.set_tokens(1);
    if (any_field) {
-        if (!s->writer->Write(reply)) s->aborted = true;
+        s->request->ObserveStreamWrite(s->writer->Write(reply));
    }
    s->tokens++;
 }
@@ -757,21 +796,30 @@ public:
        return GStatus::OK;
    }

-    GStatus Predict(ServerContext *, const backend::PredictOptions *request,
+    GStatus Predict(ServerContext *context, const backend::PredictOptions *request,
                   backend::Reply *reply) override {
        std::unique_lock<std::mutex> lock(g_engine_mu);
        if (!g_engine || !g_session) {
            return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
        }
        if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
-            return GStatus(StatusCode::UNAVAILABLE, route_err);
+        RouteWaitResult route = wait_route_ready(lock, context);
+        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
+            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
+        }
+        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
+            return GStatus(StatusCode::UNAVAILABLE, route.error);
        }
        ds4_tokens prompt = {};
        build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;

-        CollectCtx collect = {g_engine, "", {}, reply, 0, {}, "", ""};
+        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
+        const bool starts_in_thinking = think_enabled &&
+            request->usetokenizertemplate() && request->messages_size() > 0;
+        CollectCtx collect = {
+            g_engine, "", ds4cpp::DsmlParser(starts_in_thinking),
+            reply, 0, {}, "", ""};
+        ds4cpp::RequestLifecycle lifecycle;
        std::string cache_key = render_prompt_text(request);
        size_t cache_hit = maybe_load_cache(cache_key);
        (void)cache_hit; // future: skip prompt prefix if hit covers full prompt
@@ -783,15 +831,27 @@ public:
        // Either way g_session advances so the disk KV cache picks up a
        // real checkpoint after the call (see maybe_save_cache below).
        char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        int rc;
+        {
+            ds4cpp::CancelCallbackScope cancel_scope(
+                g_session, set_session_cancel, server_context_cancelled, context);
+            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        }
        int prompt_len = prompt.len;
        ds4_tokens_free(&prompt);
-        if (rc == 0) {
+        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
+            lifecycle.ObserveContextCancellation(true);
+        }
+        const bool generation_started = rc == 0;
+        if (generation_started) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
            const int eos = ds4_token_eos(g_engine);
            const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
            int produced = 0;
            while (produced < n_predict) {
+                if (!request_should_continue(&lifecycle, context)) break;
                SampleParams sp = compute_sample_params(request, collect.parser, think_enabled);
                int first;
                if (sp.temperature <= 0.0f) {
@@ -806,13 +866,20 @@ public:
                if (draft_max > 0 && sp.temperature <= 0.0f) {
                    constexpr int kAcceptedMax = 8;
                    int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                    int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                        accepted, cap, err, sizeof(err));
                    if (n < 0) { rc = -1; break; }
                    bool stop = false;
                    for (int j = 0; j < n; ++j) {
+                        if (!request_should_continue(&lifecycle, context)) {
+                            stop = true;
+                            break;
+                        }
                        if (accepted[j] == eos) { stop = true; break; }
                        collect_emit(&collect, accepted[j]);
                        if (++produced >= n_predict) { stop = true; break; }
@@ -821,12 +888,26 @@ public:
                } else {
                    collect_emit(&collect, first);
                    if (++produced >= n_predict) break;
+                    if (!request_should_continue(&lifecycle, context)) break;
                    rc = ds4_session_eval(g_session, first, err, sizeof(err));
                    if (rc != 0) break;
                }
            }
-            collect_done(&collect);
        }
+
+        request_should_continue(&lifecycle, context);
+        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
+            !lifecycle.ShouldFinalize());
+        if (!terminal.should_finalize) {
+            if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
+                return GStatus(StatusCode::INTERNAL,
+                              std::string("ds4 generation failed: ") + err);
+            }
+            return GStatus(StatusCode::CANCELLED,
+                          "ds4 request cancelled");
+        }
+        if (generation_started) collect_done(&collect);
        maybe_save_cache(cache_key);

        // Flush any buffered parser state.
@@ -834,7 +915,7 @@ public:
        collect.parser.Flush(events);
        apply_events(&collect, events);

-        if (rc != 0) {
+        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
            return GStatus(StatusCode::INTERNAL,
                          std::string("ds4 generation failed: ") + err);
        }
@@ -857,21 +938,30 @@ public:
        return GStatus::OK;
    }

-    GStatus PredictStream(ServerContext *, const backend::PredictOptions *request,
+    GStatus PredictStream(ServerContext *context, const backend::PredictOptions *request,
                         ServerWriter<backend::Reply> *writer) override {
        std::unique_lock<std::mutex> lock(g_engine_mu);
        if (!g_engine || !g_session) {
            return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
        }
        if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
-            return GStatus(StatusCode::UNAVAILABLE, route_err);
+        RouteWaitResult route = wait_route_ready(lock, context);
+        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
+            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
+        }
+        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
+            return GStatus(StatusCode::UNAVAILABLE, route.error);
        }
        ds4_tokens prompt = {};
        build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;

-        StreamCtx s = {g_engine, writer, {}, 0, false, {}};
+        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
+        const bool starts_in_thinking = think_enabled &&
+            request->usetokenizertemplate() && request->messages_size() > 0;
+        ds4cpp::RequestLifecycle lifecycle;
+        StreamCtx s = {
+            g_engine, writer, &lifecycle,
+            ds4cpp::DsmlParser(starts_in_thinking), 0, {}};
        std::string cache_key = render_prompt_text(request);
        size_t cache_hit = maybe_load_cache(cache_key);
        (void)cache_hit;
@@ -879,14 +969,26 @@ public:
        // Manual loop on g_session - see Predict() above for the rationale.
        // MTP speculative path used when ds4_engine_mtp_draft_tokens > 0.
        char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        int rc;
+        {
+            ds4cpp::CancelCallbackScope cancel_scope(
+                g_session, set_session_cancel, server_context_cancelled, context);
+            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        }
        ds4_tokens_free(&prompt);
-        if (rc == 0) {
+        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
+            lifecycle.ObserveContextCancellation(true);
+        }
+        const bool generation_started = rc == 0;
+        if (generation_started) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
            const int eos = ds4_token_eos(g_engine);
            const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
            int produced = 0;
-            while (produced < n_predict && !s.aborted) {
+            while (produced < n_predict) {
+                if (!request_should_continue(&lifecycle, context)) break;
                SampleParams sp = compute_sample_params(request, s.parser, think_enabled);
                int first;
                if (sp.temperature <= 0.0f) {
@@ -900,50 +1002,77 @@ public:
                if (draft_max > 0 && sp.temperature <= 0.0f) {
                    constexpr int kAcceptedMax = 8;
                    int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                    int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                        accepted, cap, err, sizeof(err));
                    if (n < 0) { rc = -1; break; }
                    bool stop = false;
                    for (int j = 0; j < n; ++j) {
+                        if (!request_should_continue(&lifecycle, context)) {
+                            stop = true;
+                            break;
+                        }
                        if (accepted[j] == eos) { stop = true; break; }
                        stream_emit(&s, accepted[j]);
-                        if (s.aborted) { stop = true; break; }
+                        if (!lifecycle.ShouldContinue()) { stop = true; break; }
                        if (++produced >= n_predict) { stop = true; break; }
                    }
                    if (stop) break;
                } else {
                    stream_emit(&s, first);
-                    if (s.aborted || ++produced >= n_predict) break;
+                    if (!lifecycle.ShouldContinue() || ++produced >= n_predict) break;
+                    if (!request_should_continue(&lifecycle, context)) break;
                    rc = ds4_session_eval(g_session, first, err, sizeof(err));
                    if (rc != 0) break;
                }
            }
-            stream_done(&s);
        }
-        maybe_save_cache(cache_key);

-        // Flush parser state.
-        std::vector<ds4cpp::ParserEvent> events;
-        s.parser.Flush(events);
-        if (!events.empty() && !s.aborted) {
-            backend::Reply reply;
-            auto *delta = reply.add_chat_deltas();
-            for (const auto &e : events) {
-                if (e.type == ds4cpp::ParserEvent::CONTENT) {
-                    delta->set_content(delta->content() + e.text);
-                } else if (e.type == ds4cpp::ParserEvent::REASONING) {
-                    delta->set_reasoning_content(delta->reasoning_content() + e.text);
+        request_should_continue(&lifecycle, context);
+        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
+            !lifecycle.ShouldFinalize());
+        terminal = ds4cpp::RunPostlude(
+            terminal,
+            [&]() {
+                ds4cpp::DsmlParser staged_parser = s.parser;
+                std::vector<ds4cpp::ParserEvent> events;
+                staged_parser.Flush(events);
+                bool write_succeeded = true;
+                if (!events.empty()) {
+                    backend::Reply reply;
+                    auto *delta = reply.add_chat_deltas();
+                    for (const auto &e : events) {
+                        if (e.type == ds4cpp::ParserEvent::CONTENT) {
+                            delta->set_content(delta->content() + e.text);
+                        } else if (e.type == ds4cpp::ParserEvent::REASONING) {
+                            delta->set_reasoning_content(
+                                delta->reasoning_content() + e.text);
+                        }
+                    }
+                    write_succeeded = s.writer->Write(reply);
                }
-            }
-            s.writer->Write(reply);
-        }
+                lifecycle.ObserveStreamWrite(write_succeeded);
+                request_should_continue(&lifecycle, context);
+                if (!lifecycle.ShouldFinalize()) return false;
+                s.parser = std::move(staged_parser);
+                if (generation_started) stream_done(&s);
+                return true;
+            },
+            [&]() { maybe_save_cache(cache_key); });

-        if (rc != 0 && !s.aborted) {
+        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
            return GStatus(StatusCode::INTERNAL,
                          std::string("ds4 generation failed: ") + err);
        }
+        if (terminal.cause == ds4cpp::TerminalCause::Cancelled) {
+            return GStatus(StatusCode::CANCELLED,
+                          "ds4 request cancelled");
+        }
        return GStatus::OK;
    }

@@ -0,0 +1,111 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+namespace ds4cpp {
+
+using CancelCallback = bool (*)(void *);
+using CancelSetter = void (*)(void *, CancelCallback, void *) noexcept;
+
+class CancelCallbackScope {
+public:
+    CancelCallbackScope(void *target, CancelSetter setter,
+                        CancelCallback callback, void *userdata) noexcept
+        : target_(target), setter_(setter) {
+        setter_(target_, callback, userdata);
+    }
+
+    ~CancelCallbackScope() noexcept {
+        setter_(target_, nullptr, nullptr);
+    }
+
+    CancelCallbackScope(const CancelCallbackScope &) = delete;
+    CancelCallbackScope &operator=(const CancelCallbackScope &) = delete;
+
+private:
+    void *target_;
+    CancelSetter setter_;
+};
+
+enum class RouteWaitDecision {
+    Pending,
+    Ready,
+    Error,
+    Cancelled,
+};
+
+inline RouteWaitDecision DecideRouteWait(int route_status, bool cancelled) {
+    if (cancelled) return RouteWaitDecision::Cancelled;
+    if (route_status > 0) return RouteWaitDecision::Ready;
+    if (route_status < 0) return RouteWaitDecision::Error;
+    return RouteWaitDecision::Pending;
+}
+
+enum class TerminalCause {
+    Success,
+    Cancelled,
+    EngineError,
+};
+
+inline TerminalCause DecideTerminalCause(bool sync_interrupted,
+                                         bool engine_error,
+                                         bool abandoned) {
+    if (sync_interrupted) return TerminalCause::Cancelled;
+    if (engine_error) return TerminalCause::EngineError;
+    if (abandoned) return TerminalCause::Cancelled;
+    return TerminalCause::Success;
+}
+
+struct TerminalDecision {
+    TerminalCause cause;
+    bool should_finalize;
+};
+
+inline TerminalDecision ResolveTerminalDecision(bool sync_interrupted,
+                                                 bool engine_error,
+                                                 bool abandoned) {
+    return {
+        DecideTerminalCause(sync_interrupted, engine_error, abandoned),
+        !sync_interrupted && !abandoned,
+    };
+}
+
+template <typename Finalize, typename Persist>
+TerminalDecision RunPostlude(TerminalDecision terminal,
+                             Finalize transactional_finalize,
+                             Persist persist) {
+    if (!terminal.should_finalize) return terminal;
+    if (!transactional_finalize()) {
+        terminal.should_finalize = false;
+        if (terminal.cause != TerminalCause::EngineError) {
+            terminal.cause = TerminalCause::Cancelled;
+        }
+        return terminal;
+    }
+    persist();
+    return terminal;
+}
+
+class RequestLifecycle {
+public:
+    void ObserveContextCancellation(bool cancelled) {
+        context_cancelled_ = context_cancelled_ || cancelled;
+    }
+
+    void ObserveStreamWrite(bool succeeded) {
+        stream_write_aborted_ = stream_write_aborted_ || !succeeded;
+    }
+
+    bool ShouldContinue() const {
+        return !context_cancelled_ && !stream_write_aborted_;
+    }
+
+    bool ShouldFinalize() const {
+        return ShouldContinue();
+    }
+
+private:
+    bool context_cancelled_ = false;
+    bool stream_write_aborted_ = false;
+};
+
+} // namespace ds4cpp
@@ -0,0 +1,414 @@
+// SPDX-License-Identifier: MIT
+// Standalone regression tests for DS4 request cancellation policy.
+
+#include "request_lifecycle.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+struct FakeCancelTarget {
+    ds4cpp::CancelCallback callback = nullptr;
+    void *userdata = nullptr;
+    int installs = 0;
+    int clears = 0;
+};
+
+struct PostludeCounts {
+    int finalize_attempts = 0;
+    int finalize_commits = 0;
+    int cache_persists = 0;
+    bool cache_followed_commit = true;
+};
+
+ds4cpp::TerminalDecision run_fake_postlude(
+        ds4cpp::TerminalDecision terminal, bool finalize_succeeds,
+        PostludeCounts *counts) {
+    return ds4cpp::RunPostlude(
+        terminal,
+        [=]() {
+            counts->finalize_attempts++;
+            if (!finalize_succeeds) return false;
+            counts->finalize_commits++;
+            return true;
+        },
+        [=]() {
+            counts->cache_followed_commit = counts->finalize_commits == 1;
+            counts->cache_persists++;
+        });
+}
+
+bool fake_cancel(void *) {
+    return false;
+}
+
+void fake_set_cancel(void *target, ds4cpp::CancelCallback callback,
+                     void *userdata) noexcept {
+    auto *fake = static_cast<FakeCancelTarget *>(target);
+    fake->callback = callback;
+    fake->userdata = userdata;
+    if (callback) {
+        fake->installs++;
+    } else {
+        fake->clears++;
+    }
+}
+
+void check(bool condition, const char *name) {
+    if (condition) return;
+    std::fprintf(stderr, "FAIL %s\n", name);
+    failures++;
+}
+
+// Production mutation caught: treating an active request as abandoned would
+// skip its parser finalization and cache save.
+void test_active_request_continues_and_finalizes() {
+    ds4cpp::RequestLifecycle request;
+
+    check(request.ShouldContinue(), "active:continue");
+    check(request.ShouldFinalize(), "active:finalize");
+}
+
+// Production mutation caught: omitting the ServerContext cancellation branch
+// would continue decoding and finalize a partial response.
+void test_context_cancellation_stops_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+
+    request.ObserveContextCancellation(true);
+
+    check(!request.ShouldContinue(), "context_cancelled:stop");
+    check(!request.ShouldFinalize(), "context_cancelled:no_finalize");
+}
+
+// Production mutation caught: ignoring ServerWriter::Write failure would keep
+// streaming and finalize a response whose client has gone away.
+void test_stream_write_abort_stops_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+
+    request.ObserveStreamWrite(false);
+
+    check(!request.ShouldContinue(), "write_abort:stop");
+    check(!request.ShouldFinalize(), "write_abort:no_finalize");
+}
+
+// Production mutation caught: combining cancellation and write failure with
+// AND would fail to stop when either signal occurs on its own.
+void test_cancellation_and_write_abort_are_independent_or_conditions() {
+    ds4cpp::RequestLifecycle cancelled;
+    cancelled.ObserveContextCancellation(true);
+    cancelled.ObserveStreamWrite(true);
+
+    ds4cpp::RequestLifecycle write_aborted;
+    write_aborted.ObserveContextCancellation(false);
+    write_aborted.ObserveStreamWrite(false);
+
+    check(!cancelled.ShouldContinue(), "or:context_only");
+    check(!write_aborted.ShouldContinue(), "or:write_only");
+}
+
+// Production mutation caught: treating an incomplete distributed route as an
+// error would return before workers have time to connect.
+void test_route_wait_pending() {
+    check(ds4cpp::DecideRouteWait(0, false) ==
+              ds4cpp::RouteWaitDecision::Pending,
+          "route_wait:pending");
+}
+
+// Production mutation caught: failing to recognize a complete route would
+// keep a ready inference request in the polling loop.
+void test_route_wait_ready() {
+    check(ds4cpp::DecideRouteWait(1, false) ==
+              ds4cpp::RouteWaitDecision::Ready,
+          "route_wait:ready");
+}
+
+// Production mutation caught: ignoring a route probe error would poll until a
+// misleading timeout instead of returning UNAVAILABLE promptly.
+void test_route_wait_error() {
+    check(ds4cpp::DecideRouteWait(-1, false) ==
+              ds4cpp::RouteWaitDecision::Error,
+          "route_wait:error");
+}
+
+// Production mutation caught: omitting cancellation from route waiting would
+// leave an abandoned request blocked until the distributed timeout.
+void test_route_wait_cancellation() {
+    check(ds4cpp::DecideRouteWait(0, true) ==
+              ds4cpp::RouteWaitDecision::Cancelled,
+          "route_wait:cancelled");
+}
+
+// Production mutation caught: checking route errors before cancellation would
+// report UNAVAILABLE for a request the client already abandoned.
+void test_route_wait_cancellation_precedes_error() {
+    check(ds4cpp::DecideRouteWait(-1, true) ==
+              ds4cpp::RouteWaitDecision::Cancelled,
+          "route_wait:cancellation_precedence");
+}
+
+// Production mutation caught: classifying a successful active request as a
+// terminal failure would suppress its normal response finalization.
+void test_terminal_success() {
+    check(ds4cpp::DecideTerminalCause(false, false, false) ==
+              ds4cpp::TerminalCause::Success,
+          "terminal:success");
+}
+
+// Production mutation caught: treating DS4's cooperative sync interruption
+// as an ordinary engine error would return INTERNAL instead of CANCELLED.
+void test_terminal_sync_interruption_is_cancelled() {
+    check(ds4cpp::DecideTerminalCause(true, true, true) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:sync_interrupted");
+}
+
+// Production mutation caught: treating every nonzero engine result as client
+// abandonment would hide genuine DS4 failures behind CANCELLED.
+void test_terminal_engine_error() {
+    check(ds4cpp::DecideTerminalCause(false, true, false) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error");
+}
+
+// Production mutation caught: ignoring an rc==0 context cancellation would
+// finalize and cache an abandoned request.
+void test_terminal_context_abandonment() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveContextCancellation(true);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, false, !request.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:context_abandonment");
+}
+
+// Production mutation caught: ignoring an rc==0 stream write failure would
+// finalize and cache an abandoned streaming request.
+void test_terminal_write_abandonment() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveStreamWrite(false);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, false, !request.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:write_abandonment");
+}
+
+// Production mutation caught: checking late cancellation or write failure
+// before a determined ordinary DS4 error would replace INTERNAL with CANCELLED.
+void test_terminal_engine_error_precedes_late_abandonment() {
+    ds4cpp::RequestLifecycle cancelled;
+    cancelled.ObserveContextCancellation(true);
+    ds4cpp::RequestLifecycle write_aborted;
+    write_aborted.ObserveStreamWrite(false);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, true, !cancelled.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error_precedes_cancellation");
+    check(ds4cpp::DecideTerminalCause(
+              false, true, !write_aborted.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error_precedes_write_abort");
+}
+
+// Production mutation caught: using status precedence alone to gate side
+// effects would finalize and persist an engine-error request abandoned later.
+void test_abandoned_engine_error_keeps_internal_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveContextCancellation(true);
+
+    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+        false, true, !request.ShouldFinalize());
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "terminal_decision:abandoned_engine_error_status");
+    check(!terminal.should_finalize,
+          "terminal_decision:abandoned_engine_error_no_finalize");
+}
+
+// Production mutation caught: suppressing side effects for every engine error
+// would change the existing finalization and cache behavior of active failures.
+void test_active_engine_error_still_finalizes() {
+    ds4cpp::RequestLifecycle request;
+
+    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+        false, true, !request.ShouldFinalize());
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "terminal_decision:active_engine_error_status");
+    check(terminal.should_finalize,
+          "terminal_decision:active_engine_error_finalize");
+}
+
+// Production mutation caught: persisting before committed finalization would
+// cache a state whose final buffered stream reply was never completed.
+void test_postlude_active_success_commits_then_persists() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Success, true}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Success,
+          "postlude:success_outcome");
+    check(terminal.should_finalize, "postlude:success_committed");
+    check(counts.finalize_attempts == 1, "postlude:success_attempts");
+    check(counts.finalize_commits == 1, "postlude:success_commits");
+    check(counts.cache_persists == 1, "postlude:success_cache");
+    check(counts.cache_followed_commit, "postlude:success_cache_order");
+}
+
+// Production mutation caught: starting the postlude for an already-cancelled
+// request would flush buffered parser state or persist an abandoned session.
+void test_postlude_cancellation_skips_all_side_effects() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Cancelled, false}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
+          "postlude:cancelled_outcome");
+    check(counts.finalize_attempts == 0, "postlude:cancelled_attempts");
+    check(counts.finalize_commits == 0, "postlude:cancelled_commits");
+    check(counts.cache_persists == 0, "postlude:cancelled_cache");
+}
+
+// Production mutation caught: committing the live parser or cache after a
+// failed final Write would publish an abandoned streaming postlude.
+void test_postlude_finalize_failure_cancels_without_commit_or_cache() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Success, true}, false, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
+          "postlude:write_failure_outcome");
+    check(!terminal.should_finalize, "postlude:write_failure_not_committed");
+    check(counts.finalize_attempts == 1, "postlude:write_failure_attempts");
+    check(counts.finalize_commits == 0, "postlude:write_failure_commits");
+    check(counts.cache_persists == 0, "postlude:write_failure_cache");
+}
+
+// Production mutation caught: skipping the postlude for every engine error
+// would change active internal-error finalization and cache behavior.
+void test_postlude_active_engine_error_finalizes_and_persists() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, true}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:engine_error_outcome");
+    check(counts.finalize_attempts == 1, "postlude:engine_error_attempts");
+    check(counts.finalize_commits == 1, "postlude:engine_error_commits");
+    check(counts.cache_persists == 1, "postlude:engine_error_cache");
+    check(counts.cache_followed_commit, "postlude:engine_error_cache_order");
+}
+
+// Production mutation caught: replacing every failed transactional finalize
+// with cancellation would hide an already-determined engine error.
+void test_postlude_engine_error_finalize_failure_preserves_internal() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, true}, false, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:engine_error_write_failure_outcome");
+    check(!terminal.should_finalize,
+          "postlude:engine_error_write_failure_not_committed");
+    check(counts.finalize_attempts == 1,
+          "postlude:engine_error_write_failure_attempts");
+    check(counts.finalize_commits == 0,
+          "postlude:engine_error_write_failure_commits");
+    check(counts.cache_persists == 0,
+          "postlude:engine_error_write_failure_cache");
+}
+
+// Production mutation caught: status precedence must not grant side-effect
+// permission to an engine-error request that was also abandoned.
+void test_postlude_abandoned_engine_error_skips_all_side_effects() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, false}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:abandoned_engine_error_outcome");
+    check(counts.finalize_attempts == 0,
+          "postlude:abandoned_engine_error_attempts");
+    check(counts.finalize_commits == 0,
+          "postlude:abandoned_engine_error_commits");
+    check(counts.cache_persists == 0,
+          "postlude:abandoned_engine_error_cache");
+}
+
+// Production mutation caught: failing to install the request callback would
+// make DS4 prompt synchronization unable to observe client cancellation.
+void test_cancel_callback_scope_installs_callback() {
+    FakeCancelTarget target;
+    int request_context = 42;
+
+    {
+        ds4cpp::CancelCallbackScope scope(
+            &target, fake_set_cancel, fake_cancel, &request_context);
+        check(target.callback == fake_cancel, "cancel_scope:callback_installed");
+        check(target.userdata == &request_context, "cancel_scope:userdata_installed");
+        check(target.installs == 1, "cancel_scope:installed_once");
+    }
+}
+
+// Production mutation caught: failing to clear the callback at every scope
+// exit would leave DS4 pointing at a destroyed stack-owned ServerContext.
+void test_cancel_callback_scope_clears_callback() {
+    FakeCancelTarget target;
+    int request_context = 42;
+
+    {
+        ds4cpp::CancelCallbackScope scope(
+            &target, fake_set_cancel, fake_cancel, &request_context);
+    }
+
+    check(target.callback == nullptr, "cancel_scope:callback_cleared");
+    check(target.userdata == nullptr, "cancel_scope:userdata_cleared");
+    check(target.clears == 1, "cancel_scope:cleared_once");
+}
+
+} // namespace
+
+int main() {
+    test_active_request_continues_and_finalizes();
+    test_context_cancellation_stops_without_finalizing();
+    test_stream_write_abort_stops_without_finalizing();
+    test_cancellation_and_write_abort_are_independent_or_conditions();
+    test_route_wait_pending();
+    test_route_wait_ready();
+    test_route_wait_error();
+    test_route_wait_cancellation();
+    test_route_wait_cancellation_precedes_error();
+    test_terminal_success();
+    test_terminal_sync_interruption_is_cancelled();
+    test_terminal_engine_error();
+    test_terminal_context_abandonment();
+    test_terminal_write_abandonment();
+    test_terminal_engine_error_precedes_late_abandonment();
+    test_abandoned_engine_error_keeps_internal_without_finalizing();
+    test_active_engine_error_still_finalizes();
+    test_postlude_active_success_commits_then_persists();
+    test_postlude_cancellation_skips_all_side_effects();
+    test_postlude_finalize_failure_cancels_without_commit_or_cache();
+    test_postlude_active_engine_error_finalizes_and_persists();
+    test_postlude_engine_error_finalize_failure_preserves_internal();
+    test_postlude_abandoned_engine_error_skips_all_side_effects();
+    test_cancel_callback_scope_installs_callback();
+    test_cancel_callback_scope_clears_callback();
+
+    if (failures == 0) {
+        std::fprintf(stderr, "all request_lifecycle checks passed\n");
+        return 0;
+    }
+    std::fprintf(stderr, "%d check(s) failed\n", failures);
+    return 1;
+}
@@ -1,5 +1,5 @@

-IK_LLAMA_VERSION?=c46ffaa5665cfb2d6cf372c9a054dbab896e14fe
+IK_LLAMA_VERSION?=3bb386eb68ffee0a5dc7db21da0735d594929eeb
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp

 CMAKE_ARGS?=
@@ -2565,6 +2565,7 @@ public:
    grpc::Status Embedding(ServerContext* context, const backend::PredictOptions* request, backend::EmbeddingResult* embeddingResult) {
        auto identity = checkModelIdentity(request);
        if (!identity.ok()) return identity;
+        embeddingResult->set_layout(backend::EMBEDDING_LAYOUT_FINAL);
        json data = parse_options(false, request, llama);
        const int task_id = llama.queue_tasks.get_new_id();
        llama.queue_results.add_waiting_task_id(task_id);
@@ -120,4 +120,20 @@ if(LLAMA_GRPC_BUILD_TESTS)
    target_include_directories(tts_request_options_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
    target_compile_features(tts_request_options_test PRIVATE cxx_std_17)
    add_test(NAME tts_request_options_test COMMAND tts_request_options_test)
+
+    add_executable(thread_params_test thread_params_test.cpp thread_params.h)
+    target_include_directories(thread_params_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_compile_features(thread_params_test PRIVATE cxx_std_17)
+    add_test(NAME thread_params_test COMMAND thread_params_test)
+
+    add_executable(model_load_error_test model_load_error_test.cpp model_load_error.h)
+    target_include_directories(model_load_error_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_compile_features(model_load_error_test PRIVATE cxx_std_17)
+    add_test(NAME model_load_error_test COMMAND model_load_error_test)
+
+    # Dead-stream tracker (standard library only).
+    add_executable(stream_peer_test stream_peer_test.cpp stream_peer.h)
+    target_include_directories(stream_peer_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_compile_features(stream_peer_test PRIVATE cxx_std_17)
+    add_test(NAME stream_peer_test COMMAND stream_peer_test)
 endif()
@@ -1,5 +1,5 @@

-LLAMA_VERSION?=030ebb558a5820b444a8f836ed5cdd46c9b4bd7a
+LLAMA_VERSION?=df03399b885831b2a1603b3abb0d8c156808e363
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp

 CMAKE_ARGS?=
@@ -53,8 +53,11 @@
 #include "arg.h"
 #include "chat-auto-parser.h"
 #include "llama_compat.h" // fork-skew switches, generated by prepare.sh
+#include "model_load_error.h"
+#include "thread_params.h"
 #include "message_content.h"
 #include "passthrough_options.h"
+#include "stream_peer.h"
 #include "tts_request_options.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
@@ -87,6 +90,12 @@ using grpc::ServerBuilder;
 using grpc::ServerContext;
 using grpc::Status;

+#if LOCALAI_HAS_MTMD_INIT_OPT
+#define LOCALAI_MTMD_INIT_OPT_ARG(value) , value
+#else
+#define LOCALAI_MTMD_INIT_OPT_ARG(value)
+#endif
+
 // gRPC bearer token auth for distributed mode.
 // Reads LOCALAI_GRPC_AUTH_TOKEN from the environment. When set, rejects
 // requests without a matching "authorization: Bearer <token>" metadata header.
@@ -293,7 +302,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
            } else {
                SRV_WRN("[TOOLS DEBUG] parse_options: Parsed tools JSON is not an array: %s\n", tools_json.dump().c_str());
            }
-        } catch (const json::parse_error& e) {
+        } catch (const common_json_error& e) {
            SRV_WRN("Failed to parse tools JSON from proto: %s\n", e.what());
            SRV_WRN("[TOOLS DEBUG] parse_options: Tools string that failed to parse: %s\n", predict->tools().c_str());
        }
@@ -323,7 +332,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
                SRV_DBG("[TOOLS DEBUG] Received tool_choice object from Go layer: %s\n", tool_choice_json.dump().c_str());
            }
            SRV_INF("Extracted tool_choice from proto: %s\n", predict->toolchoice().c_str());
-        } catch (const json::parse_error& e) {
+        } catch (const common_json_error& e) {
            // If parsing fails, treat as string
            data["tool_choice"] = predict->toolchoice();
            SRV_INF("Extracted tool_choice as string: %s\n", predict->toolchoice().c_str());
@@ -352,7 +361,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
            // Add to data - llama.cpp server expects it as an object (map)
            data["logit_bias"] = logit_bias_json;
            SRV_INF("Using logit_bias: %s\n", predict->logitbias().c_str());
-        } catch (const json::parse_error& e) {
+        } catch (const common_json_error& e) {
            SRV_ERR("Failed to parse logit_bias JSON from proto: %s\n", e.what());
        }
    }
@@ -397,7 +406,10 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
            });
    }

-    data["stop"] = predict->stopprompts();
+    data["stop"] = json::array();
+    for (const auto & stop : predict->stopprompts()) {
+        data["stop"].push_back(stop);
+    }
    // data["n_probs"] = predict->nprobs();
    //TODO: images,

@@ -1115,14 +1127,16 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                try {
                    int n = std::stoi(optval_str);
                    if (n < 0) n = 0;
-                    // Keep override-name storage alive for the lifetime of the params struct
-                    // (mirrors upstream arg.cpp behavior with a function-local static).
+#if LOCALAI_HAS_N_CPU_FFN_HELPER
+                    llm_add_n_cpu_ffn_overrides(n, LLM_FFN_EXPS_REGEX, params.speculative.draft.tensor_buft_overrides);
+#else
                    static std::list<std::string> buft_overrides_draft;
                    for (int i = 0; i < n; ++i) {
                        buft_overrides_draft.push_back(llm_ffn_exps_block_regex(i));
                        params.speculative.draft.tensor_buft_overrides.push_back(
                            {buft_overrides_draft.back().c_str(), ggml_backend_cpu_buffer_type()});
                    }
+#endif
                } catch (...) {}
            }

@@ -1140,14 +1154,16 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                try {
                    int n = std::stoi(optval_str);
                    if (n < 0) n = 0;
-                    // Keep override-name storage alive for the lifetime of the
-                    // params struct (mirrors upstream arg.cpp's function-local static).
+#if LOCALAI_HAS_N_CPU_FFN_HELPER
+                    llm_add_n_cpu_ffn_overrides(n, LLM_FFN_EXPS_REGEX, params.tensor_buft_overrides);
+#else
                    static std::list<std::string> buft_overrides_main;
                    for (int i = 0; i < n; ++i) {
                        buft_overrides_main.push_back(llm_ffn_exps_block_regex(i));
                        params.tensor_buft_overrides.push_back(
                            {buft_overrides_main.back().c_str(), ggml_backend_cpu_buffer_type()});
                    }
+#endif
                } catch (...) {}
            }

@@ -1412,6 +1428,12 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
            passthrough_draft_gpu_layers);
    }

+    // The library initializer now creates both threadpools before the server
+    // can apply llama_context's fallback for the -1 batch-thread sentinel.
+    params.cpuparams_batch.n_threads = llama_grpc::resolve_batch_threads(
+        params.cpuparams_batch.n_threads,
+        params.cpuparams.n_threads);
+
 #ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
    // Score-task suffix forking: reserve seq ids (and recurrent-state cells)
    // beyond the slots so one scoring call decodes all candidate tails in a
@@ -1608,7 +1630,8 @@ public:
            {
                std::lock_guard<std::mutex> lock(error_capture_data.error_mutex);
                if (!error_capture_data.captured_error.empty()) {
-                    error_msg += ". Error: " + error_capture_data.captured_error;
+                    error_msg += ". Error: " +
+                                 localai::model_load_error_with_hint(error_capture_data.captured_error);
                } else {
                    error_msg += ". Model file may not exist or be invalid.";
                }
@@ -1788,7 +1811,7 @@ public:
                        for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j));
                        for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j));
                    }
-                    messages_json.push_back(llama_grpc::build_reconstructed_message(rin));
+                    messages_json.push_back(json::parse(llama_grpc::build_reconstructed_message(rin).dump()));
                }

                // Final safety check: Ensure no message has null content (Jinja templates require strings)
@@ -1981,7 +2004,7 @@ public:
                            if (!body_json.contains("chat_template_kwargs")) {
                                body_json["chat_template_kwargs"] = json::object();
                            }
-                            for (auto& el : ctk.items()) {
+                            for (auto el : ctk.items()) {
                                body_json["chat_template_kwargs"][el.key()] = el.value();
                            }
                        }
@@ -2067,30 +2090,27 @@ public:
            // If not using chat templates, extract files from image_data/audio_data fields
            // (If using chat templates, files were already extracted by oaicompat_chat_params_parse)
            if (!request->usetokenizertemplate() || request->messages_size() == 0 || ctx_server.impl->chat_params.tmpls == nullptr) {
-                const auto &images_data = data.find("image_data");
-                if (images_data != data.end() && images_data->is_array())
+                if (data.contains("image_data") && data.at("image_data").is_array())
                {
-                    for (const auto &img : *images_data)
+                    for (const auto &img : data.at("image_data"))
                    {
                        auto decoded_data = base64_decode(img["data"].get<std::string>());
                        files.push_back(decoded_data);
                    }
                }

-                const auto &audio_data = data.find("audio_data");
-                if (audio_data != data.end() && audio_data->is_array())
+                if (data.contains("audio_data") && data.at("audio_data").is_array())
                {
-                    for (const auto &audio : *audio_data)
+                    for (const auto &audio : data.at("audio_data"))
                    {
                        auto decoded_data = base64_decode(audio["data"].get<std::string>());
                        files.push_back(decoded_data);
                    }
                }

-                const auto &video_data = data.find("video_data");
-                if (video_data != data.end() && video_data->is_array())
+                if (data.contains("video_data") && data.at("video_data").is_array())
                {
-                    for (const auto &video : *video_data)
+                    for (const auto &video : data.at("video_data"))
                    {
                        auto decoded_data = base64_decode(video["data"].get<std::string>());
                        files.push_back(decoded_data);
@@ -2104,10 +2124,10 @@ public:
            std::vector<server_tokens> inputs;
            if (has_mtmd) {
                // multimodal
-                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files));
+                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)));
            } else {
                 // Everything else, including multimodal completions.
-                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true);
+                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
            }

            tasks.reserve(inputs.size());
@@ -2249,6 +2269,11 @@ public:
        // such concept, so there is nothing to emit — the real tokens arrive in
        // the loop below. Feeding this null into build_reply_from_json would
        // throw (uncaught) and surface as a generic RPC error.
+        // A write that returns false means the peer is gone for good. Track it
+        // so the loop below stops decoding instead of feeding a dead stream —
+        // see stream_peer.h for why that matters to everyone else's requests.
+        llama_grpc::StreamPeer peer;
+
        if (first_res_json.is_null()) {
            // skip the begin-of-stream marker
        } else if (first_res_json.is_array()) {
@@ -2261,17 +2286,21 @@ public:
                if (!is_role_init) {
                    attach_chat_deltas(reply, first_result.get());
                }
-                writer->Write(reply);
+                peer.observe_write(writer->Write(reply));
+                if (peer.gone()) {
+                    break;
+                }
            }
        } else {
            auto reply = build_reply_from_json(first_res_json, first_result.get());
            attach_chat_deltas(reply, first_result.get());
-            writer->Write(reply);
+            peer.observe_write(writer->Write(reply));
        }

        // Process subsequent results
        while (rd.has_next()) {
-            if (context->IsCancelled()) {
+            peer.observe_cancelled(context->IsCancelled());
+            if (peer.gone()) {
                break;
            }

@@ -2292,17 +2321,22 @@ public:
                    if (!is_role_init) {
                        attach_chat_deltas(reply, result.get());
                    }
-                    writer->Write(reply);
+                    peer.observe_write(writer->Write(reply));
+                    if (peer.gone()) {
+                        break;
+                    }
                }
            } else {
                auto reply = build_reply_from_json(res_json, result.get());
                attach_chat_deltas(reply, result.get());
-                writer->Write(reply);
+                peer.observe_write(writer->Write(reply));
            }
        }

-        // Check if context was cancelled during processing
-        if (context->IsCancelled()) {
+        // Returning here is what releases the slot: ~server_response_reader()
+        // posts SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
+        peer.observe_cancelled(context->IsCancelled());
+        if (peer.gone()) {
            return grpc::Status(grpc::StatusCode::CANCELLED, "Request cancelled by client");
        }

@@ -2363,7 +2397,7 @@ public:
                        for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j));
                        for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j));
                    }
-                    messages_json.push_back(llama_grpc::build_reconstructed_message(rin));
+                    messages_json.push_back(json::parse(llama_grpc::build_reconstructed_message(rin).dump()));
                }

                // Final safety check: Ensure no message has null content (Jinja templates require strings)
@@ -2556,7 +2590,7 @@ public:
                            if (!body_json.contains("chat_template_kwargs")) {
                                body_json["chat_template_kwargs"] = json::object();
                            }
-                            for (auto& el : ctk.items()) {
+                            for (auto el : ctk.items()) {
                                body_json["chat_template_kwargs"][el.key()] = el.value();
                            }
                        }
@@ -2642,11 +2676,10 @@ public:
            // If not using chat templates, extract files from image_data/audio_data fields
            // (If using chat templates, files were already extracted by oaicompat_chat_params_parse)
            if (!request->usetokenizertemplate() || request->messages_size() == 0 || ctx_server.impl->chat_params.tmpls == nullptr) {
-                const auto &images_data = data.find("image_data");
-                if (images_data != data.end() && images_data->is_array())
+                if (data.contains("image_data") && data.at("image_data").is_array())
                {
-                    std::cout << "[PREDICT] Processing " << images_data->size() << " images" << std::endl;
-                    for (const auto &img : *images_data)
+                    std::cout << "[PREDICT] Processing " << data.at("image_data").size() << " images" << std::endl;
+                    for (const auto &img : data.at("image_data"))
                    {
                        std::cout << "[PREDICT] Processing image" << std::endl;
                        auto decoded_data = base64_decode(img["data"].get<std::string>());
@@ -2654,20 +2687,18 @@ public:
                    }
                }

-                const auto &audio_data = data.find("audio_data");
-                if (audio_data != data.end() && audio_data->is_array())
+                if (data.contains("audio_data") && data.at("audio_data").is_array())
                {
-                    for (const auto &audio : *audio_data)
+                    for (const auto &audio : data.at("audio_data"))
                    {
                        auto decoded_data = base64_decode(audio["data"].get<std::string>());
                        files.push_back(decoded_data);
                    }
                }

-                const auto &video_data = data.find("video_data");
-                if (video_data != data.end() && video_data->is_array())
+                if (data.contains("video_data") && data.at("video_data").is_array())
                {
-                    for (const auto &video : *video_data)
+                    for (const auto &video : data.at("video_data"))
                    {
                        auto decoded_data = base64_decode(video["data"].get<std::string>());
                        files.push_back(decoded_data);
@@ -2682,10 +2713,10 @@ public:
            std::vector<server_tokens> inputs;
            if (has_mtmd) {
                // multimodal
-                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files));
+                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)));
            } else {
                 // Everything else, including multimodal completions.
-                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true);
+                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
            }

            tasks.reserve(inputs.size());
@@ -2872,7 +2903,7 @@ public:
        json prompt = body.at("embeddings");


-        auto tokenized_prompts = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt, true, true);
+        auto tokenized_prompts = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
        for (const auto & tokens : tokenized_prompts) {
            // this check is necessary for models that do not add BOS token to the input
            if (tokens.empty()) {
@@ -2911,42 +2942,40 @@ public:
            return grpc::Status(grpc::StatusCode::INTERNAL, all_results.error->to_json().value("message", "Error in receiving results"));
        }

-        // Collect responses
-        json responses = json::array();
+        // Extract the embeddings typed, straight from the task results (no
+        // JSON round-trip), and report the payload shape alongside the same
+        // flat float array as before: dim is the embedding width, tokens the
+        // number of vectors packed into `embeddings` (1 per prompt when the
+        // server pooled, one per token with pooling:none; summed across
+        // prompts if the request carried several), prompt_tokens the prompt
+        // tokens evaluated, for usage accounting. Consumers seeing 0/0 know
+        // the backend predates shape reporting.
+        int32_t n_vectors = 0;
+        int32_t dim = 0;
+        int32_t prompt_tokens = 0;
        for (auto & res : all_results.results) {
-            GGML_ASSERT(dynamic_cast<server_task_result_embd*>(res.get()) != nullptr);
-            responses.push_back(res->to_json());
-        }
-
-        std::cout << "[DEBUG] Responses size: " << responses.size() << std::endl;
-
-        // Process the responses and extract embeddings
-        for (const auto & response_elem : responses) {
-            // Check if the response has an "embedding" field
-            if (response_elem.contains("embedding")) {
-                json embedding_data = json_value(response_elem, "embedding", json::array());
-
-                if (embedding_data.is_array() && !embedding_data.empty()) {
-                    for (const auto & embedding_vector : embedding_data) {
-                        if (embedding_vector.is_array()) {
-                            for (const auto & embedding_value : embedding_vector) {
-                                embeddingResult->add_embeddings(embedding_value.get<float>());
-                            }
-                        }
-                    }
+            auto * embd_res = dynamic_cast<server_task_result_embd*>(res.get());
+            GGML_ASSERT(embd_res != nullptr);
+            prompt_tokens += embd_res->n_tokens;
+            for (const auto & vec : embd_res->embedding) {
+                for (const float value : vec) {
+                    embeddingResult->add_embeddings(value);
                }
-            } else {
-                // Check if the response itself contains the embedding data directly
-                if (response_elem.is_array()) {
-                    for (const auto & embedding_value : response_elem) {
-                        embeddingResult->add_embeddings(embedding_value.get<float>());
-                    }
+                if (!vec.empty()) {
+                    n_vectors++;
+                    dim = (int32_t) vec.size();
                }
            }
        }
+        embeddingResult->set_tokens(n_vectors);
+        embeddingResult->set_dim(dim);
+        embeddingResult->set_prompt_tokens(prompt_tokens);
+        embeddingResult->set_layout(
+            llama_pooling_type(ctx_server.get_llama_context()) == LLAMA_POOLING_TYPE_NONE
+                ? backend::EMBEDDING_LAYOUT_PER_TOKEN
+                : backend::EMBEDDING_LAYOUT_FINAL);

-
-
+        std::cout << "[DEBUG] Embedding vectors: " << n_vectors << " x " << dim << std::endl;

        return grpc::Status::OK;
    }
@@ -2979,7 +3008,7 @@ public:

            tasks.reserve(documents.size());
            for (size_t i = 0; i < documents.size(); i++) {
-                auto tmp = format_prompt_rerank(ctx_server.impl->model_tgt, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i]);
+                auto tmp = format_prompt_rerank(ctx_server.impl->model_tgt, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i] LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
                server_task task = server_task(SERVER_TASK_TYPE_RERANK);
                task.id = rd.queue_tasks.get_new_id();
                task.index = i;
@@ -3000,7 +3029,7 @@ public:
        }

        // Collect responses
-        json responses = json::array();
+        std::vector<json> responses;
        for (auto & res : all_results.results) {
            GGML_ASSERT(dynamic_cast<server_task_result_rerank*>(res.get()) != nullptr);
            responses.push_back(res->to_json());
@@ -3013,7 +3042,7 @@ public:
        // Crop results by request.top_n if specified
        int top_n = request->top_n();
        if (top_n > 0 && top_n < static_cast<int>(responses.size())) {
-            responses = json(responses.begin(), responses.begin() + top_n);
+            responses.resize(top_n);
        }
        // Set usage information
        backend::Usage* usage = rerankResult->mutable_usage();
@@ -3060,7 +3089,7 @@ public:
            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, opts.error);
        }

-        auto wrapper = mtmd_helper_bitmap_init_from_file(ctx_server.impl->mctx, opts.voice_path.c_str(), false);
+        auto wrapper = mtmd_helper_bitmap_init_from_file(ctx_server.impl->mctx, opts.voice_path.c_str(), false LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
        if (!wrapper.bitmap) {
            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
                "failed to read speaker reference audio: " + opts.voice_path);
@@ -3597,9 +3626,15 @@ public:
        // Populate the response with metrics
        response->set_slot_id(0);
        response->set_prompt_json_for_slot("");
+#if LOCALAI_HAS_SERVER_METRICS
+        response->set_tokens_per_second(res_metrics->metrics.prompt_bucket.n_per_second());
+        response->set_tokens_generated(res_metrics->metrics.predict.count);
+        response->set_prompt_tokens_processed(res_metrics->metrics.prompt.count);
+#else
        response->set_tokens_per_second(res_metrics->n_prompt_tokens_processed ? 1.e3 / res_metrics->t_prompt_processing * res_metrics->n_prompt_tokens_processed : 0.);
        response->set_tokens_generated(res_metrics->n_tokens_predicted_total);
        response->set_prompt_tokens_processed(res_metrics->n_prompt_tokens_processed_total);
+#endif


        return grpc::Status::OK;
@@ -52,14 +52,15 @@ inline nlohmann::ordered_json normalize_message_content(const std::string& role,
 // (#7528). A multimodal user message legitimately carries a typed-part array
 // ({type:text}, {type:image_url}, ...), which must be left intact. Shared by the
 // streaming and non-streaming paths so this invariant cannot drift between them.
-inline void normalize_template_message(nlohmann::ordered_json& msg) {
+template <typename Json>
+inline void normalize_template_message(Json& msg) {
    if (!msg.contains("content")) {
        msg["content"] = ""; // templates expect the field to exist
        return;
    }
-    nlohmann::ordered_json& content = msg["content"];
+    auto& content = msg["content"];
    const std::string role = (msg.contains("role") && msg["role"].is_string())
-                                 ? msg["role"].get<std::string>()
+                                 ? msg["role"].template get<std::string>()
                                 : std::string();
    if (content.is_null()) {
        content = ""; // #7324: null would crash content[:N] slicing
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include <string>
+
+namespace localai {
+
+inline std::string model_load_error_with_hint(const std::string& error) {
+  const std::string mismatch = "wrong number of tensors; expected ";
+  const std::string got = ", got ";
+  const std::string::size_type mismatch_pos = error.find(mismatch);
+  if (mismatch_pos == std::string::npos ||
+      error.find(got, mismatch_pos + mismatch.size()) == std::string::npos) {
+    return error;
+  }
+
+  return error +
+         " Hint: the model may be incompatible with this llama.cpp backend "
+         "or the GGUF file may be corrupt. Try a newer compatible backend "
+         "and verify or re-download the model file.";
+}
+
+}  // namespace localai
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: MIT
+
+#include "model_load_error.h"
+
+#include <cassert>
+#include <string>
+
+int main() {
+  const std::string issue_error =
+      "llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 2131, got 720; "
+      "llama_model_load_from_file_impl: failed to load model";
+  const std::string issue_result = localai::model_load_error_with_hint(issue_error);
+  assert(issue_result.compare(0, issue_error.size(), issue_error) == 0);
+  assert(issue_result.find("incompatible") != std::string::npos);
+  assert(issue_result.find("corrupt") != std::string::npos);
+
+  const std::string generic_error =
+      "wrong number of tensors; expected 42, got 17";
+  const std::string generic_result = localai::model_load_error_with_hint(generic_error);
+  assert(generic_result.compare(0, generic_error.size(), generic_error) == 0);
+  assert(generic_result.size() > generic_error.size());
+
+  const std::string unrelated_error = "failed to open GGUF file";
+  assert(localai::model_load_error_with_hint(unrelated_error) == unrelated_error);
+}
@@ -6,10 +6,9 @@ Subject: [PATCH 1/2] score-patch
 ---
 common/common.cpp               |   6 +-
 common/common.h                 |   3 +
- tools/CMakeLists.txt            |   1 +
 tools/server/server-context.cpp | 358 +++++++++++++++++++++++++++++++-
 tools/server/server-task.h      |  47 +++++
- 5 files changed, 406 insertions(+), 9 deletions(-)
+ 4 files changed, 405 insertions(+), 9 deletions(-)

 diff --git a/common/common.cpp b/common/common.cpp
 index 2e3f14c..0cec0dc 100644
@@ -42,15 +41,6 @@ index 878534d..4001df2 100644
     int32_t n_sequences           =     1; // number of sequences to decode
     int32_t n_outputs_max         =     0; // max outputs in a batch (0 = n_batch)
     int32_t n_outputs_max_per_seq =     1; // max outputs per sequence
-diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt
-index 780df32..1d2fe8f 100644
--- a/tools/CMakeLists.txt
-+++ b/tools/CMakeLists.txt
-@@ -41,3 +41,4 @@ else()
-     add_subdirectory(fit-params)
-     add_subdirectory(results)
- endif()
-+add_subdirectory(grpc-server)
 diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
 index 3b5f6a1..d0e18e6 100644
 --- a/tools/server/server-context.cpp
@@ -96,7 +86,7 @@ index 3b5f6a1..d0e18e6 100644
     bool has_next_token = true;
     bool has_new_line   = false;
     bool truncated      = false;
-@@ -351,6 +378,10 @@ struct server_slot {
+@@ -341,6 +368,10 @@ struct server_slot {
         }
         generated_tokens.clear();
         generated_token_probs.clear();
@@ -106,7 +96,7 @@ index 3b5f6a1..d0e18e6 100644
 +        score_divergence = -1;
         json_schema = json();

-         // clear speculative decoding stats
+         task_prev = std::move(task);
@@ -2271,6 +2302,229 @@ private:
         queue_results.send(std::move(res));
     }
@@ -391,7 +381,7 @@ index 3b5f6a1..d0e18e6 100644

                     // make a checkpoint of the parts of the memory that cannot be rolled back.
                     // checkpoints are created only if:
-@@ -3523,10 +3799,17 @@ private:
+@@ -3444,9 +3720,16 @@ private:
                         // embedding requires all tokens in the batch to be output;
                         // MTP also wants logits at every prompt position so the
                         // streaming hook can mirror t_h_nextn into ctx_dft.
@@ -404,16 +394,12 @@ index 3b5f6a1..d0e18e6 100644
 +                            slot.prompt.n_tokens() + 1 <  slot.task->n_tokens();
                         add_ok &= batch.add(slot.id,
                             cur_tok,
-                             slot.prompt.tokens.pos_next(),
-                            slot.need_embd());
-+                            slot.need_embd() || need_score_logit);
+                             /* pos       = */ slot.prompt.tokens.pos_next(),
+-                            /* output    = */ slot.need_embd(),
+                            /* output    = */ slot.need_embd() || need_score_logit,
+                             /* is_prompt = */ true);
                         slot.prompt.tokens.push_back(cur_tok);
-
-                         slot.n_prompt_tokens_processed++;
-@@ -3541,6 +3824,32 @@ private:
-                             }
-                         }
-
+@@ -3454,2 +3737,28 @@ private:
 +                        // score tasks: break at the shared-prompt boundary so the checkpoint
 +                        // below lands exactly there — the other candidates of the same
 +                        // scoring call re-process only their own tokens. Also break at the
@@ -440,9 +426,8 @@ index 3b5f6a1..d0e18e6 100644
 +                            }
 +                        }
 +
-                         // process the last few tokens of the prompt separately in order to allow for a checkpoint to be created.
-                         // create checkpoints that many tokens before the end of the prompt:
-                         //  - 4 + n_ubatch
+                         // break at the last user message, or at user messages at least min step past the last checkpoint
+                         if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
@@ -3573,6 +3882,15 @@ private:
                     const bool is_user_start = spans.is_user_start(n_tokens_start);
                     const bool is_last_user_message = n_tokens_start == last_user_pos;
@@ -1,22 +1,8 @@
-From 861fb06531e770fcea65e86296f80eba830dac30 Mon Sep 17 00:00:00 2001
-From: Codex <codex@local>
-Date: Mon, 10 Aug 2026 23:05:53 +0000
-Subject: [PATCH 2/2] tts-patch
-
---
- tools/mtmd/mtmd-helper-gen.cpp  | 118 +++++++++---
- tools/mtmd/mtmd-helper.h        |  47 ++++-
- tools/server/server-context.cpp | 322 +++++++++++++++++++++++++++++++-
- tools/server/server-context.h   |   3 +
- tools/server/server-task.cpp    |  11 ++
- tools/server/server-task.h      |  16 ++
- 6 files changed, 481 insertions(+), 36 deletions(-)
-
 diff --git a/tools/mtmd/mtmd-helper-gen.cpp b/tools/mtmd/mtmd-helper-gen.cpp
-index 85671d1..66ccf0a 100644
+index 1c58d3ae1..196cbd433 100644
 --- a/tools/mtmd/mtmd-helper-gen.cpp
 +++ b/tools/mtmd/mtmd-helper-gen.cpp
-@@ -48,29 +48,38 @@ static llama_token find_special_token(const llama_vocab * vocab, const std::stri
+@@ -50,29 +50,38 @@ static llama_token find_special_token(const llama_vocab * vocab, const std::stri
     return LLAMA_TOKEN_NULL;
 }

@@ -73,16 +59,16 @@ index 85671d1..66ccf0a 100644
     return true;
 }

-@@ -89,6 +98,8 @@ public:
-     // those read what they need from h_state_in instead
-     virtual int32_t step_gen(llama_token sampled, const float * h_state_in, const float ** h_state_out) = 0;
+@@ -92,6 +101,8 @@ public:
+     // set out_stop on end-of-speech, h_state_out must be null if no frame is generated
+     virtual int32_t step_gen(llama_token sampled, const float * h_state_in, const float ** h_state_out, bool * out_stop) = 0;
     virtual int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples) = 0;
 +    // forces any buffered codes through code2wav now, regardless of window_frames
-+    virtual int32_t flush() = 0;
+    virtual int32_t flush() { return 0; }

 protected:
     llama_context * lctx;
-@@ -118,6 +129,9 @@ public:
+@@ -121,6 +132,9 @@ public:
         prompt_batch.reset();
         n_prompt = 0;
         prompt_pos = 0;
@@ -92,15 +78,15 @@ index 85671d1..66ccf0a 100644
     }

     int32_t set_input(const mtmd_helper_gen_audio_inp * inp) override {
-@@ -203,6 +217,7 @@ public:
-         top_k = inp->top_k > 0 ? inp->top_k : 50;
-         top_p = inp->top_p > 0 ? inp->top_p : 1.0f;
+@@ -208,6 +222,7 @@ public:
+         top_p = inp->top_p > 0 ? inp->top_p : def.top_p;
+         seed  = inp->seed;
         out_type = inp->out_type;
 +        stream = inp->stream;

         // the prompt above holds the whole text stream up to tts_eos, so every generated
         // frame adds tts_pad on top of the codes embedding
-@@ -284,31 +299,60 @@ public:
+@@ -302,31 +317,60 @@ public:
     }

     int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples) override {
@@ -170,7 +156,7 @@ index 85671d1..66ccf0a 100644
 private:
     bool ensure_cache() {
         if (specials_ok) {
-@@ -352,7 +396,7 @@ private:
+@@ -370,7 +414,7 @@ private:
             LOG_ERR("mtmd_helper_gen_audio: mmproj has no speaker/audio encoder\n");
             return false;
         }
@@ -179,7 +165,7 @@ index 85671d1..66ccf0a 100644
         mtmd_input_text     text{ marker.c_str(), marker.size(), false, true };
         mtmd_input_chunks * chunks = mtmd_input_chunks_init();
         const mtmd_bitmap * bptr = bitmap;
-@@ -436,6 +480,9 @@ private:
+@@ -456,6 +500,9 @@ private:
     std::vector<float> h_state_buf;
     mtmd_helper_gen_audio_outtype out_type = MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV;
     std::vector<char> out_buf;
@@ -188,8 +174,8 @@ index 85671d1..66ccf0a 100644
 +    bool   wav_header_sent = false;
 };

- static std::unique_ptr<mtmd_gen_audio_pipeline> make_pipeline(llama_context * lctx, mtmd_context * mctx) {
-@@ -467,6 +514,14 @@ void mtmd_helper_gen_audio_reset(mtmd_helper_gen_audio * ctx) {
+ // settings that only live in the reference's per-pack yaml, not in the checkpoint
+@@ -1024,6 +1071,14 @@ void mtmd_helper_gen_audio_reset(mtmd_helper_gen_audio * ctx) {
     }
 }

@@ -204,7 +190,7 @@ index 85671d1..66ccf0a 100644
 int32_t mtmd_helper_gen_audio_set_input(mtmd_helper_gen_audio * ctx, const mtmd_helper_gen_audio_inp * inp) {
     if (!ctx->pipeline) {
         LOG_ERR("mtmd_helper_gen_audio: unsupported or missing gen-audio pipeline\n");
-@@ -497,3 +552,10 @@ int32_t mtmd_helper_gen_audio_get_output(mtmd_helper_gen_audio * ctx, int32_t *
+@@ -1060,3 +1115,10 @@ int32_t mtmd_helper_gen_audio_get_output(mtmd_helper_gen_audio * ctx, int32_t *
     }
     return ctx->pipeline->get_output(out_sample_rate, out_data, out_data_len, out_n_samples);
 }
@@ -216,7 +202,7 @@ index 85671d1..66ccf0a 100644
 +    return ctx->pipeline->flush();
 +}
 diff --git a/tools/mtmd/mtmd-helper.h b/tools/mtmd/mtmd-helper.h
-index 7e5cf9b..1f3ec01 100644
+index 832f7171a..3eaa01aab 100644
 --- a/tools/mtmd/mtmd-helper.h
 +++ b/tools/mtmd/mtmd-helper.h
@@ -175,6 +175,7 @@ enum mtmd_helper_gen_audio_outtype {
@@ -227,7 +213,7 @@ index 7e5cf9b..1f3ec01 100644
     llama_seq_id seq_id;

     const char * prompt;
-@@ -189,6 +190,8 @@ struct mtmd_helper_gen_audio_inp {
+@@ -190,6 +191,8 @@ struct mtmd_helper_gen_audio_inp {
     enum mtmd_helper_gen_audio_outtype out_type;
 };

@@ -236,7 +222,7 @@ index 7e5cf9b..1f3ec01 100644
 MTMD_API mtmd_helper_gen_audio * mtmd_helper_gen_audio_init(
                                     struct llama_context * lctx,
                                     struct mtmd_context * mctx);
-@@ -217,6 +220,8 @@ MTMD_API int32_t mtmd_helper_gen_audio_step_gen(
+@@ -221,6 +224,8 @@ MTMD_API int32_t mtmd_helper_gen_audio_step_gen(

 // out_data valid until next get_output() or reset() call
 // out_n_samples (optional, can be NULL) receives the number of generated PCM samples
@@ -245,7 +231,7 @@ index 7e5cf9b..1f3ec01 100644
 MTMD_API int32_t mtmd_helper_gen_audio_get_output(
                         mtmd_helper_gen_audio * ctx,
                         int32_t * out_sample_rate,
-@@ -224,6 +229,10 @@ MTMD_API int32_t mtmd_helper_gen_audio_get_output(
+@@ -228,6 +233,10 @@ MTMD_API int32_t mtmd_helper_gen_audio_get_output(
                         size_t * out_data_len,
                         int64_t * out_n_samples);

@@ -256,7 +242,7 @@ index 7e5cf9b..1f3ec01 100644
 #ifdef __cplusplus
 } // extern "C"
 #endif
-@@ -250,8 +259,41 @@ struct mtmd_helper_gen_audio_deleter {
+@@ -254,8 +263,41 @@ struct mtmd_helper_gen_audio_deleter {
 };
 using gen_audio_ptr = std::unique_ptr<mtmd_helper_gen_audio, mtmd_helper_gen_audio_deleter>;
 struct gen_audio {
@@ -299,7 +285,7 @@ index 7e5cf9b..1f3ec01 100644
     void reset() {
         mtmd_helper_gen_audio_reset(ctx.get());
     }
-@@ -267,6 +309,9 @@ struct gen_audio {
+@@ -271,6 +313,9 @@ struct gen_audio {
     int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples = nullptr) {
         return mtmd_helper_gen_audio_get_output(ctx.get(), out_sample_rate, out_data, out_data_len, out_n_samples);
     }
@@ -310,7 +296,7 @@ index 7e5cf9b..1f3ec01 100644

 } // namespace mtmd_helper
 diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
-index d0e18e6..0765065 100644
+index 9069463fe..b7fa1e534 100644
 --- a/tools/server/server-context.cpp
 +++ b/tools/server/server-context.cpp
@@ -16,6 +16,7 @@
@@ -333,7 +319,7 @@ index d0e18e6..0765065 100644
     }

     auto result = common_speculative_get_output_limits(
-@@ -211,6 +213,30 @@ struct server_slot {
+@@ -212,6 +214,30 @@ struct server_slot {
     mtmd_context * mctx = nullptr;
     mtmd::batch_ptr mbatch = nullptr;

@@ -364,7 +350,7 @@ index d0e18e6..0765065 100644
     // speculative decoding
     common_speculative * spec;

-@@ -400,6 +426,8 @@ struct server_slot {
+@@ -391,6 +417,8 @@ struct server_slot {

         // clear multimodal state
         mbatch.reset();
@@ -373,7 +359,7 @@ index d0e18e6..0765065 100644
     }

     void init_sampler() const {
-@@ -946,6 +974,14 @@ public:
+@@ -829,6 +857,14 @@ public:
     mtmd_context * mctx = nullptr;
     const llama_vocab * vocab = nullptr;

@@ -388,7 +374,7 @@ index d0e18e6..0765065 100644
     server_queue    queue_tasks;
     server_response queue_results;

-@@ -1399,6 +1435,10 @@ private:
+@@ -1288,6 +1324,10 @@ private:
             slot.mctx                   = mctx;
             slot.prompt.tokens.has_mtmd = mctx != nullptr;

@@ -399,7 +385,7 @@ index d0e18e6..0765065 100644
             SLT_TRC(slot, "new slot, n_ctx = %d\n", slot.n_ctx);

             slot.callback_on_release = [this](int id_slot) {
-@@ -1852,6 +1892,28 @@ private:
+@@ -1748,6 +1788,28 @@ private:

         SLT_DBG(slot, "launching slot : %s\n", safe_json_to_str(slot.to_json()).c_str());

@@ -428,7 +414,7 @@ index d0e18e6..0765065 100644
         // initialize samplers
         if (task.need_sampling()) {
             try {
-@@ -1869,6 +1931,9 @@ private:
+@@ -1765,6 +1827,9 @@ private:
             // TODO: getting pre sampling logits is not yet supported with backend sampling
             use_backend_sampling &= !need_pre_sample_logits;

@@ -438,7 +424,7 @@ index d0e18e6..0765065 100644
             // TODO: tmp until backend sampling is fully implemented
             if (use_backend_sampling) {
                 llama_set_sampler(ctx_tgt, slot.id, common_sampler_get(slot.smpl.get()));
-@@ -1884,9 +1949,13 @@ private:
+@@ -1783,9 +1848,13 @@ private:

         slot.task = std::make_unique<const server_task>(std::move(task));

@@ -455,7 +441,7 @@ index d0e18e6..0765065 100644

         // reset server kill-switch counter
         n_empty_consecutive = 0;
-@@ -2163,6 +2232,18 @@ private:
+@@ -2050,6 +2119,18 @@ private:
         queue_results.send(std::move(res));
     }

@@ -474,7 +460,7 @@ index d0e18e6..0765065 100644
     void send_final_response(server_slot & slot) {
         auto res = std::make_unique<server_task_result_cmpl_final>();

-@@ -2662,6 +2743,7 @@ private:
+@@ -2556,6 +2637,7 @@ private:
             case SERVER_TASK_TYPE_EMBEDDING:
             case SERVER_TASK_TYPE_RERANK:
             case SERVER_TASK_TYPE_SCORE:
@@ -482,10 +468,7 @@ index d0e18e6..0765065 100644
                 {
                     // special case: if input is provided via CLI, tokenize it first
                     // otherwise, no need to tokenize as it's already done inside the HTTP thread
-@@ -3097,6 +3179,14 @@ private:
-             abort_all_slots("pre_decode() failed: " + std::string(e.what()));
-         }
-
+@@ -3007,1 +3089,9 @@ private:
 +        // note: TTS slots bypass the shared batch entirely
 +        try {
 +            process_tts_slots();
@@ -495,9 +478,7 @@ index d0e18e6..0765065 100644
 +        }
 +
         GGML_ASSERT(batch.slot_batched || batch.size() == 0);
-
-         if (batch.slot_batched) {
-@@ -3167,10 +3257,77 @@ private:
+@@ -3074,10 +3164,77 @@ private:
         }
     }

@@ -575,7 +556,7 @@ index d0e18e6..0765065 100644
             if (slot.state == SLOT_STATE_GENERATING && slot.prompt.n_tokens() + 1 >= slot.n_ctx) {
                 if (!params_base.ctx_shift) {
                     // this check is redundant (for good)
-@@ -3243,7 +3400,7 @@ private:
+@@ -3150,7 +3307,7 @@ private:

         // determine which slots are generating and drafting
         iterate(slots, [&](server_slot & slot) {
@@ -584,7 +565,7 @@ index d0e18e6..0765065 100644
                 return;
             }

-@@ -3375,7 +3532,7 @@ private:
+@@ -3284,7 +3441,7 @@ private:
                     return; // batch is full, skip remaining slots
                 }

@@ -593,7 +574,7 @@ index d0e18e6..0765065 100644
                     return;
                 }

-@@ -4384,6 +4541,8 @@ server_context_meta server_context::get_meta() const {
+@@ -4433,6 +4590,8 @@ server_context_meta server_context::get_meta() const {
         /* has_inp_image          */ impl->chat_params.allow_image,
         /* has_inp_audio          */ impl->chat_params.allow_audio,
         /* has_inp_video          */ impl->chat_params.allow_video,
@@ -602,7 +583,7 @@ index d0e18e6..0765065 100644
         /* json_ui_settings       */ impl->json_ui_settings,
         /* slot_n_ctx             */ impl->get_slot_n_ctx(),
         /* pooling_type           */ llama_pooling_type(impl->ctx_tgt),
-@@ -4463,6 +4622,11 @@ std::unique_ptr<server_res_generator> server_routes::handle_completions_impl(
+@@ -4512,6 +4671,11 @@ std::unique_ptr<server_res_generator> server_routes::handle_completions_impl(

     res->set_req(&req); // will also set spipe if needed

@@ -614,7 +595,7 @@ index d0e18e6..0765065 100644
     int32_t sse_ping_interval = params.sse_ping_interval;

     try {
-@@ -5440,6 +5604,150 @@ void server_routes::init_routes() {
+@@ -5399,6 +5563,150 @@ void server_routes::init_routes() {
         return res;
     };

@@ -678,7 +659,7 @@ index d0e18e6..0765065 100644
 +        }
 +
 +        if (speaker_ref_len > 0) {
-+            auto wrapper = mtmd_helper_bitmap_init_from_buf(ctx_server.mctx, speaker_ref_data, speaker_ref_len, false);
+            auto wrapper = mtmd_helper_bitmap_init_from_buf(ctx_server.mctx, speaker_ref_data, speaker_ref_len, false, ctx_server.init_opt);
 +            if (!wrapper.bitmap) {
 +                res->error(format_error_response("failed to decode \"speaker_ref\"", ERROR_TYPE_INVALID_REQUEST));
 +                return res;
@@ -766,7 +747,7 @@ index d0e18e6..0765065 100644
         auto res = create_response();

 diff --git a/tools/server/server-context.h b/tools/server/server-context.h
-index f9ab113..6105126 100644
+index f9ab1132b..610512678 100644
 --- a/tools/server/server-context.h
 +++ b/tools/server/server-context.h
@@ -22,6 +22,8 @@ struct server_context_meta {
@@ -787,10 +768,10 @@ index f9ab113..6105126 100644
     server_http_context::handler_t post_lora_adapters;

 diff --git a/tools/server/server-task.cpp b/tools/server/server-task.cpp
-index 1ee6775..939630b 100644
+index 1ee677553..939630b8b 100644
 --- a/tools/server/server-task.cpp
 +++ b/tools/server/server-task.cpp
-@@ -1523,6 +1523,17 @@ json server_task_result_rerank::to_json() {
+@@ -1497,6 +1497,17 @@ json server_task_result_rerank::to_json() {
     };
 }

@@ -809,7 +790,7 @@ index 1ee6775..939630b 100644
 // server_task_result_error
 //
 diff --git a/tools/server/server-task.h b/tools/server/server-task.h
-index 5bedf19..e6ca67a 100644
+index 5bedf1987..e6ca67a65 100644
 --- a/tools/server/server-task.h
 +++ b/tools/server/server-task.h
@@ -10,6 +10,7 @@
@@ -846,7 +827,7 @@ index 5bedf19..e6ca67a 100644
                 return true;
             default:
                 return false;
-@@ -514,6 +520,16 @@ struct server_task_result_embd : server_task_result {
+@@ -494,5 +500,15 @@ struct server_task_result_embd : server_task_result {
     json to_json_oaicompat();
 };

@@ -862,6 +843,3 @@ index 5bedf19..e6ca67a 100644
 +
 struct server_task_result_rerank : server_task_result {
     float score = -1e6;
-
--
-2.39.5
@@ -15,12 +15,40 @@ if [ -d "patches" ]; then
    done
 fi

+## Apple RDMA link fixup.
+
+## ggml-rpc hands Apple's librdma to the linker with
+## target_link_options(ggml-rpc PRIVATE "LINKER:-weak_library,..."). Link options are not
+## a usage requirement of a static library, so in our BUILD_SHARED_LIBS=OFF build the flag
+## dies with libggml-rpc.a and every ibv_* symbol transport-apple.cpp reaches for comes out
+## undefined when grpc-server and ggml-rpc-server link. Re-declare the same weak link as
+## INTERFACE so it travels to whoever links the static library.
+##
+## Guarded on the marker so a second prepare.sh over the same checkout is a no-op, and on
+## GGML_RPC_RDMA_APPLE so forks that branched before the Apple RDMA transport (turboquant,
+## bonsai) are left alone.
+RPC_CMAKE=llama.cpp/ggml/src/ggml-rpc/CMakeLists.txt
+if [ -f "$RPC_CMAKE" ] && grep -q "GGML_RPC_RDMA_APPLE" "$RPC_CMAKE" && ! grep -q "LOCALAI_RDMA_IFACE" "$RPC_CMAKE"; then
+    echo "==> ggml-rpc carries the Apple RDMA transport, re-declaring its weak librdma link as INTERFACE"
+    cat >> "$RPC_CMAKE" <<'EOF'
+
+# LOCALAI_RDMA_IFACE: added by backend/cpp/llama-cpp/prepare.sh
+if (GGML_RPC_RDMA AND APPLE AND NOT BUILD_SHARED_LIBS)
+    target_link_options(ggml-rpc INTERFACE "LINKER:-weak_library,${RDMA_LIB}")
+endif()
+EOF
+fi
+
 for file in $(ls llama.cpp/tools/server/); do
    cp -rfv llama.cpp/tools/server/$file llama.cpp/tools/grpc-server/
 done

 cp -r CMakeLists.txt llama.cpp/tools/grpc-server/
 cp -r grpc-server.cpp llama.cpp/tools/grpc-server/
+# Model-load diagnostics (included by grpc-server.cpp) and their standalone
+# regression test.
+cp -r model_load_error.h llama.cpp/tools/grpc-server/
+cp -r model_load_error_test.cpp llama.cpp/tools/grpc-server/
 # Shared message-reconstruction helpers (included by grpc-server.cpp) and their
 # unit test (compiled only when -DLLAMA_GRPC_BUILD_TESTS=ON).
 cp -r message_content.h llama.cpp/tools/grpc-server/
@@ -32,10 +60,17 @@ cp -r passthrough_options_test.cpp llama.cpp/tools/grpc-server/
 # regression test.
 cp -r tts_request_options.h llama.cpp/tools/grpc-server/
 cp -r tts_request_options_test.cpp llama.cpp/tools/grpc-server/
+# Thread-count default normalization and its standalone regression test.
+cp -r thread_params.h llama.cpp/tools/grpc-server/
+cp -r thread_params_test.cpp llama.cpp/tools/grpc-server/
 # Parent-death watcher (included by grpc-server.cpp) and its standalone unit
 # test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest).
 cp -r parent_watch.h llama.cpp/tools/grpc-server/
 cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
+# Dead-stream tracker (included by grpc-server.cpp) and its standalone unit
+# test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest).
+cp -r stream_peer.h llama.cpp/tools/grpc-server/
+cp -r stream_peer_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/

@@ -53,10 +88,28 @@ else
    echo "==> llama.cpp predates the load-mode enum, using the legacy mmap/mlock/direct-io booleans"
    LEGACY_LOAD_MODE=1
 fi
+if grep -q "server_metrics metrics;" llama.cpp/tools/server/server-task.h; then
+    HAS_SERVER_METRICS=1
+else
+    HAS_SERVER_METRICS=0
+fi
+if grep -q "mtmd_helper_init_opt" llama.cpp/tools/mtmd/mtmd-helper.h; then
+    HAS_MTMD_INIT_OPT=1
+else
+    HAS_MTMD_INIT_OPT=0
+fi
+if grep -q "llm_add_n_cpu_ffn_overrides" llama.cpp/common/common.h; then
+    HAS_N_CPU_FFN_HELPER=1
+else
+    HAS_N_CPU_FFN_HELPER=0
+fi
 cat > llama.cpp/tools/grpc-server/llama_compat.h <<EOF
 // Generated by backend/cpp/llama-cpp/prepare.sh. Do not edit.
 #pragma once
 #define LOCALAI_LEGACY_LOAD_MODE ${LEGACY_LOAD_MODE}
+#define LOCALAI_HAS_SERVER_METRICS ${HAS_SERVER_METRICS}
+#define LOCALAI_HAS_MTMD_INIT_OPT ${HAS_MTMD_INIT_OPT}
+#define LOCALAI_HAS_N_CPU_FFN_HELPER ${HAS_N_CPU_FFN_HELPER}
 EOF

 set +e
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+namespace llama_grpc {
+
+// Tracks whether a server-streaming RPC still has somewhere to send tokens.
+//
+// grpc::ServerWriter::Write() returns false once the peer is gone, and a
+// stream never recovers afterwards. Ignoring that result is not harmless: the
+// handler goes on draining decoded tokens into a dead stream, so the llama.cpp
+// slot stays busy for the rest of the request's token budget. A model config
+// with no max_tokens and a large context turns that into tens of minutes per
+// abandoned request, and the slots are exactly what every other request queues
+// behind.
+//
+// Returning as soon as the peer is gone is what frees the slot: the handler's
+// server_response_reader then goes out of scope and its destructor posts
+// SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
+class StreamPeer {
+public:
+    // Records the outcome of a Write(). Once a write has failed the peer stays
+    // gone -- a later write cannot succeed on a broken stream.
+    void observe_write(bool ok) noexcept {
+        if (!ok) {
+            gone_ = true;
+        }
+    }
+
+    // Folds in the RPC's own cancellation flag, so callers have a single
+    // predicate to test rather than two that can disagree.
+    void observe_cancelled(bool cancelled) noexcept {
+        if (cancelled) {
+            gone_ = true;
+        }
+    }
+
+    bool gone() const noexcept { return gone_; }
+    bool alive() const noexcept { return !gone_; }
+
+private:
+    bool gone_ = false;
+};
+
+} // namespace llama_grpc
@@ -0,0 +1,67 @@
+#include "stream_peer.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+void check(bool condition, const char *what) {
+    if (!condition) {
+        std::fprintf(stderr, "FAIL: %s\n", what);
+        ++failures;
+    }
+}
+
+} // namespace
+
+int main() {
+    {
+        llama_grpc::StreamPeer peer;
+        check(peer.alive(), "a fresh peer is alive");
+        check(!peer.gone(), "a fresh peer is not gone");
+    }
+
+    {
+        llama_grpc::StreamPeer peer;
+        peer.observe_write(true);
+        peer.observe_write(true);
+        check(peer.alive(), "successful writes keep the peer alive");
+    }
+
+    {
+        llama_grpc::StreamPeer peer;
+        peer.observe_write(false);
+        check(peer.gone(), "a failed write marks the peer gone");
+    }
+
+    {
+        // The whole point of the guard: a stream never comes back, so a later
+        // success must not resurrect a peer an earlier failure retired.
+        llama_grpc::StreamPeer peer;
+        peer.observe_write(false);
+        peer.observe_write(true);
+        check(peer.gone(), "a failed write is sticky across later writes");
+    }
+
+    {
+        llama_grpc::StreamPeer peer;
+        peer.observe_cancelled(false);
+        check(peer.alive(), "an uncancelled RPC keeps the peer alive");
+        peer.observe_cancelled(true);
+        check(peer.gone(), "cancellation marks the peer gone");
+    }
+
+    {
+        llama_grpc::StreamPeer peer;
+        peer.observe_cancelled(true);
+        peer.observe_cancelled(false);
+        check(peer.gone(), "cancellation is sticky across later checks");
+    }
+
+    if (failures != 0) {
+        std::fprintf(stderr, "%d check(s) failed\n", failures);
+        return 1;
+    }
+    return 0;
+}
@@ -0,0 +1,11 @@
+#pragma once
+
+#include <cstdint>
+
+namespace llama_grpc {
+
+inline int32_t resolve_batch_threads(int32_t batch_threads, int32_t inference_threads) {
+    return batch_threads < 0 ? inference_threads : batch_threads;
+}
+
+} // namespace llama_grpc
@@ -0,0 +1,15 @@
+#include "thread_params.h"
+
+#include <cstdio>
+
+int main() {
+    if (llama_grpc::resolve_batch_threads(-1, 4) != 4) {
+        std::fprintf(stderr, "default batch threads did not inherit inference threads\n");
+        return 1;
+    }
+    if (llama_grpc::resolve_batch_threads(2, 4) != 2) {
+        std::fprintf(stderr, "explicit batch threads were overwritten\n");
+        return 1;
+    }
+    return 0;
+}
@@ -8,6 +8,8 @@
 #      so the grpc-server option parser skips the two references to
 #      common_params::checkpoint_min_step (the default and the option handler).
 #      That field does not exist in the fork yet; drop this once it does.
+#   3. Use nlohmann's parse_error type in JSON catch clauses because the fork
+#      predates upstream's common_json_error wrapper.
 #
 # The fork used to lag upstream on the whole common_params_speculative refactor
 # (ggml-org/llama.cpp#22397/#22838/#22964), the model_tgt rename (#22838) and
@@ -100,4 +102,16 @@ else
    echo "==> LOCALAI_TURBOQUANT_NO_CHECKPOINT_MIN_STEP define OK"
 fi

+# 3. The shared source follows current upstream and catches common_json_error.
+#    TurboQuant still exposes nlohmann::json directly, so its equivalent parse
+#    failures use json::parse_error instead.
+if grep -q 'common_json_error' "$SRC"; then
+    echo "==> patching $SRC to use the TurboQuant JSON exception type"
+    awk '{ gsub(/common_json_error/, "json::parse_error"); print }' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+    echo "==> TurboQuant JSON exception patch OK"
+else
+    echo "==> $SRC already uses a TurboQuant-compatible JSON exception type, skipping"
+fi
+
 echo "==> all patches applied"
@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)

 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=ce521ee178867ceaa5fdc11803616578c8936c19
+CRISPASR_VERSION?=301acd87b036764973b8bfba71e0a21818036d33
 SO_TARGET?=libgocrispasr.so

 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
@@ -615,10 +615,10 @@ func (w *CrispASR) TTSStream(req *pb.TTSRequest, results chan []byte) error {
 		return fmt.Errorf("crispasr: tempfile: %w", err)
 	}
 	dst := tmp.Name()
+	defer func() { _ = os.Remove(dst) }()
 	if err := tmp.Close(); err != nil {
 		return fmt.Errorf("crispasr: close tempfile: %w", err)
 	}
-	defer func() { _ = os.Remove(dst) }()

 	if err := writeWAV(dst, pcm, w.sampleRate); err != nil {
 		return err
@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=54abd5c0abfd1f394e01cb3c38f2e3af4daedf85
+DEPTHANYTHING_VERSION?=14f7461d1f704761a038ac9f50dbde8fdb7275e2

 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF
@@ -38,8 +38,9 @@ type Store struct {
 	// keysAreNormalized stays true until any non-unit-magnitude key
 	// is added; once false, the magnitude-aware fallback path is
 	// used by Find. Re-evaluated only at Set time, never again on
-	// its own — a deletion of the offending key does NOT flip it
-	// back to true (the bookkeeping cost would dominate the gain).
+	// its own — a partial deletion of the offending key does NOT flip
+	// it back to true (the bookkeeping cost would dominate the gain).
+	// An empty store returns to its initial state.
 	keysAreNormalized bool

 	// keyLen is the dimension of every stored key. -1 means "no
@@ -142,6 +143,10 @@ func (s *Store) StoresDelete(opts *pb.StoresDeleteOptions) error {
 	mergedV = append(mergedV, tailV...)
 	s.keys = mergedK
 	s.values = mergedV
+	if len(s.keys) == 0 {
+		s.keyLen = -1
+		s.keysAreNormalized = true
+	}
 	assert(slices.IsSortedFunc(s.keys, slices.Compare[[]float32]), "Delete: s.keys not sorted post-merge")
 	assert(len(s.keys) == len(s.values), "Delete: keys/values length skew")
 	return nil
@@ -105,6 +105,46 @@ var _ = Describe("StoresDelete", func() {
 		})).To(Succeed(), "delete of missing key should succeed")
 		Expect(s.keys).To(HaveLen(1))
 	})
+
+	It("reopens the dimension after deleting every key", func() {
+		s := NewStore()
+		oldKey := []float32{2, 0, 0}
+		mustSet(s, [][]float32{oldKey}, [][]byte{[]byte("3d")})
+		Expect(s.keysAreNormalized).To(BeFalse())
+
+		Expect(s.StoresDelete(&pb.StoresDeleteOptions{
+			Keys: wrapKeys([][]float32{oldKey}),
+		})).To(Succeed())
+		Expect(s.keys).To(BeEmpty())
+		Expect(s.keyLen).To(Equal(-1))
+		Expect(s.keysAreNormalized).To(BeTrue())
+
+		newKey := normalizeVec([]float32{1, 1})
+		mustSet(s, [][]float32{newKey}, [][]byte{[]byte("2d")})
+		res, err := s.StoresFind(&pb.StoresFindOptions{
+			Key:  &pb.StoresKey{Floats: newKey},
+			TopK: 1,
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(res.Values).To(HaveLen(1))
+		Expect(string(res.Values[0].Bytes)).To(Equal("2d"))
+	})
+
+	It("retains the dimension after a partial delete", func() {
+		s := NewStore()
+		mustSet(s,
+			[][]float32{{1, 0, 0}, {0, 1, 0}},
+			[][]byte{[]byte("x"), []byte("y")},
+		)
+		Expect(s.StoresDelete(&pb.StoresDeleteOptions{
+			Keys: wrapKeys([][]float32{{1, 0, 0}}),
+		})).To(Succeed())
+		Expect(s.keyLen).To(Equal(3))
+		Expect(s.StoresSet(&pb.StoresSetOptions{
+			Keys:   wrapKeys([][]float32{{1, 0}}),
+			Values: wrapValues([][]byte{[]byte("2d")}),
+		})).NotTo(Succeed())
+	})
 })

 var _ = Describe("StoresFind", func() {
@@ -11,6 +11,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"unsafe"

 	"github.com/mudler/LocalAI/pkg/grpc/base"
@@ -109,30 +110,25 @@ func (r *LocateAnythingCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, e
 		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: a text prompt is required (open-vocabulary detection)")
 	}

-	// Decode base64 image and write to temp file.
 	imgData, err := base64.StdEncoding.DecodeString(opts.Src)
 	if err != nil {
 		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to decode base64 image: %w", err)
 	}
-
-	tmpFile, err := os.CreateTemp("", "locate-anything-*.img")
-	if err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to create temp file: %w", err)
-	}
-	defer func() { _ = os.Remove(tmpFile.Name()) }()
-
-	if _, err := tmpFile.Write(imgData); err != nil {
-		_ = tmpFile.Close()
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to write temp file: %w", err)
-	}
-	if err := tmpFile.Close(); err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to close temp file: %w", err)
+	if len(imgData) == 0 {
+		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: decoded image is empty")
 	}

 	// mode 0 = hybrid (Parallel Box Decoding). The JSON return value is unused:
 	// structured detections are read via the accessor functions. Still must
 	// free the returned string.
-	jsonPtr := CapiLocatePath(r.handle, tmpFile.Name(), prompt, 0)
+	jsonPtr := CapiLocateBuffer(
+		r.handle,
+		uintptr(unsafe.Pointer(unsafe.SliceData(imgData))),
+		uintptr(len(imgData)),
+		prompt,
+		0,
+	)
+	runtime.KeepAlive(imgData)
 	if jsonPtr != 0 {
 		CapiFreeString(jsonPtr)
 	}
@@ -0,0 +1,54 @@
+package main
+
+import (
+	"encoding/base64"
+	"path/filepath"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("LocateAnythingCpp detection input", func() {
+	It("detects from memory when the temporary directory is unavailable", func() {
+		originalLocateBuffer := CapiLocateBuffer
+		originalLocatePath := CapiLocatePath
+		originalGetNDetections := CapiGetNDetections
+		defer func() {
+			CapiLocateBuffer = originalLocateBuffer
+			CapiLocatePath = originalLocatePath
+			CapiGetNDetections = originalGetNDetections
+		}()
+
+		image := []byte("encoded-image")
+		var receivedData uintptr
+		var receivedLength uintptr
+		CapiLocateBuffer = func(_ uintptr, data uintptr, length uintptr, _ string, _ int32) uintptr {
+			receivedData = data
+			receivedLength = length
+			return 0
+		}
+		CapiLocatePath = func(_ uintptr, _ string, _ string, _ int32) uintptr {
+			Fail("path-based detection must not be called")
+			return 0
+		}
+		CapiGetNDetections = func(uintptr) int32 { return 0 }
+		GinkgoT().Setenv("TMPDIR", filepath.Join(GinkgoT().TempDir(), "missing"))
+
+		result, err := (&LocateAnythingCpp{handle: 1}).Detect(&pb.DetectOptions{
+			Src:    base64.StdEncoding.EncodeToString(image),
+			Prompt: "the object",
+		})
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(result.Detections).To(BeEmpty())
+		Expect(receivedData).NotTo(BeZero())
+		Expect(receivedLength).To(Equal(uintptr(len(image))))
+	})
+
+	It("rejects an empty decoded image", func() {
+		_, err := (&LocateAnythingCpp{handle: 1}).Detect(&pb.DetectOptions{Prompt: "the object"})
+
+		Expect(err).To(MatchError("locate-anything-cpp: decoded image is empty"))
+	})
+})
@@ -12,7 +12,7 @@
 # runs 'make -C backend/go/$(BACKEND) build' and then copies package/), so it
 # has to produce the binary and the package, not just the shared libraries.

-NEMO_SPEECH_VERSION?=2e12e2def8a98ed06666f7ee3ca94e7193e04be4
+NEMO_SPEECH_VERSION?=a5b6953c4a579a2bbd1c0913ad8a85c2a4d99953
 NEMO_SPEECH_REPO?=https://github.com/NVIDIA/NeMo-Speech.cpp

 GOCMD?=go
@@ -88,6 +88,18 @@ ITN_LIB_DIR=$(ITN_PREFIX)/lib
 ITN_MARKER=$(ITN_LIB_DIR)/libsparrowhawk.so
 ITN_FST_HEADER=$(ITN_PREFIX)/include/fst/fst.h

+# SentencePiece became a core ASR dependency in 5be7bfb: RNNT context biasing
+# uses it even when Flashlight and text normalization are disabled. Build the
+# pinned static archive provided by upstream so every platform gets the same
+# dependency instead of relying on an undeclared system package.
+SENTENCEPIECE_PREFIX=sources/NeMo-Speech.cpp/.deps/sentencepiece
+SENTENCEPIECE_MARKER=$(SENTENCEPIECE_PREFIX)/lib/libsentencepiece.a
+
+# Linux's ASR CMake block looks in NEMO_SPEECH_DEPENDENCY_PREFIX directly, but
+# the Apple branch uses generic find_library()/find_path(). Put the same private
+# prefix on CMake's search path so Darwin consumes the archive built above too.
+CMAKE_ARGS+=-DCMAKE_PREFIX_PATH=$(abspath $(SENTENCEPIECE_PREFIX))
+
 ITN_CC?=gcc-12
 ITN_CXX?=g++-12

@@ -152,7 +164,7 @@ else
 endif
 CMAKE_ARGS+=-DNEMO_SPEECH_GGML_PATCHED=$(GGML_PATCHED)

-.PHONY: nemo-speech-cpp-grpc package build clean purge test all stage-libs patch-ggml engine itn patch-itn-headers
+.PHONY: nemo-speech-cpp-grpc package build clean purge test all stage-libs patch-ggml engine itn sentencepiece patch-itn-headers

 all: nemo-speech-cpp-grpc package

@@ -266,11 +278,28 @@ patch-itn-headers:

 itn: $(ITN_MARKER)

+$(SENTENCEPIECE_MARKER): | sources/NeMo-Speech.cpp
+	# Upstream's license copies use GNU install's -D flag, which BSD install
+	# does not support. Homebrew CMake 4 also rejects SentencePiece's old policy
+	# floor. Patch both incompatibilities before running the helper on Darwin.
+	@if [ "$(shell uname -s)" = Darwin ]; then \
+		cd sources/NeMo-Speech.cpp && \
+		mkdir -p .deps/sentencepiece/share/licenses/nemo-speech/third_party/sentencepiece && \
+		perl -pi \
+			-e 's/install -Dm0644/install -m 0644/g;' \
+			-e 's/-DCMAKE_BUILD_TYPE=Release /-DCMAKE_BUILD_TYPE=Release -DCMAKE_POLICY_VERSION_MINIMUM=3.5 /;' \
+			scripts/build_sentencepiece_static.sh; \
+	fi
+	cd sources/NeMo-Speech.cpp && JOBS=$(JOBS) scripts/build_sentencepiece_static.sh
+
+sentencepiece: $(SENTENCEPIECE_MARKER)
+
 # Only a WITH_NORM=ON build needs the ITN stack, and it must exist before cmake
 # configures, since the WITH_NORM cmake block find_library()s into the prefix
 # with REQUIRED.
+NEMO_RUNTIME_PREREQS=$(SENTENCEPIECE_MARKER)
 ifeq ($(WITH_NORM),ON)
-NEMO_RUNTIME_PREREQS=$(ITN_MARKER)
+NEMO_RUNTIME_PREREQS+=$(ITN_MARKER)
 endif

 # Upstream sets CMAKE_LIBRARY_OUTPUT_DIRECTORY to ${CMAKE_BINARY_DIR}/bin, so the
@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)

 # omnivoice.cpp version
 OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=4f33af825d66e6ef1cb185e87b4589cacf747291
+OMNIVOICE_VERSION?=040c8b344d8c670ce1475194751d119b5ef82c78
 SO_TARGET?=libgomnivoicecpp.so

 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
@@ -1,6 +1,6 @@
 # parakeet-cpp backend Makefile.
 #
-# Upstream pin lives below as PARAKEET_VERSION?=1bfbebfaaf493866f49597cd3b7901959d395c60
+# Upstream pin lives below as PARAKEET_VERSION?=e75de9b6b9b688fd293aa22f7e27aa724ea286f8
 # (.github/bump_deps.sh) can find and update it - matches the
 # whisper.cpp / ds4 / vibevoice-cpp convention.
 #
@@ -15,7 +15,7 @@
 # That's what the L0 smoke test uses. The default target below does the
 # proper clone-at-pin + cmake build so CI doesn't need a side-checkout.

-PARAKEET_VERSION?=1bfbebfaaf493866f49597cd3b7901959d395c60
+PARAKEET_VERSION?=e75de9b6b9b688fd293aa22f7e27aa724ea286f8
 PARAKEET_REPO?=https://github.com/mudler/parakeet.cpp

 GOCMD?=go
@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"strconv"
 	"unsafe"

@@ -102,24 +103,12 @@ func (r *RFDetrCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, error) {
 		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: model not loaded")
 	}

-	// Decode base64 image and write to temp file.
 	imgData, err := base64.StdEncoding.DecodeString(opts.Src)
 	if err != nil {
 		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to decode base64 image: %w", err)
 	}
-
-	tmpFile, err := os.CreateTemp("", "rfdetr-*.img")
-	if err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to create temp file: %w", err)
-	}
-	defer func() { _ = os.Remove(tmpFile.Name()) }()
-
-	if _, err := tmpFile.Write(imgData); err != nil {
-		_ = tmpFile.Close()
-		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to write temp file: %w", err)
-	}
-	if err := tmpFile.Close(); err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to close temp file: %w", err)
+	if len(imgData) == 0 {
+		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: decoded image is empty")
 	}

 	threshold := opts.Threshold
@@ -127,10 +116,18 @@ func (r *RFDetrCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, error) {
 		threshold = 0.5
 	}

-	// JSON output from detect_path is unused: we read structured detections via
+	// JSON output from the detection ABI is unused: we read structured detections via
 	// the accessor functions. Still must free the returned string.
 	var jsonPtr uintptr
-	rc := CapiDetectPath(r.handle, tmpFile.Name(), threshold, uint32(defaultTopK), &jsonPtr)
+	rc := CapiDetectBuffer(
+		r.handle,
+		uintptr(unsafe.Pointer(unsafe.SliceData(imgData))),
+		uintptr(len(imgData)),
+		threshold,
+		uint32(defaultTopK),
+		&jsonPtr,
+	)
+	runtime.KeepAlive(imgData)
 	if jsonPtr != 0 {
 		CapiFreeString(jsonPtr)
 	}
@@ -0,0 +1,56 @@
+package main
+
+import (
+	"encoding/base64"
+	"path/filepath"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("RFDetrCpp detection input", func() {
+	It("detects from memory when the temporary directory is unavailable", func() {
+		originalDetectBuffer := CapiDetectBuffer
+		originalDetectPath := CapiDetectPath
+		originalFreeString := CapiFreeString
+		originalGetNDetections := CapiGetNDetections
+		defer func() {
+			CapiDetectBuffer = originalDetectBuffer
+			CapiDetectPath = originalDetectPath
+			CapiFreeString = originalFreeString
+			CapiGetNDetections = originalGetNDetections
+		}()
+
+		image := []byte("encoded-image")
+		var receivedData uintptr
+		var receivedLength uintptr
+		CapiDetectBuffer = func(_ uintptr, data uintptr, length uintptr, _ float32, _ uint32, _ *uintptr) int32 {
+			receivedData = data
+			receivedLength = length
+			return 0
+		}
+		CapiDetectPath = func(_ uintptr, _ string, _ float32, _ uint32, _ *uintptr) int32 {
+			Fail("path-based detection must not be called")
+			return -1
+		}
+		CapiFreeString = func(uintptr) {}
+		CapiGetNDetections = func(uintptr) int32 { return 0 }
+		GinkgoT().Setenv("TMPDIR", filepath.Join(GinkgoT().TempDir(), "missing"))
+
+		result, err := (&RFDetrCpp{handle: 1}).Detect(&pb.DetectOptions{
+			Src: base64.StdEncoding.EncodeToString(image),
+		})
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(result.Detections).To(BeEmpty())
+		Expect(receivedData).NotTo(BeZero())
+		Expect(receivedLength).To(Equal(uintptr(len(image))))
+	})
+
+	It("rejects an empty decoded image", func() {
+		_, err := (&RFDetrCpp{handle: 1}).Detect(&pb.DetectOptions{})
+
+		Expect(err).To(MatchError("rfdetr-cpp: decoded image is empty"))
+	})
+})
@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)

 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=de298c225bed97c3f9026b73cd7b71e7879bd41b
+STABLEDIFFUSION_GGML_VERSION?=b68d58624d227682eb4b95ef8bcf569cd1311eb5

 CMAKE_ARGS+=-DGGML_MAX_NAME=128

@@ -38,17 +38,16 @@ else ifeq ($(BUILD_TYPE),hipblas)
 	ROCM_PATH ?= /opt/rocm
 	export CXX=$(ROCM_HOME)/llvm/bin/clang++
 	export CC=$(ROCM_HOME)/llvm/bin/clang
-	AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1200,gfx1201
-	CMAKE_ARGS+=-DSD_HIPBLAS=ON -DGGML_HIPBLAS=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
+	AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	# SD_HIPBLAS turns on ggml's HIP backend itself; GGML_HIPBLAS is the name ggml
+	# used before it was renamed to GGML_HIP, so passing it here only produced an
+	# unused-variable warning.
+	CMAKE_ARGS+=-DSD_HIPBLAS=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DSD_VULKAN=ON -DGGML_VULKAN=ON
-else ifeq ($(OS),Darwin)
-	ifneq ($(BUILD_TYPE),metal)
-		CMAKE_ARGS+=-DSD_METAL=OFF -DGGML_METAL=OFF
-	else
-		CMAKE_ARGS+=-DSD_METAL=ON -DGGML_METAL=ON
-		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
-	endif
+else ifeq ($(BUILD_TYPE),metal)
+	CMAKE_ARGS+=-DSD_METAL=ON -DGGML_METAL=ON
+	CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
 endif

 ifeq ($(BUILD_TYPE),sycl_f16)
@@ -72,7 +71,6 @@ sources/stablediffusion-ggml.cpp:
 	git checkout $(STABLEDIFFUSION_GGML_VERSION) && \
 	git submodule update --init --recursive --depth 1 --single-branch

-# Detect OS
 UNAME_S := $(shell uname -s)

 # Only build CPU variants on Linux
@@ -134,4 +132,4 @@ libgosd-custom: CMakeLists.txt cpp/gosd.cpp cpp/gosd.h
 	(mv build-$(SO_TARGET)/libgosd.so ./$(SO_TARGET) 2>/dev/null || \
 	 mv build-$(SO_TARGET)/libgosd.dylib ./$(SO_TARGET) 2>/dev/null)

-all: stablediffusion-ggml package
+all: stablediffusion-ggml package
@@ -401,7 +401,6 @@ int load_model(const char *model, char *model_path, char* options[], int threads
    const char *params_backend_arg = "";
    const char *rpc_servers_arg = "";
    const char *max_vram_arg = "";
-    bool stream_layers = false;

    int n_threads = threads;
    enum sd_type_t wtype = SD_TYPE_COUNT;
@@ -510,7 +509,10 @@ int load_model(const char *model, char *model_path, char* options[], int threads
        if (!strcmp(optname, "params_backend")) params_backend_arg = strdup(optval);
        if (!strcmp(optname, "rpc_servers")) rpc_servers_arg = strdup(optval);
        if (!strcmp(optname, "max_vram")) max_vram_arg = strdup(optval);
-        if (!strcmp(optname, "stream_layers")) stream_layers = (strcmp(optval, "true") == 0 || strcmp(optval, "1") == 0);
+        if (!strcmp(optname, "stream_layers")) {
+            // Retained as a no-op for existing configurations. Upstream now
+            // selects segmented weight streaming automatically.
+        }

        // vae_decode_only is still accepted for backwards compatibility with
        // existing gallery configs, but upstream dropped the option (the model
@@ -650,11 +652,9 @@ int load_model(const char *model, char *model_path, char* options[], int threads
            ctx_params.rpc_servers = env_rpc_servers;
        }
    }
-    // max_vram: GiB budget or per-backend spec for graph-cut segmented param
-    // offload ("0" = disabled, "-1" = auto). stream_layers only has effect when
-    // max_vram is set.
+    // max_vram is an optional GiB budget or per-backend spec for automatic
+    // graph-cut execution. A zero value uses the live free-VRAM budget.
    if (strlen(max_vram_arg) > 0) ctx_params.max_vram = max_vram_arg;
-    ctx_params.stream_layers = stream_layers;
    ctx_params.diffusion_flash_attn = diffusion_flash_attn;
    ctx_params.tae_preview_only = tae_preview_only;
    ctx_params.diffusion_conv_direct = diffusion_conv_direct;
@@ -1144,17 +1144,25 @@ static uint8_t* load_and_resize_image(const char* path, int target_width, int ta
 // Write sd.cpp's audio buffer to a temp WAV file (IEEE float, interleaved).
 // sd_audio_t.data is planar (all channel 0 samples, then channel 1, etc.) — we
 // interleave on the fly so ffmpeg's standard wav demuxer can read it directly.
-// Returns 0 on success and fills wav_path (must be at least 64 bytes).
+// Returns 0 on success and fills wav_path.
 static int write_planar_float_wav(const sd_audio_t* a, char* wav_path, size_t wav_path_sz) {
    if (!a || !a->data || a->sample_count == 0 || a->channels == 0 || a->sample_rate == 0) {
        return -1;
    }

-    snprintf(wav_path, wav_path_sz, "/tmp/gosd-audio-XXXXXX.wav");
+    const char* temp_dir = getenv("TMPDIR");
+    if (!temp_dir || temp_dir[0] == '\0') {
+        temp_dir = "/tmp";
+    }
+    int path_len = snprintf(wav_path, wav_path_sz, "%s/gosd-audio-XXXXXX.wav", temp_dir);
+    if (path_len < 0 || (size_t)path_len >= wav_path_sz) {
+        fprintf(stderr, "temporary directory path is too long\n");
+        return -1;
+    }
    int fd = mkstemps(wav_path, 4);
    if (fd < 0) { perror("mkstemps wav"); return -1; }
    FILE* f = fdopen(fd, "wb");
-    if (!f) { perror("fdopen wav"); close(fd); return -1; }
+    if (!f) { perror("fdopen wav"); close(fd); unlink(wav_path); return -1; }

    uint64_t frames = a->sample_count;
    uint32_t channels = a->channels;
@@ -1221,7 +1229,7 @@ static int ffmpeg_mux_raw_to_mp4(sd_image_t* frames, int num_frames, int fps,
    snprintf(fps_str, sizeof(fps_str), "%d", fps);

    // Optional audio: write a temp WAV file if the model produced audio.
-    char wav_path[64] = {0};
+    char wav_path[4096] = {0};
    bool have_audio = false;
    if (audio && audio->data && audio->sample_count > 0 && audio->channels > 0 && audio->sample_rate > 0) {
        if (write_planar_float_wav(audio, wav_path, sizeof(wav_path)) == 0) {
@@ -1438,4 +1446,3 @@ int unload() {
    free_sd_ctx(sd_c);
    return 0;
 }
-
@@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e

 # vllm.cpp version
 VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
-VLLM_CPP_VERSION?=9fd9e8f34408d5dd21d7f9385e96fc755708950b
+VLLM_CPP_VERSION?=6bf3abb580982f4fd2e4525ef37802ee0ce28981

 # MLX GEMM provider (darwin/metal only; see the metal branch below for why).
 # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun
@@ -47,26 +47,35 @@ CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
 UNAME_M := $(shell uname -m)

 ifeq ($(BUILD_TYPE),cublas)
-	# Blackwell-family targets only: other CUDA arches are build-supported
-	# upstream but have no runtime-proven fast path. amd64 gets the consumer
-	# (120a) + GB10 (121a) fat binary; arm64 CUDA (l4t-style images, DGX
-	# Spark) is GB10 only. Triton-AOT GDN cubins are vendored per-arch, no
-	# Python needed to consume them.
+	# Every CUDA architecture upstream builds that the platform can actually
+	# host, split by where the silicon exists: Jetson (87 Orin, 110 Thor) is
+	# arm64-only, desktop 120a is amd64-only, and 90a/100a appear on both
+	# because of the SBSA parts (GH200, GB200).
+	#
+	# This deliberately matches vllm.cpp's own release archive rather than
+	# narrowing to the boxes we benchmark on. A narrower list does not degrade
+	# on an unlisted card, it dies at the first request with "no kernel image
+	# is available for execution on the device", long after `backends install`
+	# reported success -- so an arch we merely lack numbers for still belongs
+	# in the binary.
+	#
+	# Triton-AOT stays ON for both. A fat build is supported on the BUILDER
+	# path: it embeds every vendored cubin tree (sm_80/86/89/90a/100a/121a) and
+	# selects by exact SM at runtime, so the arches with no tree (87, 103a,
+	# 110, 120a) take the portable CUDA kernels and can never load a
+	# neighbouring cubin. Only maintainer REGEN needs a single pinned arch.
+	# See vllm.cpp cmake/TritonAOT.cmake `_triton_aot_arch_names`.
+	#
+	# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks compute_121a
+	# (GB10) and its ptxas rejects the sm_120a NVFP4 MMA kernels ("Vector type
+	# too large"), so no cuda-12 variant is shipped.
+	ifeq ($(CUDA_MAJOR_VERSION),12)
+    $(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
+	endif
 	ifeq ($(UNAME_M),x86_64)
-		# NO -DVLLM_CPP_TRITON on fat builds: the vendored Triton-AOT cubin
-		# trees are per-arch and the engine refuses a multi-arch build unless
-		# pinned to one tree (unsound for the other arch). The non-AOT GDN
-		# path serves the fat binary; single-arch builds keep the cubins.
-		#
-		# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks
-		# compute_121a (GB10) and its ptxas rejects the sm_120a NVFP4 MMA
-		# kernels ("Vector type too large"), so no cuda-12 variant is shipped.
-		ifeq ($(CUDA_MAJOR_VERSION),12)
-      $(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
-		endif
-		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=120a;121a"
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=80;86;89;90a;100a;103a;120a;121a" -DVLLM_CPP_TRITON=ON
 	else
-		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=87;90a;100a;110;121a" -DVLLM_CPP_TRITON=ON
 	endif
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF
@@ -9,7 +9,7 @@ It serves two things: text generation, and MiniMax-H3 joint video+audio
 generation.

 The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
-ABI v16) through purego:
+ABI v20) through purego:

 - `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
  directory (`config.json` + safetensors). `context_size` maps to
@@ -1,6 +1,6 @@
 package main

-// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v16).
+// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v23).
 //
 // The structs below are hand-mirrored PODs of the C declarations, with
 // explicit padding so the Go layout matches the C layout on linux/darwin
@@ -21,7 +21,7 @@ import (
 // the header of the VLLM_CPP_VERSION pinned in the Makefile: the build checks
 // the two against each other, because a mismatch is only caught at runtime by
 // registerLib, where it takes the backend down on every load (issue #11379).
-const abiVersion = 17
+const abiVersion = 23

 // The ABI's tri-state toggles (enable_prefix_caching ABI v7,
 // enable_jump_forward ABI v10) share one encoding: 0 is NOT "off", it is
@@ -69,6 +69,7 @@ type cModelParams struct {
 	MaxNumBatchedTokens int32   // <= 0 = per-arch default (ABI v9)
 	SchedulingPolicy    uintptr // const char*; NULL = "fcfs" (ABI v9)
 	KVTransferConfig    uintptr // const char* JSON; NULL = no connector (ABI v9)
+	OffloadConfig       uintptr // const char* JSON; NULL = no weight offload
 	EnableJumpForward   int32   // tri-state 0/1/2 (ABI v10)
 	// v14/v16 tail. LocalAI sets none of these (0 is "auto" for the device and
 	// "unset" for both sizing knobs, i.e. the pre-v14 engine byte for byte), but
@@ -79,6 +80,10 @@ type cModelParams struct {
 	Device             int32   // 0 auto, 1 cpu, 2 cuda (ABI v14)
 	GPUMemoryUtil      float64 // 0 => 0.92 (ABI v16)
 	KVCacheMemoryBytes int64   // 0 => unset (ABI v16)
+	LanguageModelOnly  int32   // 0 = multimodal inputs enabled (ABI v19)
+	_                  [4]byte
+	LimitMMPerPrompt   uintptr // const char* JSON; NULL = default limits (ABI v19)
+	MMProjPath         uintptr // const char*; NULL = no GGUF projector (ABI v22)
 }

 // cSamplingParams mirrors vllm_sampling_params (structured fields included).
@@ -147,6 +152,11 @@ type cVideoModelParams struct {
 	Device             int32   // 0 cpu, 1 cuda
 	DequantBf16        int32   // 0 keep-quant, 1 dequant/stream bf16
 	Fp4Resident        int32   // NVFP4+cuda: keep FP4 packed, Marlin W4A16
+	_                  [4]byte
+	Family             uintptr // const char*; NULL = detect (ABI v18)
+	ExtraKeys          uintptr // const char* const* (ABI v18)
+	ExtraValues        uintptr // const char* const* (ABI v18)
+	NExtras            int32   // 0 = none (ABI v18)
 	_                  [4]byte // trailing pad to the struct's 8-byte alignment
 }

@@ -154,22 +164,26 @@ type cVideoModelParams struct {
 // `steps` pair up into 8-byte slots; the uint64 seed forces the alignment after
 // them, and the float noise_aug leaves a pad before output_dir.
 type cVideoParams struct {
-	Prompt     uintptr // const char*
-	Width      int32
-	Height     int32
-	NumFrames  int32 // <= 1 => per-task default (124 for t2va/fl2va)
-	Steps      int32 // <= 0 => the H3 default (50)
-	Seed       uint64
-	HasSeed    int32
-	_          [4]byte
-	FirstFrame uintptr // const char*; fl2va keyframe, binary PPM (P6)
-	LastFrame  uintptr // const char*
-	RefImage   uintptr // const char*; ref2va only
-	RefVideo   uintptr // const char*; ref2va only, a frame_%06d.ppm DIRECTORY
-	RefAudio   uintptr // const char*; ref2va only, 16-bit PCM WAV
-	NoiseAug   float32 // <= 0 => 1.0
-	_          [4]byte
-	OutputDir  uintptr // const char*; REQUIRED
+	Prompt      uintptr // const char*
+	Width       int32
+	Height      int32
+	NumFrames   int32 // <= 1 => per-task default (124 for t2va/fl2va)
+	Steps       int32 // <= 0 => the H3 default (50)
+	Seed        uint64
+	HasSeed     int32
+	_           [4]byte
+	FirstFrame  uintptr // const char*; fl2va keyframe, binary PPM (P6)
+	LastFrame   uintptr // const char*
+	RefImage    uintptr // const char*; ref2va only
+	RefVideo    uintptr // const char*; ref2va only, a frame_%06d.ppm DIRECTORY
+	RefAudio    uintptr // const char*; ref2va only, 16-bit PCM WAV
+	NoiseAug    float32 // <= 0 => 1.0
+	_           [4]byte
+	OutputDir   uintptr // const char*; REQUIRED
+	ExtraKeys   uintptr // const char* const* (ABI v18)
+	ExtraValues uintptr // const char* const* (ABI v18)
+	NExtras     int32   // 0 = none (ABI v18)
+	_           [4]byte
 }

 // cVideoResult mirrors vllm_video_result. Every member is library-allocated and
@@ -128,9 +128,40 @@ func parseOptions(opts *pb.ModelOptions) loadOptions {
 	lo := loadOptions{}
 	applyOptionsList(&lo, opts.GetOptions())
 	applyEngineArgs(&lo, opts.GetEngineArgs())
+	applyDraftModelOption(&lo, opts.GetOptions())
 	return lo
 }

+// applyDraftModelOption binds a managed companion snapshot after engine_args
+// has supplied the speculative document. Companion paths do not exist until
+// LocalAI materializes the artifact, so they must replace the gallery's static
+// repository reference without disturbing the method or token budget.
+func applyDraftModelOption(lo *loadOptions, options []string) {
+	if strings.TrimSpace(lo.speculativeConfig) == "" {
+		return
+	}
+	var draftModel string
+	for _, option := range options {
+		key, value, found := strings.Cut(option, ":")
+		if found && strings.TrimSpace(key) == "draft_model" {
+			draftModel = strings.TrimSpace(value)
+		}
+	}
+	if draftModel == "" {
+		return
+	}
+
+	var spec map[string]any
+	if err := json.Unmarshal([]byte(lo.speculativeConfig), &spec); err != nil {
+		return
+	}
+	spec["model"] = draftModel
+	encoded, err := json.Marshal(spec)
+	if err == nil {
+		lo.speculativeConfig = string(encoded)
+	}
+}
+
 // applyOptionsList reads the legacy free-form "key:value" list. strings.Cut
 // splits on the FIRST colon only, so a JSON object value survives intact.
 func applyOptionsList(lo *loadOptions, options []string) {
@@ -0,0 +1,38 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+var _ = Describe("managed DFlash companion options", func() {
+	It("replaces only the draft model in an existing speculative configuration", func() {
+		managedPath := ".artifacts/huggingface/0123456789abcdef/snapshot"
+		lo := parseOptions(&pb.ModelOptions{
+			Options: []string{"draft_model:" + managedPath},
+			EngineArgs: `{
+				"speculative_config": {
+					"method": "dflash",
+					"model": "Mia-AiLab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw",
+					"num_speculative_tokens": 7
+				}
+			}`,
+		})
+
+		Expect(lo.speculativeConfig).To(MatchJSON(`{
+			"method": "dflash",
+			"model": ".artifacts/huggingface/0123456789abcdef/snapshot",
+			"num_speculative_tokens": 7
+		}`))
+	})
+
+	It("ignores a draft companion when speculative decoding is not configured", func() {
+		lo := parseOptions(&pb.ModelOptions{
+			Options: []string{"draft_model:.artifacts/huggingface/0123456789abcdef/snapshot"},
+		})
+
+		Expect(lo.speculativeConfig).To(BeEmpty())
+	})
+})
@@ -28,7 +28,11 @@ var _ = Describe("C ABI video struct mirrors", func() {
 		Expect(unsafe.Offsetof(p.Device)).To(Equal(uintptr(72)))
 		Expect(unsafe.Offsetof(p.DequantBf16)).To(Equal(uintptr(76)))
 		Expect(unsafe.Offsetof(p.Fp4Resident)).To(Equal(uintptr(80)))
-		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(88)))
+		Expect(unsafe.Offsetof(p.Family)).To(Equal(uintptr(88)))
+		Expect(unsafe.Offsetof(p.ExtraKeys)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.ExtraValues)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.NExtras)).To(Equal(uintptr(112)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
 	})

 	It("cVideoParams matches vllm_video_params", func() {
@@ -47,7 +51,10 @@ var _ = Describe("C ABI video struct mirrors", func() {
 		Expect(unsafe.Offsetof(p.RefAudio)).To(Equal(uintptr(72)))
 		Expect(unsafe.Offsetof(p.NoiseAug)).To(Equal(uintptr(80)))
 		Expect(unsafe.Offsetof(p.OutputDir)).To(Equal(uintptr(88)))
-		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.ExtraKeys)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.ExtraValues)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.NExtras)).To(Equal(uintptr(112)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
 	})

 	It("cVideoResult matches vllm_video_result", func() {
@@ -16,7 +16,7 @@ func TestVllmCpp(t *testing.T) {
 	RunSpecs(t, "vllm-cpp suite")
 }

-// The Go POD mirrors must match the C struct layout of vllm.h (ABI v16)
+// The Go POD mirrors must match the C struct layout of vllm.h (ABI v23)
 // byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
 // amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
 var _ = Describe("C ABI struct mirrors", func() {
@@ -24,7 +24,7 @@ var _ = Describe("C ABI struct mirrors", func() {
 		// VLLM_ABI_VERSION in the vllm.h of VLLM_CPP_VERSION (Makefile).
 		// Moving the pin past this without growing the mirrors below ships a
 		// backend that refuses every load at startup (issue #11379).
-		Expect(abiVersion).To(Equal(16))
+		Expect(abiVersion).To(Equal(23))
 	})

 	It("cModelParams matches vllm_model_params", func() {
@@ -42,13 +42,17 @@ var _ = Describe("C ABI struct mirrors", func() {
 		Expect(unsafe.Offsetof(p.MaxNumBatchedTokens)).To(Equal(uintptr(60)))
 		Expect(unsafe.Offsetof(p.SchedulingPolicy)).To(Equal(uintptr(64)))
 		Expect(unsafe.Offsetof(p.KVTransferConfig)).To(Equal(uintptr(72)))
-		Expect(unsafe.Offsetof(p.EnableJumpForward)).To(Equal(uintptr(80)))
-		Expect(unsafe.Offsetof(p.Device)).To(Equal(uintptr(84)))
-		// 88, not 92: gpu_memory_utilization is a double, so it takes the next
+		Expect(unsafe.Offsetof(p.OffloadConfig)).To(Equal(uintptr(80)))
+		Expect(unsafe.Offsetof(p.EnableJumpForward)).To(Equal(uintptr(88)))
+		Expect(unsafe.Offsetof(p.Device)).To(Equal(uintptr(92)))
+		// 96: gpu_memory_utilization is a double, so it takes the next
 		// 8-aligned slot after the int32 pair. Go pads identically.
-		Expect(unsafe.Offsetof(p.GPUMemoryUtil)).To(Equal(uintptr(88)))
-		Expect(unsafe.Offsetof(p.KVCacheMemoryBytes)).To(Equal(uintptr(96)))
-		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.GPUMemoryUtil)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.KVCacheMemoryBytes)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.LanguageModelOnly)).To(Equal(uintptr(112)))
+		Expect(unsafe.Offsetof(p.LimitMMPerPrompt)).To(Equal(uintptr(120)))
+		Expect(unsafe.Offsetof(p.MMProjPath)).To(Equal(uintptr(128)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(136)))
 	})

 	It("cSamplingParams matches vllm_sampling_params (ABI v8)", func() {
@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)

 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=592feef04a1802b18cbeffd0fd0eb5d02570c2ec
+WHISPER_CPP_VERSION?=c44b60b8053bbf2a5c1e014f11323fb3f2485177
 SO_TARGET?=libgowhisper.so

 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
@@ -0,0 +1,35 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// Specs for resolveAddr run under the suite bootstrap in gowhisper_test.go
+// (TestWhisper); they need no native library, so they never skip.
+var _ = Describe("resolveAddr", func() {
+	It("prefers an explicitly set -addr over a positional argument", func() {
+		Expect(resolveAddr("127.0.0.1:12345", true, []string{"127.0.0.1:59999"})).To(Equal("127.0.0.1:12345"))
+	})
+
+	It("keeps an explicit -addr equal to the default over a positional argument", func() {
+		Expect(resolveAddr(defaultAddr, true, []string{"127.0.0.1:59999"})).To(Equal(defaultAddr))
+	})
+
+	It("falls back to the positional argument when -addr is unset", func() {
+		Expect(resolveAddr(defaultAddr, false, []string{"127.0.0.1:59999"})).To(Equal("127.0.0.1:59999"))
+	})
+
+	It("keeps the default when -addr is unset and no positional argument is given", func() {
+		Expect(resolveAddr(defaultAddr, false, nil)).To(Equal(defaultAddr))
+	})
+
+	It("uses the first positional argument among several", func() {
+		Expect(resolveAddr(defaultAddr, false, []string{"127.0.0.1:59999", "extra"})).To(Equal("127.0.0.1:59999"))
+	})
+
+	It("treats an explicitly empty -addr as unset", func() {
+		Expect(resolveAddr("", true, []string{"127.0.0.1:59999"})).To(Equal("127.0.0.1:59999"))
+		Expect(resolveAddr("", true, nil)).To(Equal(defaultAddr))
+	})
+})
@@ -10,10 +10,31 @@ import (
 	grpc "github.com/mudler/LocalAI/pkg/grpc"
 )

-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+const (
+	defaultAddr = "localhost:50051"
 )

+var (
+	addr = flag.String("addr", defaultAddr, "the address to listen on")
+)
+
+// resolveAddr picks the address the gRPC server binds to. An explicitly set
+// -addr always wins. Launchers may hand us the listen address as a bare
+// positional argument, which Go's flag package silently drops — honour it
+// next so the server binds the port its caller actually allocated instead of
+// the default one (#11623). An explicitly empty -addr counts as unset:
+// binding the empty address would listen on an OS-chosen port on every
+// interface instead of the one the caller allocated.
+func resolveAddr(flagAddr string, addrSet bool, args []string) string {
+	if addrSet && flagAddr != "" {
+		return flagAddr
+	}
+	if len(args) > 0 {
+		return args[0]
+	}
+	return defaultAddr
+}
+
 type LibFuncs struct {
 	FuncPtr any
 	Name    string
@@ -61,7 +82,16 @@ func main() {

 	flag.Parse()

-	if err := grpc.StartServer(*addr, &Whisper{}); err != nil {
+	// flag.Visit reports only flags that were explicitly set, so an -addr
+	// equal to the default is still distinguished from an untouched one.
+	addrSet := false
+	flag.Visit(func(f *flag.Flag) {
+		if f.Name == "addr" {
+			addrSet = true
+		}
+	})
+
+	if err := grpc.StartServer(resolveAddr(*addr, addrSet, flag.Args()), &Whisper{}); err != nil {
 		panic(err)
 	}
 }
@@ -150,6 +150,7 @@
    - audio-transcription
    - CPU
    - CUDA
+    - HIP
    - Metal
  # No vulkan key: the vulkan image would carry a Vulkan loader with no Mesa ICD
  # (see the audio-cpp block in .github/backend-matrix.yml). Pointing a
@@ -161,6 +162,7 @@
    nvidia: "cuda12-audio-cpp"
    nvidia-cuda-12: "cuda12-audio-cpp"
    nvidia-cuda-13: "cuda13-audio-cpp"
+    amd: "rocm-audio-cpp"
    metal: "metal-audio-cpp"
    metal-darwin-arm64: "metal-audio-cpp"
 - &whispercpp
@@ -510,7 +512,7 @@
    default: "cpu-stablediffusion-ggml"
    nvidia: "cuda12-stablediffusion-ggml"
    intel: "intel-sycl-f16-stablediffusion-ggml"
-    # amd: "rocm-stablediffusion-ggml"
+    amd: "rocm-stablediffusion-ggml"
    vulkan: "vulkan-stablediffusion-ggml"
    nvidia-l4t: "nvidia-l4t-arm64-stablediffusion-ggml"
    metal: "metal-stablediffusion-ggml"
@@ -1060,6 +1062,21 @@
    nvidia-l4t: "nvidia-l4t-mlx"
    nvidia-l4t-cuda-12: "nvidia-l4t-mlx"
    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx"
+- &mlx-video
+  name: "mlx-video"
+  icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
+  urls:
+    - https://github.com/Blaizzy/mlx-video
+  license: MIT
+  description: |
+      Generate videos with LTX-2 and converted Wan2.1/Wan2.2 checkpoints using
+      MLX on Apple Silicon.
+  tags:
+    - text-to-video
+    - image-to-video
+    - MLX
+  capabilities:
+    metal: "metal-mlx-video"
 - &mlx-vlm
  name: "mlx-vlm"
  icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
@@ -1194,6 +1211,7 @@
  tags:
    - image-generation
    - video-generation
+    - sound-generation
    - diffusion-models
  license: apache-2.0
  alias: "diffusers"
@@ -1518,6 +1536,25 @@
    nvidia-cuda-12: "cuda12-faster-whisper"
    nvidia-l4t: "nvidia-l4t-arm64-faster-whisper"
    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-faster-whisper"
+- &whisper-medusa
+  description: |
+    Whisper-Medusa accelerates Whisper transcription by predicting multiple tokens per decoding step.
+    The upstream checkpoints use custom Transformers generation code and accept audio clips up to 30 seconds.
+  urls:
+    - https://github.com/aiola-lab/whisper-medusa
+    - https://huggingface.co/collections/aiola/whisper-medusa
+  tags:
+    - speech-to-text
+    - transcription
+    - Whisper
+    - Medusa
+  license: MIT
+  name: "whisper-medusa"
+  alias: "whisper-medusa"
+  capabilities:
+    default: "cpu-whisper-medusa"
+    nvidia: "cuda12-whisper-medusa"
+    nvidia-cuda-12: "cuda12-whisper-medusa"
 - &moonshine
  description: |
    Moonshine is a fast, accurate, and efficient speech-to-text transcription model using ONNX Runtime.
@@ -1785,6 +1822,32 @@
    nvidia-l4t-cuda-12: "nvidia-l4t-faster-qwen3-tts"
    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-faster-qwen3-tts"
  icon: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/-s1gyJfvbE1RgO5iBeNOi.png
+- &funasr
+  urls:
+    - https://github.com/modelscope/FunASR
+  description: |
+    FunASR is an industrial-grade speech recognition toolkit supporting 50+ languages.
+    Includes SenseVoice (170x realtime, emotion detection), Paraformer (highest Chinese
+    accuracy), and built-in VAD, punctuation restoration, and speaker diarization.
+  tags:
+    - speech-recognition
+    - ASR
+    - multilingual
+  license: mit
+  name: "funasr"
+  alias: "funasr"
+  capabilities:
+    nvidia: "cuda12-funasr"
+    intel: "intel-funasr"
+    amd: "rocm-funasr"
+    metal: "metal-funasr"
+    default: "cpu-funasr"
+    nvidia-cuda-13: "cuda13-funasr"
+    nvidia-cuda-12: "cuda12-funasr"
+    nvidia-l4t: "nvidia-l4t-funasr"
+    nvidia-l4t-cuda-12: "nvidia-l4t-funasr"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-funasr"
+  icon: https://avatars.githubusercontent.com/u/109454077
 - &qwen-asr
  urls:
    - https://github.com/QwenLM/Qwen3-ASR
@@ -2101,6 +2164,7 @@
    nvidia: "cuda12-audio-cpp-development"
    nvidia-cuda-12: "cuda12-audio-cpp-development"
    nvidia-cuda-13: "cuda13-audio-cpp-development"
+    amd: "rocm-audio-cpp-development"
    metal: "metal-audio-cpp-development"
    metal-darwin-arm64: "metal-audio-cpp-development"
 - !!merge <<: *stablediffusionggml
@@ -2109,7 +2173,7 @@
    default: "cpu-stablediffusion-ggml-development"
    nvidia: "cuda12-stablediffusion-ggml-development"
    intel: "intel-sycl-f16-stablediffusion-ggml-development"
-    # amd: "rocm-stablediffusion-ggml-development"
+    amd: "rocm-stablediffusion-ggml-development"
    vulkan: "vulkan-stablediffusion-ggml-development"
    nvidia-l4t: "nvidia-l4t-arm64-stablediffusion-ggml-development"
    metal: "metal-stablediffusion-ggml-development"
@@ -2164,6 +2228,20 @@
  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx"
  mirrors:
    - localai/localai-backends:latest-metal-darwin-arm64-mlx
+- !!merge <<: *mlx-video
+  name: "metal-mlx-video"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-video"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-mlx-video
+- !!merge <<: *mlx-video
+  name: "metal-mlx-video-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx-video"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-mlx-video
+- !!merge <<: *mlx-video
+  name: "mlx-video-development"
+  capabilities:
+    metal: "metal-mlx-video-development"
 - !!merge <<: *mlx
  name: "metal-mlx-development"
  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx"
@@ -3904,6 +3982,11 @@
  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-stablediffusion-ggml"
  mirrors:
    - localai/localai-backends:latest-gpu-nvidia-cuda-12-stablediffusion-ggml
+- !!merge <<: *stablediffusionggml
+  name: "rocm-stablediffusion-ggml"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-stablediffusion-ggml"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-stablediffusion-ggml
 - !!merge <<: *stablediffusionggml
  name: "intel-sycl-f32-stablediffusion-ggml"
  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-stablediffusion-ggml"
@@ -3917,6 +4000,11 @@
  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-stablediffusion-ggml"
  mirrors:
    - localai/localai-backends:master-gpu-nvidia-cuda-12-stablediffusion-ggml
+- !!merge <<: *stablediffusionggml
+  name: "rocm-stablediffusion-ggml-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-stablediffusion-ggml"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-stablediffusion-ggml
 - !!merge <<: *stablediffusionggml
  name: "intel-sycl-f32-stablediffusion-ggml-development"
  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-stablediffusion-ggml"
@@ -6358,6 +6446,34 @@
  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-faster-qwen3-tts"
  mirrors:
    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-faster-qwen3-tts
+## whisper-medusa
+- !!merge <<: *whisper-medusa
+  name: "whisper-medusa-development"
+  capabilities:
+    default: "cpu-whisper-medusa-development"
+    nvidia: "cuda12-whisper-medusa-development"
+    nvidia-cuda-12: "cuda12-whisper-medusa-development"
+- !!merge <<: *whisper-medusa
+  name: "cpu-whisper-medusa"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-whisper-medusa"
+  mirrors:
+    - localai/localai-backends:latest-cpu-whisper-medusa
+- !!merge <<: *whisper-medusa
+  name: "cpu-whisper-medusa-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-whisper-medusa"
+  mirrors:
+    - localai/localai-backends:master-cpu-whisper-medusa
+- !!merge <<: *whisper-medusa
+  name: "cuda12-whisper-medusa"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-whisper-medusa"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-whisper-medusa
+- !!merge <<: *whisper-medusa
+  name: "cuda12-whisper-medusa-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-whisper-medusa"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-whisper-medusa
+
 ## qwen-asr
 - !!merge <<: *qwen-asr
  name: "qwen-asr-development"
@@ -7044,6 +7160,16 @@
  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-audio-cpp"
  mirrors:
    - localai/localai-backends:master-gpu-nvidia-cuda-13-audio-cpp
+- !!merge <<: *audiocpp
+  name: "rocm-audio-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-audio-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-audio-cpp
+- !!merge <<: *audiocpp
+  name: "rocm-audio-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-audio-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-audio-cpp
 - !!merge <<: *audiocpp
  name: "metal-audio-cpp"
  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-audio-cpp"
@@ -7054,3 +7180,97 @@
  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-audio-cpp"
  mirrors:
    - localai/localai-backends:master-metal-darwin-arm64-audio-cpp
+## funasr
+- !!merge <<: *funasr
+  name: "funasr-development"
+  capabilities:
+    nvidia: "cuda12-funasr-development"
+    intel: "intel-funasr-development"
+    amd: "rocm-funasr-development"
+    nvidia-l4t: "nvidia-l4t-funasr-development"
+    metal: "metal-funasr-development"
+    default: "cpu-funasr-development"
+    nvidia-cuda-13: "cuda13-funasr-development"
+    nvidia-cuda-12: "cuda12-funasr-development"
+    nvidia-l4t-cuda-12: "nvidia-l4t-funasr-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-funasr-development"
+- !!merge <<: *funasr
+  name: "cpu-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-funasr"
+  mirrors:
+    - localai/localai-backends:latest-cpu-funasr
+- !!merge <<: *funasr
+  name: "cpu-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-funasr"
+  mirrors:
+    - localai/localai-backends:master-cpu-funasr
+- !!merge <<: *funasr
+  name: "cuda12-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-funasr"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-funasr
+- !!merge <<: *funasr
+  name: "cuda12-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-funasr"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-funasr
+- !!merge <<: *funasr
+  name: "cuda13-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-funasr"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-funasr
+- !!merge <<: *funasr
+  name: "cuda13-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-funasr"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-funasr
+- !!merge <<: *funasr
+  name: "intel-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-funasr"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-funasr
+- !!merge <<: *funasr
+  name: "intel-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-funasr"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-funasr
+- !!merge <<: *funasr
+  name: "rocm-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-funasr"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-funasr
+- !!merge <<: *funasr
+  name: "rocm-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-funasr"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-funasr
+- !!merge <<: *funasr
+  name: "nvidia-l4t-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-funasr"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-funasr
+- !!merge <<: *funasr
+  name: "nvidia-l4t-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-funasr"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-funasr
+- !!merge <<: *funasr
+  name: "cuda13-nvidia-l4t-arm64-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-funasr"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-funasr
+- !!merge <<: *funasr
+  name: "cuda13-nvidia-l4t-arm64-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-funasr"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-funasr
+- !!merge <<: *funasr
+  name: "metal-funasr"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-funasr"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-funasr
+- !!merge <<: *funasr
+  name: "metal-funasr-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-funasr"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-funasr
@@ -20,6 +20,7 @@ The Python backends use a unified build system based on `libbackend.sh` that pro
 ### Audio & Speech
 - **coqui** - Coqui TTS models
 - **faster-whisper** - Fast Whisper speech recognition
+- **funasr** - Local multilingual transcription with FunASR and SenseVoice
 - **kitten-tts** - Lightweight TTS
 - **mlx-audio** - Apple Silicon audio processing
 - **chatterbox** - TTS model
@@ -19,6 +19,7 @@ import grpc
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
 from grpc_auth import get_auth_interceptors
+from temp_utils import cleanup_paths

 import tempfile

@@ -115,11 +116,6 @@ def merge_audio_files(audio_files, output_path, sample_rate):
    # Save the merged audio
    ta.save(output_path, merged_waveform, sample_rate)
    
-    # Clean up temporary files
-    for audio_file in audio_files:
-        if os.path.exists(audio_file):
-            os.remove(audio_file)
-
 _ONE_DAY_IN_SECONDS = 60 * 60 * 24

 # If MAX_WORKERS are specified in the environment use it, otherwise default to 1
@@ -226,19 +222,20 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                text_chunks = split_text_at_word_boundary(request.text, max_length=250)
                print(f"Splitting text into chunks of 250 characters: {len(text_chunks)}", file=sys.stderr)
                # Generate audio for each chunk
-                temp_audio_files = []
-                for i, chunk in enumerate(text_chunks):
-                    # Generate audio for this chunk
-                    wav = self.model.generate(chunk, **kwargs)
-                    
-                    # Create temporary file for this chunk
-                    temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
-                    temp_file.close()
-                    ta.save(temp_file.name, wav, self.model.sr)
-                    temp_audio_files.append(temp_file.name)
-                
-                # Merge all audio files
-                merge_audio_files(temp_audio_files, request.dst, self.model.sr)
+                with cleanup_paths() as temp_audio_files:
+                    for i, chunk in enumerate(text_chunks):
+                        # Generate audio for this chunk
+                        wav = self.model.generate(chunk, **kwargs)
+
+                        # Register ownership before saving so a partial write is
+                        # removed too when generation or encoding fails.
+                        temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
+                        temp_file.close()
+                        temp_audio_files.append(temp_file.name)
+                        ta.save(temp_file.name, wav, self.model.sr)
+
+                    # Merge all audio files
+                    merge_audio_files(temp_audio_files, request.dst, self.model.sr)
            else:
                # Generate audio using ChatterboxTTS for short text
                wav = self.model.generate(request.text, **kwargs)
@@ -563,6 +563,18 @@ function startBackend() {
        echo "Added ${EDIR}/lib to LD_LIBRARY_PATH for GPU libraries"
    fi

+    # XPU wheels carry a matching SYCL/Unified Runtime set. Prefer it over
+    # packaged oneAPI or host libraries, which may expose an older loader ABI.
+    if [ "$(uname -s)" = "Linux" ]; then
+        local sycl_runtime
+        for sycl_runtime in "${EDIR}/venv/lib"/libsycl.so*; do
+            if [ -f "${sycl_runtime}" ]; then
+                export LD_LIBRARY_PATH="${EDIR}/venv/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"
+                break
+            fi
+        done
+    fi
+
    if [ ! -z "${BACKEND_FILE:-}" ]; then
        exec "${EDIR}/venv/bin/python" "${BACKEND_FILE}" "$@"
    elif [ -e "${MY_DIR}/server.py" ]; then
@@ -37,6 +37,46 @@ def parse_options(options_list):
    return opts


+def attach_media_parts(messages_dicts, n_images=0, n_videos=0):
+    """Rebuild the last user message as content *parts* carrying media markers.
+
+    Backends that let the tokenizer do the templating hand plain string content
+    to ``apply_chat_template``, but a chat template only emits the model's own
+    media tokens (``<|vision_start|><|image_pad|><|vision_end|>`` for the
+    Qwen-VL family, and the equivalents elsewhere) when the content is a list
+    of parts. Without those markers the engine's multimodal processor finds
+    nothing to substitute and silently discards the pixels, even though they
+    were forwarded correctly out of band.
+
+    Returns a new list whose last user message has
+    ``[{"type": "image"} * n_images, {"type": "video"} * n_videos, text]`` as
+    its content, or ``None`` when there is nothing to attach - no media, no
+    user turn, or content that is already a list of parts - so the caller can
+    keep using the original string-content list.
+    """
+    if not n_images and not n_videos:
+        return None
+    idx = next(
+        (
+            i
+            for i in reversed(range(len(messages_dicts)))
+            if messages_dicts[i].get("role") == "user"
+        ),
+        None,
+    )
+    if idx is None:
+        return None
+    text = messages_dicts[idx].get("content") or ""
+    if not isinstance(text, str):
+        return None
+    parts = [{"type": "image"}] * n_images + [{"type": "video"}] * n_videos
+    if text:
+        parts.append({"type": "text", "text": text})
+    patched = list(messages_dicts)
+    patched[idx] = dict(patched[idx], content=parts)
+    return patched
+
+
 def messages_to_dicts(proto_messages):
    """Convert proto ``Message`` objects to dicts suitable for ``apply_chat_template``.

@@ -14,7 +14,7 @@ import json
 import types
 import unittest

-from python_utils import messages_to_dicts, parse_options
+from python_utils import attach_media_parts, messages_to_dicts, parse_options


 def _msg(**fields):
@@ -118,5 +118,63 @@ class TestMessagesToDicts(unittest.TestCase):
        self.assertNotIn("tool_calls", out[0])


+class TestAttachMediaParts(unittest.TestCase):
+    def test_image_marker_added_to_last_user_turn(self):
+        messages = [
+            {"role": "system", "content": "be brief"},
+            {"role": "user", "content": "first"},
+            {"role": "assistant", "content": "ok"},
+            {"role": "user", "content": "how high is the water?"},
+        ]
+        out = attach_media_parts(messages, n_images=1)
+        self.assertEqual(
+            out[3]["content"],
+            [{"type": "image"}, {"type": "text", "text": "how high is the water?"}],
+        )
+        # Earlier turns and the input list itself are untouched.
+        self.assertEqual(out[:3], messages[:3])
+        self.assertEqual(messages[3]["content"], "how high is the water?")
+
+    def test_counts_and_order_images_then_videos(self):
+        out = attach_media_parts(
+            [{"role": "user", "content": "describe"}], n_images=2, n_videos=1
+        )
+        self.assertEqual(
+            out[0]["content"],
+            [
+                {"type": "image"},
+                {"type": "image"},
+                {"type": "video"},
+                {"type": "text", "text": "describe"},
+            ],
+        )
+
+    def test_empty_text_yields_media_only_parts(self):
+        out = attach_media_parts([{"role": "user", "content": ""}], n_images=1)
+        self.assertEqual(out[0]["content"], [{"type": "image"}])
+
+    def test_other_message_keys_are_preserved(self):
+        out = attach_media_parts(
+            [{"role": "user", "content": "hi", "name": "bob"}], n_images=1
+        )
+        self.assertEqual(out[0]["name"], "bob")
+
+    def test_no_media_is_a_no_op(self):
+        self.assertIsNone(attach_media_parts([{"role": "user", "content": "hi"}]))
+
+    def test_no_user_turn_is_a_no_op(self):
+        self.assertIsNone(
+            attach_media_parts([{"role": "system", "content": "hi"}], n_images=1)
+        )
+
+    def test_content_already_parts_is_a_no_op(self):
+        self.assertIsNone(
+            attach_media_parts(
+                [{"role": "user", "content": [{"type": "text", "text": "hi"}]}],
+                n_images=1,
+            )
+        )
+
+
 if __name__ == "__main__":
    unittest.main()
@@ -0,0 +1,36 @@
+import base64
+import contextlib
+import os
+import tempfile
+
+
+@contextlib.contextmanager
+def materialize_base64(data, suffix=""):
+    """Materialize base64 data for a path-only library and always remove it."""
+    descriptor, path = tempfile.mkstemp(prefix="localai-media-", suffix=suffix)
+    try:
+        with os.fdopen(descriptor, "wb") as output:
+            descriptor = None
+            output.write(base64.b64decode(data))
+        yield path
+    finally:
+        if descriptor is not None:
+            os.close(descriptor)
+        try:
+            os.remove(path)
+        except OSError:
+            pass
+
+
+@contextlib.contextmanager
+def cleanup_paths():
+    """Collect temporary paths and remove them on success or failure."""
+    paths = []
+    try:
+        yield paths
+    finally:
+        for path in paths:
+            try:
+                os.remove(path)
+            except OSError:
+                pass
@@ -0,0 +1,41 @@
+import os
+import tempfile
+import unittest
+from unittest import mock
+
+from temp_utils import cleanup_paths, materialize_base64
+
+
+class MaterializeBase64Test(unittest.TestCase):
+    def test_removes_materialized_file_after_success(self):
+        with tempfile.TemporaryDirectory() as directory:
+            with mock.patch.object(tempfile, "tempdir", directory):
+                with materialize_base64("aGVsbG8=", suffix=".data") as path:
+                    with open(path, "rb") as materialized:
+                        self.assertEqual(materialized.read(), b"hello")
+                self.assertFalse(os.path.exists(path))
+
+    def test_removes_materialized_file_when_consumer_fails(self):
+        with tempfile.TemporaryDirectory() as directory:
+            with mock.patch.object(tempfile, "tempdir", directory):
+                with self.assertRaisesRegex(RuntimeError, "decode failed"):
+                    with materialize_base64("aGVsbG8="):
+                        raise RuntimeError("decode failed")
+                self.assertEqual(os.listdir(directory), [])
+
+
+class CleanupPathsTest(unittest.TestCase):
+    def test_removes_every_registered_path_after_failure(self):
+        with tempfile.TemporaryDirectory() as directory:
+            paths = [os.path.join(directory, name) for name in ("one.wav", "two.wav")]
+            with self.assertRaisesRegex(RuntimeError, "merge failed"):
+                with cleanup_paths() as registered:
+                    for path in paths:
+                        open(path, "wb").close()
+                        registered.append(path)
+                    raise RuntimeError("merge failed")
+            self.assertEqual(os.listdir(directory), [])
+
+
+if __name__ == "__main__":
+    unittest.main()