test(worker): canonicalize missing upload fixture

Resolve the existing model directory before choosing a missing file. On
macOS, resolving the missing leaf fails and otherwise compares an aliased
path with a canonical allowed directory. Keep this test on the Upload
error branch instead of the containment rejection branch.

Assisted-by: Codex:GPT-6

.agents/ai-coding-assistants.md

@@ -49,6 +49,40 @@ AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
 A human reviewer owns the contribution; the AI's involvement is recorded
 via `Assisted-by` (see below).
 
+### Exception: automation operated by a maintainer
+
+The rule above addresses the common case, an AI assistant helping a human
+contributor who then signs off. It does not fit automation that a
+maintainer runs themselves, which opens pull requests with no human
+submitter to sign. Applied literally there, nothing ever signs and the
+DCO check blocks the pull request permanently.
+
+A maintainer-operated bot MUST therefore add a `Signed-off-by` trailer
+naming **the maintainer who operates it**, not the bot and not the model:
+
+```
+Assisted-by: Codex:gpt-5
+Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
+```
+
+This is not the AI certifying the DCO. The maintainer is, exactly as they
+do for a commit they typed by hand: they configured the automation, they
+own its output, and they take responsibility for it when they merge it.
+The `Assisted-by` trailer still records that a model produced the code, so
+the provenance trail is unchanged.
+
+The exception is narrow and does not widen the rule for anyone else:
+
+- It applies only to automation a LocalAI maintainer operates and whose
+  output that maintainer reviews before merge.
+- The sign-off names a real person who accepts DCO responsibility.
+- An AI assistant helping an outside contributor still MUST NOT sign off.
+  That contributor adds their own trailer.
+- A bot MUST NOT sign off on behalf of anyone other than its operator, and
+  MUST NOT add a trailer for a contributor whose branch it pushes to. If
+  automation contributes to someone else's branch, it leaves the sign-off
+  to that contributor.
+
 ## Attribution
 
 When AI tools contribute to LocalAI development, proper attribution helps

.agents/api-endpoints-and-auth.md

@@ -236,6 +236,58 @@ Use these HTTP status codes:
 
 If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
 
+## Control-plane database health metrics
+
+In distributed mode the frontend registers three OpenTelemetry gauges over the
+PostgreSQL control-plane database (`core/services/monitoring/control_plane_db.go`,
+wired in `core/application/distributed.go`). They reach `/metrics` through the
+same Prometheus exporter as the rest of the API metrics.
+
+| Metric | Meaning | Page when |
+|--------|---------|-----------|
+| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising |
+| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 |
+| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table |
+
+A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on.
+While it grows, autovacuum can reclaim nothing anywhere in the database no
+matter how often it runs, so the dead tuple ratio keeps climbing and a six-row
+registry table can reach hundreds of megabytes. Tuning autovacuum does not help.
+The fix is to find the transaction holding the horizon open and clear it:
+
+```sql
+SELECT pid, state, age(backend_xmin) AS xmin_age, now() - xact_start AS xact_age, query
+FROM pg_stat_activity
+WHERE backend_xmin IS NOT NULL
+ORDER BY age(backend_xmin) DESC;
+```
+
+Then `pg_terminate_backend(pid)` on the offenders, and `VACUUM (VERBOSE)` the
+bloated tables once the horizon has moved.
+
+**A healthy-looking xmin age does not on its own prove the horizon is free.**
+The gauge reads `pg_stat_activity`, which only sees live backends. Two other
+things pin the very same horizon and are invisible there, so either one can hold
+vacuum back while the gauge reads 0:
+
+```sql
+SELECT gid, prepared, database, transaction FROM pg_prepared_xacts;
+SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;
+```
+
+An orphaned prepared transaction is cleared with `ROLLBACK PREPARED '<gid>'`,
+and a stale slot with `pg_drop_replication_slot('<slot_name>')`. Check both
+before concluding that a bloated table has some other cause.
+
+Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not
+translate into database load. Failed and timed-out samples cost the same interval
+as successful ones, so a database that is already struggling is not retried on
+every scrape. A failed sample reports the last good values rather than failing the
+scrape, because these gauges matter most when the database is struggling. Before
+the first successful sample the gauges are absent rather than zero, since a zero
+xmin age would read as a healthy horizon: alert on `absent()` too if you need to
+distinguish "healthy" from "never sampled".
+
 ## Advertising surfaces — where to register a new capability
 
 Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.

.agents/backend-signing.md

@@ -67,7 +67,9 @@ entry (`backend/index.yaml`):
 
 ```yaml
 - name: localai
-  url: github:mudler/LocalAI/backend/index.yaml@master
+  url: https://index.localai.io/backends
+  mirrors:
+    - github:mudler/LocalAI/backend/index.yaml@master
   verification:
     issuer: "https://token.actions.githubusercontent.com"
     identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/(heads/master|tags/.+)$"

.agents/building-and-testing.md

@@ -45,3 +45,26 @@ Rules (both gates):
 - **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
 - If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
 - The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.
+
+## Distributed-mode test suites
+
+Two suites cover distributed mode (frontend replicas, worker nodes, PostgreSQL), split by a Ginkgo label. Neither starts a message broker: a distributed deployment needs PostgreSQL and the frontends' own HTTP listener, and the `nats-io` modules are not in the build.
+
+- `make test-e2e-distributed` runs `Distributed && !VLLMMultinode && !Cluster` over `./tests/e2e/distributed` recursively. Services are wired directly into the test binary. 231 specs in about 2m20s (measured 133.3s for the main suite plus the 8 argument-validation specs in the subpackage).
+- `make test-e2e-cluster` runs `Cluster` and spawns real `local-ai` child processes through the `tests/e2e/distributed/cluster` helper package. It **builds `./local-ai` itself** (the `e2e-binary` target), so you do not run `make build` first and cannot forget to. 26 specs, 933.8s of specs in a 15m37s target run (four measurements; the suite grew from 6 specs and roughly 509s, then 21 specs and 802.5s, then 24 specs and 897 to 907s, so budget on the newest number).
+
+Both jobs live in `.github/workflows/tests-e2e-distributed.yml`, with `timeout-minutes: 45` each. They trigger on pull requests *and* on every push to `master`; the `paths-ignore` filter (see [.agents/ci-caching.md](ci-caching.md)) sits on the pull-request trigger only, so a master push always runs both. They are advisory only because `master` carries no branch protection, which is a repository setting and not a YAML key: `continue-on-error: true` would flip the run's *conclusion* to success and hide the failure, so it is not used.
+
+- **The container is suite-scoped, not spec-scoped.** `SetupInfra` used to start a PostgreSQL (~10s) and a broker (~3.5s) per spec. Across the 213 specs behind it that was roughly **48 minutes of pure container startup per run**, which is why this suite was never in CI. (213 rather than the ~240 above: the larger number is everything the label filter selects, the smaller one is just the specs that call `SetupInfra`.) The broker container is gone entirely, and the PostgreSQL one now starts once in `BeforeSuite` with each spec getting its own database via `CREATE DATABASE` (~67ms), which is what the `dbName` argument was always describing. Adding a spec needs no change: call `SetupInfra("some-name")` as before, the name is a prefix and a counter keeps it unique.
+- **Consequence for new specs:** a fan-out carrier is *shared* within a Ginkgo process, so a wildcard subscriber can observe another spec's traffic. Filter assertions on an identifier your spec owns (a node ID, a job ID) instead of counting everything on `jobs.*.progress`, and verify the spec with `--randomize-all`.
+- **`BeforeSuite`, not `SynchronizedBeforeSuite`.** Under `ginkgo -p` each process then gets its own container, keeping each process's carrier isolated. A single shared database across parallel processes would let specs on different processes see each other's messages on the same subject.
+- **The label split.** The 8 argument-validation specs under `tests/e2e/distributed/cluster/` carry `Label("Distributed")` only, on purpose: they need no binary and no PostgreSQL, so they belong in the fast job. That is why `test-e2e-distributed` keeps `-r` (it must reach the subpackage) and `test-e2e-cluster` deliberately does **not** (the subpackage is out of its scope).
+- **The binary is built, and a stale one is refused.** `test-e2e-cluster` used to take `./local-ai` as given and only check that a file by that name existed. An edit that was never rebuilt therefore ran the entire suite against an older build while reporting on the working tree, which surfaced once as a spec failing on caller line numbers that matched no line in any source file. Two things close it. The target depends on `e2e-binary`, which is `go build -o local-ai ./cmd/local-ai` and not `make build` (that one also builds the React UI, which this suite never touches and CI stubs). And `localAIBinary` in the harness **fails**, on a developer box as well as under CI, when the binary is older than the newest non-test Go source in the tree, which covers a run started with `LOCALAI_E2E_BINARY` or by invoking ginkgo directly. `_test.go` files are excluded from that scan on purpose: they compile into the ginkgo suite and never into `local-ai`, so counting them would fire on every edit to the spec being run. Set `LOCALAI_E2E_BINARY` to opt out of the build; there is no opt-out of the staleness check, because a rebuild is the remedy.
+- **`--fail-on-empty` is load-bearing on both targets.** Ginkgo exits 0 when a label filter selects nothing, so without it a refactor that renames or drops `Label("Cluster")` leaves the target reporting "Test Suite Passed" having started no cluster at all. `LOCALAI_E2E_REQUIRE_BINARIES` does not cover this case: it only fires inside a spec that is actually running.
+- **The binary gate.** `localAIBinary()` and `mockBackendBinary()` **fail** rather than skip when `CI` is set, or when `LOCALAI_E2E_REQUIRE_BINARIES` is truthy; `LOCALAI_E2E_REQUIRE_BINARIES=0` (also `off`, `no`, `n`, `disabled`, and anything `strconv.ParseBool` reads as false) forces skipping even under CI. **Any value that parses as neither reads as ON**, not off: setting the variable to something meaningless means someone meant to turn the gate on, and reading it as false would quietly restore the silent skip the flag exists to remove. The whole polarity is deliberate, because in CI a skipped cluster spec is indistinguishable from a passing one: Ginkgo exits 0 on skips. Locally a missing binary still just skips, since `CI` is unset in an ordinary shell.
+- **Flake budget: no retries at all.** `--flake-attempts` is *total attempts*, not retries (ginkgo v2.29.0 `internal/group.go` sets `maxAttempts = FlakeAttempts` and loops `attempt < maxAttempts`; the flag's own usage string reads "0 - failed tests are not retried"). `DISTRIBUTED_TEST_FLAKES` defaults to **1**, so each spec runs once and a failure is a failure, and `test-e2e-cluster` pins `--flake-attempts 1` outright rather than reading the variable. The repo-wide `TEST_FLAKES=5` means up to five attempts, so up to four retries. These suites exist to surface nondeterminism, and a retry converts exactly that signal into a green run. Raise it locally when bisecting something unrelated, not in the Makefile.
+- **Coverage:** `tests/e2e/distributed` is excluded from the coverage roots (`COVERAGE_E2E_ROOTS = ./tests/e2e`, run non-recursively), and so is the `cluster` helper package beneath it. Neither suite moves the baseline, so production code that these suites are the only cover for reads as **uncovered**. Unit tests for such code belong under `./core/...` with `testutil.SetupTestDB()`.
+- **The cluster job builds against a stubbed React UI.** `core/http/react-ui/dist` is gitignored and built by Node, so the workflow writes a one-line `index.html` there to satisfy the `//go:embed react-ui/dist/*` in `core/http/app.go` and skips a full Node and Vite install. That holds only while the suite drives the HTTP API and never the UI, which has its own e2e suite. A spec that ever asserts on a UI asset would pass locally, where a real `dist/` exists, and be served the stub in CI: if you write one, the stub step has to go and the real build come back.
+- **The topology that matters is two frontends and two workers.** `cluster_busless_test.go` runs the whole programme on it, and it is the only shape in which the OWNER path and the RELAY path are live at the same time: with the two workers' tunnels landing on different replicas, every request either goes to the replica holding that worker's tunnel or is relayed through the one that does. A one-worker cluster cannot tell those apart, and cannot state that killing a replica re-homes only ITS worker. Which worker served a request is read back from `node_models` rather than assumed, and the two models are pinned to one worker each through the admin API (`PUT /api/nodes/:id/labels` plus `POST /api/nodes/scheduling`), so a selector that failed open reddens the spec instead of hiding in it.
+- **The fan-out carrier is proven under real binaries by `cluster_fanout_test.go`.** Both specs run two frontends and no workers against one PostgreSQL, publish at frontend 0 and read at frontend 1, and cost 7.0s together. They drive a gallery operation on purpose: `GET /models/jobs/<id>` answers from `galleryop`'s in-memory statuses map, which on a peer replica is filled by the `gallery.*.progress` subscriber and by nothing else (`Hydrate` runs once at startup, long before these operations exist), so a peer that answers was told over the carrier. Every other broadcast family has a durable table behind it that a peer would converge through anyway, and a spec on one of those cannot separate "the broadcast arrived" from "the row was read". The specs make that checkable rather than argued: the `gallery_operations` row is written when the gallery worker DEQUEUES an operation, so an operation still queued has none, and both specs assert zero rows while the peer is already answering with the operation's own bytes. Holding the queue is what `cluster.Options.Galleries` is for: it points every frontend at one gated index and turns the startup estimate warmer off, so the operation the spec submits is parked inside a fetch only the spec can complete. The second spec adds the oversized path: an element name of 9000 bytes puts the encoded notification past PostgreSQL's 8000-byte cap (exclusive), so the broadcast becomes a `bus_messages` row plus a notification carrying its id, and the spec asserts the row exists, decodes it, and compares its element name byte for byte with what frontend 1 answers. A one-byte control operation in the same run is asserted to leave NO row, which is what keeps "a row exists" a statement about size rather than one that is true of every operation.
+- **Do not shorten the cluster suite's waits.** Three of its six specs sit at ~167s each because they wait out a 60s staleness threshold plus a 15s health-check tick. That wait is what stops the assertions from passing before the system could have reacted, which was a real false green earlier on. If the job has to get faster, the levers are CI concurrency or making the thresholds configurable, not shorter waits.

.agents/ci-caching.md

@@ -153,7 +153,7 @@ This is worth more than it looks. Measured over the week to 2026-07-30, **97% of
 
 The volume is real: 13 gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs opened were bot-generated.
 
-`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20. The excluded set:
+`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20, measured before `tests-e2e-distributed.yml` (2 jobs) landed. That workflow carries the same exclusion set for the same reason: its dependency graph is 99 packages, so an allowlist of paths would silently stop guarding the moment code moved, while a diff confined to the paths below provably cannot reach it. The excluded set:
 
 | Path | Why no image or Go build can see it |
 |---|---|
@@ -192,13 +192,13 @@ What still runs, and why it has to:
 Two properties this relies on:
 
 - `paths-ignore` skips a run only when **every** changed file matches, so a PR touching the gallery *and* Go code still runs everything. That is what makes the exclusion safe rather than a hole.
-- `master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these four entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
+- `master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these five entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
 
 ### `image.yml` on master push is gated too, by a job rather than a path filter
 
 The same reasoning applies to master pushes, and the volume is larger there: on 2026-07-30, **12 of the 23 queued `image.yml` runs** were commits like "add 1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
 
-`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `github:mudler/LocalAI/gallery/index.yaml@master`. A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users through GitHub immediately whether or not an image is rebuilt.
+`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `https://index.localai.io/models` (a caching mirror of `gallery/index.yaml` on master, with `github:mudler/LocalAI/gallery/index.yaml@master` as the fallback mirror). A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users over the network immediately whether or not an image is rebuilt.
 
 Two properties to preserve if you touch it:
 

.agents/ds4-backend.md

@@ -77,6 +77,56 @@ spectrum. **Metal (Darwin) only** - it is a no-op on CUDA/CPU. Enable with
 budget). Gallery entries built on this: `deepseek-v4-flash-q4-ssd` (153 GB Flash
 on a 128 GB Mac) and `deepseek-v4-pro-q2-ssd` (433 GB Pro, experimental).
 
+## CUDA architecture (do not build without one)
+
+`backend/cpp/ds4/Makefile` drives upstream's **object targets** directly
+(`$(MAKE) -C ds4 ds4.o ds4_cuda.o ...`), which bypasses upstream's own guard:
+its `cuda` target refuses to build unless `CUDA_ARCH` is set, and offers
+`cuda-spark` (sm_121, DGX Spark / GB10) and `cuda-generic` (native) instead.
+Built with no `-arch`, nvcc targets its default architecture and the kernels run
+as JIT'd PTX. On GB10 that silently corrupted every prefill batch of >=128
+tokens - the model emitted text unrelated to the prompt and never closed its
+thinking block, so `content` came back empty - and cost close to two orders of
+magnitude of prefill throughput (4.21 t/s vs 325.70 t/s, same box, same model).
+Short prompts stayed correct, which is why it went unnoticed.
+
+The Makefile therefore picks a gencode list from `CUDA_MAJOR_VERSION` (a build
+arg the backend matrix already declares, forwarded by `Dockerfile.ds4`) and
+`uname -m`, and passes it as `NVCC_ARCH_FLAGS` to the sub-make. Upstream's
+`CUDA_ARCH` accepts a single value, so it cannot express the fat binary the
+shipped images need; a command-line assignment beats its `:=`. An empty
+`CUDA_MAJOR_VERSION` falls back to upstream's `native` for local developer
+builds, and an unrecognised one is a hard error - no CI runner has a GPU, so a
+silent `native` there is exactly the failure mode this guards against.
+
+`DS4_CUDA_HAVE_MXF4` is deliberately unset: upstream defines it only for
+single-arch sm_120/sm_121 builds and guards it with a plain `#ifdef` rather than
+`__CUDA_ARCH__`, so it cannot be combined with older archs. It gates an optional
+MXFP4 indexer fast path whose `#ifndef` branch returns 0, so omitting it costs
+speed, not correctness.
+
+### Verifying a build
+
+Check which flags a configuration resolves to, without compiling anything:
+
+```
+make -C backend/cpp/ds4 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 NATIVE=false \
+  --eval='show: ; @echo [$(DS4_ARCH_MAKEVARS)]' show
+```
+
+Do not use `make -n` for this: the recipe is `+$(MAKE) ...`, and the `+` prefix
+makes it run even under `-n`.
+
+Then exercise the failure mode itself against a built backend. It only appears
+above one prefill batch, so the ordinary `predict` spec cannot catch it:
+
+```
+BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \
+BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \
+BACKEND_TEST_CAPS=health,load,predict,long_prefill \
+go test -count=1 -timeout=30m -v ./tests/e2e-backends/...
+```
+
 ## Build matrix
 
 | Build | Where | Notes |

.agents/localai-assistant-mcp.md

@@ -94,4 +94,4 @@ The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.m
 
 ## Distributed mode
 
-The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period.
+The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the PostgreSQL-backed node registry). MCP tools are not routed anywhere: the admin surface lives on the head, period.

.dockerignore

@@ -59,7 +59,9 @@ backend/rust/*/target
 backend-images
 local-backends
 local-ai
+.claude
 .crush
+.tools
 protoc
 tests
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -5,7 +5,7 @@ This PR fixes #
 **Notes for Reviewers**
 
 
-**[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
+**[Signed commits](../CONTRIBUTING.md#commit-messages)**
 - [ ] Yes, I signed my commits.
 - [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable
  

.github/backend-matrix.yml

@@ -860,6 +860,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-nemo-speech-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1911,6 +1924,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-nemo-speech-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1963,6 +1989,24 @@ include:
     backend: "parakeet-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
+  # The CUDA-13 counterpart to the JetPack r36.4.0 row in the nemo-speech-cpp
+  # block below. A Jetson whose CUDA 13 runtime is present reports the
+  # nvidia-l4t-cuda-13 capability, and pointing that key at the JetPack image
+  # would hand it a ggml linked against CUDA 12 whose libcudart.so.12 is not
+  # there to dlopen. Same base and runner as the parakeet-cpp row above.
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-nemo-speech-cpp'
+    base-image: "ubuntu:24.04"
+    ubuntu-version: '2404'
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2540,7 +2584,7 @@ include:
     tag-latest: 'auto'
     tag-suffix: '-gpu-intel-vllm'
     runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
     skip-drivers: 'false'
     backend: "vllm"
     dockerfile: "./backend/Dockerfile.python"
@@ -3710,6 +3754,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-stablediffusion-ggml'
+    runs-on: 'ubuntu-latest'
+    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+    skip-drivers: 'false'
+    backend: "stablediffusion-ggml"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4183,6 +4240,86 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # nemo-speech-cpp
+  #
+  # No hipblas and no sycl rows, unlike the parakeet-cpp block above: upstream
+  # NeMo-Speech.cpp builds ggml with CUDA, Vulkan or Metal only, so a ROCm or
+  # SYCL image would be a CPU build wearing a GPU tag.
+  #
+  # cpu and vulkan are per-arch pairs sharing one tag-suffix, so
+  # backend-merge-jobs assembles a multi-arch manifest from the two digests.
+  # The arm64 legs are not redundant with the Jetson image below: an ARM server
+  # with no NVIDIA GPU reports the "default" capability and would otherwise pull
+  # an amd64-only manifest.
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-nemo-speech-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-nemo-speech-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-nemo-speech-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-nemo-speech-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-arm64-nemo-speech-cpp'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "nemo-speech-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2204'
   # moss-transcribe-cpp
   - build-type: ''
     cuda-major-version: ""
@@ -6226,6 +6363,10 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-moss-transcribe-cpp"
     build-type: "metal"
     lang: "go"
+  - backend: "nemo-speech-cpp"
+    tag-suffix: "-metal-darwin-arm64-nemo-speech-cpp"
+    build-type: "metal"
+    lang: "go"
   - backend: "ced"
     tag-suffix: "-metal-darwin-arm64-ced"
     build-type: "metal"

.github/bump_vllm_metal.sh

@@ -3,9 +3,9 @@
 # darwin (Apple Silicon) install path. The macOS/Metal build
 # (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is
 # version-locked to a specific vLLM source release. install.sh derives that vLLM
-# version at build time from vllm-metal's own installer (`vllm_v=`) at the pinned
-# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh,
-# which bumps the Linux cu130 wheel pin.
+# version, and the wheel asset name, at build time from the pinned tag, so there
+# is only ONE value to bump here -- mirroring bump_vllm_wheel.sh, which bumps the
+# Linux cu130 wheel pin.
 #
 # This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the
 # darwin build can only use the exact vLLM version vllm-metal supports, so it may
@@ -23,19 +23,24 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
     exit 1
 fi
 
-# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
-# /releases/latest returns the newest one (with its cp312 wheel asset).
+# vllm-metal ships frequent .dev releases, flagged as prereleases, alongside the
+# stable ones. /releases/latest skips the prereleases and returns the newest
+# stable tag, which is what darwin should pin: upstream deletes and re-cuts .dev
+# tags, and a pin to a deleted tag 404s the whole build.
 LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
     "https://api.github.com/repos/$REPO/releases/latest" \
     | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
 
-# The coupled vLLM source version lives in vllm-metal's installer at that tag.
-NEW_VLLM_VERSION=$(gh_curl \
-    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
-    | grep -oE 'vllm_v="[0-9]+\.[0-9]+\.[0-9]+"' | head -1 | cut -d'"' -f2)
+# The coupled vLLM release lives in .github/vllm-release-tag.commit at that tag
+# (since vllm-metal 0.28); releases predating that file pinned it inline in their
+# own install.sh. The extractor reads both forms.
+NEW_VLLM_VERSION=$( { gh_curl \
+    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/.github/vllm-release-tag.commit" \
+    || gh_curl "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh"; } \
+    | "$(dirname "${BASH_SOURCE[0]}")/../scripts/lib/extract-vllm-metal-version.sh")
 
 if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then
-    echo "Could not resolve vllm-metal tag ($LATEST_TAG) or its vllm_v ($NEW_VLLM_VERSION)." >&2
+    echo "Could not resolve vllm-metal tag ($LATEST_TAG) or its vLLM version ($NEW_VLLM_VERSION)." >&2
     exit 1
 fi
 

.github/ci/test-website-review-143.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+python3 - <<'PY'
+from pathlib import Path
+
+home = Path("website/layouts/index.html").read_text()
+css = Path("website/static/css/site.css").read_text()
+install = Path("docs/content/getting-started/install.md").read_text()
+containers = Path("docs/content/getting-started/containers.md").read_text()
+
+def require(condition, message):
+    if not condition:
+        raise SystemExit(f"FAIL: {message}")
+
+require("Drop-in replacement for most upstream APIs." in home,
+        "homepage must use the requested drop-in API heading")
+require("Everything else plugs into LocalAI." not in home,
+        "old runtime heading must be removed")
+require("When the engine we need" not in home,
+        "hero must describe user outcomes instead of team implementation")
+require('href="mailto:contact@localai.io"' in home and "business" in home.lower(),
+        "homepage must provide a direct business contact action")
+require(home.index('id="localai"') < home.index('id="proof-quotes"') < home.index('id="mission"'),
+        "headline testimonials must directly follow the runtime section")
+require(home.count('id="proof-quotes"') == 1,
+        "headline testimonials must appear exactly once")
+require('id="engines"' not in home and "Engines we build" not in home,
+        "homepage engine showcase must be removed")
+require('href="/docs/installation/index.html"' in home,
+        "installation guide action must use the direct installation URL")
+require('<iframe' in install and "youtube.com/embed/cMVNnlqwfw4" in install,
+        "installation page must embed the walkthrough video")
+require("## Quick Start" not in install,
+        "installation landing page must not duplicate Quick Start")
+for text in ("CUDA 12", "CUDA 13", "ROCm", "Intel", "Jetson", "Vulkan", "fallback"):
+    require(text.lower() in containers.lower(), f"GPU chooser must explain {text}")
+require('class="sn__e"><a href="https://github.com/mudler/parakeet.cpp">parakeet.cpp</a>' in home,
+        "capability engine names must link to their repositories")
+require(".pane{min-height:" in css.replace(" ", ""),
+        "all installation panes must have a fixed minimum height")
+
+print("website review 143 source checks passed")
+PY

.github/dependabot.yml

@@ -29,10 +29,6 @@ updates:
     schedule:
       # Check for updates to GitHub Actions every weekday
       interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/bark"
-    schedule:
-      interval: "weekly"
   - package-ecosystem: "pip"
     directory: "/backend/python/common/template"
     schedule:
@@ -55,30 +51,10 @@ updates:
     ignore:
     - dependency-name: "torch"
     - dependency-name: "transformers"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/exllama"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/exllama2"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/mamba"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/openvoice"
-    schedule:
-      interval: "weekly"
   - package-ecosystem: "pip"
     directory: "/backend/python/rerankers"
     schedule:
       interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/backend/python/sentencetransformers"
-    schedule:
-      interval: "weekly"
   - package-ecosystem: "pip"
     directory: "/backend/python/transformers"
     schedule:
@@ -86,44 +62,4 @@ updates:
   - package-ecosystem: "pip"
     directory: "/backend/python/vllm"
     schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/chainlit"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/functions"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/langchain/langchainpy-localai-example"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/langchain-chroma"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "pip"
-    directory: "/examples/streamlit-bot"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/k8sgpt"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/kubernetes"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/langchain"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "gomod"
-    directory: "/examples/semantic-todo"
-    schedule:
-      interval: "weekly"
-  - package-ecosystem: "docker"
-    directory: "/examples/telegram-bot"
-    schedule:
-      interval: "weekly"
+      interval: "weekly"

.github/workflows/backend_merge.yml

@@ -71,8 +71,8 @@ jobs:
 
       # cosign signs each pushed manifest list with --recursive so the
       # index and every per-arch entry get an attached Sigstore bundle.
-      # The pinned cosign v2 release needs --new-bundle-format explicitly;
-      # the verifier only consumes OCI 1.1 Sigstore bundle referrers.
+      # Cosign v2.4.1 emits the current bundle format by default; the
+      # verifier discovers those bundles through OCI 1.1 referrers.
       - name: Install cosign
         if: github.event_name != 'pull_request'
         uses: sigstore/cosign-installer@v3
@@ -159,7 +159,6 @@ jobs:
           # manifest before checking signatures need the per-arch
           # signatures, not just the list-level one.
           cosign sign --yes --recursive \
-            --new-bundle-format \
             --registry-referrers-mode=oci-1-1 \
             "quay.io/go-skynet/local-ai-backends@${digest}"
 
@@ -186,7 +185,6 @@ jobs:
           ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
           digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}')
           cosign sign --yes --recursive \
-            --new-bundle-format \
             --registry-referrers-mode=oci-1-1 \
             "localai/localai-backends@${digest}"
 

.github/workflows/bump_deps.yaml

@@ -62,6 +62,10 @@ jobs:
             variable: "MOSS_VERSION"
             branch: "master"
             file: "backend/go/moss-transcribe-cpp/Makefile"
+          - repository: "NVIDIA/NeMo-Speech.cpp"
+            variable: "NEMO_SPEECH_VERSION"
+            branch: "main"
+            file: "backend/go/nemo-speech-cpp/Makefile"
           - repository: "localai-org/ced.cpp"
             variable: "CED_VERSION"
             branch: "main"
@@ -162,7 +166,7 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update ${{ matrix.repository }}'
           title: 'chore: :arrow_up: Update ${{ matrix.repository }} to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/${{ matrix.variable }}"
+          branch: "bump/${{ matrix.variable }}"
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
@@ -199,7 +203,7 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel'
           title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_VERSION"
+          branch: "bump/VLLM_VERSION"
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
@@ -237,6 +241,6 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update vllm-project/vllm-metal (darwin)'
           title: 'chore: :arrow_up: Update vllm-metal (darwin) to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_METAL_VERSION"
+          branch: "bump/VLLM_METAL_VERSION"
           body: ${{ steps.bump.outputs.message }}
           signoff: true

.github/workflows/notify-releases.yaml

@@ -31,13 +31,14 @@ jobs:
             messages: [
                 {
                 role: "system",
-                content: "Write a discord message with a bullet point summary of the release notes."
+                content: "Write a Discord message with a bullet point summary of the release notes. Keep the complete message under 1800 characters."
                 },
                 {
                 role: "user",
                 content: $input
                 }
-            ]
+            ],
+            max_tokens: 450
             }')
 
             # Send the request to LocalAI API
@@ -46,7 +47,7 @@ jobs:
             -d "$json_payload")
 
             # Extract the summary from the response
-            summary=$(echo $response | jq -r '.choices[0].message.content')
+            summary=$(printf '%s' "$response" | jq -er '.choices[0].message.content | strings | .[0:1800]')
 
             # Print the summary
             #  -H "Authorization: Bearer $API_KEY" \

.github/workflows/refresh-site-counters.yml

@@ -14,6 +14,7 @@ on:
 
 permissions:
   contents: write
+  pull-requests: write
 
 concurrency:
   group: refresh-site-counters
@@ -23,22 +24,32 @@ jobs:
   refresh:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v7
 
       - name: Read the counts off the GitHub API
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: ./.github/ci/refresh-site-counters.sh
 
-      - name: Commit only if something moved
+      - name: Show changes
         run: |
           if git diff --quiet -- website/data/stats.yaml; then
-            echo "counters unchanged, nothing to commit"
-            exit 0
+            echo "counters unchanged"
+          else
+            git diff --unified=0 -- website/data/stats.yaml
           fi
-          git diff --unified=0 -- website/data/stats.yaml
-          git config user.name  "github-actions[bot]"
-          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-          git add website/data/stats.yaml
-          git commit -m "chore(website): refresh the counters"
-          git push
+
+      - name: Create pull request when counters moved
+        uses: peter-evans/create-pull-request@v8
+        with:
+          token: ${{ secrets.UPDATE_BOT_TOKEN }}
+          push-to-fork: ci-forks/LocalAI
+          commit-message: "chore(website): refresh the counters"
+          title: "chore(website): refresh the counters"
+          body: |
+            Weekly refresh of the landing-page counters from the GitHub API.
+
+            This PR was created automatically by the `refresh-site-counters` workflow.
+          branch: update/site-counters
+          delete-branch: true
+          labels: automated

.github/workflows/stalebot.yml

@@ -11,7 +11,7 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@1e223db275d687790206a7acac4d1a11bd6fe629 # v9
+      - uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v9
         with:
           stale-issue-message: 'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
           stale-pr-message: 'This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.'

.github/workflows/test-extra.yml

@@ -50,6 +50,7 @@ jobs:
       sherpa-onnx: ${{ steps.detect.outputs.sherpa-onnx }}
       whisper: ${{ steps.detect.outputs.whisper }}
       parakeet-cpp: ${{ steps.detect.outputs.parakeet-cpp }}
+      nemo-speech-cpp: ${{ steps.detect.outputs.nemo-speech-cpp }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v7
@@ -525,6 +526,7 @@ jobs:
       - name: Build llama-cpp backend image and run gRPC e2e tests
         run: |
           make test-extra-backend-llama-cpp
+          make test-extra-backend-llama-cpp-embeddings
   tests-llama-cpp-grpc-transcription:
     needs: detect-changes
     if: needs.detect-changes.outputs.llama-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
@@ -900,6 +902,57 @@ jobs:
       - name: Test magpie-tts-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/magpie-tts-cpp test
+  # Per-backend unit suite for nemo-speech-cpp. This job exists for one reason
+  # above all: abi_test.go asserts the size and field offsets of every Go mirror
+  # struct against the C ABI it is dlopened into. Those assertions are the only
+  # thing standing between a purego symbol rename or an upstream header change
+  # and silent memory corruption at run time, and they are worthless unless
+  # something executes them. `make -C backend/go/nemo-speech-cpp test` sets
+  # NEMO_SPEECH_REQUIRE_LIBS=1, which turns "library missing" from a skip into a
+  # failure, so this job cannot report green having checked nothing.
+  #
+  # The backend Makefile's `test` target depends on `stage-libs`, so it clones
+  # upstream at the pinned SHA and builds the native runtime itself. There is no
+  # separate build step for that reason, and no model download: the specs are
+  # ABI and pure-Go only.
+  #
+  # WITH_NORM=OFF skips the Sparrowhawk/OpenFST inverse-text-normalization
+  # stack, which is the single most expensive leg of the build and needs a gcc-12
+  # pin because OpenFST's templates ICE on gcc-13/14. It costs no coverage here:
+  # nothing in include/nemo_speech/{asr,tts,diar,nmt}.h is conditional on it (the
+  # only preprocessor conditionals in those headers are include guards,
+  # __cplusplus and the _WIN32 export macros), so every struct layout this suite
+  # checks is identical either way. The shipped images still build WITH_NORM=ON;
+  # that path is covered by the backend image build in backend_pr.yml.
+  tests-nemo-speech-cpp:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.nemo-speech-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake ninja-build curl libopenblas-dev ffmpeg
+      - name: Setup Go
+        uses: actions/setup-go@v5
+      - name: Display Go version
+        run: go version
+      - name: Proto Dependencies
+        run: |
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Test nemo-speech-cpp
+        run: |
+          make --jobs=5 --output-sync=target -C backend/go/nemo-speech-cpp WITH_NORM=OFF test
   # Per-backend smoke for rfdetr-cpp: builds the .so + Go binary and runs
   # `make -C backend/go/rfdetr-cpp test`. test.sh fetches the small (~20 MB)
   # rfdetr-nano-q8_0 GGUF from the published mudler/rfdetr-cpp-nano HF repo

.github/workflows/test.yml

@@ -65,6 +65,12 @@ jobs:
       - name: Test (with coverage gate)
         run: |
           PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test-coverage-check
+      # tests/integration is outside the coverage roots because its store specs
+      # need a live backend. test-stores builds and installs local-store before
+      # running the complete suite, so new local-store specs are collected
+      # automatically without adding another workflow entry.
+      - name: Test local-store integration
+        run: PATH="$PATH:$HOME/go/bin" make test-stores
       - name: Upload coverage report
         if: ${{ always() }}
         uses: actions/upload-artifact@v4
@@ -74,8 +80,13 @@ jobs:
             coverage/coverage.out
             coverage/coverage.html
           if-no-files-found: ignore
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true
@@ -119,8 +130,13 @@ jobs:
           export PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
           PATH="$PATH:$HOME/go/bin" make protogen-go
           PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target test
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-aio.yml

@@ -77,8 +77,13 @@ jobs:
       - name: Test
         run: |
             PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-e2e-distributed.yml

@@ -0,0 +1,206 @@
+---
+name: 'E2E Distributed Tests'
+
+on:
+  pull_request:
+    # The suite's dependency graph is 99 packages, so an allowlist of paths
+    # silently stops guarding the moment code moves. At ~75s the job is cheap
+    # enough to run unless the diff is confined to paths it provably cannot
+    # reach. See .agents/ci-caching.md.
+    paths-ignore:
+      - 'gallery/**'
+      - 'docs/**'
+      - 'examples/**'
+      - '**/*.md'
+  push:
+    branches:
+      - master
+
+concurrency:
+  group: ci-tests-e2e-distributed-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  tests-e2e-distributed:
+    runs-on: ubuntu-latest
+    # Advisory because it is deliberately not in branch protection, so a failure
+    # is a visible red X rather than a blocked merge. Promoting it to a required
+    # check is a repository-settings change, to be made once it has a track
+    # record; a heavy suite made required on day one gets disabled instead of
+    # fixed.
+    timeout-minutes: 45
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.0'
+          cache: false
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential libopus-dev
+      - name: Proto Dependencies
+        run: |
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Pre-pull test images
+        # Pulling here rather than inside the suite keeps container-start timing
+        # out of the spec timeouts and makes a registry outage read as a
+        # setup failure instead of a test failure. This is the only image the
+        # suite needs once the testcontainers reaper is disabled below: the
+        # suite stands up no message broker, because nothing under test dials
+        # one.
+        run: |
+          docker pull postgres:16-alpine
+      - name: Distributed E2E
+        # TESTCONTAINERS_RYUK_DISABLED keeps the pre-pull above meaningful. The
+        # reaper exists to clean up leaked containers on a long-lived host, but
+        # this runner is ephemeral and every container dies with the VM. Leaving
+        # it enabled would pull a third, unpinned image (testcontainers/ryuk)
+        # from Docker Hub mid-suite: exactly the registry dependency the
+        # pre-pull step exists to remove.
+        env:
+          TESTCONTAINERS_RYUK_DISABLED: "true"
+        run: |
+          PATH="$PATH:$HOME/go/bin" make test-e2e-distributed
+      - name: Setup tmate session if tests fail
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3.23
+        with:
+          detached: true
+          connect-timeout-seconds: 180
+          limit-access-to-actor: true
+
+  tests-e2e-cluster:
+    runs-on: ubuntu-latest
+    # Advisory for the same reason as the job above: master has no branch
+    # protection, so a failure here is a visible red X rather than a blocked
+    # merge. That is a repository-settings property, not a YAML key. The key
+    # that looks like it says "advisory" instead flips the run's conclusion to
+    # success, which hides the failure rather than flagging it, so it appears in
+    # none of this repo's workflows and must not be added here.
+    #
+    # Separate job from tests-e2e-distributed so the fast in-process suite is
+    # not held behind a Go build of local-ai. Serial on purpose: each Ginkgo
+    # process would get its own PostgreSQL container and each spec spawns two or
+    # three local-ai children, so --procs on an unmeasured runner is a change to
+    # make with numbers, not by default.
+    #
+    # The two timeouts bound different things and are not alternatives. Ginkgo's
+    # --timeout=20m bounds the SUITE only; this job timeout must additionally
+    # cover setup, which here is the larger and more variable half: submodule
+    # checkout, apt, protoc plus two go installs plus protogen-go, a cold-cache
+    # module download (cache: false), a full go build of ./cmd/local-ai, and a
+    # separate ginkgo test compile. That build alone is ~316s of CPU, so on a
+    # 4-vCPU runner setup is realistically 8-12 minutes.
+    #
+    # 45 minutes therefore, matching the sibling job. A tighter number does not
+    # make a hang fail faster, it just moves the kill from Ginkgo, which prints
+    # which spec hung, to the runner, which prints nothing: a red job with no
+    # evidence, which is how a suite gets disabled rather than fixed.
+    #
+    # The suite itself is about 8m30s over three consecutive runs (509.1s /
+    # 509.8s / 512.3s, so 8m29s to 8m32s) on a developer box, and will be slower
+    # here. Three specs sit at ~167s each because they wait out a 60s staleness
+    # threshold plus a 15s health-check tick (HealthCheckInterval, in
+    # core/config/distributed_config.go; core/services/nodes/health.go runs the
+    # ticker on the unexported checkInterval, not one of the reconcilers). Do
+    # not shorten those windows to make this job faster: the wait is what stops
+    # the assertions from passing before the system could have reacted, which
+    # was a real false green earlier on.
+    timeout-minutes: 45
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.0'
+          cache: false
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential libopus-dev
+      - name: Proto Dependencies
+        run: |
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Stub the embedded React UI
+        # core/http/react-ui/dist is gitignored and built by Node, but this
+        # suite drives the HTTP API and never the UI, which has its own e2e
+        # suite. A single index.html satisfies the //go:embed react-ui/dist/*
+        # in core/http/app.go, so the job skips a full Node and Vite install.
+        # If a cluster spec ever asserts on a UI asset, this step must go and
+        # the real build come back: a developer box has a real dist/, so such a
+        # spec would pass locally and fail only here, or worse be served the
+        # stub and pass in both places.
+        run: |
+          mkdir -p core/http/react-ui/dist
+          printf '<!doctype html><title>stub</title>\n' > core/http/react-ui/dist/index.html
+      - name: Pre-pull test images
+        # Same reasoning as the job above: pulling here keeps container-start
+        # timing out of the spec timeouts and makes a registry outage read as a
+        # setup failure rather than a test failure.
+        run: |
+          docker pull postgres:16-alpine
+      - name: Cluster E2E
+        env:
+          # No LOCALAI_E2E_BINARY and no separate build step: make test-e2e-cluster
+          # builds ./local-ai itself, AFTER the protogen-go it also depends on.
+          # Building it in a step of its own put the generated .pb.go files newer
+          # than the binary, which the harness's staleness check reads, correctly,
+          # as a binary that does not contain the tree it is about to be measured
+          # against. One owner for the build is also what makes a local run and
+          # this job exercise the same bytes.
+          #
+          # Must live under the workspace so the upload step below can reach it.
+          # The harness defaults to GinkgoT().TempDir(), which lands under
+          # TMPDIR and would leave the artifact glob matching nothing.
+          LOCALAI_E2E_LOG_DIR: ${{ github.workspace }}/cluster-logs
+          # Belt and braces: the harness already fails rather than skips when CI
+          # is set, and GitHub Actions always sets CI. Stating it here means a
+          # future edit to that default cannot silently turn this job into one
+          # that passes without ever starting a cluster, since a skipped cluster
+          # spec is indistinguishable from a passing one.
+          LOCALAI_E2E_REQUIRE_BINARIES: "true"
+          # See the job above: the runner is ephemeral, so the reaper buys
+          # nothing and would pull a third, unpinned Docker Hub image mid-suite.
+          TESTCONTAINERS_RYUK_DISABLED: "true"
+        run: |
+          PATH="$PATH:$HOME/go/bin" make test-e2e-cluster
+      - name: Upload process logs
+        # The per-process logs are the only way to read a cluster failure: the
+        # Ginkgo output says which assertion failed, not what the four child
+        # processes were doing. Without this a red job is undebuggable.
+        if: ${{ failure() }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: cluster-process-logs
+          path: cluster-logs/**/*.log
+          if-no-files-found: ignore
+          retention-days: 7
+      - name: Setup tmate session if tests fail
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3.23
+        with:
+          detached: true
+          connect-timeout-seconds: 180
+          limit-access-to-actor: true

.github/workflows/tests-e2e.yml

@@ -63,8 +63,13 @@ jobs:
       - name: Test Backend E2E
         run: |
           PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-pii-ner-e2e.yml

@@ -88,8 +88,13 @@ jobs:
       # CPU and runs the token_classify capability spec (byte-offset contract).
       - name: Run live PII NER backend E2E
         run: PATH="$PATH:$HOME/go/bin" make test-extra-backend-privacy-filter
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-ui-e2e.yml

@@ -52,6 +52,8 @@ jobs:
         run: |
           sudo apt-get update
           sudo apt-get install -y build-essential libopus-dev
+      - name: Run stale chunk recovery tests
+        run: PATH="$PATH:$HOME/go/bin" make test-ui-stale-chunk
       # Builds an instrumented UI bundle, runs the Playwright specs, and fails
       # if line coverage regressed beyond the jitter tolerance (the gate is
       # in `make test-ui-coverage-check`). PLAYWRIGHT_CHROMIUM_PATH is unset
@@ -73,8 +75,13 @@ jobs:
           path: core/http/react-ui/coverage/
           if-no-files-found: ignore
           retention-days: 7
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.golangci.yml

@@ -19,7 +19,22 @@ linters:
     - staticcheck
   enable:
     - forbidigo
+    # gocritic is enabled for ONE checker: ruleguard, which runs the rules in
+    # hack/lint/. Every other gocritic check is off (disable-all below), so
+    # this adds no style noise; it is here purely as the gate that catches a
+    # gRPC backend wrapper written without Unwrap. See
+    # hack/lint/backend_wrappers.go for why that cannot be a compile-time
+    # assertion.
+    - gocritic
   settings:
+    gocritic:
+      disable-all: true
+      enabled-checks:
+        - ruleguard
+      settings:
+        ruleguard:
+          failOn: all
+          rules: '${base-path}/hack/lint/backend_wrappers.go'
     forbidigo:
       forbid:
         - pattern: '^t\.Errorf$'
@@ -126,3 +141,10 @@ linters:
       - path: ^backend/go/whisper/sources/
         text: 'http\.(DefaultClient|Get|Post|PostForm|Head)'
         linters: [forbidigo]
+      # Test doubles embed grpc.Backend to inherit the interface's method set
+      # over a NIL value; they decorate nothing, hold no inner client, and have
+      # no transport answer to forward. The rule targets production wrappers,
+      # which is where swallowing that answer deletes replica rows.
+      # gocritic here is only the backend-wrapper ruleguard rule.
+      - path: _test\.go$
+        linters: [gocritic]

.impeccable.md

@@ -0,0 +1,21 @@
+## Design Context
+
+### Users
+
+LocalAI serves both single-host users who want to install and try models quickly and experienced developers, ML engineers, system administrators, and DevOps operators who manage production hosts or distributed clusters. The interface must support first-time discovery without hiding the runtime state, configuration, and control that returning operators need.
+
+### Brand Personality
+
+Capable, easy to use, and trustworthy. The interface should make sophisticated local-AI infrastructure feel understandable and under control. It should be direct and calm rather than playful, ornamental, or intimidating.
+
+### Aesthetic Direction
+
+Use LocalAI's established technical, editorial design language: Geist typography, compact information density, sharp geometry, deep blue-black surfaces, action blue, mint for healthy/local/live state, and amber only for decisions requiring attention. Support both dark and light themes. Avoid generic card dashboards, decorative gradients, glass effects, and visual noise.
+
+### Design Principles
+
+1. Use progressive disclosure to serve newcomers and operators in the same workflow: make the common path obvious, then reveal operational depth in context.
+2. Organize navigation around user intent and lifecycle state, not implementation concepts or nested containers.
+3. Give each resource one canonical home; expose discovery, installed state, and runtime state as clear views of that resource instead of duplicating management surfaces.
+4. Keep operational status visible and trustworthy through precise labels, explicit scope, and actionable state—not decoration.
+5. Preserve information density for expert use while flattening navigation and reducing repeated summaries, tabs, rails, and panels.

ADOPTERS.md

@@ -27,6 +27,7 @@ To be removed, open a pull request deleting your row, or email
 
 | Organisation | What they use it for | Status |
 |---|---|---|
+| [walcz.de](https://walcz.de) | Self-hosted appliance for a German B2B consultancy: local-only inference on AMD Strix Halo (gfx1151/ROCm), agents with MCP tools, RAG over an internal knowledge base, and a document/bookkeeping pipeline. | Production |
 | _Your organisation here_ | | |
 
 ## What this list is not

AGENTS.md

@@ -8,7 +8,7 @@ Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development w
 
 LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:
 
-- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
+- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. One exception: automation a maintainer operates signs off with *that maintainer's* identity, since no other human submitter exists to certify it. See [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md).
 - **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
 - **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
 - **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.
@@ -33,6 +33,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 | [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) | LocalAI Assistant chat modality — adding admin tools to the in-process MCP server, editing skill prompts, keeping REST + MCP + skills in sync |
 | [.agents/backend-signing.md](.agents/backend-signing.md) | Backend OCI image signing (keyless cosign + sigstore-go) — producer-side CI setup, consumer-side gallery `verification:` block, strict mode (`LOCALAI_REQUIRE_BACKEND_INTEGRITY`), revocation via `not_before` |
 | [.agents/preparing-a-release.md](.agents/preparing-a-release.md) | Cutting a release: PR labels, `RELEASE_NOTES_vX.Y.Z.md`, the blog post under `website/content/blog/`, and the demo clips under `website/static/media/` |
+| [.impeccable.md](.impeccable.md) | Design context for UI/UX work — users, brand personality, aesthetic direction, and design principles |
 
 ## Quick Reference
 

CONTRIBUTING.md

@@ -218,7 +218,7 @@ LocalAI follows the **same guidelines as the Linux kernel project** for AI-assis
 
 The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:
 
-- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
+- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. Automation operated by a maintainer is the one exception: it signs off with that maintainer's identity, because there is no other human submitter to certify it.
 - **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
 - **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:
 
@@ -265,6 +265,37 @@ The e2e tests run LocalAI in a Docker container and exercise the API:
 make test-e2e
 ```
 
+### Running distributed-mode tests
+
+Distributed mode (several frontend replicas, worker nodes and PostgreSQL) has two suites. Both bring up their PostgreSQL with testcontainers, so Docker has to be available. There is no message broker to bring up: a distributed deployment needs PostgreSQL and the frontends' own HTTP listener, and nothing else.
+
+```bash
+make test-e2e-distributed   # in-process: services wired directly into the test binary
+make test-e2e-cluster       # process-level: real local-ai child processes
+```
+
+`make test-e2e-distributed` is the fast one (231 specs in roughly 2m20s). It starts one PostgreSQL for the whole run and gives each spec its own database. It runs each spec exactly once, with no retry: `DISTRIBUTED_TEST_FLAKES` defaults to 1 and feeds ginkgo's `--flake-attempts`, which counts *total attempts*, not retries. That is deliberately below the repo-wide `TEST_FLAKES=5`, because this suite exists to catch nondeterministic cluster behaviour and a retry hides exactly the failure it is meant to catch. Raise it locally when bisecting something unrelated.
+
+`make test-e2e-cluster` runs `local-ai` as real child processes, one per frontend replica and one per worker, so a spec can kill a replica and assert what the survivors do. Budget about 13m30s (one measurement: 802.5s of specs in a 13m26s target run, which includes building the binary): several of its 21 specs wait out real staleness and health-check windows. An earlier record of 6 specs and roughly 509s is superseded. It needs a built binary and the mock backend:
+
+```bash
+make build build-mock-backend
+make test-e2e-cluster
+```
+
+Two environment variables steer it:
+
+| Variable | Purpose |
+|---|---|
+| `LOCALAI_E2E_BINARY` | path to the `local-ai` binary (default: `local-ai` in the repository root) |
+| `LOCALAI_E2E_LOG_DIR` | directory for the per-process logs (default: a Ginkgo temp dir) |
+
+Set `LOCALAI_E2E_LOG_DIR` when debugging. A cluster failure is unreadable without the individual frontend and worker logs, and Ginkgo only tells you which assertion failed.
+
+A missing binary skips the cluster specs locally but fails them whenever `CI` is set, so a build problem cannot turn the CI job green without ever starting a cluster. `LOCALAI_E2E_REQUIRE_BINARIES=1` forces that failing behaviour anywhere; `LOCALAI_E2E_REQUIRE_BINARIES=0` forces the skip back on even under CI.
+
+Both suites run in `.github/workflows/tests-e2e-distributed.yml`, on pull requests and on every push to `master`. The `paths-ignore` filter is on the pull-request trigger only, so a master push always runs them.
+
 ### React UI tests and coverage
 
 The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -34,6 +34,11 @@ TEST_FLAKES?=5
 RANDOM := $(shell bash -c 'echo $$RANDOM')
 
 VERSION?=$(shell git describe --always --tags || echo "dev" )
+# fyne package only accepts numeric x[.y[.z]] app versions, so reduce git
+# describe output (v4.9.0, v4.9.0-14-gabc1234, or a bare sha on untagged
+# checkouts) to its numeric core; anything non-numeric falls back to 0.0.0.
+# Without this the packaged launcher reports itself as version 0.0.0 (#11673).
+LAUNCHER_APP_VERSION?=$(shell v=$$(echo "$(VERSION)" | sed -E 's/^v//; s/[+-].*$$//'); echo "$$v" | grep -qE '^[0-9]+(\.[0-9]+){0,2}$$' && echo "$$v" || echo "0.0.0")
 # go tool nm ./local-ai | grep Commit
 LD_FLAGS?=-s -w
 override LD_FLAGS += -X "github.com/mudler/LocalAI/internal.Version=$(VERSION)"
@@ -103,7 +108,7 @@ COVERAGE_E2E_LABELS?=!real-models
 COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go
 
 
-.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
+.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-stale-chunk test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
 
 all: help
 
@@ -335,12 +340,108 @@ run-e2e-aio: protogen-go
 	@echo 'Running e2e AIO tests'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e-aio
 
-# Distributed architecture e2e (PostgreSQL + NATS via testcontainers).
-# Includes NatsJWT specs (JWT-enabled NATS). Requires Docker.
+# Total ginkgo attempts per spec for the distributed suite: --flake-attempts counts
+# attempts, not retries. Defaults to 1, so each spec runs once and is never retried,
+# unlike TEST_FLAKES=5. This suite exists to catch nondeterministic cluster behaviour,
+# and a retry hides exactly the failures it is meant to surface. Raise it locally if
+# you are bisecting something unrelated.
+DISTRIBUTED_TEST_FLAKES?=1
+
+# Distributed architecture e2e (PostgreSQL via testcontainers). Requires Docker.
+# There is no broker container and no NatsJWT label any more: a distributed
+# deployment needs PostgreSQL and the frontends' own HTTP listener, and this
+# suite stands up exactly that.
 # VLLMMultinode is excluded here; use test-e2e-vllm-multinode for that.
+# Cluster is excluded too and runs in test-e2e-cluster below, which needs a
+# built binary. The argument-validation specs under tests/e2e/distributed/cluster
+# carry Label("Distributed") only, so they run here and not there, on purpose.
+# -r stays because of those: they are in a subpackage this target must reach.
+# --fail-on-empty because ginkgo exits 0 when a label filter matches nothing, so
+# without it a rename of the label would turn this target into a silent no-op
+# that still reports "Test Suite Passed".
+#
+# This target needs NO built binary and therefore has no stale-binary hole:
+# ginkgo compiles the suite from the working tree on every run, and everything
+# these specs exercise runs in that process against containers. The one target
+# in this file that does exec a prebuilt binary is test-e2e-cluster below, and
+# the reason it now BUILDS it is written there.
 test-e2e-distributed: protogen-go
-	@echo 'Running distributed e2e tests (label Distributed, incl. NatsJWT)'
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='Distributed && !VLLMMultinode' --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e/distributed
+	@echo 'Running distributed e2e tests (label Distributed)'
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='Distributed && !VLLMMultinode && !Cluster' --fail-on-empty --flake-attempts $(DISTRIBUTED_TEST_FLAKES) --timeout=40m -v -r ./tests/e2e/distributed
+
+# The local-ai binary the cluster e2e execs as its frontend and worker
+# processes.
+#
+# Deliberately not `make build`: that target also builds the embedded React UI,
+# which this suite never touches, and CI stubs core/http/react-ui/dist for
+# exactly that reason. What is left is the go build, which is incremental and
+# costs seconds when nothing changed, so making it a prerequisite of the suite
+# is cheap enough that nobody has a reason to route around it.
+#
+# LOCALAI_E2E_BINARY means the caller has its own binary and owns keeping it
+# current, so this builds nothing. The staleness check in the harness still
+# applies to it, which is the point: a binary this target did not produce is
+# precisely the one nothing else vouches for.
+.PHONY: e2e-binary
+e2e-binary: protogen-go
+	@if [ -n "$$LOCALAI_E2E_BINARY" ]; then \
+	  echo "LOCALAI_E2E_BINARY=$$LOCALAI_E2E_BINARY is set; not building $(BINARY_NAME)"; \
+	else \
+	  echo "Building $(BINARY_NAME) for the cluster e2e"; \
+	  CGO_LDFLAGS="$(CGO_LDFLAGS)" $(GOCMD) build -ldflags "$(LD_FLAGS)" -tags "$(GO_TAGS)" -o $(BINARY_NAME) ./cmd/local-ai; \
+	fi
+
+# Cluster e2e: runs local-ai as real child processes (frontend replicas +
+# workers) against PostgreSQL, and kills them to assert failover.
+# It BUILDS that binary rather than checking that a file by that name exists,
+# and that is a correctness fix rather than a convenience. This target used to
+# take ./local-ai as given, so an edit to core/ that was never rebuilt left the
+# whole suite exercising an older build while reporting on the working tree. The
+# failure mode is not theoretical and not loud: it surfaced as a spec failing on
+# caller line numbers that matched no line in any source file, and it would
+# otherwise have surfaced as a green run that proved nothing. build-mock-backend
+# already rebuilds unconditionally, so ./local-ai was the only gap.
+#
+# The harness carries the other half. Building here cannot help a run started
+# with LOCALAI_E2E_BINARY or by invoking ginkgo directly, so localAIBinary also
+# FAILS when the binary it is about to exec is older than the newest non-test Go
+# source in the tree. Together: this target makes the common path correct, and
+# the harness makes every other path loud.
+#
+# The argument-validation specs in tests/e2e/distributed/cluster deliberately
+# stay in test-e2e-distributed above: they need no binary and no PostgreSQL, so
+# no -r here and that package is simply out of scope.
+#
+# --fail-on-empty is load-bearing, not tidiness. Ginkgo exits 0 when a label
+# filter selects nothing, so without it a refactor that renames or drops
+# Label("Cluster") leaves this target reporting "Test Suite Passed" having
+# started no cluster at all. LOCALAI_E2E_REQUIRE_BINARIES does not cover this:
+# it only fires inside a spec that is actually running.
+#
+# --flake-attempts is pinned to 1 rather than $(DISTRIBUTED_TEST_FLAKES), and
+# should stay there: this suite exists to catch nondeterministic cluster
+# behaviour, and a retry turns exactly that signal into a green run.
+#
+# Budget: 26 specs, measured at 933.8 seconds of Ginkgo time (15m37s wall
+# including the compile) on a fast developer box. It was 591 to 612 seconds
+# before the phase 3 control-plane specs and 800 to 830 after them; the three
+# two-frontend two-worker specs in cluster_busless_test.go added 118 to 127
+# seconds (3s, 70s and 46 to 53s), nearly all of it in the churn spec, which
+# cannot be shortened: it waits for a killed replica to leave the live set
+# (cluster.InstanceLiveness is 30s, measured at 27s) before it may assert
+# anything, and then holds a window inside the reconnect grace. The two
+# cross-replica fan-out specs added 7.0 seconds of spec time (5.0s and 2.0s):
+# they run two frontends and no workers, so they pay for no registration, and
+# what they wait on is a broadcast rather than a threshold.
+#
+# --timeout is 30m rather than 20m because of that. The margin is not slack: a
+# Ginkgo timeout kills the suite mid-spec and reports a spec name rather than a
+# cause, and 20m on a loaded CI runner was one slow health tick away from
+# turning a green suite into an unreadable red one. At 15 minutes measured, 30m
+# is still twice the budget.
+test-e2e-cluster: protogen-go build-mock-backend e2e-binary
+	@echo 'Running cluster e2e tests (label Cluster, real local-ai processes)'
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='Cluster' --fail-on-empty --flake-attempts 1 --timeout=30m -v ./tests/e2e/distributed
 
 # vLLM multi-node DP smoke (CPU). Builds local-ai:tests and the
 # cpu-vllm backend from the current working tree, then drives a
@@ -378,19 +479,29 @@ prepare-e2e:
 run-e2e-image:
 	docker run -p 5390:8080 -e MODELS_PATH=/models -e THREADS=1 -e DEBUG=true -d --rm -v $(TEST_DIR):/models --name e2e-tests-$(RANDOM) localai-tests
 
+# Distributed subpackages have separate targets that provision PostgreSQL and
+# build the host binaries; this target exercises the container-backed API only.
 test-e2e: build-mock-backend build-cloud-proxy-backend prepare-e2e run-e2e-image
 	@echo 'Running e2e tests'
 	BUILD_TYPE=$(BUILD_TYPE) \
 	LOCALAI_API=http://$(E2E_BRIDGE_IP):5390 \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v ./tests/e2e
 	$(MAKE) clean-mock-backend
 	$(MAKE) clean-cloud-proxy-backend
 	$(MAKE) teardown-e2e
 	docker rmi localai-tests
 
+# `docker stop` returns as soon as the container exits, but Docker reaps a
+# `--rm` container asynchronously after that. The `docker rmi localai-tests` in
+# test-e2e then loses the race against the reaper and fails on a still
+# referenced image, turning a green suite red. Removing the container ourselves
+# is synchronous, so the image reference is gone before we return. It also
+# covers the case where nothing is running, which `docker stop` could not
+# because it rejects an empty argument list.
 teardown-e2e:
 	rm -rf $(TEST_DIR) || true
-	docker stop $$(docker ps -q --filter ancestor=localai-tests)
+	@CONTAINERS=$$(docker ps -aq --filter ancestor=localai-tests 2>/dev/null); \
+	if [ -n "$$CONTAINERS" ]; then docker rm -f $$CONTAINERS || true; fi
 
 ########################################################
 ## Integration and unit tests
@@ -654,6 +765,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/go/depth-anything-cpp test
 	$(MAKE) -C backend/go/supertonic test
 	$(MAKE) -C backend/go/vllm-cpp test
+	$(MAKE) -C backend/go/nemo-speech-cpp test
 	$(MAKE) -C backend/go/trellis2cpp test
 	$(MAKE) -C backend/go/valkey-store test
 
@@ -675,6 +787,7 @@ test-extra: prepare-test-extra
 ##   BACKEND_TEST_PROMPT      Override the prompt used in predict/stream specs.
 ##   BACKEND_TEST_OPTIONS     Comma-separated Options[] entries forwarded to LoadModel,
 ##                            e.g. "tool_parser:hermes,reasoning_parser:qwen3".
+##   BACKEND_TEST_EMBEDDING_LAYOUT Expected EmbeddingResult layout: "final" or "per_token".
 ##
 ## Direct usage (image already built, no docker-build-* dependency):
 ##
@@ -704,6 +817,7 @@ test-extra-backend: protogen-go
 	BACKEND_TEST_CAPS="$$BACKEND_TEST_CAPS" \
 	BACKEND_TEST_PROMPT="$$BACKEND_TEST_PROMPT" \
 	BACKEND_TEST_OPTIONS="$$BACKEND_TEST_OPTIONS" \
+	BACKEND_TEST_EMBEDDING_LAYOUT="$$BACKEND_TEST_EMBEDDING_LAYOUT" \
 	BACKEND_TEST_TOOL_PROMPT="$$BACKEND_TEST_TOOL_PROMPT" \
 	BACKEND_TEST_TOOL_NAME="$$BACKEND_TEST_TOOL_NAME" \
 	BACKEND_TEST_CACHE_TYPE_K="$$BACKEND_TEST_CACHE_TYPE_K" \
@@ -723,6 +837,15 @@ test-extra-backend-llama-cpp: docker-build-llama-cpp
 	BACKEND_TEST_CAPS=health,load,predict,stream,logprobs,logit_bias \
 	$(MAKE) test-extra-backend
 
+## Raw llama.cpp embeddings are required by Go-side pooling. This exercises the
+## real C++ backend and verifies that it marks the flattened matrix per-token.
+test-extra-backend-llama-cpp-embeddings: docker-build-llama-cpp
+	BACKEND_IMAGE=local-ai-backend:llama-cpp \
+	BACKEND_TEST_CAPS=health,load,embeddings \
+	BACKEND_TEST_OPTIONS=pooling:none \
+	BACKEND_TEST_EMBEDDING_LAYOUT=per_token \
+	$(MAKE) test-extra-backend
+
 test-extra-backend-ik-llama-cpp: docker-build-ik-llama-cpp
 	BACKEND_IMAGE=local-ai-backend:ik-llama-cpp $(MAKE) test-extra-backend
 
@@ -812,6 +935,7 @@ test-extra-backend-tinygrad-embeddings: docker-build-tinygrad
 	BACKEND_IMAGE=local-ai-backend:tinygrad \
 	BACKEND_TEST_MODEL_NAME=Qwen/Qwen3-0.6B \
 	BACKEND_TEST_CAPS=health,load,embeddings \
+	BACKEND_TEST_EMBEDDING_LAYOUT=final \
 	$(MAKE) test-extra-backend
 
 ## tinygrad — Stable Diffusion 1.5. The original CompVis/runwayml repos have
@@ -1298,6 +1422,7 @@ BACKEND_WHISPER = whisper|golang|.|false|true
 BACKEND_CRISPASR = crispasr|golang|.|false|true
 BACKEND_PARAKEET_CPP = parakeet-cpp|golang|.|false|true
 BACKEND_MOSS_TRANSCRIBE_CPP = moss-transcribe-cpp|golang|.|false|true
+BACKEND_NEMO_SPEECH_CPP = nemo-speech-cpp|golang|.|false|true
 BACKEND_DEPTH_ANYTHING_CPP = depth-anything-cpp|golang|.|false|true
 BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
@@ -1400,6 +1525,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_WHISPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_CRISPASR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PARAKEET_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TRANSCRIBE_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_NEMO_SPEECH_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DEPTH_ANYTHING_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VOXTRAL)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OPUS)))
@@ -1456,7 +1582,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-audio-cpp
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-audio-cpp
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1502,6 +1628,13 @@ test-ui: build-mock-backend protogen-go
 	$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui
 	cd core/http/react-ui && sh $(CURDIR)/scripts/ensure-playwright-browser.sh && bunx playwright test $(PLAYWRIGHT_WORKERS_FLAG)
 
+## The stale-chunk specs need the production code-split bundle. The V8 coverage
+## bundle below inlines dynamic imports to keep every page in its denominator.
+test-ui-stale-chunk: build-mock-backend protogen-go
+	cd core/http/react-ui && bun install && bun run build
+	$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui
+	cd core/http/react-ui && sh $(CURDIR)/scripts/ensure-playwright-browser.sh && bunx playwright test --grep @production-chunks --workers=1
+
 ## React UI code coverage from the Playwright e2e suite. Builds a
 ## NON-instrumented bundle with source maps (COVERAGE_V8=true), re-embeds it
 ## into the ui-test-server (the dist is //go:embed'ed at compile time), runs the
@@ -1517,7 +1650,7 @@ test-ui-coverage: build-mock-backend protogen-go
 	$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui && \
 	( cd core/http/react-ui && rm -rf .nyc_output coverage && \
 	    sh $(CURDIR)/scripts/ensure-playwright-browser.sh && \
-	    PW_V8_COVERAGE=1 bunx playwright test $(PLAYWRIGHT_WORKERS_FLAG) && bun run coverage:report )
+	    PW_V8_COVERAGE=1 bunx playwright test --grep-invert @production-chunks $(PLAYWRIGHT_WORKERS_FLAG) && bun run coverage:report )
 
 ## UI coverage baseline (committed) and the strict gate that compares against
 ## it — the React mirror of test-coverage-baseline / test-coverage-check.
@@ -1600,7 +1733,7 @@ site-serve: site
 build-launcher-darwin:
 	rm -rf dist/LocalAI.app cmd/launcher/LocalAI.app
 	mkdir -p dist
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION)
 	mv cmd/launcher/LocalAI.app dist/LocalAI.app
 	bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app
 
@@ -1627,4 +1760,4 @@ release-launcher-darwin: notarize-launcher-darwin
 	@echo "dist/LocalAI.dmg is ready"
 
 build-launcher-linux:
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION) && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz

README.md

@@ -5,9 +5,6 @@
 </h1>
 
 <p align="center">
-<a href="https://github.com/go-skynet/LocalAI/stargazers" target="blank">
-<img src="https://img.shields.io/github/stars/go-skynet/LocalAI?style=for-the-badge" alt="LocalAI stars"/>
-</a>
 <a href='https://github.com/go-skynet/LocalAI/releases'>
 <img src='https://img.shields.io/github/release/go-skynet/LocalAI?&label=Latest&style=for-the-badge'>
 </a>
@@ -212,7 +209,7 @@ For older news and full release notes, see [GitHub Releases](https://github.com/
 - [Object Detection](https://localai.io/features/object-detection/)
 - [Reranker API](https://localai.io/features/reranker/)
 - [P2P Inferencing](https://localai.io/features/distribute/)
-- [Distributed Mode](https://localai.io/features/distributed-mode/) — Horizontal scaling with PostgreSQL + NATS
+- [Distributed Mode](https://localai.io/features/distributed-mode/): horizontal scaling on PostgreSQL, with no message broker to run
 - [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/)
 - [Built-in Agents](https://localai.io/features/agents/) — Autonomous AI agents with tool use, RAG, skills, SSE streaming, and [Agent Hub](https://agenthub.localai.io)
 - [Backend Gallery](https://localai.io/backends/) — Install/remove backends on the fly via OCI images
@@ -231,7 +228,7 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 
 | Backend | What it does |
 |---------|-------------|
-| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
+| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan. Also serves MiniMax-H3 joint video+audio generation |
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
 | [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
 | [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
@@ -318,10 +315,6 @@ Past sponsors
 
 A special thanks to individual sponsors, a full list is on [GitHub](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler). Special shout out to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!
 
-## Star history
-
-[![LocalAI Star history Chart](https://api.star-history.com/svg?repos=go-skynet/LocalAI&type=Date)](https://star-history.com/#go-skynet/LocalAI&Date)
-
 ## License
 
 LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/) and maintained by the [LocalAI team](#team).

backend/Dockerfile.ds4

@@ -10,6 +10,7 @@ FROM ${BASE_IMAGE} AS builder
 ARG BUILD_TYPE
 ARG TARGETARCH
 ARG TARGETVARIANT
+ARG CUDA_MAJOR_VERSION
 
 ENV BUILD_TYPE=${BUILD_TYPE} \
     DEBIAN_FRONTEND=noninteractive \
@@ -35,7 +36,8 @@ RUN apt-get update && \
 COPY . /LocalAI
 
 RUN --mount=type=cache,target=/root/.ccache,id=ds4-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package
+    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} \
+        CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} NATIVE=false grpc-server package
 
 FROM scratch
 COPY --from=builder /LocalAI/backend/cpp/ds4/package/. ./

backend/Dockerfile.golang

@@ -248,6 +248,134 @@ RUN <<EOT bash
     fi
 EOT
 
+# nemo-speech-cpp builds NVIDIA NeMo-Speech.cpp with text normalization enabled,
+# which compiles the Sparrowhawk/OpenFST WFST stack from source via
+# scripts/build_itn_deps.sh. That step needs gcc-12 specifically: OpenFST's
+# template-heavy translation units ICE on gcc-13 and gcc-14 at -O2, so upstream
+# pins gcc-12 for it while the runtime itself builds with the image default.
+# No update-alternatives here, so the default compiler is untouched; the backend
+# Makefile reaches gcc-12 by name for that one step.
+#
+# The rest is what build_itn_deps.sh and the WITH_NORM cmake block expect:
+# protobuf (headers plus protoc, which must come from the same apt set so the
+# generated stubs match the headers they compile against) and re2 for
+# Sparrowhawk, and autotools because OpenFST and Sparrowhawk ship autoconf
+# builds. ninja is not in the common apt list because this is the only Go
+# backend that configures with -G Ninja, and that list is a layer shared by
+# every backend image in the matrix.
+#
+# No libabsl-dev, despite upstream's Dockerfile installing it: upstream builds
+# against protobuf 25, which splits its runtime across libabsl_*, whereas every
+# base image in this matrix carries protobuf 3.21 (noble) or 3.12 (jammy), which
+# has no absl dependency. The cmake block's file(GLOB ... /usr/lib/libabsl_*.so)
+# would not match on Ubuntu anyway, since multiarch puts those under
+# /usr/lib/<triplet>/.
+#
+# Placed down here with the other per-backend gates rather than next to the
+# shared apt layer: Docker re-keys every layer below an inserted one, so adding
+# a step above the Vulkan SDK, CUDA, Go and protoc layers would force all of
+# them to re-execute once for every Go backend image, not just this one.
+# Nothing between there and here needs any of these packages (the Vulkan and
+# opus blocks install their own ninja and pkg-config, and the protoc download is
+# a release binary that needs neither libprotobuf-dev nor protoc from apt), and
+# nothing here needs anything those layers provide.
+#
+# The second half of this block backfills cmake. NeMo-Speech.cpp opens with
+# cmake_minimum_required(VERSION 3.26), which every noble base in the matrix
+# satisfies (24.04 ships 3.28) but the JetPack r36.4.0 row does not: that image
+# is jammy, whose apt cmake is 3.22, so configure aborts before it reads a
+# single one of our -D flags. This is the only Go backend that needs more than
+# jammy's cmake; parakeet-cpp and moss-transcribe-cpp share the same JetPack
+# base and both declare cmake_minimum_required(VERSION 3.18).
+#
+# Taken from Kitware's own release tarball rather than from their APT repo or
+# from pip. The tarball is a pinned URL with a published checksum, so the build
+# is reproducible and an upstream release cannot change what lands here; the
+# APT repo serves a moving 'latest', which today would be CMake 4.x, and 4.x
+# drops compatibility with cmake_minimum_required below 3.5 and so would break
+# vendored third_party subprojects that still declare one. pip would drag a
+# Python toolchain into a backend that otherwise has none. The binaries need
+# only glibc 2.17 and carry no libstdc++ DT_NEEDED, so jammy's 2.35 is far
+# above the floor. doc/, man/, ccmake and cmake-gui are left in the tarball;
+# this is a builder stage and the final image is FROM scratch, but there is no
+# reason to page 50 MB of Qt GUI and docs through the CI cache.
+#
+# Conditional on the installed cmake being too old rather than unconditional,
+# so the rows that already build green (noble cpu, vulkan, cublas and hipblas)
+# keep configuring with exactly the cmake they configure with today.
+#
+# The version test compares through two temp files and a grep on the exit
+# status rather than the obvious "$(sort -V ... | head -n1)". BuildKit delivers
+# a RUN heredoc through an outer shell with an unquoted delimiter, so the outer
+# shell expands the body before bash ever sees it: a $(...) here runs once, too
+# early, in a container where the files it reads do not exist yet, and its empty
+# output is then pasted into the script. Same reason there are no shell
+# variables below. ${BACKEND} and ${TARGETARCH} are fine because they are build
+# args, which BuildKit exports into that outer shell's environment.
+#
+# The symlink goes in /usr/local/bin, which precedes /usr/bin on PATH, so it
+# shadows apt's cmake. That is deliberate and, unlike the protoc shadowing that
+# broke Sparrowhawk earlier in this PR, it is inert: protoc has to agree with
+# the libprotobuf headers it generates against, whereas cmake is a standalone
+# build driver with no ABI relationship to anything in the image, and it locates
+# its own Modules/ tree by resolving the symlink back to /opt, so a 3.31 binary
+# can never read 3.22's modules. Scope is the ${BACKEND} gate: no other Go
+# backend image gets /opt/cmake or the symlink. Inside this image the only
+# other cmake consumers, the base apt layer and the Vulkan SDK build, both run
+# in layers above this one and have already finished.
+RUN <<EOT bash
+    if [ "${BACKEND}" = "nemo-speech-cpp" ]; then
+        set -e
+        apt-get update
+        apt-get install -y --no-install-recommends \
+            gcc-12 g++-12 \
+            ninja-build \
+            libprotobuf-dev protobuf-compiler \
+            libre2-dev \
+            autoconf automake libtool pkg-config
+        apt-get clean
+        rm -rf /var/lib/apt/lists/*
+
+        echo 3.26.0 > /tmp/cmake-required
+        cmake --version 2>/dev/null | head -n1 | cut -d' ' -f3 > /tmp/cmake-present
+        if [ ! -s /tmp/cmake-present ]; then
+            echo 0.0.0 > /tmp/cmake-present
+        fi
+        if sort -V /tmp/cmake-required /tmp/cmake-present | head -n1 | grep -qxF 3.26.0; then
+            echo "==> cmake is new enough for NeMo-Speech.cpp:"
+            cmake --version | head -n1
+        else
+            echo "==> cmake is below the 3.26 NeMo-Speech.cpp requires; installing 3.31.12. Found:"
+            cat /tmp/cmake-present
+            mkdir -p /opt/cmake
+            if [ "${TARGETARCH}" = "arm64" ]; then
+                curl -fsSL -o /tmp/cmake.tar.gz https://github.com/Kitware/CMake/releases/download/v3.31.12/cmake-3.31.12-linux-aarch64.tar.gz
+                echo "83f8fd91d2038a56556e1400390fcfe42f79602940c494f6c6f1cdae7f9e7f40  /tmp/cmake.tar.gz" | sha256sum -c -
+                tar -xzf /tmp/cmake.tar.gz -C /opt/cmake --strip-components=1 \
+                    cmake-3.31.12-linux-aarch64/bin/cmake \
+                    cmake-3.31.12-linux-aarch64/bin/cpack \
+                    cmake-3.31.12-linux-aarch64/bin/ctest \
+                    cmake-3.31.12-linux-aarch64/share
+            else
+                curl -fsSL -o /tmp/cmake.tar.gz https://github.com/Kitware/CMake/releases/download/v3.31.12/cmake-3.31.12-linux-x86_64.tar.gz
+                echo "0dc2e9a6860f06bf10bd8fadc03e35d9eeb4df46e33763a7e480e987758f385c  /tmp/cmake.tar.gz" | sha256sum -c -
+                tar -xzf /tmp/cmake.tar.gz -C /opt/cmake --strip-components=1 \
+                    cmake-3.31.12-linux-x86_64/bin/cmake \
+                    cmake-3.31.12-linux-x86_64/bin/cpack \
+                    cmake-3.31.12-linux-x86_64/bin/ctest \
+                    cmake-3.31.12-linux-x86_64/share
+            fi
+            rm -f /tmp/cmake.tar.gz
+            ln -sf /opt/cmake/bin/cmake /usr/local/bin/cmake
+            ln -sf /opt/cmake/bin/cpack /usr/local/bin/cpack
+            ln -sf /opt/cmake/bin/ctest /usr/local/bin/ctest
+            hash -r
+            cmake --version
+        fi
+        rm -f /tmp/cmake-required /tmp/cmake-present
+    fi
+EOT
+
 RUN git config --global --add safe.directory /LocalAI
 
 # Prebuild the native engine from a layer that depends on this backend's own

backend/backend.proto

@@ -536,8 +536,28 @@ message Result {
   bool success = 2;
 }
 
+// EmbeddingLayout describes whether embeddings contains one final vector or
+// a matrix of per-token vectors. Go-side pooling must never infer this from
+// tokens/dim alone: a one-token raw matrix and a final vector have the same
+// shape.
+enum EmbeddingLayout {
+  EMBEDDING_LAYOUT_UNSPECIFIED = 0;
+  EMBEDDING_LAYOUT_FINAL = 1;
+  EMBEDDING_LAYOUT_PER_TOKEN = 2;
+}
+
 message EmbeddingResult {
   repeated float embeddings = 1;
+  // Shape of the payload above: dim is the embedding width, tokens is the
+  // number of vectors packed into `embeddings` (1 when the backend pooled
+  // server-side, N with pooling:none; total across prompts if a request
+  // carried several). tokens=0/dim=0 means the backend predates shape
+  // reporting. prompt_tokens is the number of prompt tokens evaluated, for
+  // usage accounting.
+  int32 tokens = 2;
+  int32 dim = 3;
+  int32 prompt_tokens = 4;
+  EmbeddingLayout layout = 5;
 }
 
 message TranscriptRequest {

backend/cpp/audio-cpp/Makefile

@@ -9,7 +9,7 @@
 # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean
 # rebuild and so the bump bot can see the pin.
 
-AUDIO_CPP_VERSION?=7efbb58def443722ea540d931dd3debee3e4d5e8
+AUDIO_CPP_VERSION?=f6277c1695a83cf388a8282c1c1a8757cf626f18
 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/audio-cpp/capability_routing.cpp

@@ -29,6 +29,7 @@ const NamedTask kTaskNames[] = {
     {Task::VoiceDesign, "vdes"},
     {Task::SpeakerRecognition, "spk"},
     {Task::Svc, "svc"},
+    {Task::Midi, "midi"},
 };
 
 // Accepted on input but never emitted. "spkrec" was this backend's own earlier

backend/cpp/audio-cpp/capability_routing.h

@@ -25,6 +25,7 @@ enum class Task {
     VoiceDesign,
     SpeakerRecognition,
     Svc,
+    Midi,
 };
 
 // Mirrors engine::runtime::RunMode.

backend/cpp/audio-cpp/capability_routing_test.cpp

@@ -361,7 +361,7 @@ static void test_names_round_trip() {
                         Task::SourceSeparation, Task::AudioGeneration, Task::Tts,
                         Task::VoiceCloning, Task::VoiceConversion,
                         Task::SpeechToSpeech, Task::Alignment, Task::VoiceDesign,
-                        Task::SpeakerRecognition, Task::Svc};
+                        Task::SpeakerRecognition, Task::Svc, Task::Midi};
     for (const Task t : all) {
         Task parsed = Task::Vad;
         const bool ok = parse_task_name(task_name(t), parsed);

backend/cpp/audio-cpp/loaded_model.cpp

@@ -69,7 +69,8 @@ static_assert(kEngine(engine::runtime::VoiceTaskKind::VoiceDesign) == 10, "Voice
 static_assert(kEngine(engine::runtime::VoiceTaskKind::SpeakerRecognition) == 11, "VoiceTaskKind drifted");
 // The last member. Pinning it pins the member count too, as long as the
 // enumerators stay contiguous and unassigned, which upstream's declaration is.
-static_assert(kEngine(engine::runtime::VoiceTaskKind::Svc) == 12,
+static_assert(kEngine(engine::runtime::VoiceTaskKind::Svc) == 12, "VoiceTaskKind drifted");
+static_assert(kEngine(engine::runtime::VoiceTaskKind::Midi) == 13,
               "engine::runtime::VoiceTaskKind gained, lost or reordered a member. "
               "audiocpp_backend::Task mirrors it positionally: update capability_routing.h, "
               "to_engine_task and from_engine_task together, then move this pin.");
@@ -87,6 +88,7 @@ static_assert(kMirror(Task::Alignment) == 9, "Task drifted from VoiceTaskKind");
 static_assert(kMirror(Task::VoiceDesign) == 10, "Task drifted from VoiceTaskKind");
 static_assert(kMirror(Task::SpeakerRecognition) == 11, "Task drifted from VoiceTaskKind");
 static_assert(kMirror(Task::Svc) == 12, "Task drifted from VoiceTaskKind");
+static_assert(kMirror(Task::Midi) == 13, "Task drifted from VoiceTaskKind");
 
 static_assert(static_cast<int>(engine::runtime::RunMode::Offline) == 0, "RunMode drifted");
 static_assert(static_cast<int>(engine::runtime::RunMode::Streaming) == 1,
@@ -241,6 +243,7 @@ engine::runtime::VoiceTaskKind to_engine_task(Task task) {
     case Task::VoiceDesign:        return K::VoiceDesign;
     case Task::SpeakerRecognition: return K::SpeakerRecognition;
     case Task::Svc:                return K::Svc;
+    case Task::Midi:               return K::Midi;
     }
     // Unreachable for any valid enumerator. No `default:` label, so -Wswitch
     // still reports a member this switch stops covering.
@@ -263,6 +266,7 @@ Task from_engine_task(engine::runtime::VoiceTaskKind kind) {
     case K::VoiceDesign:        return Task::VoiceDesign;
     case K::SpeakerRecognition: return Task::SpeakerRecognition;
     case K::Svc:                return Task::Svc;
+    case K::Midi:               return Task::Midi;
     }
     return Task::Vad;
 }

backend/cpp/bonsai/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-BONSAI_VERSION?=9ca265a57f85f2117942490f421f64a226dd9847
+BONSAI_VERSION?=312bb2a93ea2bf798333fa859614fbf913ecb9e2
 LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp
 
 CMAKE_ARGS?=
@@ -41,7 +41,9 @@ define bonsai-build
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
+	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build llama.cpp
@@ -78,7 +80,9 @@ bonsai-cpu-all:
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
+	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build llama.cpp

backend/cpp/bonsai/patch-grpc-server.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+# Adapt the shared llama.cpp gRPC source to the older JSON API in Bonsai.
+
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+    echo "usage: $0 <grpc-server.cpp>" >&2
+    exit 2
+fi
+
+SRC=$1
+if [[ ! -f "$SRC" ]]; then
+    echo "grpc-server.cpp not found at $SRC" >&2
+    exit 2
+fi
+
+if grep -q 'common_json_error' "$SRC"; then
+    echo "==> patching $SRC to use the Bonsai JSON exception type"
+    awk '{ gsub(/common_json_error/, "json::parse_error"); print }' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+    echo "==> Bonsai JSON exception patch OK"
+else
+    echo "==> $SRC already uses a Bonsai-compatible JSON exception type, skipping"
+fi

backend/cpp/ds4/CMakeLists.txt

@@ -84,9 +84,10 @@ elseif(DS4_GPU STREQUAL "cpu")
     set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
 endif()
 
-# Upstream splits distributed inference, tensor-parallel transport, the SSD
-# expert cache, and layer placement into GPU-agnostic translation units. Link
-# them regardless of DS4_GPU.
+# Upstream splits image preprocessing, distributed inference, tensor-parallel
+# transport, the SSD expert cache, and layer placement into GPU-agnostic
+# translation units. Link them regardless of DS4_GPU.
+list(APPEND DS4_OBJS "${DS4_DIR}/ds4_image.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_distributed.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_tp.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_ssd.o")

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=b0309611041655f4e45671cfd9c9886aff161406
+# Upstream pin lives below as DS4_VERSION?=9ab705347c1775e7599ede7eb81a6255ec7dccb5
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=b0309611041655f4e45671cfd9c9886aff161406
+DS4_VERSION?=9ab705347c1775e7599ede7eb81a6255ec7dccb5
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -18,21 +18,83 @@ UNAME_S := $(shell uname -s)
 
 CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
 
-# Upstream splits distributed inference, tensor-parallel transport, the SSD
-# expert cache, and layer placement into GPU-agnostic translation units. They
-# are shared by every GPU mode, so append them unconditionally below.
+# nvcc must be told the target architecture explicitly for a cublas build, and
+# this is not a tuning knob. Upstream's Makefile leaves CUDA_ARCH empty and its
+# `cuda` target REFUSES to build without one, offering `cuda-spark`
+# (CUDA_ARCH=sm_121) and `cuda-generic` (CUDA_ARCH=native) instead. We drive its
+# object targets directly, which bypasses that guard: nvcc then compiles with no
+# -arch at all, and the kernels run as JIT'd PTX for its default architecture.
+# On GB10 (sm_121) that silently produced corrupt inference output above a
+# ~128-token prefill batch and ~77x slower prefill (4.21 t/s vs 325.70 t/s,
+# measured on the same box with the same model). No CI runner has a GPU, so
+# `native` has nothing to enumerate there.
+#
+# Upstream's CUDA_ARCH takes a SINGLE value (see its sm_120/sm_121 special cases
+# and the `-arch=$(CUDA_ARCH)` fallback), so it cannot express the fat binary
+# these images need. NVCC_ARCH_FLAGS is overridden instead: a command-line
+# assignment wins over the `:=` in upstream's Makefile, and its NVCCFLAGS
+# expands whatever we pass.
+#
+# The architecture lists are copied from backend/go/vllm-cpp/Makefile rather
+# than invented, so the two CUDA images cover the same GPUs: amd64 datacenter +
+# consumer, and l4t/arm64 covering Orin (87), Thor (110) and GB10 (121a).
+#
+# -DDS4_CUDA_HAVE_MXF4=1 is deliberately NOT set. Upstream only defines it for
+# single-arch sm_120/sm_121 builds and guards the code with a plain #ifdef
+# rather than __CUDA_ARCH__, so it cannot be combined with older archs in one
+# fat binary. It gates an optional MXFP4 indexer fast path whose #ifndef branch
+# returns 0 and falls back to the generic path, so omitting it costs some speed
+# on GB10, not correctness. Revisit if upstream adds __CUDA_ARCH__ guards.
+#
+# An EMPTY CUDA_MAJOR_VERSION means a local developer build, not CI: fall back
+# to upstream's own `native` handling, which needs a GPU present but is what a
+# developer building on their own machine wants. Both variables are `?=` so an
+# explicit value on the command line always wins.
+UNAME_M := $(shell uname -m)
+CUDA_MAJOR_VERSION ?=
+ifeq ($(BUILD_TYPE),cublas)
+ifeq ($(CUDA_MAJOR_VERSION),13)
+ifeq ($(UNAME_M),aarch64)
+    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_87,code=sm_87 \
+                           -gencode arch=compute_90a,code=sm_90a \
+                           -gencode arch=compute_100a,code=sm_100a \
+                           -gencode arch=compute_110,code=sm_110 \
+                           -gencode arch=compute_121a,code=sm_121a
+else
+    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_80,code=sm_80 \
+                           -gencode arch=compute_86,code=sm_86 \
+                           -gencode arch=compute_89,code=sm_89 \
+                           -gencode arch=compute_90a,code=sm_90a \
+                           -gencode arch=compute_100a,code=sm_100a \
+                           -gencode arch=compute_103a,code=sm_103a \
+                           -gencode arch=compute_120a,code=sm_120a \
+                           -gencode arch=compute_121a,code=sm_121a
+endif
+    DS4_ARCH_MAKEVARS := NVCC_ARCH_FLAGS="$(DS4_NVCC_ARCH_FLAGS)"
+else ifeq ($(CUDA_MAJOR_VERSION),)
+    # Local build: let upstream resolve the host GPU.
+    DS4_ARCH_MAKEVARS := CUDA_ARCH=native
+else
+    $(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (13 does). Leave it empty for a native build, or pass DS4_NVCC_ARCH_FLAGS explicitly.)
+endif
+endif
+
+# Upstream splits image preprocessing, distributed inference, tensor-parallel
+# transport, the SSD expert cache, and layer placement into GPU-agnostic
+# translation units. They are shared by every GPU mode, so append them
+# unconditionally below.
 ifeq ($(BUILD_TYPE),cublas)
     CMAKE_ARGS += -DDS4_GPU=cuda
-    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
+    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
         cuda/mmq/ds4_ggml_stubs.o cuda/mmq/ds4_mmq.o cuda/mmq/ds4_mmq_d2r.o \
         cuda/mmq/quantize.o cuda/mmq/mmid.o cuda/mmq/mmvq.o cuda/mmq/ds4_repack.o
 else ifeq ($(UNAME_S),Darwin)
     CMAKE_ARGS += -DDS4_GPU=metal
-    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
     # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
     CMAKE_ARGS += -DDS4_GPU=cpu
-    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 ifneq ($(NATIVE),true)
@@ -57,11 +119,11 @@ ds4:
 # the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
 ds4/ds4.o: ds4
 ifeq ($(BUILD_TYPE),cublas)
-	+$(MAKE) -C ds4 $(DS4_OBJ_TARGET)
+	+$(MAKE) -C ds4 $(DS4_ARCH_MAKEVARS) $(DS4_OBJ_TARGET)
 else ifeq ($(UNAME_S),Darwin)
-	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
-	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 grpc-server: ds4/ds4.o

backend/cpp/ds4/dsml_parser.cpp

@@ -92,7 +92,8 @@ std::string json_escape(const std::string &in) {
 
 } // namespace
 
-DsmlParser::DsmlParser() = default;
+DsmlParser::DsmlParser(bool starts_in_thinking)
+    : state_(starts_in_thinking ? State::THINK : State::TEXT) {}
 
 bool DsmlParser::IsInDsmlStructural() const {
     switch (state_) {

backend/cpp/ds4/dsml_parser.h

@@ -17,7 +17,9 @@ struct ParserEvent {
 // Streaming parser. Stateless across instances; one per Predict call.
 class DsmlParser {
 public:
-    DsmlParser();
+    // The chat prompt may already contain the opening thinking marker, so the
+    // generated text can begin directly with reasoning bytes.
+    explicit DsmlParser(bool starts_in_thinking = false);
 
     // Feed a chunk of raw model-emitted text. Appends classified events to
     // `out`. May buffer the tail of `chunk` internally if it looks like a
@@ -43,7 +45,7 @@ public:
 
 private:
     enum class State { TEXT, THINK, TOOL_CALLS, INVOKE, PARAM_VALUE };
-    State state_ = State::TEXT;
+    State state_;
     std::string buf_;
     std::string current_tool_name_;
     int tool_index_ = -1;

backend/cpp/ds4/dsml_parser_test.cpp

@@ -0,0 +1,133 @@
+// SPDX-License-Identifier: MIT
+// Standalone regression tests for the DSML streaming parser.
+//
+// The repository's backend/cpp/run-unit-tests.sh harness compiles each
+// *_test.cpp as a single translation unit, so include the implementation here.
+
+#include "dsml_parser.cpp"
+
+#include <cstdio>
+#include <string>
+#include <type_traits>
+#include <vector>
+
+namespace {
+
+struct ParsedText {
+    std::string content;
+    std::string reasoning;
+};
+
+int failures = 0;
+
+void check_equal(const std::string &got, const std::string &want,
+                 const char *name) {
+    if (got == want) return;
+    std::fprintf(stderr, "FAIL %s: got \"%s\", want \"%s\"\n",
+                 name, got.c_str(), want.c_str());
+    failures++;
+}
+
+void collect_text(const std::vector<ds4cpp::ParserEvent> &events,
+                  ParsedText *parsed) {
+    for (const auto &event : events) {
+        if (event.type == ds4cpp::ParserEvent::CONTENT) {
+            parsed->content += event.text;
+        } else if (event.type == ds4cpp::ParserEvent::REASONING) {
+            parsed->reasoning += event.text;
+        }
+    }
+}
+
+ParsedText parse_chunks(ds4cpp::DsmlParser *parser,
+                        const std::vector<std::string> &chunks) {
+    ParsedText parsed;
+    for (const auto &chunk : chunks) {
+        std::vector<ds4cpp::ParserEvent> events;
+        parser->Feed(chunk, events);
+        collect_text(events, &parsed);
+    }
+    std::vector<ds4cpp::ParserEvent> events;
+    parser->Flush(events);
+    collect_text(events, &parsed);
+    return parsed;
+}
+
+template <typename Parser>
+void test_reasoning_opened_by_prompt() {
+    if constexpr (!std::is_constructible_v<Parser, bool>) {
+        std::fprintf(stderr,
+                     "FAIL reasoning_opened_by_prompt: parser cannot start in thinking state\n");
+        failures++;
+    } else {
+        Parser parser(true);
+        ParsedText parsed = parse_chunks(
+            &parser,
+            {"We need to calculate factorial recursively.</think>Here is the answer."});
+        check_equal(parsed.reasoning,
+                    "We need to calculate factorial recursively.",
+                    "reasoning_opened_by_prompt:reasoning");
+        check_equal(parsed.content, "Here is the answer.",
+                    "reasoning_opened_by_prompt:content");
+    }
+}
+
+template <typename Parser>
+Parser text_parser() {
+    if constexpr (std::is_constructible_v<Parser, bool>) {
+        return Parser(false);
+    } else {
+        return Parser();
+    }
+}
+
+void test_reasoning_disabled() {
+    auto parser = text_parser<ds4cpp::DsmlParser>();
+    ParsedText parsed = parse_chunks(&parser, {"Here is the answer."});
+    check_equal(parsed.reasoning, "", "reasoning_disabled:reasoning");
+    check_equal(parsed.content, "Here is the answer.",
+                "reasoning_disabled:content");
+}
+
+void test_explicit_think_tag() {
+    auto parser = text_parser<ds4cpp::DsmlParser>();
+    ParsedText parsed = parse_chunks(
+        &parser, {"<think>reasoning</think>answer"});
+    check_equal(parsed.reasoning, "reasoning", "explicit_think_tag:reasoning");
+    check_equal(parsed.content, "answer", "explicit_think_tag:content");
+}
+
+template <typename Parser>
+void test_split_think_close_marker() {
+    if constexpr (!std::is_constructible_v<Parser, bool>) {
+        std::fprintf(stderr,
+                     "FAIL split_think_close_marker: parser cannot start in thinking state\n");
+        failures++;
+    } else {
+        Parser parser(true);
+        ParsedText parsed = parse_chunks(
+            &parser,
+            {"We need ", "to calculate ", "factorial", "</thi", "nk>",
+             "Here is ", "the answer."});
+        check_equal(parsed.reasoning, "We need to calculate factorial",
+                    "split_think_close_marker:reasoning");
+        check_equal(parsed.content, "Here is the answer.",
+                    "split_think_close_marker:content");
+    }
+}
+
+} // namespace
+
+int main() {
+    test_reasoning_opened_by_prompt<ds4cpp::DsmlParser>();
+    test_reasoning_disabled();
+    test_explicit_think_tag();
+    test_split_think_close_marker<ds4cpp::DsmlParser>();
+
+    if (failures == 0) {
+        std::fprintf(stderr, "all dsml_parser checks passed\n");
+        return 0;
+    }
+    std::fprintf(stderr, "%d check(s) failed\n", failures);
+    return 1;
+}

backend/cpp/ds4/generation_limits.h

@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+#include <algorithm>
+
+namespace ds4cpp {
+
+inline int EffectiveGenerationLimit(int requested, int context_size,
+                                    int session_position) {
+  const int limit = requested > 0 ? requested : 256;
+  const int room = context_size - session_position;
+  if (room <= 1) return 0;
+  return std::min(limit, room - 1);
+}
+
+inline int RemainingGenerationBudget(int effective_limit, int produced) {
+  if (effective_limit <= produced) return 0;
+  return effective_limit - produced;
+}
+
+inline int SpeculativeAcceptedCapacity(int remaining, int draft_allowance,
+                                       int buffer_capacity) {
+  if (remaining <= 0 || draft_allowance < 0 || buffer_capacity <= 0) return 0;
+  return std::min({remaining, draft_allowance + 1, buffer_capacity});
+}
+
+} // namespace ds4cpp

backend/cpp/ds4/generation_limits_test.cpp

@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: MIT
+
+#include "generation_limits.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+void check_equal(int got, int want, const char *name) {
+  if (got == want) return;
+  std::fprintf(stderr, "FAIL %s: got %d, want %d\n", name, got, want);
+  failures++;
+}
+
+// Mutation caught: treating omitted or negative max_tokens as unlimited instead
+// of preserving DS4's legacy 256-token default.
+void test_nonpositive_uses_legacy_default_when_space_permits() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 4096, 100), 256,
+              "zero max_tokens uses legacy default");
+  check_equal(ds4cpp::EffectiveGenerationLimit(-1, 4096, 100), 256,
+              "negative max_tokens uses legacy default");
+}
+
+// Mutation caught: applying the legacy default without clamping it to the
+// post-prefill context room and reserved slot.
+void test_legacy_default_is_clamped_by_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 300, 100), 199,
+              "legacy default is context-clamped");
+}
+
+// Mutation caught: allowing an explicitly large request to overrun the
+// post-prefill context boundary.
+void test_large_positive_limit_is_clamped_to_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32768, 32768, 100), 32667,
+              "large positive is context-clamped");
+}
+
+// Mutation caught: replacing every positive request with the legacy default
+// rather than preserving a smaller configured limit.
+void test_smaller_positive_limit_is_preserved() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(64, 4096, 100), 64,
+              "smaller positive is preserved");
+}
+
+// Mutation caught: consuming the final context slot instead of reserving it as
+// required by DS4's generation loop.
+void test_no_usable_room_returns_zero() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32, 100, 99), 0,
+              "one remaining context slot is not usable");
+}
+
+// Mutation caught: sending the original generation limit to a later
+// speculative cycle instead of subtracting tokens already produced.
+void test_remaining_budget_accounts_for_produced_tokens() {
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 4), 6,
+              "remaining budget subtracts produced tokens");
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 12), 0,
+              "remaining budget never becomes negative");
+}
+
+// Mutation caught: giving speculative evaluation capacity beyond either the
+// output budget, the draft allowance plus its first target token, or the fixed
+// accepted-token buffer.
+void test_speculative_capacity_obeys_all_bounds() {
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(3, 8, 8), 3,
+              "capacity respects remaining output budget");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 4, 8), 5,
+              "capacity includes one target token beyond draft allowance");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 8, 6), 6,
+              "capacity respects fixed buffer");
+}
+
+} // namespace
+
+int main() {
+  test_nonpositive_uses_legacy_default_when_space_permits();
+  test_legacy_default_is_clamped_by_context();
+  test_large_positive_limit_is_clamped_to_context();
+  test_smaller_positive_limit_is_preserved();
+  test_no_usable_room_returns_zero();
+  test_remaining_budget_accounts_for_produced_tokens();
+  test_speculative_capacity_obeys_all_bounds();
+
+  if (failures == 0) {
+    std::fprintf(stderr, "all generation limit checks passed\n");
+    return 0;
+  }
+  std::fprintf(stderr, "%d check(s) failed\n", failures);
+  return 1;
+}

backend/cpp/ds4/grpc-server.cpp

@@ -10,7 +10,9 @@
 
 #include "dsml_parser.h"   // populated in Task 12
 #include "dsml_renderer.h" // populated in Task 16
+#include "generation_limits.h"
 #include "kv_cache.h"      // populated in Task 17
+#include "request_lifecycle.h"
 
 extern "C" {
 #include "ds4.h"
@@ -35,6 +37,7 @@ extern "C" {
 #include <mutex>
 #include <string>
 #include <thread>
+#include <utility>
 #include <vector>
 
 using grpc::Server;
@@ -69,6 +72,21 @@ int g_route_timeout_sec = 60;
 
 std::atomic<Server *> g_server{nullptr};
 
+static bool server_context_cancelled(void *ud) {
+    return static_cast<ServerContext *>(ud)->IsCancelled();
+}
+
+static void set_session_cancel(void *target, ds4cpp::CancelCallback callback,
+                               void *userdata) noexcept {
+    ds4_session_set_cancel(static_cast<ds4_session *>(target), callback, userdata);
+}
+
+static bool request_should_continue(ds4cpp::RequestLifecycle *request,
+                                    ServerContext *context) {
+    request->ObserveContextCancellation(context->IsCancelled());
+    return request->ShouldContinue();
+}
+
 // Parse a "key:value" option string. Returns empty when no colon.
 static std::pair<std::string, std::string> split_option(const std::string &opt) {
     auto colon = opt.find(':');
@@ -238,37 +256,58 @@ static bool apply_engine_option(ds4_engine_options *opt, const std::string &key,
 
 // When acting as a distributed coordinator, block until the worker route
 // covers all layers (ds4_session_distributed_route_ready == 1) or the timeout
-// elapses. Returns an empty string on success, or an error message to return
-// to the client. No-op when not distributed.
+// elapses. No-op when not distributed.
 //
 // Takes the g_engine_mu lock by reference and RELEASES it during each poll
 // sleep. The wait can span up to g_route_timeout_sec seconds while workers
 // connect; holding g_engine_mu the whole time would block the Status/Health
 // readiness probes (they also lock g_engine_mu), making LocalAI's loader treat
 // a still-starting worker as hung.
-static std::string wait_route_ready(std::unique_lock<std::mutex> &lock) {
-    if (!g_distributed) return "";
+struct RouteWaitResult {
+    ds4cpp::RouteWaitDecision decision;
+    std::string error;
+};
+
+static RouteWaitResult wait_route_ready(std::unique_lock<std::mutex> &lock,
+                                        ServerContext *context) {
+    if (!g_distributed) return {ds4cpp::RouteWaitDecision::Ready, ""};
     char err[256] = {0};
     const int deadline_polls = g_route_timeout_sec * 10; // 100ms per poll
     for (int i = 0; i <= deadline_polls; ++i) {
         int ready = ds4_session_distributed_route_ready(g_session, err, sizeof(err));
-        if (ready == 1) return "";
-        if (ready < 0) {
-            return std::string("ds4 distributed route error: ") +
-                   (err[0] ? err : "unknown");
+        switch (ds4cpp::DecideRouteWait(ready, context->IsCancelled())) {
+        case ds4cpp::RouteWaitDecision::Ready:
+            return {ds4cpp::RouteWaitDecision::Ready, ""};
+        case ds4cpp::RouteWaitDecision::Error:
+            return {ds4cpp::RouteWaitDecision::Error,
+                    std::string("ds4 distributed route error: ") +
+                        (err[0] ? err : "unknown")};
+        case ds4cpp::RouteWaitDecision::Cancelled:
+            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+        case ds4cpp::RouteWaitDecision::Pending:
+            break;
         }
+        if (i == deadline_polls) break;
         // Release the lock while sleeping so Status/Health and other RPCs can
         // interleave during worker startup.
         lock.unlock();
         struct timespec ts = {0, 100L * 1000L * 1000L}; // 100ms
         nanosleep(&ts, nullptr);
         lock.lock();
+        if (context->IsCancelled()) {
+            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+        }
         // A concurrent Free() may have torn down the engine while we slept.
         if (!g_engine || !g_session) {
-            return "ds4: model unloaded while waiting for distributed route";
+            return {ds4cpp::RouteWaitDecision::Error,
+                    "ds4: model unloaded while waiting for distributed route"};
         }
     }
-    return "ds4 distributed route incomplete: workers not connected (layers uncovered)";
+    if (context->IsCancelled()) {
+        return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+    }
+    return {ds4cpp::RouteWaitDecision::Error,
+            "ds4 distributed route incomplete: workers not connected (layers uncovered)"};
 }
 
 static void append_token_text(ds4_engine *engine, int token, std::string &out) {
@@ -341,9 +380,9 @@ static void collect_done(void *) {}
 struct StreamCtx {
     ds4_engine *engine;
     ServerWriter<backend::Reply> *writer;
+    ds4cpp::RequestLifecycle *request;
     ds4cpp::DsmlParser parser;
     int tokens;
-    bool aborted;
     // Track which tool indices we've seen TOOL_START for, so subsequent
     // ARGS deltas can elide the redundant id/name fields.
     std::vector<bool> tool_started;
@@ -351,7 +390,7 @@ struct StreamCtx {
 
 static void stream_emit(void *ud, int token) {
     auto *s = static_cast<StreamCtx *>(ud);
-    if (s->aborted) return;
+    if (!s->request->ShouldContinue()) return;
     if (token == ds4_token_eos(s->engine)) return;
     size_t len = 0;
     const char *text = ds4_token_text(s->engine, token, &len);
@@ -401,7 +440,7 @@ static void stream_emit(void *ud, int token) {
     reply.set_message(chunk);
     reply.set_tokens(1);
     if (any_field) {
-        if (!s->writer->Write(reply)) s->aborted = true;
+        s->request->ObserveStreamWrite(s->writer->Write(reply));
     }
     s->tokens++;
 }
@@ -757,21 +796,30 @@ public:
         return GStatus::OK;
     }
 
-    GStatus Predict(ServerContext *, const backend::PredictOptions *request,
+    GStatus Predict(ServerContext *context, const backend::PredictOptions *request,
                    backend::Reply *reply) override {
         std::unique_lock<std::mutex> lock(g_engine_mu);
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
         if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
-            return GStatus(StatusCode::UNAVAILABLE, route_err);
+        RouteWaitResult route = wait_route_ready(lock, context);
+        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
+            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
+        }
+        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
+            return GStatus(StatusCode::UNAVAILABLE, route.error);
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
-        CollectCtx collect = {g_engine, "", {}, reply, 0, {}, "", ""};
+        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
+        const bool starts_in_thinking = think_enabled &&
+            request->usetokenizertemplate() && request->messages_size() > 0;
+        CollectCtx collect = {
+            g_engine, "", ds4cpp::DsmlParser(starts_in_thinking),
+            reply, 0, {}, "", ""};
+        ds4cpp::RequestLifecycle lifecycle;
         std::string cache_key = render_prompt_text(request);
         size_t cache_hit = maybe_load_cache(cache_key);
         (void)cache_hit; // future: skip prompt prefix if hit covers full prompt
@@ -783,15 +831,27 @@ public:
         // Either way g_session advances so the disk KV cache picks up a
         // real checkpoint after the call (see maybe_save_cache below).
         char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        int rc;
+        {
+            ds4cpp::CancelCallbackScope cancel_scope(
+                g_session, set_session_cancel, server_context_cancelled, context);
+            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        }
         int prompt_len = prompt.len;
         ds4_tokens_free(&prompt);
-        if (rc == 0) {
+        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
+            lifecycle.ObserveContextCancellation(true);
+        }
+        const bool generation_started = rc == 0;
+        if (generation_started) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
             int produced = 0;
             while (produced < n_predict) {
+                if (!request_should_continue(&lifecycle, context)) break;
                 SampleParams sp = compute_sample_params(request, collect.parser, think_enabled);
                 int first;
                 if (sp.temperature <= 0.0f) {
@@ -806,13 +866,20 @@ public:
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;
                     for (int j = 0; j < n; ++j) {
+                        if (!request_should_continue(&lifecycle, context)) {
+                            stop = true;
+                            break;
+                        }
                         if (accepted[j] == eos) { stop = true; break; }
                         collect_emit(&collect, accepted[j]);
                         if (++produced >= n_predict) { stop = true; break; }
@@ -821,12 +888,26 @@ public:
                 } else {
                     collect_emit(&collect, first);
                     if (++produced >= n_predict) break;
+                    if (!request_should_continue(&lifecycle, context)) break;
                     rc = ds4_session_eval(g_session, first, err, sizeof(err));
                     if (rc != 0) break;
                 }
             }
-            collect_done(&collect);
         }
+
+        request_should_continue(&lifecycle, context);
+        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
+            !lifecycle.ShouldFinalize());
+        if (!terminal.should_finalize) {
+            if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
+                return GStatus(StatusCode::INTERNAL,
+                              std::string("ds4 generation failed: ") + err);
+            }
+            return GStatus(StatusCode::CANCELLED,
+                          "ds4 request cancelled");
+        }
+        if (generation_started) collect_done(&collect);
         maybe_save_cache(cache_key);
 
         // Flush any buffered parser state.
@@ -834,7 +915,7 @@ public:
         collect.parser.Flush(events);
         apply_events(&collect, events);
 
-        if (rc != 0) {
+        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
             return GStatus(StatusCode::INTERNAL,
                           std::string("ds4 generation failed: ") + err);
         }
@@ -857,21 +938,30 @@ public:
         return GStatus::OK;
     }
 
-    GStatus PredictStream(ServerContext *, const backend::PredictOptions *request,
+    GStatus PredictStream(ServerContext *context, const backend::PredictOptions *request,
                          ServerWriter<backend::Reply> *writer) override {
         std::unique_lock<std::mutex> lock(g_engine_mu);
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
         if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
-            return GStatus(StatusCode::UNAVAILABLE, route_err);
+        RouteWaitResult route = wait_route_ready(lock, context);
+        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
+            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
+        }
+        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
+            return GStatus(StatusCode::UNAVAILABLE, route.error);
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
-        StreamCtx s = {g_engine, writer, {}, 0, false, {}};
+        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
+        const bool starts_in_thinking = think_enabled &&
+            request->usetokenizertemplate() && request->messages_size() > 0;
+        ds4cpp::RequestLifecycle lifecycle;
+        StreamCtx s = {
+            g_engine, writer, &lifecycle,
+            ds4cpp::DsmlParser(starts_in_thinking), 0, {}};
         std::string cache_key = render_prompt_text(request);
         size_t cache_hit = maybe_load_cache(cache_key);
         (void)cache_hit;
@@ -879,14 +969,26 @@ public:
         // Manual loop on g_session - see Predict() above for the rationale.
         // MTP speculative path used when ds4_engine_mtp_draft_tokens > 0.
         char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        int rc;
+        {
+            ds4cpp::CancelCallbackScope cancel_scope(
+                g_session, set_session_cancel, server_context_cancelled, context);
+            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        }
         ds4_tokens_free(&prompt);
-        if (rc == 0) {
+        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
+            lifecycle.ObserveContextCancellation(true);
+        }
+        const bool generation_started = rc == 0;
+        if (generation_started) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
             int produced = 0;
-            while (produced < n_predict && !s.aborted) {
+            while (produced < n_predict) {
+                if (!request_should_continue(&lifecycle, context)) break;
                 SampleParams sp = compute_sample_params(request, s.parser, think_enabled);
                 int first;
                 if (sp.temperature <= 0.0f) {
@@ -900,50 +1002,77 @@ public:
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;
                     for (int j = 0; j < n; ++j) {
+                        if (!request_should_continue(&lifecycle, context)) {
+                            stop = true;
+                            break;
+                        }
                         if (accepted[j] == eos) { stop = true; break; }
                         stream_emit(&s, accepted[j]);
-                        if (s.aborted) { stop = true; break; }
+                        if (!lifecycle.ShouldContinue()) { stop = true; break; }
                         if (++produced >= n_predict) { stop = true; break; }
                     }
                     if (stop) break;
                 } else {
                     stream_emit(&s, first);
-                    if (s.aborted || ++produced >= n_predict) break;
+                    if (!lifecycle.ShouldContinue() || ++produced >= n_predict) break;
+                    if (!request_should_continue(&lifecycle, context)) break;
                     rc = ds4_session_eval(g_session, first, err, sizeof(err));
                     if (rc != 0) break;
                 }
             }
-            stream_done(&s);
         }
-        maybe_save_cache(cache_key);
 
-        // Flush parser state.
-        std::vector<ds4cpp::ParserEvent> events;
-        s.parser.Flush(events);
-        if (!events.empty() && !s.aborted) {
-            backend::Reply reply;
-            auto *delta = reply.add_chat_deltas();
-            for (const auto &e : events) {
-                if (e.type == ds4cpp::ParserEvent::CONTENT) {
-                    delta->set_content(delta->content() + e.text);
-                } else if (e.type == ds4cpp::ParserEvent::REASONING) {
-                    delta->set_reasoning_content(delta->reasoning_content() + e.text);
+        request_should_continue(&lifecycle, context);
+        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
+            !lifecycle.ShouldFinalize());
+        terminal = ds4cpp::RunPostlude(
+            terminal,
+            [&]() {
+                ds4cpp::DsmlParser staged_parser = s.parser;
+                std::vector<ds4cpp::ParserEvent> events;
+                staged_parser.Flush(events);
+                bool write_succeeded = true;
+                if (!events.empty()) {
+                    backend::Reply reply;
+                    auto *delta = reply.add_chat_deltas();
+                    for (const auto &e : events) {
+                        if (e.type == ds4cpp::ParserEvent::CONTENT) {
+                            delta->set_content(delta->content() + e.text);
+                        } else if (e.type == ds4cpp::ParserEvent::REASONING) {
+                            delta->set_reasoning_content(
+                                delta->reasoning_content() + e.text);
+                        }
+                    }
+                    write_succeeded = s.writer->Write(reply);
                 }
-            }
-            s.writer->Write(reply);
-        }
+                lifecycle.ObserveStreamWrite(write_succeeded);
+                request_should_continue(&lifecycle, context);
+                if (!lifecycle.ShouldFinalize()) return false;
+                s.parser = std::move(staged_parser);
+                if (generation_started) stream_done(&s);
+                return true;
+            },
+            [&]() { maybe_save_cache(cache_key); });
 
-        if (rc != 0 && !s.aborted) {
+        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
             return GStatus(StatusCode::INTERNAL,
                           std::string("ds4 generation failed: ") + err);
         }
+        if (terminal.cause == ds4cpp::TerminalCause::Cancelled) {
+            return GStatus(StatusCode::CANCELLED,
+                          "ds4 request cancelled");
+        }
         return GStatus::OK;
     }
 

backend/cpp/ds4/request_lifecycle.h

@@ -0,0 +1,111 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+namespace ds4cpp {
+
+using CancelCallback = bool (*)(void *);
+using CancelSetter = void (*)(void *, CancelCallback, void *) noexcept;
+
+class CancelCallbackScope {
+public:
+    CancelCallbackScope(void *target, CancelSetter setter,
+                        CancelCallback callback, void *userdata) noexcept
+        : target_(target), setter_(setter) {
+        setter_(target_, callback, userdata);
+    }
+
+    ~CancelCallbackScope() noexcept {
+        setter_(target_, nullptr, nullptr);
+    }
+
+    CancelCallbackScope(const CancelCallbackScope &) = delete;
+    CancelCallbackScope &operator=(const CancelCallbackScope &) = delete;
+
+private:
+    void *target_;
+    CancelSetter setter_;
+};
+
+enum class RouteWaitDecision {
+    Pending,
+    Ready,
+    Error,
+    Cancelled,
+};
+
+inline RouteWaitDecision DecideRouteWait(int route_status, bool cancelled) {
+    if (cancelled) return RouteWaitDecision::Cancelled;
+    if (route_status > 0) return RouteWaitDecision::Ready;
+    if (route_status < 0) return RouteWaitDecision::Error;
+    return RouteWaitDecision::Pending;
+}
+
+enum class TerminalCause {
+    Success,
+    Cancelled,
+    EngineError,
+};
+
+inline TerminalCause DecideTerminalCause(bool sync_interrupted,
+                                         bool engine_error,
+                                         bool abandoned) {
+    if (sync_interrupted) return TerminalCause::Cancelled;
+    if (engine_error) return TerminalCause::EngineError;
+    if (abandoned) return TerminalCause::Cancelled;
+    return TerminalCause::Success;
+}
+
+struct TerminalDecision {
+    TerminalCause cause;
+    bool should_finalize;
+};
+
+inline TerminalDecision ResolveTerminalDecision(bool sync_interrupted,
+                                                 bool engine_error,
+                                                 bool abandoned) {
+    return {
+        DecideTerminalCause(sync_interrupted, engine_error, abandoned),
+        !sync_interrupted && !abandoned,
+    };
+}
+
+template <typename Finalize, typename Persist>
+TerminalDecision RunPostlude(TerminalDecision terminal,
+                             Finalize transactional_finalize,
+                             Persist persist) {
+    if (!terminal.should_finalize) return terminal;
+    if (!transactional_finalize()) {
+        terminal.should_finalize = false;
+        if (terminal.cause != TerminalCause::EngineError) {
+            terminal.cause = TerminalCause::Cancelled;
+        }
+        return terminal;
+    }
+    persist();
+    return terminal;
+}
+
+class RequestLifecycle {
+public:
+    void ObserveContextCancellation(bool cancelled) {
+        context_cancelled_ = context_cancelled_ || cancelled;
+    }
+
+    void ObserveStreamWrite(bool succeeded) {
+        stream_write_aborted_ = stream_write_aborted_ || !succeeded;
+    }
+
+    bool ShouldContinue() const {
+        return !context_cancelled_ && !stream_write_aborted_;
+    }
+
+    bool ShouldFinalize() const {
+        return ShouldContinue();
+    }
+
+private:
+    bool context_cancelled_ = false;
+    bool stream_write_aborted_ = false;
+};
+
+} // namespace ds4cpp

backend/cpp/ds4/request_lifecycle_test.cpp

@@ -0,0 +1,414 @@
+// SPDX-License-Identifier: MIT
+// Standalone regression tests for DS4 request cancellation policy.
+
+#include "request_lifecycle.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+struct FakeCancelTarget {
+    ds4cpp::CancelCallback callback = nullptr;
+    void *userdata = nullptr;
+    int installs = 0;
+    int clears = 0;
+};
+
+struct PostludeCounts {
+    int finalize_attempts = 0;
+    int finalize_commits = 0;
+    int cache_persists = 0;
+    bool cache_followed_commit = true;
+};
+
+ds4cpp::TerminalDecision run_fake_postlude(
+        ds4cpp::TerminalDecision terminal, bool finalize_succeeds,
+        PostludeCounts *counts) {
+    return ds4cpp::RunPostlude(
+        terminal,
+        [=]() {
+            counts->finalize_attempts++;
+            if (!finalize_succeeds) return false;
+            counts->finalize_commits++;
+            return true;
+        },
+        [=]() {
+            counts->cache_followed_commit = counts->finalize_commits == 1;
+            counts->cache_persists++;
+        });
+}
+
+bool fake_cancel(void *) {
+    return false;
+}
+
+void fake_set_cancel(void *target, ds4cpp::CancelCallback callback,
+                     void *userdata) noexcept {
+    auto *fake = static_cast<FakeCancelTarget *>(target);
+    fake->callback = callback;
+    fake->userdata = userdata;
+    if (callback) {
+        fake->installs++;
+    } else {
+        fake->clears++;
+    }
+}
+
+void check(bool condition, const char *name) {
+    if (condition) return;
+    std::fprintf(stderr, "FAIL %s\n", name);
+    failures++;
+}
+
+// Production mutation caught: treating an active request as abandoned would
+// skip its parser finalization and cache save.
+void test_active_request_continues_and_finalizes() {
+    ds4cpp::RequestLifecycle request;
+
+    check(request.ShouldContinue(), "active:continue");
+    check(request.ShouldFinalize(), "active:finalize");
+}
+
+// Production mutation caught: omitting the ServerContext cancellation branch
+// would continue decoding and finalize a partial response.
+void test_context_cancellation_stops_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+
+    request.ObserveContextCancellation(true);
+
+    check(!request.ShouldContinue(), "context_cancelled:stop");
+    check(!request.ShouldFinalize(), "context_cancelled:no_finalize");
+}
+
+// Production mutation caught: ignoring ServerWriter::Write failure would keep
+// streaming and finalize a response whose client has gone away.
+void test_stream_write_abort_stops_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+
+    request.ObserveStreamWrite(false);
+
+    check(!request.ShouldContinue(), "write_abort:stop");
+    check(!request.ShouldFinalize(), "write_abort:no_finalize");
+}
+
+// Production mutation caught: combining cancellation and write failure with
+// AND would fail to stop when either signal occurs on its own.
+void test_cancellation_and_write_abort_are_independent_or_conditions() {
+    ds4cpp::RequestLifecycle cancelled;
+    cancelled.ObserveContextCancellation(true);
+    cancelled.ObserveStreamWrite(true);
+
+    ds4cpp::RequestLifecycle write_aborted;
+    write_aborted.ObserveContextCancellation(false);
+    write_aborted.ObserveStreamWrite(false);
+
+    check(!cancelled.ShouldContinue(), "or:context_only");
+    check(!write_aborted.ShouldContinue(), "or:write_only");
+}
+
+// Production mutation caught: treating an incomplete distributed route as an
+// error would return before workers have time to connect.
+void test_route_wait_pending() {
+    check(ds4cpp::DecideRouteWait(0, false) ==
+              ds4cpp::RouteWaitDecision::Pending,
+          "route_wait:pending");
+}
+
+// Production mutation caught: failing to recognize a complete route would
+// keep a ready inference request in the polling loop.
+void test_route_wait_ready() {
+    check(ds4cpp::DecideRouteWait(1, false) ==
+              ds4cpp::RouteWaitDecision::Ready,
+          "route_wait:ready");
+}
+
+// Production mutation caught: ignoring a route probe error would poll until a
+// misleading timeout instead of returning UNAVAILABLE promptly.
+void test_route_wait_error() {
+    check(ds4cpp::DecideRouteWait(-1, false) ==
+              ds4cpp::RouteWaitDecision::Error,
+          "route_wait:error");
+}
+
+// Production mutation caught: omitting cancellation from route waiting would
+// leave an abandoned request blocked until the distributed timeout.
+void test_route_wait_cancellation() {
+    check(ds4cpp::DecideRouteWait(0, true) ==
+              ds4cpp::RouteWaitDecision::Cancelled,
+          "route_wait:cancelled");
+}
+
+// Production mutation caught: checking route errors before cancellation would
+// report UNAVAILABLE for a request the client already abandoned.
+void test_route_wait_cancellation_precedes_error() {
+    check(ds4cpp::DecideRouteWait(-1, true) ==
+              ds4cpp::RouteWaitDecision::Cancelled,
+          "route_wait:cancellation_precedence");
+}
+
+// Production mutation caught: classifying a successful active request as a
+// terminal failure would suppress its normal response finalization.
+void test_terminal_success() {
+    check(ds4cpp::DecideTerminalCause(false, false, false) ==
+              ds4cpp::TerminalCause::Success,
+          "terminal:success");
+}
+
+// Production mutation caught: treating DS4's cooperative sync interruption
+// as an ordinary engine error would return INTERNAL instead of CANCELLED.
+void test_terminal_sync_interruption_is_cancelled() {
+    check(ds4cpp::DecideTerminalCause(true, true, true) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:sync_interrupted");
+}
+
+// Production mutation caught: treating every nonzero engine result as client
+// abandonment would hide genuine DS4 failures behind CANCELLED.
+void test_terminal_engine_error() {
+    check(ds4cpp::DecideTerminalCause(false, true, false) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error");
+}
+
+// Production mutation caught: ignoring an rc==0 context cancellation would
+// finalize and cache an abandoned request.
+void test_terminal_context_abandonment() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveContextCancellation(true);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, false, !request.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:context_abandonment");
+}
+
+// Production mutation caught: ignoring an rc==0 stream write failure would
+// finalize and cache an abandoned streaming request.
+void test_terminal_write_abandonment() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveStreamWrite(false);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, false, !request.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:write_abandonment");
+}
+
+// Production mutation caught: checking late cancellation or write failure
+// before a determined ordinary DS4 error would replace INTERNAL with CANCELLED.
+void test_terminal_engine_error_precedes_late_abandonment() {
+    ds4cpp::RequestLifecycle cancelled;
+    cancelled.ObserveContextCancellation(true);
+    ds4cpp::RequestLifecycle write_aborted;
+    write_aborted.ObserveStreamWrite(false);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, true, !cancelled.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error_precedes_cancellation");
+    check(ds4cpp::DecideTerminalCause(
+              false, true, !write_aborted.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error_precedes_write_abort");
+}
+
+// Production mutation caught: using status precedence alone to gate side
+// effects would finalize and persist an engine-error request abandoned later.
+void test_abandoned_engine_error_keeps_internal_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveContextCancellation(true);
+
+    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+        false, true, !request.ShouldFinalize());
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "terminal_decision:abandoned_engine_error_status");
+    check(!terminal.should_finalize,
+          "terminal_decision:abandoned_engine_error_no_finalize");
+}
+
+// Production mutation caught: suppressing side effects for every engine error
+// would change the existing finalization and cache behavior of active failures.
+void test_active_engine_error_still_finalizes() {
+    ds4cpp::RequestLifecycle request;
+
+    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+        false, true, !request.ShouldFinalize());
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "terminal_decision:active_engine_error_status");
+    check(terminal.should_finalize,
+          "terminal_decision:active_engine_error_finalize");
+}
+
+// Production mutation caught: persisting before committed finalization would
+// cache a state whose final buffered stream reply was never completed.
+void test_postlude_active_success_commits_then_persists() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Success, true}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Success,
+          "postlude:success_outcome");
+    check(terminal.should_finalize, "postlude:success_committed");
+    check(counts.finalize_attempts == 1, "postlude:success_attempts");
+    check(counts.finalize_commits == 1, "postlude:success_commits");
+    check(counts.cache_persists == 1, "postlude:success_cache");
+    check(counts.cache_followed_commit, "postlude:success_cache_order");
+}
+
+// Production mutation caught: starting the postlude for an already-cancelled
+// request would flush buffered parser state or persist an abandoned session.
+void test_postlude_cancellation_skips_all_side_effects() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Cancelled, false}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
+          "postlude:cancelled_outcome");
+    check(counts.finalize_attempts == 0, "postlude:cancelled_attempts");
+    check(counts.finalize_commits == 0, "postlude:cancelled_commits");
+    check(counts.cache_persists == 0, "postlude:cancelled_cache");
+}
+
+// Production mutation caught: committing the live parser or cache after a
+// failed final Write would publish an abandoned streaming postlude.
+void test_postlude_finalize_failure_cancels_without_commit_or_cache() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Success, true}, false, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
+          "postlude:write_failure_outcome");
+    check(!terminal.should_finalize, "postlude:write_failure_not_committed");
+    check(counts.finalize_attempts == 1, "postlude:write_failure_attempts");
+    check(counts.finalize_commits == 0, "postlude:write_failure_commits");
+    check(counts.cache_persists == 0, "postlude:write_failure_cache");
+}
+
+// Production mutation caught: skipping the postlude for every engine error
+// would change active internal-error finalization and cache behavior.
+void test_postlude_active_engine_error_finalizes_and_persists() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, true}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:engine_error_outcome");
+    check(counts.finalize_attempts == 1, "postlude:engine_error_attempts");
+    check(counts.finalize_commits == 1, "postlude:engine_error_commits");
+    check(counts.cache_persists == 1, "postlude:engine_error_cache");
+    check(counts.cache_followed_commit, "postlude:engine_error_cache_order");
+}
+
+// Production mutation caught: replacing every failed transactional finalize
+// with cancellation would hide an already-determined engine error.
+void test_postlude_engine_error_finalize_failure_preserves_internal() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, true}, false, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:engine_error_write_failure_outcome");
+    check(!terminal.should_finalize,
+          "postlude:engine_error_write_failure_not_committed");
+    check(counts.finalize_attempts == 1,
+          "postlude:engine_error_write_failure_attempts");
+    check(counts.finalize_commits == 0,
+          "postlude:engine_error_write_failure_commits");
+    check(counts.cache_persists == 0,
+          "postlude:engine_error_write_failure_cache");
+}
+
+// Production mutation caught: status precedence must not grant side-effect
+// permission to an engine-error request that was also abandoned.
+void test_postlude_abandoned_engine_error_skips_all_side_effects() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, false}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:abandoned_engine_error_outcome");
+    check(counts.finalize_attempts == 0,
+          "postlude:abandoned_engine_error_attempts");
+    check(counts.finalize_commits == 0,
+          "postlude:abandoned_engine_error_commits");
+    check(counts.cache_persists == 0,
+          "postlude:abandoned_engine_error_cache");
+}
+
+// Production mutation caught: failing to install the request callback would
+// make DS4 prompt synchronization unable to observe client cancellation.
+void test_cancel_callback_scope_installs_callback() {
+    FakeCancelTarget target;
+    int request_context = 42;
+
+    {
+        ds4cpp::CancelCallbackScope scope(
+            &target, fake_set_cancel, fake_cancel, &request_context);
+        check(target.callback == fake_cancel, "cancel_scope:callback_installed");
+        check(target.userdata == &request_context, "cancel_scope:userdata_installed");
+        check(target.installs == 1, "cancel_scope:installed_once");
+    }
+}
+
+// Production mutation caught: failing to clear the callback at every scope
+// exit would leave DS4 pointing at a destroyed stack-owned ServerContext.
+void test_cancel_callback_scope_clears_callback() {
+    FakeCancelTarget target;
+    int request_context = 42;
+
+    {
+        ds4cpp::CancelCallbackScope scope(
+            &target, fake_set_cancel, fake_cancel, &request_context);
+    }
+
+    check(target.callback == nullptr, "cancel_scope:callback_cleared");
+    check(target.userdata == nullptr, "cancel_scope:userdata_cleared");
+    check(target.clears == 1, "cancel_scope:cleared_once");
+}
+
+} // namespace
+
+int main() {
+    test_active_request_continues_and_finalizes();
+    test_context_cancellation_stops_without_finalizing();
+    test_stream_write_abort_stops_without_finalizing();
+    test_cancellation_and_write_abort_are_independent_or_conditions();
+    test_route_wait_pending();
+    test_route_wait_ready();
+    test_route_wait_error();
+    test_route_wait_cancellation();
+    test_route_wait_cancellation_precedes_error();
+    test_terminal_success();
+    test_terminal_sync_interruption_is_cancelled();
+    test_terminal_engine_error();
+    test_terminal_context_abandonment();
+    test_terminal_write_abandonment();
+    test_terminal_engine_error_precedes_late_abandonment();
+    test_abandoned_engine_error_keeps_internal_without_finalizing();
+    test_active_engine_error_still_finalizes();
+    test_postlude_active_success_commits_then_persists();
+    test_postlude_cancellation_skips_all_side_effects();
+    test_postlude_finalize_failure_cancels_without_commit_or_cache();
+    test_postlude_active_engine_error_finalizes_and_persists();
+    test_postlude_engine_error_finalize_failure_preserves_internal();
+    test_postlude_abandoned_engine_error_skips_all_side_effects();
+    test_cancel_callback_scope_installs_callback();
+    test_cancel_callback_scope_clears_callback();
+
+    if (failures == 0) {
+        std::fprintf(stderr, "all request_lifecycle checks passed\n");
+        return 0;
+    }
+    std::fprintf(stderr, "%d check(s) failed\n", failures);
+    return 1;
+}

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=cf1aa57e1a0fabfd015831718fc99d1aec01ada5
+IK_LLAMA_VERSION?=fe215a8ccdce6b844d2a3a3bbde08ae76a6284bf
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/ik-llama-cpp/grpc-server.cpp

@@ -2565,6 +2565,7 @@ public:
     grpc::Status Embedding(ServerContext* context, const backend::PredictOptions* request, backend::EmbeddingResult* embeddingResult) {
         auto identity = checkModelIdentity(request);
         if (!identity.ok()) return identity;
+        embeddingResult->set_layout(backend::EMBEDDING_LAYOUT_FINAL);
         json data = parse_options(false, request, llama);
         const int task_id = llama.queue_tasks.get_new_id();
         llama.queue_results.add_waiting_task_id(task_id);

backend/cpp/llama-cpp/CMakeLists.txt

@@ -115,4 +115,14 @@ if(LLAMA_GRPC_BUILD_TESTS)
     target_include_directories(passthrough_options_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
     target_compile_features(passthrough_options_test PRIVATE cxx_std_17)
     add_test(NAME passthrough_options_test COMMAND passthrough_options_test)
+
+    add_executable(tts_request_options_test tts_request_options_test.cpp tts_request_options.h)
+    target_include_directories(tts_request_options_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_compile_features(tts_request_options_test PRIVATE cxx_std_17)
+    add_test(NAME tts_request_options_test COMMAND tts_request_options_test)
+
+    add_executable(thread_params_test thread_params_test.cpp thread_params.h)
+    target_include_directories(thread_params_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_compile_features(thread_params_test PRIVATE cxx_std_17)
+    add_test(NAME thread_params_test COMMAND thread_params_test)
 endif()

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=221f0f6356efe2260023208365705ec5d5a7c8f5
+LLAMA_VERSION?=465e49b9cea78a68b9c244ffb48d0ee24a82873d
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/disable-tts-task.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Mark a copied gRPC server as targeting a llama.cpp fork that does not carry
+# LocalAI's SERVER_TASK_TYPE_TTS patch. The RPCs remain present in the shared
+# protobuf service, but respond with UNIMPLEMENTED instead of referencing
+# server task types and mtmd gen-audio APIs absent from those forks.
+
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+    echo "usage: $0 <grpc-server.cpp>" >&2
+    exit 2
+fi
+
+SRC=$1
+
+if [[ ! -f "$SRC" ]]; then
+    echo "grpc-server.cpp not found at $SRC" >&2
+    exit 2
+fi
+
+if grep -q '^#define LOCALAI_LLAMA_CPP_NO_TTS_TASK' "$SRC"; then
+    echo "==> $SRC already disables the LocalAI TTS task, skipping"
+    exit 0
+fi
+
+awk '
+    !done && /^#include/ {
+        print "#define LOCALAI_LLAMA_CPP_NO_TTS_TASK 1"
+        print "// ^ injected by disable-tts-task.sh for an unpatched llama.cpp fork"
+        print ""
+        done = 1
+    }
+    { print }
+    END {
+        if (!done) {
+            print "disable-tts-task.sh: no #include anchor found" > "/dev/stderr"
+            exit 1
+        }
+    }
+' "$SRC" > "$SRC.tmp"
+mv "$SRC.tmp" "$SRC"
+
+echo "==> LocalAI TTS task disabled in $SRC"

backend/cpp/llama-cpp/grpc-server.cpp

@@ -53,8 +53,10 @@
 #include "arg.h"
 #include "chat-auto-parser.h"
 #include "llama_compat.h" // fork-skew switches, generated by prepare.sh
+#include "thread_params.h"
 #include "message_content.h"
 #include "passthrough_options.h"
+#include "tts_request_options.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
 #include <grpcpp/grpcpp.h>
@@ -65,6 +67,7 @@
 #include <atomic>
 #include <cmath>
 #include <cstdlib>
+#include <cstring>
 #include <fstream>
 #include <iterator>
 #include <list>
@@ -85,6 +88,12 @@ using grpc::ServerBuilder;
 using grpc::ServerContext;
 using grpc::Status;
 
+#if LOCALAI_HAS_MTMD_INIT_OPT
+#define LOCALAI_MTMD_INIT_OPT_ARG(value) , value
+#else
+#define LOCALAI_MTMD_INIT_OPT_ARG(value)
+#endif
+
 // gRPC bearer token auth for distributed mode.
 // Reads LOCALAI_GRPC_AUTH_TOKEN from the environment. When set, rejects
 // requests without a matching "authorization: Bearer <token>" metadata header.
@@ -233,7 +242,15 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
     data["typical_p"] = predict->typicalp();
     data["temperature"] = predict->temperature();
     data["repeat_last_n"] = predict->repeat();
-    data["repeat_penalty"] = predict->penalty();
+    // PredictOptions.Penalty is a bare proto float, so a caller that names no
+    // repetition penalty sends 0 rather than omitting the field. Since
+    // llama.cpp 9de0fcf2b, common_sampler_init() rejects a non-positive
+    // penalty_repeat outright (it would divide logits by zero), which turned
+    // every such request into "Failed to initialize samplers". Treat 0 as
+    // "unset" and leave llama.cpp's own neutral default in place.
+    if (predict->penalty() > 0.0f) {
+        data["repeat_penalty"] = predict->penalty();
+    }
     data["frequency_penalty"] = predict->frequencypenalty();
     data["presence_penalty"] = predict->presencepenalty();
     data["mirostat"] = predict->mirostat();
@@ -283,7 +300,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
             } else {
                 SRV_WRN("[TOOLS DEBUG] parse_options: Parsed tools JSON is not an array: %s\n", tools_json.dump().c_str());
             }
-        } catch (const json::parse_error& e) {
+        } catch (const common_json_error& e) {
             SRV_WRN("Failed to parse tools JSON from proto: %s\n", e.what());
             SRV_WRN("[TOOLS DEBUG] parse_options: Tools string that failed to parse: %s\n", predict->tools().c_str());
         }
@@ -313,7 +330,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
                 SRV_DBG("[TOOLS DEBUG] Received tool_choice object from Go layer: %s\n", tool_choice_json.dump().c_str());
             }
             SRV_INF("Extracted tool_choice from proto: %s\n", predict->toolchoice().c_str());
-        } catch (const json::parse_error& e) {
+        } catch (const common_json_error& e) {
             // If parsing fails, treat as string
             data["tool_choice"] = predict->toolchoice();
             SRV_INF("Extracted tool_choice as string: %s\n", predict->toolchoice().c_str());
@@ -342,7 +359,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
             // Add to data - llama.cpp server expects it as an object (map)
             data["logit_bias"] = logit_bias_json;
             SRV_INF("Using logit_bias: %s\n", predict->logitbias().c_str());
-        } catch (const json::parse_error& e) {
+        } catch (const common_json_error& e) {
             SRV_ERR("Failed to parse logit_bias JSON from proto: %s\n", e.what());
         }
     }
@@ -387,7 +404,10 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
             });
     }
 
-    data["stop"] = predict->stopprompts();
+    data["stop"] = json::array();
+    for (const auto & stop : predict->stopprompts()) {
+        data["stop"].push_back(stop);
+    }
     // data["n_probs"] = predict->nprobs();
     //TODO: images,
 
@@ -1105,14 +1125,16 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                 try {
                     int n = std::stoi(optval_str);
                     if (n < 0) n = 0;
-                    // Keep override-name storage alive for the lifetime of the params struct
-                    // (mirrors upstream arg.cpp behavior with a function-local static).
+#if LOCALAI_HAS_N_CPU_FFN_HELPER
+                    llm_add_n_cpu_ffn_overrides(n, LLM_FFN_EXPS_REGEX, params.speculative.draft.tensor_buft_overrides);
+#else
                     static std::list<std::string> buft_overrides_draft;
                     for (int i = 0; i < n; ++i) {
                         buft_overrides_draft.push_back(llm_ffn_exps_block_regex(i));
                         params.speculative.draft.tensor_buft_overrides.push_back(
                             {buft_overrides_draft.back().c_str(), ggml_backend_cpu_buffer_type()});
                     }
+#endif
                 } catch (...) {}
             }
 
@@ -1130,14 +1152,16 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                 try {
                     int n = std::stoi(optval_str);
                     if (n < 0) n = 0;
-                    // Keep override-name storage alive for the lifetime of the
-                    // params struct (mirrors upstream arg.cpp's function-local static).
+#if LOCALAI_HAS_N_CPU_FFN_HELPER
+                    llm_add_n_cpu_ffn_overrides(n, LLM_FFN_EXPS_REGEX, params.tensor_buft_overrides);
+#else
                     static std::list<std::string> buft_overrides_main;
                     for (int i = 0; i < n; ++i) {
                         buft_overrides_main.push_back(llm_ffn_exps_block_regex(i));
                         params.tensor_buft_overrides.push_back(
                             {buft_overrides_main.back().c_str(), ggml_backend_cpu_buffer_type()});
                     }
+#endif
                 } catch (...) {}
             }
 
@@ -1402,6 +1426,12 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             passthrough_draft_gpu_layers);
     }
 
+    // The library initializer now creates both threadpools before the server
+    // can apply llama_context's fallback for the -1 batch-thread sentinel.
+    params.cpuparams_batch.n_threads = llama_grpc::resolve_batch_threads(
+        params.cpuparams_batch.n_threads,
+        params.cpuparams.n_threads);
+
 #ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
     // Score-task suffix forking: reserve seq ids (and recurrent-state cells)
     // beyond the slots so one scoring call decodes all candidate tails in a
@@ -1445,6 +1475,26 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
 }
 
 
+#ifndef LOCALAI_LLAMA_CPP_NO_TTS_TASK
+// MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back raw float32 samples, but the
+// WAV header core/backend/tts.go builds around the streamed chunks announces
+// 16-bit samples, so the wire has to carry s16 or the client decodes floats as
+// integers and hears noise. The scaling matches write_wav16() in
+// tools/mtmd/mtmd-helper-gen.cpp, which is what the non-streaming path writes.
+static std::string tts_pcm_f32_to_s16(const std::string & samples) {
+    const size_t n = samples.size() / sizeof(float);
+    std::string out;
+    out.resize(n * sizeof(int16_t));
+    for (size_t i = 0; i < n; i++) {
+        float v = 0.0f;
+        std::memcpy(&v, samples.data() + i * sizeof(float), sizeof(float));
+        const int16_t s = (int16_t) (std::max(-1.0f, std::min(1.0f, v)) * 32767.0f);
+        std::memcpy(&out[i * sizeof(int16_t)], &s, sizeof(int16_t));
+    }
+    return out;
+}
+#endif
+
 // GRPC Server start
 class BackendServiceImpl final : public backend::Backend::Service {
 private:
@@ -1758,7 +1808,7 @@ public:
                         for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j));
                         for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j));
                     }
-                    messages_json.push_back(llama_grpc::build_reconstructed_message(rin));
+                    messages_json.push_back(json::parse(llama_grpc::build_reconstructed_message(rin).dump()));
                 }
 
                 // Final safety check: Ensure no message has null content (Jinja templates require strings)
@@ -1951,7 +2001,7 @@ public:
                             if (!body_json.contains("chat_template_kwargs")) {
                                 body_json["chat_template_kwargs"] = json::object();
                             }
-                            for (auto& el : ctk.items()) {
+                            for (auto el : ctk.items()) {
                                 body_json["chat_template_kwargs"][el.key()] = el.value();
                             }
                         }
@@ -2037,30 +2087,27 @@ public:
             // If not using chat templates, extract files from image_data/audio_data fields
             // (If using chat templates, files were already extracted by oaicompat_chat_params_parse)
             if (!request->usetokenizertemplate() || request->messages_size() == 0 || ctx_server.impl->chat_params.tmpls == nullptr) {
-                const auto &images_data = data.find("image_data");
-                if (images_data != data.end() && images_data->is_array())
+                if (data.contains("image_data") && data.at("image_data").is_array())
                 {
-                    for (const auto &img : *images_data)
+                    for (const auto &img : data.at("image_data"))
                     {
                         auto decoded_data = base64_decode(img["data"].get<std::string>());
                         files.push_back(decoded_data);
                     }
                 }
 
-                const auto &audio_data = data.find("audio_data");
-                if (audio_data != data.end() && audio_data->is_array())
+                if (data.contains("audio_data") && data.at("audio_data").is_array())
                 {
-                    for (const auto &audio : *audio_data)
+                    for (const auto &audio : data.at("audio_data"))
                     {
                         auto decoded_data = base64_decode(audio["data"].get<std::string>());
                         files.push_back(decoded_data);
                     }
                 }
 
-                const auto &video_data = data.find("video_data");
-                if (video_data != data.end() && video_data->is_array())
+                if (data.contains("video_data") && data.at("video_data").is_array())
                 {
-                    for (const auto &video : *video_data)
+                    for (const auto &video : data.at("video_data"))
                     {
                         auto decoded_data = base64_decode(video["data"].get<std::string>());
                         files.push_back(decoded_data);
@@ -2074,10 +2121,10 @@ public:
             std::vector<server_tokens> inputs;
             if (has_mtmd) {
                 // multimodal
-                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files));
+                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)));
             } else {
                  // Everything else, including multimodal completions.
-                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true);
+                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
             }
 
             tasks.reserve(inputs.size());
@@ -2089,15 +2136,23 @@ public:
 
                 task.tokens    = std::move(inputs[i]);
 #ifdef LOCALAI_HAS_SERVER_SCHEMA
+                // The schema evaluator no longer takes the per-slot n_ctx: upstream
+                // dropped the parameter and server-schema stopped consulting n_ctx at
+                // all, leaving the context bound to the slot. Forks that predate the
+                // server-schema split still expect it, so only this branch loses it.
                 task.params           = server_schema::eval_llama_cmpl_schema(
+                        ctx_server.impl->vocab,
+                        params_base,
+                        ctx_server.get_meta().logit_bias_eog,
+                        data);
 #else
                 task.params           = server_task::params_from_json_cmpl(
-#endif
                         ctx_server.impl->vocab,
                         params_base,
                         ctx_server.get_meta().slot_n_ctx,
                         ctx_server.get_meta().logit_bias_eog,
                         data);
+#endif
                 task.id_slot = json_value(data, "id_slot", -1);
 
                 // OAI-compat: enable autoparser (PEG-based chat parsing) so that
@@ -2325,7 +2380,7 @@ public:
                         for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j));
                         for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j));
                     }
-                    messages_json.push_back(llama_grpc::build_reconstructed_message(rin));
+                    messages_json.push_back(json::parse(llama_grpc::build_reconstructed_message(rin).dump()));
                 }
 
                 // Final safety check: Ensure no message has null content (Jinja templates require strings)
@@ -2518,7 +2573,7 @@ public:
                             if (!body_json.contains("chat_template_kwargs")) {
                                 body_json["chat_template_kwargs"] = json::object();
                             }
-                            for (auto& el : ctk.items()) {
+                            for (auto el : ctk.items()) {
                                 body_json["chat_template_kwargs"][el.key()] = el.value();
                             }
                         }
@@ -2604,11 +2659,10 @@ public:
             // If not using chat templates, extract files from image_data/audio_data fields
             // (If using chat templates, files were already extracted by oaicompat_chat_params_parse)
             if (!request->usetokenizertemplate() || request->messages_size() == 0 || ctx_server.impl->chat_params.tmpls == nullptr) {
-                const auto &images_data = data.find("image_data");
-                if (images_data != data.end() && images_data->is_array())
+                if (data.contains("image_data") && data.at("image_data").is_array())
                 {
-                    std::cout << "[PREDICT] Processing " << images_data->size() << " images" << std::endl;
-                    for (const auto &img : *images_data)
+                    std::cout << "[PREDICT] Processing " << data.at("image_data").size() << " images" << std::endl;
+                    for (const auto &img : data.at("image_data"))
                     {
                         std::cout << "[PREDICT] Processing image" << std::endl;
                         auto decoded_data = base64_decode(img["data"].get<std::string>());
@@ -2616,20 +2670,18 @@ public:
                     }
                 }
 
-                const auto &audio_data = data.find("audio_data");
-                if (audio_data != data.end() && audio_data->is_array())
+                if (data.contains("audio_data") && data.at("audio_data").is_array())
                 {
-                    for (const auto &audio : *audio_data)
+                    for (const auto &audio : data.at("audio_data"))
                     {
                         auto decoded_data = base64_decode(audio["data"].get<std::string>());
                         files.push_back(decoded_data);
                     }
                 }
 
-                const auto &video_data = data.find("video_data");
-                if (video_data != data.end() && video_data->is_array())
+                if (data.contains("video_data") && data.at("video_data").is_array())
                 {
-                    for (const auto &video : *video_data)
+                    for (const auto &video : data.at("video_data"))
                     {
                         auto decoded_data = base64_decode(video["data"].get<std::string>());
                         files.push_back(decoded_data);
@@ -2644,10 +2696,10 @@ public:
             std::vector<server_tokens> inputs;
             if (has_mtmd) {
                 // multimodal
-                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files));
+                inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)));
             } else {
                  // Everything else, including multimodal completions.
-                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true);
+                inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
             }
 
             tasks.reserve(inputs.size());
@@ -2659,15 +2711,23 @@ public:
 
                 task.tokens    = std::move(inputs[i]);
 #ifdef LOCALAI_HAS_SERVER_SCHEMA
+                // The schema evaluator no longer takes the per-slot n_ctx: upstream
+                // dropped the parameter and server-schema stopped consulting n_ctx at
+                // all, leaving the context bound to the slot. Forks that predate the
+                // server-schema split still expect it, so only this branch loses it.
                 task.params           = server_schema::eval_llama_cmpl_schema(
+                        ctx_server.impl->vocab,
+                        params_base,
+                        ctx_server.get_meta().logit_bias_eog,
+                        data);
 #else
                 task.params           = server_task::params_from_json_cmpl(
-#endif
                         ctx_server.impl->vocab,
                         params_base,
                         ctx_server.get_meta().slot_n_ctx,
                         ctx_server.get_meta().logit_bias_eog,
                         data);
+#endif
                 task.id_slot = json_value(data, "id_slot", -1);
 
                 // OAI-compat: enable autoparser (PEG-based chat parsing) so that
@@ -2826,7 +2886,7 @@ public:
         json prompt = body.at("embeddings");
 
 
-        auto tokenized_prompts = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt, true, true);
+        auto tokenized_prompts = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
         for (const auto & tokens : tokenized_prompts) {
             // this check is necessary for models that do not add BOS token to the input
             if (tokens.empty()) {
@@ -2865,42 +2925,40 @@ public:
             return grpc::Status(grpc::StatusCode::INTERNAL, all_results.error->to_json().value("message", "Error in receiving results"));
         }
 
-        // Collect responses
-        json responses = json::array();
+        // Extract the embeddings typed, straight from the task results (no
+        // JSON round-trip), and report the payload shape alongside the same
+        // flat float array as before: dim is the embedding width, tokens the
+        // number of vectors packed into `embeddings` (1 per prompt when the
+        // server pooled, one per token with pooling:none; summed across
+        // prompts if the request carried several), prompt_tokens the prompt
+        // tokens evaluated, for usage accounting. Consumers seeing 0/0 know
+        // the backend predates shape reporting.
+        int32_t n_vectors = 0;
+        int32_t dim = 0;
+        int32_t prompt_tokens = 0;
         for (auto & res : all_results.results) {
-            GGML_ASSERT(dynamic_cast<server_task_result_embd*>(res.get()) != nullptr);
-            responses.push_back(res->to_json());
-        }
-
-        std::cout << "[DEBUG] Responses size: " << responses.size() << std::endl;
-
-        // Process the responses and extract embeddings
-        for (const auto & response_elem : responses) {
-            // Check if the response has an "embedding" field
-            if (response_elem.contains("embedding")) {
-                json embedding_data = json_value(response_elem, "embedding", json::array());
-
-                if (embedding_data.is_array() && !embedding_data.empty()) {
-                    for (const auto & embedding_vector : embedding_data) {
-                        if (embedding_vector.is_array()) {
-                            for (const auto & embedding_value : embedding_vector) {
-                                embeddingResult->add_embeddings(embedding_value.get<float>());
-                            }
-                        }
-                    }
+            auto * embd_res = dynamic_cast<server_task_result_embd*>(res.get());
+            GGML_ASSERT(embd_res != nullptr);
+            prompt_tokens += embd_res->n_tokens;
+            for (const auto & vec : embd_res->embedding) {
+                for (const float value : vec) {
+                    embeddingResult->add_embeddings(value);
                 }
-            } else {
-                // Check if the response itself contains the embedding data directly
-                if (response_elem.is_array()) {
-                    for (const auto & embedding_value : response_elem) {
-                        embeddingResult->add_embeddings(embedding_value.get<float>());
-                    }
+                if (!vec.empty()) {
+                    n_vectors++;
+                    dim = (int32_t) vec.size();
                 }
             }
         }
+        embeddingResult->set_tokens(n_vectors);
+        embeddingResult->set_dim(dim);
+        embeddingResult->set_prompt_tokens(prompt_tokens);
+        embeddingResult->set_layout(
+            llama_pooling_type(ctx_server.get_llama_context()) == LLAMA_POOLING_TYPE_NONE
+                ? backend::EMBEDDING_LAYOUT_PER_TOKEN
+                : backend::EMBEDDING_LAYOUT_FINAL);
 
-
-
+        std::cout << "[DEBUG] Embedding vectors: " << n_vectors << " x " << dim << std::endl;
 
         return grpc::Status::OK;
     }
@@ -2933,7 +2991,7 @@ public:
 
             tasks.reserve(documents.size());
             for (size_t i = 0; i < documents.size(); i++) {
-                auto tmp = format_prompt_rerank(ctx_server.impl->model_tgt, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i]);
+                auto tmp = format_prompt_rerank(ctx_server.impl->model_tgt, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i] LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
                 server_task task = server_task(SERVER_TASK_TYPE_RERANK);
                 task.id = rd.queue_tasks.get_new_id();
                 task.index = i;
@@ -2954,7 +3012,7 @@ public:
         }
 
         // Collect responses
-        json responses = json::array();
+        std::vector<json> responses;
         for (auto & res : all_results.results) {
             GGML_ASSERT(dynamic_cast<server_task_result_rerank*>(res.get()) != nullptr);
             responses.push_back(res->to_json());
@@ -2967,7 +3025,7 @@ public:
         // Crop results by request.top_n if specified
         int top_n = request->top_n();
         if (top_n > 0 && top_n < static_cast<int>(responses.size())) {
-            responses = json(responses.begin(), responses.begin() + top_n);
+            responses.resize(top_n);
         }
         // Set usage information
         backend::Usage* usage = rerankResult->mutable_usage();
@@ -2994,6 +3052,229 @@ public:
         return grpc::Status::OK;
     }
 
+#ifndef LOCALAI_LLAMA_CPP_NO_TTS_TASK
+    // Builds the shared TTS task from a request. Returns a non-OK status and
+    // leaves `task` untouched when the request is malformed or the loaded model
+    // cannot synthesise audio.
+    grpc::Status prepareTTSTask(const backend::TTSRequest* request, bool stream, server_task & task) {
+        if (!ctx_server.get_meta().has_cap_tts) {
+            return grpc::Status(grpc::StatusCode::UNIMPLEMENTED,
+                "the loaded model does not support audio generation (no gen-audio mmproj)");
+        }
+
+        std::map<std::string, std::string> params(request->params().begin(), request->params().end());
+        const auto opts = llama_grpc::parse_tts_request_options(
+            request->text(),
+            request->voice(),
+            request->has_language() ? request->language() : std::string(),
+            params);
+        if (!opts.ok) {
+            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, opts.error);
+        }
+
+        auto wrapper = mtmd_helper_bitmap_init_from_file(ctx_server.impl->mctx, opts.voice_path.c_str(), false LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt));
+        if (!wrapper.bitmap) {
+            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                "failed to read speaker reference audio: " + opts.voice_path);
+        }
+
+        task.tts_inp.set_prompt(opts.text);
+        // core/backend/tts.go always sets TTSRequest.language, so has_language()
+        // is true even when the caller named no language and the string is empty.
+        // gen_audio::inp::get() already maps a stored blank to nullptr, so this
+        // guard is behavior-preserving rather than behavior-fixing. It is kept
+        // so the "unset" intent is visible at the call site instead of resting
+        // on a detail of the helper.
+        if (!opts.language.empty()) {
+            task.tts_inp.set_lang(opts.language);
+        }
+        task.tts_inp.set_speaker_ref(mtmd::bitmap_ptr(wrapper.bitmap));
+        task.tts_inp.data.top_k    = opts.top_k;
+        task.tts_inp.data.top_p    = opts.top_p;
+        task.tts_inp.data.stream   = stream;
+        task.tts_inp.data.out_type = stream
+            ? MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM   // Go prepends its own WAV header, see core/backend/tts.go
+            : MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV;
+
+        task.params.stream    = stream;
+        // -1 keeps upstream's 512-frame default. The model does not always emit
+        // its codec EOS, so a short input can otherwise generate the full cap.
+        task.params.n_predict = opts.max_frames > 0 ? opts.max_frames : -1;
+        task.params.sampling  = params_base.sampling;
+        // Both values mirror upstream's draft POST /tts handler. Note that the
+        // pair is INERT at this pin: llama_sampler_init_penalties() clamps
+        // penalty_last_n with std::max(penalty_last_n, 0), so -1 means "off",
+        // not "the whole generation", and the penalty sampler is then built
+        // disabled. No repetition penalty is actually applied.
+        //
+        // That is deliberate. Dropping the second line lets the sampling
+        // default of 64 apply and genuinely engages the 1.05 penalty, which was
+        // measured here against the model's habit of never emitting its codec
+        // EOS and running to the frame cap: 0 of 15 short requests ran away
+        // with the penalty inert, 1 of 15 with it active over the last 64
+        // tokens. It does not fix the runaway, so the line stays for parity
+        // with the draft. Use max_frames to bound the output instead.
+        task.params.sampling.penalty_repeat = 1.05f;
+        task.params.sampling.penalty_last_n = -1;
+        if (opts.top_k > 0) {
+            task.params.sampling.top_k = opts.top_k;
+        }
+        if (opts.top_p > 0) {
+            task.params.sampling.top_p = opts.top_p;
+        }
+
+        return grpc::Status::OK;
+    }
+
+    grpc::Status TTS(ServerContext* context, const backend::TTSRequest* request, backend::Result* result) override {
+        auto auth = checkAuth(context);
+        if (!auth.ok()) return auth;
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
+        if (params_base.model.path.empty()) {
+            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
+        }
+        if (request->dst().empty()) {
+            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "dst must name an output file path");
+        }
+
+        server_task task(SERVER_TASK_TYPE_TTS);
+        auto prepared = prepareTTSTask(request, /* stream= */ false, task);
+        if (!prepared.ok()) return prepared;
+
+        auto rd = ctx_server.get_response_reader();
+        task.id = rd.get_new_id();
+        rd.post_task(std::move(task));
+
+        auto should_stop = [context]() { return context->IsCancelled(); };
+
+        std::string audio;
+        while (true) {
+            auto res = rd.next(should_stop);
+            if (!res) {
+                return grpc::Status(grpc::StatusCode::CANCELLED, "TTS request cancelled");
+            }
+            if (res->is_error()) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, res->to_json().dump());
+            }
+            auto * tts_res = dynamic_cast<server_task_result_tts *>(res.get());
+            if (tts_res == nullptr) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "unexpected result type for a TTS task");
+            }
+            audio.append(tts_res->audio);
+            if (tts_res->final) {
+                break;
+            }
+        }
+
+        std::ofstream out(request->dst(), std::ios::binary | std::ios::trunc);
+        if (!out) {
+            return grpc::Status(grpc::StatusCode::INTERNAL, "failed to open output file: " + request->dst());
+        }
+        out.write(audio.data(), (std::streamsize) audio.size());
+        if (!out) {
+            return grpc::Status(grpc::StatusCode::INTERNAL, "failed to write output file: " + request->dst());
+        }
+        // Buffered data is flushed here, so a full disk or a failing device can
+        // surface for the first time on close. Reporting success then would
+        // leave a truncated file behind under the name the caller will read.
+        out.close();
+        if (!out) {
+            return grpc::Status(grpc::StatusCode::INTERNAL, "failed to close output file: " + request->dst());
+        }
+
+        result->set_success(true);
+        result->set_message("TTS audio generated");
+        return grpc::Status::OK;
+    }
+
+    grpc::Status TTSStream(ServerContext* context, const backend::TTSRequest* request, grpc::ServerWriter<backend::Reply>* writer) override {
+        auto auth = checkAuth(context);
+        if (!auth.ok()) return auth;
+        auto identity = checkModelIdentity(request);
+        if (!identity.ok()) return identity;
+        if (params_base.model.path.empty()) {
+            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
+        }
+
+        server_task task(SERVER_TASK_TYPE_TTS);
+        auto prepared = prepareTTSTask(request, /* stream= */ true, task);
+        if (!prepared.ok()) return prepared;
+
+        auto rd = ctx_server.get_response_reader();
+        task.id = rd.get_new_id();
+        rd.post_task(std::move(task));
+
+        auto should_stop = [context]() { return context->IsCancelled(); };
+
+        // core/backend/tts.go:ModelTTSStream builds the WAV header itself from
+        // the sample rate in the first reply's Message, then concatenates every
+        // Reply.Audio verbatim. So the rate goes out once, up front, and the
+        // chunks stay raw PCM.
+        //
+        // Send it before draining rather than off the first audio result: a
+        // chunk needs a whole 72-frame window, about 5.8 s of audio and far
+        // longer in wall time on CPU, and the Go side cannot emit the WAV
+        // header until this reply lands. Waiting would hold the client at zero
+        // bytes for that entire stretch. The rate is a property of the loaded
+        // model, available synchronously, so there is nothing to wait for.
+        {
+            backend::Reply header;
+            const json info = { {"sample_rate", mtmd_gen_audio_get_info(ctx_server.impl->mctx).sample_rate} };
+            header.set_message(info.dump());
+            if (!writer->Write(header)) {
+                return grpc::Status(grpc::StatusCode::CANCELLED, "client closed the TTS stream");
+            }
+        }
+
+        while (true) {
+            auto res = rd.next(should_stop);
+            if (!res) {
+                return grpc::Status(grpc::StatusCode::CANCELLED, "TTS request cancelled");
+            }
+            if (res->is_error()) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, res->to_json().dump());
+            }
+            auto * tts_res = dynamic_cast<server_task_result_tts *>(res.get());
+            if (tts_res == nullptr) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "unexpected result type for a TTS task");
+            }
+
+            if (!tts_res->audio.empty()) {
+                backend::Reply chunk;
+                chunk.set_audio(tts_pcm_f32_to_s16(tts_res->audio));
+                if (!writer->Write(chunk)) {
+                    return grpc::Status(grpc::StatusCode::CANCELLED, "client closed the TTS stream");
+                }
+            }
+
+            if (tts_res->final) {
+                break;
+            }
+        }
+
+        return grpc::Status::OK;
+    }
+#else
+    grpc::Status TTS(ServerContext* context, const backend::TTSRequest* request, backend::Result* result) override {
+        auto auth = checkAuth(context);
+        if (!auth.ok()) return auth;
+        (void) request;
+        (void) result;
+        return grpc::Status(grpc::StatusCode::UNIMPLEMENTED,
+            "TTS is unavailable in this llama.cpp fork backend");
+    }
+
+    grpc::Status TTSStream(ServerContext* context, const backend::TTSRequest* request, grpc::ServerWriter<backend::Reply>* writer) override {
+        auto auth = checkAuth(context);
+        if (!auth.ok()) return auth;
+        (void) request;
+        (void) writer;
+        return grpc::Status(grpc::StatusCode::UNIMPLEMENTED,
+            "TTSStream is unavailable in this llama.cpp fork backend");
+    }
+#endif
+
     // Score returns the model's joint log-probability of each candidate
     // continuation given a shared prompt.
     //
@@ -3328,9 +3609,15 @@ public:
         // Populate the response with metrics
         response->set_slot_id(0);
         response->set_prompt_json_for_slot("");
+#if LOCALAI_HAS_SERVER_METRICS
+        response->set_tokens_per_second(res_metrics->metrics.prompt_bucket.n_per_second());
+        response->set_tokens_generated(res_metrics->metrics.predict.count);
+        response->set_prompt_tokens_processed(res_metrics->metrics.prompt.count);
+#else
         response->set_tokens_per_second(res_metrics->n_prompt_tokens_processed ? 1.e3 / res_metrics->t_prompt_processing * res_metrics->n_prompt_tokens_processed : 0.);
         response->set_tokens_generated(res_metrics->n_tokens_predicted_total);
         response->set_prompt_tokens_processed(res_metrics->n_prompt_tokens_processed_total);
+#endif
 
 
         return grpc::Status::OK;

backend/cpp/llama-cpp/message_content.h

@@ -52,14 +52,15 @@ inline nlohmann::ordered_json normalize_message_content(const std::string& role,
 // (#7528). A multimodal user message legitimately carries a typed-part array
 // ({type:text}, {type:image_url}, ...), which must be left intact. Shared by the
 // streaming and non-streaming paths so this invariant cannot drift between them.
-inline void normalize_template_message(nlohmann::ordered_json& msg) {
+template <typename Json>
+inline void normalize_template_message(Json& msg) {
     if (!msg.contains("content")) {
         msg["content"] = ""; // templates expect the field to exist
         return;
     }
-    nlohmann::ordered_json& content = msg["content"];
+    auto& content = msg["content"];
     const std::string role = (msg.contains("role") && msg["role"].is_string())
-                                 ? msg["role"].get<std::string>()
+                                 ? msg["role"].template get<std::string>()
                                  : std::string();
     if (content.is_null()) {
         content = ""; // #7324: null would crash content[:N] slicing

backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch

@@ -1,8 +1,20 @@
+From 75220a0d74892e3315f4042274b1efa6195868d8 Mon Sep 17 00:00:00 2001
+From: Codex <codex@local>
+Date: Mon, 10 Aug 2026 23:05:52 +0000
+Subject: [PATCH 1/2] score-patch
+
+---
+ common/common.cpp               |   6 +-
+ common/common.h                 |   3 +
+ tools/server/server-context.cpp | 358 +++++++++++++++++++++++++++++++-
+ tools/server/server-task.h      |  47 +++++
+ 4 files changed, 405 insertions(+), 9 deletions(-)
+
 diff --git a/common/common.cpp b/common/common.cpp
-index 8f13217..fc584e1 100644
+index 2e3f14c..0cec0dc 100644
 --- a/common/common.cpp
 +++ b/common/common.cpp
-@@ -1591,8 +1591,10 @@ struct llama_context_params common_context_params_to_llama(const common_params &
+@@ -1636,8 +1636,10 @@ struct llama_context_params common_context_params_to_llama(const common_params &
      auto cparams = llama_context_default_params();
 
      cparams.n_ctx             = params.n_ctx;
@@ -13,13 +25,13 @@ index 8f13217..fc584e1 100644
 +    cparams.n_seq_max         = params.n_parallel + params.n_seq_score_forks;
 +    cparams.n_rs_seq          = std::max(params.speculative.need_n_rs_seq(), (uint32_t) std::max(0, params.n_rs_seq));
      cparams.n_outputs_max     = std::max(params.n_outputs_max, 0);
+     cparams.n_outputs_max_per_seq = std::max(params.n_outputs_max_per_seq, 0);
      cparams.n_batch           = params.n_batch;
-     cparams.n_ubatch          = params.n_ubatch;
 diff --git a/common/common.h b/common/common.h
-index bffc176..e313bd6 100644
+index 878534d..4001df2 100644
 --- a/common/common.h
 +++ b/common/common.h
-@@ -455,6 +455,9 @@ struct common_params {
+@@ -445,6 +445,9 @@ struct common_params {
      int32_t n_keep                =     0; // number of tokens to keep from initial prompt
      int32_t n_chunks              =    -1; // max number of chunks to process (-1 = unlimited)
      int32_t n_parallel            =     1; // number of parallel sequences to decode
@@ -28,39 +40,26 @@ index bffc176..e313bd6 100644
 +    bool    score_enabled          = false; // reserve server resources for the Score task type
      int32_t n_sequences           =     1; // number of sequences to decode
      int32_t n_outputs_max         =     0; // max outputs in a batch (0 = n_batch)
-     int32_t grp_attn_n            =     1; // group-attention factor
-diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt
-index 780df32..1d2fe8f 100644
---- a/tools/CMakeLists.txt
-+++ b/tools/CMakeLists.txt
-@@ -41,3 +41,4 @@ else()
-     add_subdirectory(fit-params)
-     add_subdirectory(results)
- endif()
-+add_subdirectory(grpc-server)
+     int32_t n_outputs_max_per_seq =     1; // max outputs per sequence
 diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
-index 715477e..de5bed8 100644
+index 3b5f6a1..d0e18e6 100644
 --- a/tools/server/server-context.cpp
 +++ b/tools/server/server-context.cpp
-@@ -49,7 +49,16 @@ static uint32_t server_n_outputs_max(const common_params & params) {
+@@ -48,6 +48,13 @@ static common_speculative_output_limits server_output_limits(const common_params
+     auto result = common_speculative_get_output_limits(
+             params.n_batch, params.n_parallel, common_speculative_n_max(&params.speculative));
 
-     const uint32_t n_outputs_per_seq = 1 + common_speculative_n_max(&params.speculative);
-
--    const uint64_t n_outputs = (uint64_t) params.n_parallel * n_outputs_per_seq;
-+    // score tasks (SERVER_TASK_TYPE_SCORE) output logits for every candidate
-+    // token, so reserve room for a bounded candidate tail per parallel slot
-+    if (!params.score_enabled) {
-+        return std::max<uint32_t>(1, std::min<uint64_t>(n_batch,
-+            (uint64_t) params.n_parallel * n_outputs_per_seq));
++    // Score tasks output logits for every candidate token, so reserve room
++    // for a bounded candidate tail per parallel slot.
++    if (params.score_enabled) {
++        result.per_seq = std::max<int32_t>(result.per_seq, 1 + SERVER_SCORE_MAX_CAND_TOKENS);
++        result.total   = std::min<int32_t>(params.n_batch, params.n_parallel * result.per_seq);
 +    }
 +
-+    const uint32_t n_outputs_score_seq = 1 + SERVER_SCORE_MAX_CAND_TOKENS;
-+
-+    const uint64_t n_outputs = (uint64_t) params.n_parallel * std::max(n_outputs_per_seq, n_outputs_score_seq);
-
-     return std::max<uint32_t>(1, std::min<uint64_t>(n_batch, n_outputs));
- }
-@@ -202,6 +211,26 @@ struct server_slot {
+     result.total   = std::max<int32_t>(1, result.total);
+     result.per_seq = std::max<int32_t>(1, result.per_seq);
+     return result;
+@@ -239,6 +246,26 @@ struct server_slot {
 
      std::vector<completion_token_output> generated_token_probs;
 
@@ -87,7 +86,7 @@ index 715477e..de5bed8 100644
      bool has_next_token = true;
      bool has_new_line   = false;
      bool truncated      = false;
-@@ -311,6 +340,10 @@ struct server_slot {
+@@ -341,6 +368,10 @@ struct server_slot {
          }
          generated_tokens.clear();
          generated_token_probs.clear();
@@ -97,8 +96,8 @@ index 715477e..de5bed8 100644
 +        score_divergence = -1;
          json_schema = json();
 
-         // clear speculative decoding stats
-@@ -2205,6 +2238,229 @@ private:
+         task_prev = std::move(task);
+@@ -2271,6 +2302,229 @@ private:
          queue_results.send(std::move(res));
      }
 
@@ -328,7 +327,7 @@ index 715477e..de5bed8 100644
      //
      // Functions to process the task
      //
-@@ -2341,6 +2597,7 @@ private:
+@@ -2407,6 +2661,7 @@ private:
              case SERVER_TASK_TYPE_INFILL:
              case SERVER_TASK_TYPE_EMBEDDING:
              case SERVER_TASK_TYPE_RERANK:
@@ -336,7 +335,7 @@ index 715477e..de5bed8 100644
                  {
                      // special case: if input is provided via CLI, tokenize it first
                      // otherwise, no need to tokenize as it's already done inside the HTTP thread
-@@ -2832,6 +3089,13 @@ private:
+@@ -2903,6 +3158,13 @@ private:
                  break; // stop any further processing
              }
          }
@@ -350,7 +349,7 @@ index 715477e..de5bed8 100644
      }
 
      void pre_decode() {
-@@ -3154,6 +3418,16 @@ private:
+@@ -3222,6 +3484,16 @@ private:
                                      n_past = std::min(n_past, slot.alora_invocation_start - 1);
                                  }
 
@@ -367,7 +366,7 @@ index 715477e..de5bed8 100644
                                  const auto n_cache_reuse = slot.task->params.n_cache_reuse;
 
                                  const bool can_cache_reuse =
-@@ -3395,8 +3669,12 @@ private:
+@@ -3455,8 +3727,12 @@ private:
 
                      bool do_checkpoint = params_base.n_ctx_checkpoints > 0;
 
@@ -382,7 +381,7 @@ index 715477e..de5bed8 100644
 
                      // make a checkpoint of the parts of the memory that cannot be rolled back.
                      // checkpoints are created only if:
-@@ -3463,10 +3741,17 @@ private:
+@@ -3444,9 +3720,16 @@ private:
                          // embedding requires all tokens in the batch to be output;
                          // MTP also wants logits at every prompt position so the
                          // streaming hook can mirror t_h_nextn into ctx_dft.
@@ -395,16 +394,12 @@ index 715477e..de5bed8 100644
 +                            slot.prompt.n_tokens() + 1 <  slot.task->n_tokens();
                          add_ok &= batch.add(slot.id,
                              cur_tok,
-                             slot.prompt.tokens.pos_next(),
--                            slot.need_embd());
-+                            slot.need_embd() || need_score_logit);
+                             /* pos       = */ slot.prompt.tokens.pos_next(),
+-                            /* output    = */ slot.need_embd(),
++                            /* output    = */ slot.need_embd() || need_score_logit,
+                             /* is_prompt = */ true);
                          slot.prompt.tokens.push_back(cur_tok);
-
-                         slot.n_prompt_tokens_processed++;
-@@ -3481,6 +3766,32 @@ private:
-                             }
-                         }
-
+@@ -3454,2 +3737,28 @@ private:
 +                        // score tasks: break at the shared-prompt boundary so the checkpoint
 +                        // below lands exactly there — the other candidates of the same
 +                        // scoring call re-process only their own tokens. Also break at the
@@ -431,10 +426,9 @@ index 715477e..de5bed8 100644
 +                            }
 +                        }
 +
-                         // process the last few tokens of the prompt separately in order to allow for a checkpoint to be created.
-                         // create checkpoints that many tokens before the end of the prompt:
-                         //  - 4 + n_ubatch
-@@ -3513,6 +3824,15 @@ private:
+                         // break at the last user message, or at user messages at least min step past the last checkpoint
+                         if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
+@@ -3573,6 +3882,15 @@ private:
                      const bool is_user_start = spans.is_user_start(n_tokens_start);
                      const bool is_last_user_message = n_tokens_start == last_user_pos;
 
@@ -450,7 +444,7 @@ index 715477e..de5bed8 100644
                      // entire prompt has been processed
                      if (slot.prompt.n_tokens() == slot.task->n_tokens()) {
                          slot.state = SLOT_STATE_DONE_PROMPT;
-@@ -3528,8 +3848,8 @@ private:
+@@ -3588,8 +3906,8 @@ private:
                          slot.init_sampler();
                      } else {
                          // skip ordinary mid-prompt checkpoints, unless the batch starts a user
@@ -461,7 +455,7 @@ index 715477e..de5bed8 100644
                              do_checkpoint = false;
                          }
                      }
-@@ -3546,10 +3866,10 @@ private:
+@@ -3606,10 +3924,10 @@ private:
                      // do not checkpoint after mtmd chunks
                      do_checkpoint = do_checkpoint && !has_mtmd;
 
@@ -474,7 +468,7 @@ index 715477e..de5bed8 100644
                              n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
                      SLT_DBG(slot, "main/do_checkpoint = %s, pos_min = %d, pos_max = %d\n", do_checkpoint ? "yes" : "no", pos_min, pos_max);
 
-@@ -3703,6 +4023,13 @@ private:
+@@ -3772,6 +4090,13 @@ private:
                  }
              }
 
@@ -488,7 +482,7 @@ index 715477e..de5bed8 100644
              if (!is_inside_view(slot.i_batch)) {
                  // the required token not in this sub-batch, skip
                  return;
-@@ -3724,6 +4051,25 @@ private:
+@@ -3793,6 +4118,25 @@ private:
                      return;
                  }
 
@@ -515,7 +509,7 @@ index 715477e..de5bed8 100644
 
                  // prompt evaluated for next-token prediction
 diff --git a/tools/server/server-task.h b/tools/server/server-task.h
-index c3eea2e..fb3c178 100644
+index 6275ec7..5bedf19 100644
 --- a/tools/server/server-task.h
 +++ b/tools/server/server-task.h
 @@ -13,10 +13,25 @@
@@ -597,3 +591,5 @@ index c3eea2e..fb3c178 100644
  struct server_task_result_error : server_task_result {
      error_type err_type = ERROR_TYPE_SERVER;
      std::string err_msg;
+--
+2.39.5

backend/cpp/llama-cpp/patches/0002-add-server-task-type-tts.patch

@@ -0,0 +1,845 @@
+diff --git a/tools/mtmd/mtmd-helper-gen.cpp b/tools/mtmd/mtmd-helper-gen.cpp
+index 1c58d3ae1..196cbd433 100644
+--- a/tools/mtmd/mtmd-helper-gen.cpp
++++ b/tools/mtmd/mtmd-helper-gen.cpp
+@@ -50,29 +50,38 @@ static llama_token find_special_token(const llama_vocab * vocab, const std::stri
+     return LLAMA_TOKEN_NULL;
+ }
+
++static void put_bytes(std::vector<char> & buf, const void * p, size_t n) {
++    const char * c = (const char *) p;
++    buf.insert(buf.end(), c, c + n);
++}
++
++// data_sz == UINT32_MAX writes the "unknown length" sentinel (streaming), same as ffmpeg does on a pipe
++static void write_wav16_header(std::vector<char> & buf, uint32_t data_sz, int32_t rate) {
++    const uint32_t riff_sz   = data_sz == UINT32_MAX ? UINT32_MAX : 36 + data_sz;
++    const uint32_t fmt_sz    = 16, byte_rate = (uint32_t) rate * 2;
++    const uint16_t fmt = 1, ch = 1, align = 2, bits = 16;
++    const uint32_t rate32    = (uint32_t) rate;
++    put_bytes(buf, "RIFF", 4); put_bytes(buf, &riff_sz, 4); put_bytes(buf, "WAVE", 4);
++    put_bytes(buf, "fmt ", 4); put_bytes(buf, &fmt_sz, 4);
++    put_bytes(buf, &fmt, 2); put_bytes(buf, &ch, 2); put_bytes(buf, &rate32, 4);
++    put_bytes(buf, &byte_rate, 4); put_bytes(buf, &align, 2); put_bytes(buf, &bits, 2);
++    put_bytes(buf, "data", 4); put_bytes(buf, &data_sz, 4);
++}
++
++static void append_wav16_pcm(std::vector<char> & buf, const float * pcm, size_t n) {
++    for (size_t i = 0; i < n; i++) {
++        int16_t s = (int16_t) (std::max(-1.0f, std::min(1.0f, pcm[i])) * 32767.0f);
++        put_bytes(buf, &s, 2);
++    }
++}
++
+ static bool write_wav16(std::vector<char> & buf, const std::vector<float> & pcm, int32_t rate) {
+     // RIFF chunk sizes are 32-bit; refuse to emit a file with a truncated header
+     if (pcm.size() > ((size_t) UINT32_MAX - 36) / 2) {
+         return false;
+     }
+-    const uint32_t data_sz   = (uint32_t) (pcm.size() * 2);
+-    const uint32_t riff_sz   = 36 + data_sz;
+-    const uint32_t fmt_sz    = 16, byte_rate = (uint32_t) rate * 2;
+-    const uint16_t fmt = 1, ch = 1, align = 2, bits = 16;
+-    const uint32_t rate32    = (uint32_t) rate;
+-    auto put = [&](const void * p, size_t n) {
+-        const char * c = (const char *) p;
+-        buf.insert(buf.end(), c, c + n);
+-    };
+-    put("RIFF", 4); put(&riff_sz, 4); put("WAVE", 4);
+-    put("fmt ", 4); put(&fmt_sz, 4);
+-    put(&fmt, 2); put(&ch, 2); put(&rate32, 4);
+-    put(&byte_rate, 4); put(&align, 2); put(&bits, 2);
+-    put("data", 4); put(&data_sz, 4);
+-    for (float v : pcm) {
+-        int16_t s = (int16_t) (std::max(-1.0f, std::min(1.0f, v)) * 32767.0f);
+-        put(&s, 2);
+-    }
++    write_wav16_header(buf, (uint32_t) (pcm.size() * 2), rate);
++    append_wav16_pcm(buf, pcm.data(), pcm.size());
+     return true;
+ }
+
+@@ -92,6 +101,8 @@ public:
+     // set out_stop on end-of-speech, h_state_out must be null if no frame is generated
+     virtual int32_t step_gen(llama_token sampled, const float * h_state_in, const float ** h_state_out, bool * out_stop) = 0;
+     virtual int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples) = 0;
++    // forces any buffered codes through code2wav now, regardless of window_frames
++    virtual int32_t flush() { return 0; }
+
+ protected:
+     llama_context * lctx;
+@@ -121,6 +132,9 @@ public:
+         prompt_batch.reset();
+         n_prompt = 0;
+         prompt_pos = 0;
++        stream = false;
++        pcm_sent = 0;
++        wav_header_sent = false;
+     }
+
+     int32_t set_input(const mtmd_helper_gen_audio_inp * inp) override {
+@@ -208,6 +222,7 @@ public:
+         top_p = inp->top_p > 0 ? inp->top_p : def.top_p;
+         seed  = inp->seed;
+         out_type = inp->out_type;
++        stream = inp->stream;
+
+         // the prompt above holds the whole text stream up to tts_eos, so every generated
+         // frame adds tts_pad on top of the codes embedding
+@@ -302,31 +317,60 @@ public:
+     }
+
+     int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples) override {
+-        if (!flush_gen_wav()) {
+-            return 1;
++        *out_sample_rate = info.sample_rate;
++
++        if (!stream) {
++            // one-shot call: force out whatever's left, regardless of window_frames
++            if (!flush_gen_wav()) {
++                return 1;
++            }
++            if (out_n_samples) {
++                *out_n_samples = (int64_t) audio_pcm.size();
++            }
++            if (out_type == MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM) {
++                *out_data     = (const char *) audio_pcm.data();
++                *out_data_len = audio_pcm.size() * sizeof(float);
++                return 0;
++            }
++            out_buf.clear();
++            if (!write_wav16(out_buf, audio_pcm, info.sample_rate)) {
++                LOG_ERR("mtmd_helper_gen_audio: output too large for WAV\n");
++                return 1;
++            }
++            *out_data     = out_buf.data();
++            *out_data_len = out_buf.size();
++            return 0;
+         }
+
+-        *out_sample_rate = info.sample_rate;
++        // streaming: only return audio produced since the previous call
++        const size_t n_new = audio_pcm.size() - pcm_sent;
+         if (out_n_samples) {
+-            *out_n_samples = (int64_t) audio_pcm.size();
++            *out_n_samples = (int64_t) n_new;
+         }
+
+         if (out_type == MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM) {
+-            *out_data     = (const char *) audio_pcm.data();
+-            *out_data_len = audio_pcm.size() * sizeof(float);
++            *out_data     = (const char *) (audio_pcm.data() + pcm_sent);
++            *out_data_len = n_new * sizeof(float);
++            pcm_sent = audio_pcm.size();
+             return 0;
+         }
+
+         out_buf.clear();
+-        if (!write_wav16(out_buf, audio_pcm, info.sample_rate)) {
+-            LOG_ERR("mtmd_helper_gen_audio: output too large for WAV\n");
+-            return 1;
++        if (!wav_header_sent) {
++            write_wav16_header(out_buf, UINT32_MAX, info.sample_rate);
++            wav_header_sent = true;
+         }
++        append_wav16_pcm(out_buf, audio_pcm.data() + pcm_sent, n_new);
++        pcm_sent = audio_pcm.size();
+         *out_data     = out_buf.data();
+         *out_data_len = out_buf.size();
+         return 0;
+     }
+
++    int32_t flush() override {
++        return flush_gen_wav() ? 0 : 1;
++    }
++
+ private:
+     bool ensure_cache() {
+         if (specials_ok) {
+@@ -370,7 +414,7 @@ private:
+             LOG_ERR("mtmd_helper_gen_audio: mmproj has no speaker/audio encoder\n");
+             return false;
+         }
+-        const std::string  marker = mtmd_default_marker();
++        const std::string  marker = mtmd_get_marker(mctx);
+         mtmd_input_text     text{ marker.c_str(), marker.size(), false, true };
+         mtmd_input_chunks * chunks = mtmd_input_chunks_init();
+         const mtmd_bitmap * bptr = bitmap;
+@@ -456,6 +500,9 @@ private:
+     std::vector<float> h_state_buf;
+     mtmd_helper_gen_audio_outtype out_type = MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV;
+     std::vector<char> out_buf;
++    bool   stream          = false;
++    size_t pcm_sent        = 0; // samples already returned by get_output()
++    bool   wav_header_sent = false;
+ };
+
+ // settings that only live in the reference's per-pack yaml, not in the checkpoint
+@@ -1024,6 +1071,14 @@ void mtmd_helper_gen_audio_reset(mtmd_helper_gen_audio * ctx) {
+     }
+ }
+
++struct mtmd_helper_gen_audio_inp mtmd_helper_gen_audio_inp_default(void) {
++    mtmd_helper_gen_audio_inp inp{};
++    inp.top_k    = 50;
++    inp.top_p    = 1.0f;
++    inp.out_type = MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV;
++    return inp;
++}
++
+ int32_t mtmd_helper_gen_audio_set_input(mtmd_helper_gen_audio * ctx, const mtmd_helper_gen_audio_inp * inp) {
+     if (!ctx->pipeline) {
+         LOG_ERR("mtmd_helper_gen_audio: unsupported or missing gen-audio pipeline\n");
+@@ -1060,3 +1115,10 @@ int32_t mtmd_helper_gen_audio_get_output(mtmd_helper_gen_audio * ctx, int32_t *
+     }
+     return ctx->pipeline->get_output(out_sample_rate, out_data, out_data_len, out_n_samples);
+ }
++
++int32_t mtmd_helper_gen_audio_flush(mtmd_helper_gen_audio * ctx) {
++    if (!ctx->pipeline) {
++        return 1;
++    }
++    return ctx->pipeline->flush();
++}
+diff --git a/tools/mtmd/mtmd-helper.h b/tools/mtmd/mtmd-helper.h
+index 832f7171a..3eaa01aab 100644
+--- a/tools/mtmd/mtmd-helper.h
++++ b/tools/mtmd/mtmd-helper.h
+@@ -175,6 +175,7 @@ enum mtmd_helper_gen_audio_outtype {
+     MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV, // WAV PCM 16-bit LE, mono
+ };
+ struct mtmd_helper_gen_audio_inp {
++    bool stream; // if true, output() must be called after each step_gen()
+     llama_seq_id seq_id;
+
+     const char * prompt;
+@@ -190,6 +191,8 @@ struct mtmd_helper_gen_audio_inp {
+     enum mtmd_helper_gen_audio_outtype out_type;
+ };
+
++MTMD_API struct mtmd_helper_gen_audio_inp mtmd_helper_gen_audio_inp_default(void);
++
+ MTMD_API mtmd_helper_gen_audio * mtmd_helper_gen_audio_init(
+                                     struct llama_context * lctx,
+                                     struct mtmd_context * mctx);
+@@ -221,6 +224,8 @@ MTMD_API int32_t mtmd_helper_gen_audio_step_gen(
+
+ // out_data valid until next get_output() or reset() call
+ // out_n_samples (optional, can be NULL) receives the number of generated PCM samples
++// if inp->stream is true: returns only audio produced since the previous call, and
++// *out_data_len == 0 whenever a full window_frames batch hasn't accumulated yet
+ MTMD_API int32_t mtmd_helper_gen_audio_get_output(
+                         mtmd_helper_gen_audio * ctx,
+                         int32_t * out_sample_rate,
+@@ -228,6 +233,10 @@ MTMD_API int32_t mtmd_helper_gen_audio_get_output(
+                         size_t * out_data_len,
+                         int64_t * out_n_samples);
+
++// forces any buffered codes through code2wav now, regardless of window_frames;
++// call once when generation has ended, before the last get_output() in stream mode
++MTMD_API int32_t mtmd_helper_gen_audio_flush(mtmd_helper_gen_audio * ctx);
++
+ #ifdef __cplusplus
+ } // extern "C"
+ #endif
+@@ -254,8 +263,41 @@ struct mtmd_helper_gen_audio_deleter {
+ };
+ using gen_audio_ptr = std::unique_ptr<mtmd_helper_gen_audio, mtmd_helper_gen_audio_deleter>;
+ struct gen_audio {
++
++    // sub-struct, RAII wrapper for mtmd_helper_gen_audio_inp
++    struct inp {
++        mtmd_helper_gen_audio_inp data = mtmd_helper_gen_audio_inp_default();
++        std::string prompt_str;
++        std::string lang_str;
++        mtmd::bitmap_ptr speaker_ref_ptr;
++
++        inp()                             = default;
++        inp(inp &&)                       = default;
++        inp & operator=(inp &&)           = default;
++        inp(const inp &)                  = delete;
++        inp & operator=(const inp &)      = delete;
++
++        void set_prompt     (std::string p)        { prompt_str = std::move(p); }
++        void set_lang       (std::string l)        { lang_str   = std::move(l); }
++        void set_speaker_ref(mtmd::bitmap_ptr bmp) { speaker_ref_ptr = std::move(bmp); }
++
++        // pointers are only valid as long as *this is alive
++        const mtmd_helper_gen_audio_inp * get() {
++            data.prompt      = prompt_str.c_str();
++            data.prompt_len  = prompt_str.size();
++            data.lang        = lang_str.empty() ? nullptr : lang_str.c_str();
++            data.speaker_ref = speaker_ref_ptr.get();
++            return &data;
++        }
++    };
++
+     gen_audio_ptr ctx;
+-    gen_audio(struct llama_context * lctx, struct mtmd_context * mctx) : ctx(mtmd_helper_gen_audio_init(lctx, mctx)) {}
++    void init(struct llama_context * lctx, struct mtmd_context * mctx) {
++        ctx.reset(mtmd_helper_gen_audio_init(lctx, mctx));
++    }
++    bool valid() const {
++        return ctx.get() != nullptr;
++    }
+     void reset() {
+         mtmd_helper_gen_audio_reset(ctx.get());
+     }
+@@ -271,6 +313,9 @@ struct gen_audio {
+     int32_t get_output(int32_t * out_sample_rate, const char ** out_data, size_t * out_data_len, int64_t * out_n_samples = nullptr) {
+         return mtmd_helper_gen_audio_get_output(ctx.get(), out_sample_rate, out_data, out_data_len, out_n_samples);
+     }
++    int32_t flush() {
++        return mtmd_helper_gen_audio_flush(ctx.get());
++    }
+ };
+
+ } // namespace mtmd_helper
+diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
+index 9069463fe..b7fa1e534 100644
+--- a/tools/server/server-context.cpp
++++ b/tools/server/server-context.cpp
+@@ -16,6 +16,7 @@
+ #include "speculative.h"
+ #include "mtmd.h"
+ #include "mtmd-helper.h"
++#include "base64.hpp"
+
+ #include <algorithm>
+ #include <cstddef>
+@@ -41,8 +42,9 @@ constexpr int HTTP_POLLING_SECONDS = 1;
+
+ static common_speculative_output_limits server_output_limits(const common_params & params) {
+     if (params.embedding ||
+-            (params.pooling_type != LLAMA_POOLING_TYPE_UNSPECIFIED && params.pooling_type != LLAMA_POOLING_TYPE_NONE)) {
+-        return { params.n_batch, 1 };
++            (params.pooling_type != LLAMA_POOLING_TYPE_UNSPECIFIED && params.pooling_type != LLAMA_POOLING_TYPE_NONE) ||
++            !params.mmproj.path.empty()) { // gen-audio (TTS) capability isn't known until the mmproj loads, size generously
++        return { params.n_batch, params.n_batch };
+     }
+
+     auto result = common_speculative_get_output_limits(
+@@ -212,6 +214,30 @@ struct server_slot {
+     mtmd_context * mctx = nullptr;
+     mtmd::batch_ptr mbatch = nullptr;
+
++    struct tts_ctx {
++        mtmd_helper::gen_audio ctx;
++        const float * h_state;
++        llama_token   sampled;
++        int32_t       n_decoded;
++        bool is_supported() const {
++            return ctx.valid();
++        }
++        void reset() {
++            // mtmd_helper_gen_audio_reset() dereferences its argument before it
++            // null-checks the pipeline, and the pipeline is only allocated for
++            // models that actually carry a gen-audio mmproj. server_slot::reset()
++            // runs for every slot of every model, so without this guard any
++            // non-TTS model segfaults during slot initialization.
++            if (is_supported()) {
++                ctx.reset();
++            }
++            h_state   = nullptr;
++            sampled   = LLAMA_TOKEN_NULL;
++            n_decoded = 0;
++        }
++    };
++    tts_ctx tts;
++
+     // speculative decoding
+     common_speculative * spec;
+
+@@ -391,6 +417,8 @@ struct server_slot {
+
+         // clear multimodal state
+         mbatch.reset();
++
++        tts.reset();
+     }
+
+     void init_sampler() const {
+@@ -829,6 +857,14 @@ public:
+     mtmd_context * mctx = nullptr;
+     const llama_vocab * vocab = nullptr;
+
++    bool has_cap_tts() const {
++        return mctx != nullptr && mtmd_gen_audio_get_info(mctx).type != MTMD_GEN_AUDIO_TYPE_NONE;
++    }
++
++    bool has_cap_chat() const {
++        return mctx == nullptr || mtmd_helper_model_can_chat(ctx_tgt, mctx);
++    }
++
+     server_queue    queue_tasks;
+     server_response queue_results;
+
+@@ -1288,6 +1324,10 @@ private:
+             slot.mctx                   = mctx;
+             slot.prompt.tokens.has_mtmd = mctx != nullptr;
+
++            if (has_cap_tts()) {
++                slot.tts.ctx.init(ctx_tgt, mctx);
++            }
++
+             SLT_TRC(slot, "new slot, n_ctx = %d\n", slot.n_ctx);
+
+             slot.callback_on_release = [this](int id_slot) {
+@@ -1748,6 +1788,28 @@ private:
+
+         SLT_DBG(slot, "launching slot : %s\n", safe_json_to_str(slot.to_json()).c_str());
+
++        if (task.type == SERVER_TASK_TYPE_TTS) {
++            GGML_ASSERT(has_cap_tts()); // should already checked in route handler
++            if (!slot.tts.is_supported()) {
++                slot.tts.ctx.init(ctx_tgt, slot.mctx);
++            }
++
++            // TTS slots never enter the shared batch: pre_decode() returns early for
++            // them and process_tts_slots() drives them instead, so they skip the
++            // prompt-cache bookkeeping that clears this sequence between requests.
++            // The gen-audio pipeline always decodes from position 0, and its own
++            // reset() only clears host-side buffers, so without this the second and
++            // later tasks on a slot decode over the previous request's tokens and
++            // step_prompt() fails immediately.
++            slot.prompt_clear();
++
++            task.tts_inp.data.seq_id = slot.id;
++            if (slot.tts.ctx.set_input(task.tts_inp.get()) != 0) {
++                send_error(task, "failed to process TTS prompt", ERROR_TYPE_SERVER);
++                return false;
++            }
++        }
++
+         // initialize samplers
+         if (task.need_sampling()) {
+             try {
+@@ -1765,6 +1827,9 @@ private:
+             // TODO: getting pre sampling logits is not yet supported with backend sampling
+             use_backend_sampling &= !need_pre_sample_logits;
+
++            // TODO: check verify if this actually works with TTS
++            use_backend_sampling &= task.type != SERVER_TASK_TYPE_TTS;
++
+             // TODO: tmp until backend sampling is fully implemented
+             if (use_backend_sampling) {
+                 llama_set_sampler(ctx_tgt, slot.id, common_sampler_get(slot.smpl.get()));
+@@ -1783,9 +1848,13 @@ private:
+
+         slot.task = std::make_unique<const server_task>(std::move(task));
+
+-        slot.state = slot.task->is_child()
+-            ? SLOT_STATE_WAIT_OTHER // wait for the parent to process prompt
+-            : SLOT_STATE_STARTED;
++        if (slot.task->type == SERVER_TASK_TYPE_TTS) {
++            slot.state = SLOT_STATE_PROCESSING_PROMPT;
++        } else {
++            slot.state = slot.task->is_child()
++                ? SLOT_STATE_WAIT_OTHER // wait for the parent to process prompt
++                : SLOT_STATE_STARTED;
++        }
+
+         // reset server kill-switch counter
+         n_empty_consecutive = 0;
+@@ -2050,6 +2119,18 @@ private:
+         queue_results.send(std::move(res));
+     }
+
++    void send_tts_result(server_slot & slot, int32_t sample_rate, const char * data, size_t data_len, bool final) {
++        auto res = std::make_unique<server_task_result_tts>();
++
++        res->id          = slot.task->id;
++        res->index       = slot.task->index;
++        res->sample_rate = sample_rate;
++        res->audio.assign(data, data_len);
++        res->final       = final;
++
++        queue_results.send(std::move(res));
++    }
++
+     void send_final_response(server_slot & slot) {
+         auto res = std::make_unique<server_task_result_cmpl_final>();
+
+@@ -2556,6 +2637,7 @@ private:
+             case SERVER_TASK_TYPE_EMBEDDING:
+             case SERVER_TASK_TYPE_RERANK:
+             case SERVER_TASK_TYPE_SCORE:
++            case SERVER_TASK_TYPE_TTS:
+                 {
+                     // special case: if input is provided via CLI, tokenize it first
+                     // otherwise, no need to tokenize as it's already done inside the HTTP thread
+@@ -3007,1 +3089,9 @@ private:
++        // note: TTS slots bypass the shared batch entirely
++        try {
++            process_tts_slots();
++        } catch (const std::exception & e) {
++            SRV_ERR("process_tts_slots() failed: %s\n", e.what());
++            abort_all_slots("process_tts_slots() failed: " + std::string(e.what()));
++        }
++
+         GGML_ASSERT(batch.slot_batched || batch.size() == 0);
+@@ -3074,10 +3164,77 @@ private:
+         }
+     }
+
++    void process_tts_slots() {
++        iterate(slots, [&](server_slot & slot) {
++            if (!slot.is_processing() || slot.task->type != SERVER_TASK_TYPE_TTS) {
++                return;
++            }
++
++            llama_set_embeddings(ctx_tgt, true);
++
++            if (slot.state == SLOT_STATE_PROCESSING_PROMPT) {
++                const int32_t ret = slot.tts.ctx.step_prompt(llama_n_batch(ctx_tgt));
++                if (ret < 0) {
++                    send_error(slot, "TTS prompt processing failed", ERROR_TYPE_SERVER);
++                    slot.release();
++                } else if (ret == 0) {
++                    slot.tts.sampled = common_sampler_sample(slot.smpl.get(), ctx_tgt, -1);
++                    common_sampler_accept(slot.smpl.get(), slot.tts.sampled, true);
++                    slot.tts.h_state = llama_get_embeddings_ith(ctx_tgt, -1);
++                    slot.state = SLOT_STATE_GENERATING;
++                }
++                return;
++            }
++
++            const int32_t n_predict = slot.task->params.n_predict > 0 ? slot.task->params.n_predict : 512;
++            if (slot.tts.n_decoded >= n_predict || llama_vocab_is_eog(vocab, slot.tts.sampled)) {
++                int32_t      sample_rate = 0;
++                const char * data        = nullptr;
++                size_t       data_len    = 0;
++                // generation truly ends here: force out any sub-window remainder still buffered
++                if (slot.tts.ctx.flush() != 0 || slot.tts.ctx.get_output(&sample_rate, &data, &data_len) != 0) {
++                    send_error(slot, "failed to finalize TTS output", ERROR_TYPE_SERVER);
++                } else {
++                    send_tts_result(slot, sample_rate, data, data_len, true);
++                }
++                slot.release();
++                return;
++            }
++
++            const float * h_state_next = nullptr;
++            if (slot.tts.ctx.step_gen(slot.tts.sampled, slot.tts.h_state, &h_state_next) != 0) {
++                send_error(slot, "TTS generation failed", ERROR_TYPE_SERVER);
++                slot.release();
++                return;
++            }
++            slot.tts.h_state = h_state_next;
++            slot.tts.n_decoded++;
++
++            slot.tts.sampled = common_sampler_sample(slot.smpl.get(), ctx_tgt, -1);
++            common_sampler_accept(slot.smpl.get(), slot.tts.sampled, true);
++
++            if (slot.task->params.stream) {
++                int32_t      sample_rate = 0;
++                const char * data        = nullptr;
++                size_t       data_len    = 0;
++                if (slot.tts.ctx.get_output(&sample_rate, &data, &data_len) != 0) {
++                    send_error(slot, "TTS streaming output failed", ERROR_TYPE_SERVER);
++                    slot.release();
++                } else if (data_len > 0) {
++                    send_tts_result(slot, sample_rate, data, data_len, false);
++                }
++            }
++        });
++    }
++
+     void pre_decode() {
+         // apply context-shift if needed
+         // TODO: simplify and improve
+         iterate(slots, [&](server_slot & slot) {
++            if (slot.task && slot.task->type == SERVER_TASK_TYPE_TTS) {
++                // TTS slots drive their own decode loop in process_tts_slots(), never enter the shared batch
++                return;
++            }
+             if (slot.state == SLOT_STATE_GENERATING && slot.prompt.n_tokens() + 1 >= slot.n_ctx) {
+                 if (!params_base.ctx_shift) {
+                     // this check is redundant (for good)
+@@ -3150,7 +3307,7 @@ private:
+
+         // determine which slots are generating and drafting
+         iterate(slots, [&](server_slot & slot) {
+-            if (slot.state != SLOT_STATE_GENERATING) {
++            if (slot.state != SLOT_STATE_GENERATING || slot.task->type == SERVER_TASK_TYPE_TTS) {
+                 return;
+             }
+
+@@ -3284,7 +3441,7 @@ private:
+                     return; // batch is full, skip remaining slots
+                 }
+
+-                if (!slot.is_processing()) {
++                if (!slot.is_processing() || slot.task->type == SERVER_TASK_TYPE_TTS) {
+                     return;
+                 }
+
+@@ -4433,6 +4590,8 @@ server_context_meta server_context::get_meta() const {
+         /* has_inp_image          */ impl->chat_params.allow_image,
+         /* has_inp_audio          */ impl->chat_params.allow_audio,
+         /* has_inp_video          */ impl->chat_params.allow_video,
++        /* has_cap_chat           */ impl->has_cap_chat(),
++        /* has_cap_tts            */ impl->has_cap_tts(),
+         /* json_ui_settings       */ impl->json_ui_settings,
+         /* slot_n_ctx             */ impl->get_slot_n_ctx(),
+         /* pooling_type           */ llama_pooling_type(impl->ctx_tgt),
+@@ -4512,6 +4671,11 @@ std::unique_ptr<server_res_generator> server_routes::handle_completions_impl(
+
+     res->set_req(&req); // will also set spipe if needed
+
++    if (!ctx_server.has_cap_chat()) {
++        res->error(format_error_response("this server does not support chat/completions", ERROR_TYPE_NOT_SUPPORTED));
++        return res;
++    }
++
+     int32_t sse_ping_interval = params.sse_ping_interval;
+
+     try {
+@@ -5399,6 +5563,150 @@ void server_routes::init_routes() {
+         return res;
+     };
+
++    this->post_tts = [this](const server_http_req & req) {
++        auto res = create_response();
++        res->set_req(&req); // will also set spipe if needed
++
++        if (!ctx_server.has_cap_tts()) {
++            res->error(format_error_response("this server does not support audio generation", ERROR_TYPE_NOT_SUPPORTED));
++            return res;
++        }
++
++        const json body = json::parse(req.body);
++
++        std::string prompt = json_value(body, "input", json_value(body, "prompt", std::string()));
++        if (prompt.empty()) {
++            res->error(format_error_response("\"input\" must be a non-empty string", ERROR_TYPE_INVALID_REQUEST));
++            return res;
++        }
++
++        const std::string response_format = json_value(body, "response_format", std::string("wav"));
++        const bool stream = json_value(body, "stream", false);
++
++        server_task task(SERVER_TASK_TYPE_TTS);
++        task.tts_inp.set_prompt(prompt);
++        task.tts_inp.set_lang(json_value(body, "lang", std::string()));
++        task.tts_inp.data.top_k     = json_value(body, "top_k", 0);
++        task.tts_inp.data.top_p     = json_value(body, "top_p", 0.0f);
++        task.tts_inp.data.stream    = stream;
++        task.tts_inp.data.out_type  = response_format == "pcm"
++            ? MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM
++            : MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV;
++        task.params.stream    = stream;
++        task.params.n_predict = json_value(body, "n_predict", -1);
++        task.params.sampling  = params.sampling; // baseline defaults, then apply overrides below
++        task.params.sampling.penalty_repeat  = json_value(body, "repeat_penalty", 1.05f);
++        task.params.sampling.penalty_last_n  = -1;
++        if (task.tts_inp.data.top_k > 0) {
++            task.params.sampling.top_k = task.tts_inp.data.top_k;
++        }
++        if (task.tts_inp.data.top_p > 0) {
++            task.params.sampling.top_p = task.tts_inp.data.top_p;
++        }
++
++        // speaker reference: either an uploaded form file ("speaker_ref") or a base64 JSON field ("speaker_ref_b64")
++        const unsigned char * speaker_ref_data = nullptr;
++        size_t speaker_ref_len  = 0;
++        std::string speaker_ref_b64_decoded;
++
++        auto speaker_ref_file = req.files.find("speaker_ref");
++        if (speaker_ref_file != req.files.end()) {
++            speaker_ref_data = speaker_ref_file->second.data.data();
++            speaker_ref_len  = speaker_ref_file->second.data.size();
++        } else {
++            std::string speaker_ref_b64 = json_value(body, "speaker_ref_b64", std::string());
++            if (!speaker_ref_b64.empty()) {
++                speaker_ref_b64_decoded = base64::decode(speaker_ref_b64);
++                speaker_ref_data = (const unsigned char *) speaker_ref_b64_decoded.data();
++                speaker_ref_len  = speaker_ref_b64_decoded.size();
++            }
++        }
++
++        if (speaker_ref_len > 0) {
++            auto wrapper = mtmd_helper_bitmap_init_from_buf(ctx_server.mctx, speaker_ref_data, speaker_ref_len, false, ctx_server.init_opt);
++            if (!wrapper.bitmap) {
++                res->error(format_error_response("failed to decode \"speaker_ref\"", ERROR_TYPE_INVALID_REQUEST));
++                return res;
++            }
++            task.tts_inp.set_speaker_ref(mtmd::bitmap_ptr(wrapper.bitmap));
++        } else {
++            // SRV_WRN expands __VA_ARGS__ without the GNU comma-elision extension,
++            // so a bare format string leaves a trailing comma and will not compile
++            SRV_WRN("%s", "no speaker reference provided, the model may behave randomly\n");
++        }
++
++        auto & rd = res->rd;
++        task.id = rd.get_new_id();
++        rd.post_task(std::move(task));
++
++        const std::string content_type = response_format == "pcm" ? "audio/L16" : "audio/wav";
++
++        if (!stream) {
++            auto result = rd.next(req.should_stop);
++            if (!result) {
++                GGML_ASSERT(req.should_stop());
++                return res; // connection is closed
++            }
++            if (result->is_error()) {
++                res->error(result->to_json());
++                return res;
++            }
++            auto * tts_res = dynamic_cast<server_task_result_tts *>(result.get());
++            GGML_ASSERT(tts_res != nullptr);
++            res->status       = 200;
++            res->content_type = content_type;
++            res->data         = std::move(tts_res->audio);
++            return res;
++        } else {
++            auto first_result = rd.next(req.should_stop);
++            if (!first_result) {
++                GGML_ASSERT(req.should_stop());
++                return res; // connection is closed
++            }
++            if (first_result->is_error()) {
++                res->error(first_result->to_json());
++                return res;
++            }
++            auto * first_tts_res = dynamic_cast<server_task_result_tts *>(first_result.get());
++            GGML_ASSERT(first_tts_res != nullptr);
++
++            res->status       = 200;
++            res->content_type = content_type;
++            res->data          = std::move(first_tts_res->audio);
++            bool is_done       = first_tts_res->final;
++
++            res->set_next([res_this = res.get(), is_done](std::string & output) mutable -> bool {
++                if (is_done) {
++                    return false;
++                }
++                if (res_this->should_stop()) {
++                    return false;
++                }
++                if (!res_this->data.empty()) {
++                    output = std::move(res_this->data);
++                    res_this->data.clear();
++                    return true;
++                }
++
++                server_response_reader & rd = res_this->rd;
++                if (!rd.has_next()) {
++                    return false;
++                }
++                auto result = rd.next([&res_this]() { return res_this->should_stop(); });
++                if (!result || result->is_error()) {
++                    return false;
++                }
++                auto * tts_res = dynamic_cast<server_task_result_tts *>(result.get());
++                GGML_ASSERT(tts_res != nullptr);
++                output  = std::move(tts_res->audio);
++                is_done = tts_res->final;
++                return true;
++            });
++        }
++
++        return res;
++    };
++
+     this->get_lora_adapters = [this](const server_http_req & req) {
+         auto res = create_response();
+
+diff --git a/tools/server/server-context.h b/tools/server/server-context.h
+index f9ab1132b..610512678 100644
+--- a/tools/server/server-context.h
++++ b/tools/server/server-context.h
+@@ -22,6 +22,8 @@ struct server_context_meta {
+     bool has_inp_image;
+     bool has_inp_audio;
+     bool has_inp_video;
++    bool has_cap_chat;
++    bool has_cap_tts;
+     json json_ui_settings;
+     int slot_n_ctx;
+     enum llama_pooling_type pooling_type;
+@@ -151,6 +153,7 @@ struct server_routes {
+     server_http_context::handler_t post_embeddings;
+     server_http_context::handler_t post_embeddings_oai;
+     server_http_context::handler_t post_rerank;
++    server_http_context::handler_t post_tts;
+     server_http_context::handler_t get_lora_adapters;
+     server_http_context::handler_t post_lora_adapters;
+
+diff --git a/tools/server/server-task.cpp b/tools/server/server-task.cpp
+index 1ee677553..939630b8b 100644
+--- a/tools/server/server-task.cpp
++++ b/tools/server/server-task.cpp
+@@ -1497,6 +1497,17 @@ json server_task_result_rerank::to_json() {
+     };
+ }
+
++//
++// server_task_result_tts
++//
++json server_task_result_tts::to_json() {
++    return json {
++        {"sample_rate", sample_rate},
++        {"n_bytes",     audio.size()},
++        {"final",       final},
++    };
++}
++
+ //
+ // server_task_result_error
+ //
+diff --git a/tools/server/server-task.h b/tools/server/server-task.h
+index 5bedf1987..e6ca67a65 100644
+--- a/tools/server/server-task.h
++++ b/tools/server/server-task.h
+@@ -10,6 +10,7 @@
+
+ // TODO: prevent including the whole server-common.h as we only use server_tokens
+ #include "server-common.h"
++#include "mtmd-helper.h"
+
+ using json = nlohmann::ordered_json;
+
+@@ -42,6 +43,7 @@ enum server_task_type {
+     SERVER_TASK_TYPE_SLOT_ERASE,
+     SERVER_TASK_TYPE_GET_LORA,
+     SERVER_TASK_TYPE_SET_LORA,
++    SERVER_TASK_TYPE_TTS,
+ };
+
+ // TODO: change this to more generic "response_format" to replace the "format_response_*" in server-common
+@@ -202,6 +204,9 @@ struct server_task {
+     // used by SERVER_TASK_TYPE_SET_LORA
+     std::map<int, float> set_lora; // mapping adapter ID -> scale
+
++    // used by SERVER_TASK_TYPE_TTS
++    mtmd_helper::gen_audio::inp tts_inp;
++
+     server_task() = default;
+
+     server_task(server_task_type type) : type(type) {}
+@@ -235,6 +240,7 @@ struct server_task {
+         switch (type) {
+             case SERVER_TASK_TYPE_COMPLETION:
+             case SERVER_TASK_TYPE_INFILL:
++            case SERVER_TASK_TYPE_TTS:
+                 return true;
+             default:
+                 return false;
+@@ -494,5 +500,15 @@ struct server_task_result_embd : server_task_result {
+     json to_json_oaicompat();
+ };
+
++struct server_task_result_tts : server_task_result {
++    std::string audio; // raw bytes for this chunk (WAV or PCM, per request's out_type)
++    int32_t     sample_rate = 0;
++    bool        final = false; // true for the last chunk of a request
++
++    virtual bool is_stop() override { return final; }
++
++    virtual json to_json() override;
++};
++
+ struct server_task_result_rerank : server_task_result {
+     float score = -1e6;

backend/cpp/llama-cpp/prepare.sh

@@ -15,6 +15,30 @@ if [ -d "patches" ]; then
     done
 fi
 
+## Apple RDMA link fixup.
+
+## ggml-rpc hands Apple's librdma to the linker with
+## target_link_options(ggml-rpc PRIVATE "LINKER:-weak_library,..."). Link options are not
+## a usage requirement of a static library, so in our BUILD_SHARED_LIBS=OFF build the flag
+## dies with libggml-rpc.a and every ibv_* symbol transport-apple.cpp reaches for comes out
+## undefined when grpc-server and ggml-rpc-server link. Re-declare the same weak link as
+## INTERFACE so it travels to whoever links the static library.
+##
+## Guarded on the marker so a second prepare.sh over the same checkout is a no-op, and on
+## GGML_RPC_RDMA_APPLE so forks that branched before the Apple RDMA transport (turboquant,
+## bonsai) are left alone.
+RPC_CMAKE=llama.cpp/ggml/src/ggml-rpc/CMakeLists.txt
+if [ -f "$RPC_CMAKE" ] && grep -q "GGML_RPC_RDMA_APPLE" "$RPC_CMAKE" && ! grep -q "LOCALAI_RDMA_IFACE" "$RPC_CMAKE"; then
+    echo "==> ggml-rpc carries the Apple RDMA transport, re-declaring its weak librdma link as INTERFACE"
+    cat >> "$RPC_CMAKE" <<'EOF'
+
+# LOCALAI_RDMA_IFACE: added by backend/cpp/llama-cpp/prepare.sh
+if (GGML_RPC_RDMA AND APPLE AND NOT BUILD_SHARED_LIBS)
+    target_link_options(ggml-rpc INTERFACE "LINKER:-weak_library,${RDMA_LIB}")
+endif()
+EOF
+fi
+
 for file in $(ls llama.cpp/tools/server/); do
     cp -rfv llama.cpp/tools/server/$file llama.cpp/tools/grpc-server/
 done
@@ -28,6 +52,13 @@ cp -r message_content_test.cpp llama.cpp/tools/grpc-server/
 # Generic passthrough parser staging and its standalone regression test.
 cp -r passthrough_options.h llama.cpp/tools/grpc-server/
 cp -r passthrough_options_test.cpp llama.cpp/tools/grpc-server/
+# TTS request validation (included by grpc-server.cpp) and its standalone
+# regression test.
+cp -r tts_request_options.h llama.cpp/tools/grpc-server/
+cp -r tts_request_options_test.cpp llama.cpp/tools/grpc-server/
+# Thread-count default normalization and its standalone regression test.
+cp -r thread_params.h llama.cpp/tools/grpc-server/
+cp -r thread_params_test.cpp llama.cpp/tools/grpc-server/
 # Parent-death watcher (included by grpc-server.cpp) and its standalone unit
 # test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest).
 cp -r parent_watch.h llama.cpp/tools/grpc-server/
@@ -49,10 +80,28 @@ else
     echo "==> llama.cpp predates the load-mode enum, using the legacy mmap/mlock/direct-io booleans"
     LEGACY_LOAD_MODE=1
 fi
+if grep -q "server_metrics metrics;" llama.cpp/tools/server/server-task.h; then
+    HAS_SERVER_METRICS=1
+else
+    HAS_SERVER_METRICS=0
+fi
+if grep -q "mtmd_helper_init_opt" llama.cpp/tools/mtmd/mtmd-helper.h; then
+    HAS_MTMD_INIT_OPT=1
+else
+    HAS_MTMD_INIT_OPT=0
+fi
+if grep -q "llm_add_n_cpu_ffn_overrides" llama.cpp/common/common.h; then
+    HAS_N_CPU_FFN_HELPER=1
+else
+    HAS_N_CPU_FFN_HELPER=0
+fi
 cat > llama.cpp/tools/grpc-server/llama_compat.h <<EOF
 // Generated by backend/cpp/llama-cpp/prepare.sh. Do not edit.
 #pragma once
 #define LOCALAI_LEGACY_LOAD_MODE ${LEGACY_LOAD_MODE}
+#define LOCALAI_HAS_SERVER_METRICS ${HAS_SERVER_METRICS}
+#define LOCALAI_HAS_MTMD_INIT_OPT ${HAS_MTMD_INIT_OPT}
+#define LOCALAI_HAS_N_CPU_FFN_HELPER ${HAS_N_CPU_FFN_HELPER}
 EOF
 
 set +e

backend/cpp/llama-cpp/thread_params.h

@@ -0,0 +1,11 @@
+#pragma once
+
+#include <cstdint>
+
+namespace llama_grpc {
+
+inline int32_t resolve_batch_threads(int32_t batch_threads, int32_t inference_threads) {
+    return batch_threads < 0 ? inference_threads : batch_threads;
+}
+
+} // namespace llama_grpc

backend/cpp/llama-cpp/thread_params_test.cpp

@@ -0,0 +1,15 @@
+#include "thread_params.h"
+
+#include <cstdio>
+
+int main() {
+    if (llama_grpc::resolve_batch_threads(-1, 4) != 4) {
+        std::fprintf(stderr, "default batch threads did not inherit inference threads\n");
+        return 1;
+    }
+    if (llama_grpc::resolve_batch_threads(2, 4) != 2) {
+        std::fprintf(stderr, "explicit batch threads were overwritten\n");
+        return 1;
+    }
+    return 0;
+}

backend/cpp/llama-cpp/tts_request_options.h

@@ -0,0 +1,149 @@
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include <cstdint>
+#include <exception>
+#include <map>
+#include <string>
+
+namespace llama_grpc {
+
+// Validated, parsed form of a backend::TTSRequest, kept free of llama.cpp,
+// mtmd and gRPC headers so backend/cpp/run-unit-tests.sh can compile it as a
+// standalone translation unit. grpc-server.cpp turns this into a
+// mtmd_helper::gen_audio::inp.
+struct tts_request_options {
+    bool        ok = false;
+    std::string error;
+
+    std::string text;
+    std::string voice_path;
+    std::string language;
+
+    // 0 / 0.0f mean "unset": upstream only overrides the sampler defaults when
+    // the value is strictly positive.
+    int32_t top_k = 0;
+    float   top_p = 0.0f;
+
+    // Upper bound on generated audio frames, exposed because the model does not
+    // always emit its codec EOS and will otherwise run to the 512-frame default,
+    // which is roughly 41 s at the 12.5 Hz frame rate. 0 means unset, leaving
+    // that default in place.
+    int32_t max_frames = 0;
+};
+
+namespace detail {
+
+// Strict whole-string numeric parsing. std::stoi/stof accept trailing garbage
+// ("40abc" -> 40), which would silently honour a typo'd request.
+inline bool parse_whole_int32(const std::string & value, int32_t & out) {
+    if (value.empty()) {
+        return false;
+    }
+    try {
+        size_t     consumed = 0;
+        const long parsed   = std::stol(value, &consumed);
+        if (consumed != value.size()) {
+            return false;
+        }
+        if (parsed < INT32_MIN || parsed > INT32_MAX) {
+            return false;
+        }
+        out = static_cast<int32_t>(parsed);
+        return true;
+    } catch (const std::exception &) {
+        return false;
+    }
+}
+
+inline bool parse_whole_float(const std::string & value, float & out) {
+    if (value.empty()) {
+        return false;
+    }
+    try {
+        size_t      consumed = 0;
+        const float parsed   = std::stof(value, &consumed);
+        if (consumed != value.size()) {
+            return false;
+        }
+        out = parsed;
+        return true;
+    } catch (const std::exception &) {
+        return false;
+    }
+}
+
+inline tts_request_options reject(const std::string & message) {
+    tts_request_options opts;
+    opts.ok    = false;
+    opts.error = message;
+    return opts;
+}
+
+} // namespace detail
+
+inline tts_request_options parse_tts_request_options(
+        const std::string & text,
+        const std::string & voice,
+        const std::string & language,
+        const std::map<std::string, std::string> & params) {
+    if (text.empty()) {
+        return detail::reject("text must be a non-empty string");
+    }
+
+    // The Qwen3-TTS Base checkpoints have no built-in speaker. Without a
+    // reference clip the model picks an arbitrary voice, so an unset voice is
+    // a request error rather than a defaulted one.
+    if (voice.empty()) {
+        return detail::reject("voice must name a speaker reference audio file");
+    }
+
+    tts_request_options opts;
+    opts.text       = text;
+    opts.voice_path = voice;
+    opts.language   = language;
+
+    // Both values are range-checked here rather than left to the caller: the
+    // consumer copies them straight into mtmd_helper::gen_audio::inp, and only
+    // its separate sampler assignment is guarded by "> 0". An out-of-range or
+    // non-finite value would slip past that guard and reach llama.cpp.
+    const auto top_k_it = params.find("top_k");
+    if (top_k_it != params.end()) {
+        if (!detail::parse_whole_int32(top_k_it->second, opts.top_k)) {
+            return detail::reject("top_k must be an integer, got \"" + top_k_it->second + "\"");
+        }
+        if (opts.top_k < 0) {
+            return detail::reject("top_k must be >= 0, got \"" + top_k_it->second + "\"");
+        }
+    }
+
+    const auto top_p_it = params.find("top_p");
+    if (top_p_it != params.end()) {
+        if (!detail::parse_whole_float(top_p_it->second, opts.top_p)) {
+            return detail::reject("top_p must be a number, got \"" + top_p_it->second + "\"");
+        }
+        // Phrased as a negated in-range test, not "p < 0.0f || p > 1.0f",
+        // because every comparison against NaN is false: the obvious form
+        // would accept NaN, and NaN then defeats the consumer's "> 0" guard
+        // too, since that comparison is false as well.
+        if (!(opts.top_p >= 0.0f && opts.top_p <= 1.0f)) {
+            return detail::reject("top_p must be between 0.0 and 1.0, got \"" + top_p_it->second + "\"");
+        }
+    }
+
+    const auto max_frames_it = params.find("max_frames");
+    if (max_frames_it != params.end()) {
+        if (!detail::parse_whole_int32(max_frames_it->second, opts.max_frames)) {
+            return detail::reject("max_frames must be an integer, got \"" + max_frames_it->second + "\"");
+        }
+        if (opts.max_frames < 0) {
+            return detail::reject("max_frames must be >= 0, got \"" + max_frames_it->second + "\"");
+        }
+    }
+
+    opts.ok = true;
+    return opts;
+}
+
+} // namespace llama_grpc

backend/cpp/llama-cpp/tts_request_options_test.cpp

@@ -0,0 +1,209 @@
+// SPDX-License-Identifier: MIT
+
+#include <cstdio>
+#include <map>
+#include <string>
+
+#include "tts_request_options.h"
+
+static int failures = 0;
+
+static void check(bool ok, const char * name) {
+    if (!ok) {
+        ++failures;
+        std::fprintf(stderr, "FAIL: %s\n", name);
+    }
+}
+
+static void test_accepts_a_minimal_valid_request() {
+    const auto opts = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "en", {});
+
+    check(opts.ok, "minimal request is accepted");
+    check(opts.error.empty(), "minimal request has no error");
+    check(opts.text == "Hello world", "text passes through");
+    check(opts.voice_path == "/models/voices/ref.wav", "voice path passes through");
+    check(opts.language == "en", "language passes through");
+    check(opts.top_k == 0, "top_k defaults to the unset sentinel");
+    check(opts.top_p == 0.0f, "top_p defaults to the unset sentinel");
+    check(opts.max_frames == 0, "max_frames defaults to the unset sentinel");
+}
+
+static void test_rejects_empty_text() {
+    const auto opts = llama_grpc::parse_tts_request_options(
+        "", "/models/voices/ref.wav", "en", {});
+
+    check(!opts.ok, "empty text is rejected");
+    check(opts.error.find("text") != std::string::npos, "empty-text error names the field");
+}
+
+static void test_rejects_missing_speaker_reference() {
+    // Qwen3-TTS Base has no built-in speaker; without a reference it produces
+    // an arbitrary voice, so this must be a hard error rather than a surprise.
+    const auto opts = llama_grpc::parse_tts_request_options(
+        "Hello world", "", "en", {});
+
+    check(!opts.ok, "missing voice is rejected");
+    check(opts.error.find("voice") != std::string::npos, "missing-voice error names the field");
+}
+
+static void test_parses_sampling_params() {
+    const std::map<std::string, std::string> params{
+        {"top_k", "40"},
+        {"top_p", "0.85"},
+    };
+    const auto opts = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", params);
+
+    check(opts.ok, "sampling params are accepted");
+    check(opts.top_k == 40, "top_k is parsed");
+    check(opts.top_p > 0.849f && opts.top_p < 0.851f, "top_p is parsed");
+    check(opts.language.empty(), "absent language stays empty");
+}
+
+static void test_rejects_malformed_sampling_params() {
+    const auto bad_top_k = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_k", "forty"}});
+    check(!bad_top_k.ok, "non-numeric top_k is rejected");
+    check(bad_top_k.error.find("top_k") != std::string::npos, "top_k error names the field");
+
+    const auto bad_top_p = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", ""}});
+    check(!bad_top_p.ok, "empty top_p is rejected");
+
+    const auto trailing = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_k", "40abc"}});
+    check(!trailing.ok, "top_k with trailing garbage is rejected");
+
+    const auto trailing_float = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", "0.8abc"}});
+    check(!trailing_float.ok, "top_p with trailing garbage is rejected");
+
+    // std::stol returns a long, which is wider than int32_t on 64-bit hosts, so
+    // an in-range-for-long value still has to be caught before the narrowing.
+    const auto overflow_top_k = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_k", "99999999999"}});
+    check(!overflow_top_k.ok, "top_k beyond int32 range is rejected");
+    check(overflow_top_k.error.find("top_k") != std::string::npos,
+          "top_k overflow error names the field");
+}
+
+static void test_rejects_out_of_range_sampling_params() {
+    // These reach mtmd_helper::gen_audio::inp unconditionally downstream, where
+    // the "> 0" sampler guard does not screen them, so they must die here.
+    const auto negative_top_k = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_k", "-5"}});
+    check(!negative_top_k.ok, "negative top_k is rejected");
+    check(negative_top_k.error.find("top_k") != std::string::npos,
+          "negative top_k error names the field");
+
+    const auto negative_top_p = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", "-0.1"}});
+    check(!negative_top_p.ok, "negative top_p is rejected");
+    check(negative_top_p.error.find("top_p") != std::string::npos,
+          "negative top_p error names the field");
+
+    const auto large_top_p = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", "1.5"}});
+    check(!large_top_p.ok, "top_p above 1.0 is rejected");
+
+    // NaN survives a naive "p < 0.0f || p > 1.0f" range test because every
+    // comparison against NaN is false. This case pins the correct form.
+    const auto nan_top_p = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", "nan"}});
+    check(!nan_top_p.ok, "NaN top_p is rejected");
+
+    const auto inf_top_p = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", "inf"}});
+    check(!inf_top_p.ok, "infinite top_p is rejected");
+}
+
+static void test_accepts_sampling_param_boundaries() {
+    const auto zero_top_p = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", "0.0"}});
+    check(zero_top_p.ok, "top_p of 0.0 is accepted");
+    check(zero_top_p.top_p == 0.0f, "top_p of 0.0 round-trips");
+
+    const auto one_top_p = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_p", "1.0"}});
+    check(one_top_p.ok, "top_p of 1.0 is accepted");
+    check(one_top_p.top_p == 1.0f, "top_p of 1.0 round-trips");
+
+    const auto zero_top_k = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_k", "0"}});
+    check(zero_top_k.ok, "top_k of 0 is accepted");
+}
+
+static void test_parses_max_frames() {
+    // The consumer maps a positive value onto n_predict and leaves upstream's
+    // 512-frame default in place when it is unset, so the sentinel matters as
+    // much as the parsed value.
+    const auto opts = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "120"}});
+
+    check(opts.ok, "max_frames is accepted");
+    check(opts.max_frames == 120, "max_frames is parsed");
+
+    const auto absent = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"top_k", "40"}});
+    check(absent.ok, "a request without max_frames is accepted");
+    check(absent.max_frames == 0, "absent max_frames leaves the unset sentinel");
+
+    const auto zero = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "0"}});
+    check(zero.ok, "max_frames of 0 is accepted");
+    check(zero.max_frames == 0, "max_frames of 0 means unset");
+}
+
+static void test_rejects_malformed_max_frames() {
+    const auto negative = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "-1"}});
+    check(!negative.ok, "negative max_frames is rejected");
+    check(negative.error.find("max_frames") != std::string::npos,
+          "negative max_frames error names the field");
+
+    const auto non_numeric = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "many"}});
+    check(!non_numeric.ok, "non-numeric max_frames is rejected");
+    check(non_numeric.error.find("max_frames") != std::string::npos,
+          "non-numeric max_frames error names the field");
+
+    const auto trailing = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "120abc"}});
+    check(!trailing.ok, "max_frames with trailing garbage is rejected");
+
+    const auto empty = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"max_frames", ""}});
+    check(!empty.ok, "empty max_frames is rejected");
+
+    const auto overflow = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"max_frames", "99999999999"}});
+    check(!overflow.ok, "max_frames beyond int32 range is rejected");
+}
+
+static void test_ignores_unknown_params() {
+    // Unknown keys are backend-specific knobs meant for other TTS engines. A
+    // request routed here must not fail just because it carries them.
+    const auto opts = llama_grpc::parse_tts_request_options(
+        "Hello world", "/models/voices/ref.wav", "", {{"exaggeration", "0.7"}});
+
+    check(opts.ok, "unknown params are ignored, not rejected");
+}
+
+int main() {
+    test_accepts_a_minimal_valid_request();
+    test_rejects_empty_text();
+    test_rejects_missing_speaker_reference();
+    test_parses_sampling_params();
+    test_rejects_malformed_sampling_params();
+    test_rejects_out_of_range_sampling_params();
+    test_accepts_sampling_param_boundaries();
+    test_parses_max_frames();
+    test_rejects_malformed_max_frames();
+    test_ignores_unknown_params();
+
+    if (failures == 0) {
+        std::printf("tts_request_options_test: all checks passed\n");
+    }
+    return failures;
+}

backend/cpp/turboquant/Makefile

@@ -48,6 +48,7 @@ define turboquant-build
 	# stays compiling against vanilla upstream.
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build llama.cpp
@@ -86,6 +87,7 @@ turboquant-cpu-all:
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build purge
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
 	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build llama.cpp

backend/cpp/turboquant/patch-grpc-server.sh

@@ -8,6 +8,8 @@
 #      so the grpc-server option parser skips the two references to
 #      common_params::checkpoint_min_step (the default and the option handler).
 #      That field does not exist in the fork yet; drop this once it does.
+#   3. Use nlohmann's parse_error type in JSON catch clauses because the fork
+#      predates upstream's common_json_error wrapper.
 #
 # The fork used to lag upstream on the whole common_params_speculative refactor
 # (ggml-org/llama.cpp#22397/#22838/#22964), the model_tgt rename (#22838) and
@@ -100,4 +102,16 @@ else
     echo "==> LOCALAI_TURBOQUANT_NO_CHECKPOINT_MIN_STEP define OK"
 fi
 
+# 3. The shared source follows current upstream and catches common_json_error.
+#    TurboQuant still exposes nlohmann::json directly, so its equivalent parse
+#    failures use json::parse_error instead.
+if grep -q 'common_json_error' "$SRC"; then
+    echo "==> patching $SRC to use the TurboQuant JSON exception type"
+    awk '{ gsub(/common_json_error/, "json::parse_error"); print }' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+    echo "==> TurboQuant JSON exception patch OK"
+else
+    echo "==> $SRC already uses a TurboQuant-compatible JSON exception type, skipping"
+fi
+
 echo "==> all patches applied"

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=21901d3f7c23554f072964828363e49ddbc2dc68
+CRISPASR_VERSION?=c6d4265ae2ee2b8931b09d7d25d5c65c75c36a41
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/depth-anything-cpp/Makefile

@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=2028b47ac75a8659c6a9aa617baf09be193eb55f
+DEPTHANYTHING_VERSION?=02ba082274e001a63e50de5a1eb0ccc50c6af4b1
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/local-store/store.go

@@ -38,8 +38,9 @@ type Store struct {
 	// keysAreNormalized stays true until any non-unit-magnitude key
 	// is added; once false, the magnitude-aware fallback path is
 	// used by Find. Re-evaluated only at Set time, never again on
-	// its own — a deletion of the offending key does NOT flip it
-	// back to true (the bookkeeping cost would dominate the gain).
+	// its own — a partial deletion of the offending key does NOT flip
+	// it back to true (the bookkeeping cost would dominate the gain).
+	// An empty store returns to its initial state.
 	keysAreNormalized bool
 
 	// keyLen is the dimension of every stored key. -1 means "no
@@ -142,6 +143,10 @@ func (s *Store) StoresDelete(opts *pb.StoresDeleteOptions) error {
 	mergedV = append(mergedV, tailV...)
 	s.keys = mergedK
 	s.values = mergedV
+	if len(s.keys) == 0 {
+		s.keyLen = -1
+		s.keysAreNormalized = true
+	}
 	assert(slices.IsSortedFunc(s.keys, slices.Compare[[]float32]), "Delete: s.keys not sorted post-merge")
 	assert(len(s.keys) == len(s.values), "Delete: keys/values length skew")
 	return nil

backend/go/local-store/store_test.go

@@ -105,6 +105,46 @@ var _ = Describe("StoresDelete", func() {
 		})).To(Succeed(), "delete of missing key should succeed")
 		Expect(s.keys).To(HaveLen(1))
 	})
+
+	It("reopens the dimension after deleting every key", func() {
+		s := NewStore()
+		oldKey := []float32{2, 0, 0}
+		mustSet(s, [][]float32{oldKey}, [][]byte{[]byte("3d")})
+		Expect(s.keysAreNormalized).To(BeFalse())
+
+		Expect(s.StoresDelete(&pb.StoresDeleteOptions{
+			Keys: wrapKeys([][]float32{oldKey}),
+		})).To(Succeed())
+		Expect(s.keys).To(BeEmpty())
+		Expect(s.keyLen).To(Equal(-1))
+		Expect(s.keysAreNormalized).To(BeTrue())
+
+		newKey := normalizeVec([]float32{1, 1})
+		mustSet(s, [][]float32{newKey}, [][]byte{[]byte("2d")})
+		res, err := s.StoresFind(&pb.StoresFindOptions{
+			Key:  &pb.StoresKey{Floats: newKey},
+			TopK: 1,
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(res.Values).To(HaveLen(1))
+		Expect(string(res.Values[0].Bytes)).To(Equal("2d"))
+	})
+
+	It("retains the dimension after a partial delete", func() {
+		s := NewStore()
+		mustSet(s,
+			[][]float32{{1, 0, 0}, {0, 1, 0}},
+			[][]byte{[]byte("x"), []byte("y")},
+		)
+		Expect(s.StoresDelete(&pb.StoresDeleteOptions{
+			Keys: wrapKeys([][]float32{{1, 0, 0}}),
+		})).To(Succeed())
+		Expect(s.keyLen).To(Equal(3))
+		Expect(s.StoresSet(&pb.StoresSetOptions{
+			Keys:   wrapKeys([][]float32{{1, 0}}),
+			Values: wrapValues([][]byte{[]byte("2d")}),
+		})).NotTo(Succeed())
+	})
 })
 
 var _ = Describe("StoresFind", func() {

backend/go/nemo-speech-cpp/.gitignore

@@ -0,0 +1,22 @@
+# Fetched upstream sources
+sources/
+
+# CMake build directories
+build*/
+
+# Packaging output
+package/
+
+# Compiled backend binary. The second name is what a bare `go build ./...` from
+# this directory produces (it names the binary after the directory), as opposed
+# to the -o name the Makefile asks for.
+nemo-speech-cpp-grpc
+/nemo-speech-cpp
+
+# Shared libraries staged in-tree by the Makefile (cp from sources/). The
+# SOVERSION suffix means the payload is libnemo_speech_*.so.1, hence both globs.
+*.so
+*.so.*
+*.dylib
+
+compile_commands.json

backend/go/nemo-speech-cpp/Makefile

@@ -0,0 +1,391 @@
+# nemo-speech-cpp backend Makefile.
+#
+# Upstream pin lives below as NEMO_SPEECH_VERSION so .github/bump_deps.sh can
+# find and update it, matching the parakeet-cpp / vibevoice-cpp convention.
+#
+# Bumping NEMO_SPEECH_VERSION is a no-op on an existing checkout: sources/ is a
+# directory target, so make only clones when it is missing and never re-checks
+# out an already-cloned tree. After a bump run 'make purge && make', the same
+# rule the parakeet-cpp Makefile documents.
+#
+# 'build' is the entry point the backend image calls (backend/Dockerfile.golang
+# runs 'make -C backend/go/$(BACKEND) build' and then copies package/), so it
+# has to produce the binary and the package, not just the shared libraries.
+
+NEMO_SPEECH_VERSION?=ffa38cb2408f1e832a36d46fef5e3e1e80d07e6c
+NEMO_SPEECH_REPO?=https://github.com/NVIDIA/NeMo-Speech.cpp
+
+GOCMD?=go
+GO_TAGS?=
+JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+BUILD_TYPE?=
+NATIVE?=false
+
+# NEMO_SPEECH_CUBLAS_SHIM defaults ON upstream and builds a drop-in
+# libcublas.so.13. LocalAI's CUDA images ship the real cuBLAS, so the shim would
+# shadow it with a slower native GEMM. Always OFF here.
+CMAKE_ARGS?=-DCMAKE_BUILD_TYPE=Release \
+	-DBUILD_SHARED_LIBS=OFF \
+	-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
+	-DNEMO_SPEECH_CUBLAS_SHIM=OFF \
+	-DNEMO_SPEECH_BUILD_ASR=ON \
+	-DNEMO_SPEECH_BUILD_DIAR=ON \
+	-DNEMO_SPEECH_BUILD_TTS=ON \
+	-DNEMO_SPEECH_BUILD_NMT=ON \
+	-DNEMO_SPEECH_BUILD_CLI=OFF \
+	-DNEMO_SPEECH_BUILD_HTTP=OFF \
+	-DNEMO_SPEECH_BUILD_GRPC=OFF \
+	-DNEMO_SPEECH_WITH_FLASHLIGHT=OFF \
+	-DNEMO_SPEECH_TTS_WITH_ZH=ON \
+	-DNEMO_SPEECH_TTS_WITH_JA=ON
+
+ifeq ($(NATIVE),false)
+	CMAKE_ARGS+=-DGGML_NATIVE=OFF
+endif
+
+# NEMO_SPEECH_TTS_WITH_JA=ON compiles Open JTalk's bundled MeCab, and
+# mecab/src/dictionary.cpp derives a comparator from std::binary_function, which
+# C++17 removed. libstdc++ still ships it as deprecated-but-present under
+# -std=gnu++17, so Linux never notices; libc++ compiles it out and the build dies
+# with "no template named 'binary_function' in namespace 'std'". Upstream's own
+# CMakeLists already carries the equivalent workaround for MSVC's STL
+# (_HAS_AUTO_PTR_ETC plus /FIfunctional) but has no libc++ branch, because
+# NEMO_SPEECH_TTS_WITH_JA defaults OFF upstream and only LocalAI turns it on.
+#
+# libc++ gates the two templates on _LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION,
+# and has done since LLVM 16, which is older than any clang Xcode still ships.
+# The name matters: the older _LIBCPP_ENABLE_CXX17_REMOVED_BINDERS covers
+# bind1st/bind2nd/ptr_fun/mem_fun and NOT unary_function/binary_function, and the
+# umbrella _LIBCPP_ENABLE_CXX17_REMOVED_FEATURES no longer exists at all. A wrong
+# name is silently accepted by the preprocessor and fixes nothing.
+#
+# Applied through CMAKE_CXX_FLAGS rather than to the one target because the
+# tokenizer CMakeLists is upstream's and this tree is a pinned checkout, not a
+# patched one. Project-wide is also the safer scope: the macro decides whether
+# libc++'s internal __binary_function alias resolves to std::binary_function or
+# to __binary_function_keep_layout_base, which is a base class of std::less and
+# friends, so defining it for a subset of translation units would give those
+# class templates two spellings in one binary. Both bases are empty and, at
+# C++17, carry identical members, so the project-wide define changes no layout
+# and no ABI. On Linux the macro is not a name libstdc++ knows, so the branch is
+# unreachable there and would be inert even if it were taken.
+ifeq ($(shell uname -s),Darwin)
+	CXX_COMPAT_FLAGS?=-D_LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION
+else
+	CXX_COMPAT_FLAGS?=
+endif
+ifneq ($(strip $(CXX_COMPAT_FLAGS)),)
+	CMAKE_ARGS+=-DCMAKE_CXX_FLAGS=$(CXX_COMPAT_FLAGS)
+endif
+
+# scripts/build_itn_deps.sh installs the Sparrowhawk/OpenFST runtime here.
+# NEMO_SPEECH_DEPENDENCY_PREFIX defaults to <src>/.deps upstream, and the ITN
+# stack goes under its itn/ subdirectory. ITN_MARKER is a real output of that
+# script (it prints exactly this file on success), so it can drive a make rule.
+ITN_PREFIX=sources/NeMo-Speech.cpp/.deps/itn
+ITN_LIB_DIR=$(ITN_PREFIX)/lib
+ITN_MARKER=$(ITN_LIB_DIR)/libsparrowhawk.so
+ITN_FST_HEADER=$(ITN_PREFIX)/include/fst/fst.h
+
+# SentencePiece became a core ASR dependency in 5be7bfb: RNNT context biasing
+# uses it even when Flashlight and text normalization are disabled. Build the
+# pinned static archive provided by upstream so every platform gets the same
+# dependency instead of relying on an undeclared system package.
+SENTENCEPIECE_PREFIX=sources/NeMo-Speech.cpp/.deps/sentencepiece
+SENTENCEPIECE_MARKER=$(SENTENCEPIECE_PREFIX)/lib/libsentencepiece.a
+
+# Linux's ASR CMake block looks in NEMO_SPEECH_DEPENDENCY_PREFIX directly, but
+# the Apple branch uses generic find_library()/find_path(). Put the same private
+# prefix on CMake's search path so Darwin consumes the archive built above too.
+CMAKE_ARGS+=-DCMAKE_PREFIX_PATH=$(abspath $(SENTENCEPIECE_PREFIX))
+
+ITN_CC?=gcc-12
+ITN_CXX?=g++-12
+
+# Pin protoc to the apt one. backend/Dockerfile.golang drops protoc 27.1 into
+# /usr/local/bin, which precedes /usr/bin on PATH, while libprotobuf-dev is the
+# distro's (3.21 on noble, 3.12 on jammy). Sparrowhawk resolves protoc from PATH
+# at make time (configure.ac uses AC_CHECK_PROG, so PROTOC substitutes to the
+# bare word, and src/proto/Makefile.am invokes $(PROTOC)), and it commits no
+# pregenerated stubs, so this always runs. Code generated by 27.1 includes
+# google/protobuf/runtime_version.h and a PROTOBUF_VERSION #error guard that the
+# older headers do not have, so the mismatch breaks the build. configure honours
+# a pre-set PROTOC ("Let the user override the test"), which is what this is.
+ITN_PROTOC?=/usr/bin/protoc
+
+# Text normalization is Linux-only: Sparrowhawk/OpenFST assume a GNU toolchain
+# and the gcc-12 pin has no macOS analogue. Documented gap, see the spec.
+#
+# An already-configured build tree wins over the platform default. Without that,
+# a tree configured WITH_NORM=OFF would silently try to reconfigure itself to ON
+# on the next bare `make test`, which means demanding gcc-12 from a developer who
+# deliberately built without it. An explicit WITH_NORM= on the command line still
+# overrides both, since command-line variables beat ?= assignments.
+CMAKE_CACHE=sources/NeMo-Speech.cpp/build/CMakeCache.txt
+CACHED_WITH_NORM=$(shell sed -n 's/^NEMO_SPEECH_WITH_NORM:BOOL=//p' $(CMAKE_CACHE) 2>/dev/null)
+ifeq ($(shell uname -s),Darwin)
+	WITH_NORM?=OFF
+else ifneq ($(CACHED_WITH_NORM),)
+	WITH_NORM?=$(CACHED_WITH_NORM)
+else
+	WITH_NORM?=ON
+endif
+CMAKE_ARGS+=-DNEMO_SPEECH_WITH_NORM=$(WITH_NORM)
+
+ifeq ($(BUILD_TYPE),cublas)
+	CMAKE_ARGS+=-DGGML_CUDA=ON
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DGGML_VULKAN=ON
+else ifeq ($(BUILD_TYPE),metal)
+	CMAKE_ARGS+=-DGGML_METAL=ON
+endif
+
+# ggml-patches/ is a CUDA series. Every kernel it adds lives under
+# src/ggml-cuda/; the only files it touches outside that directory are enum and
+# name-table entries in include/ggml.h and src/ggml.c plus, in ggml-cpu, a
+# supports_op returning false and an abort case for the CUDA-only op. Upstream
+# agrees: its metal-* and vulkan-* CMake presets inherit the cpu-* ones, which
+# set NEMO_SPEECH_GGML_PATCHED=OFF, and every use of a patch-only symbol in the
+# ASR sources sits behind NEMO_SPEECH_FUSED_RELPOS_ATTN /
+# NEMO_SPEECH_FASTCONFORMER_CUDA_FUSIONS (both force-OFF without GGML_CUDA) or
+# behind NEMO_SPEECH_GGML_PATCHED itself, which guards a Q8_PLANAR flag write
+# that a non-CUDA buffer already throws before reaching.
+#
+# So on macOS the series buys nothing, and it cannot be applied there anyway:
+# upstream's scripts/apply-ggml-patches.sh uses mapfile, a bash 4 builtin, and
+# macOS ships bash 3.2 as the only bash on the runner's PATH. Skip the patch
+# step and tell cmake the linked ggml is stock, which is exactly upstream's own
+# Metal configuration. Linux keeps applying the series unchanged.
+ifeq ($(shell uname -s),Darwin)
+	GGML_PATCHED?=OFF
+else
+	GGML_PATCHED?=ON
+endif
+CMAKE_ARGS+=-DNEMO_SPEECH_GGML_PATCHED=$(GGML_PATCHED)
+
+.PHONY: nemo-speech-cpp-grpc package build clean purge test all stage-libs patch-ggml engine itn sentencepiece patch-itn-headers
+
+all: nemo-speech-cpp-grpc package
+
+sources/NeMo-Speech.cpp:
+	mkdir -p sources
+	cd sources && git clone $(NEMO_SPEECH_REPO) NeMo-Speech.cpp
+	cd sources/NeMo-Speech.cpp && git checkout $(NEMO_SPEECH_VERSION)
+	# NMT links llama.cpp; ja needs open_jtalk; zh needs cppjieba. flashlight and
+	# kenlm are deliberately not initialized, they are out of scope.
+	cd sources/NeMo-Speech.cpp && git submodule update --init --recursive \
+		ggml llama.cpp third_party/open_jtalk third_party/cppjieba third_party/cpp-httplib
+
+# NEMO_SPEECH_GGML_PATCHED defaults ON and silently assumes the ggml-patches
+# series is applied. An unpatched checkout builds fine and produces wrong CUDA
+# encoder output, so a failure here must stop the build rather than warn.
+#
+# Upstream's own script is the right tool: it applies the series in filename
+# order, exits non-zero when a patch does not apply, and decides "already
+# applied" by comparing the full-series tree hash rather than a timestamp. That
+# makes it safe to run unconditionally, so there is no sentinel file to go stale
+# or to wedge the build when deleted.
+#
+# Both branches keep the order-only clone prerequisite: it is the only thing
+# that pulls sources/ in on a WITH_NORM=OFF tree, where the library rule has no
+# other prerequisite left.
+ifeq ($(GGML_PATCHED),ON)
+patch-ggml: | sources/NeMo-Speech.cpp
+	cd sources/NeMo-Speech.cpp && bash scripts/apply-ggml-patches.sh
+else
+patch-ggml: | sources/NeMo-Speech.cpp
+	@echo "[ggml-patch] skipped: NEMO_SPEECH_GGML_PATCHED=$(GGML_PATCHED), the series is CUDA-only"
+endif
+
+# The Sparrowhawk/OpenFST text-normalization stack, as a target in its own right
+# keyed on a file the build script actually produces.
+#
+# It used to be a side effect of the runtime library rule, which meant make had
+# no idea whether it existed: once the library was up to date the script could
+# never run again, so a tree built WITH_NORM=OFF could not be moved to ON, and
+# anything that needed the ITN prefix was stuck demanding a full clean. As its
+# own rule it is built on demand, rebuilt independently, and reachable directly
+# with 'make itn'.
+#
+# OpenFST's templates ICE on gcc-13/14 at -O2, hence the gcc-12 pin for this one
+# step; the runtime itself builds with the image default compiler.
+$(ITN_MARKER): | sources/NeMo-Speech.cpp
+	@command -v $(ITN_CC) >/dev/null 2>&1 && command -v $(ITN_CXX) >/dev/null 2>&1 || { \
+		echo "ERROR: $(ITN_CC)/$(ITN_CXX) not found, and text normalization needs them:" >&2; \
+		echo "       OpenFST's templates ICE on gcc-13 and gcc-14 at -O2." >&2; \
+		echo "       Install them, or build this backend with WITH_NORM=OFF." >&2; \
+		exit 1; }
+	# configure's only gate on a preset PROTOC is test -n, so a path that does not
+	# exist is accepted here and surfaces much later as a bare "No such file or
+	# directory" from inside make -C src/proto. Check it up front instead.
+	@command -v $(ITN_PROTOC) >/dev/null 2>&1 || { \
+		echo "ERROR: protoc not found at $(ITN_PROTOC)." >&2; \
+		echo "       Install the protobuf-compiler package, whose protoc matches" >&2; \
+		echo "       the libprotobuf-dev headers Sparrowhawk compiles against, or" >&2; \
+		echo "       point this at a matching one with ITN_PROTOC=/path/to/protoc." >&2; \
+		exit 1; }
+	cd sources/NeMo-Speech.cpp && CC=$(ITN_CC) CXX=$(ITN_CXX) PROTOC=$(ITN_PROTOC) \
+		JOBS=$(JOBS) scripts/build_itn_deps.sh
+	@$(MAKE) --no-print-directory patch-itn-headers
+
+# OpenFST 1.8.3's FstImpl copy-assignment operator assigns a raw SymbolTable*
+# (what SymbolTable::Copy() returns) straight to a std::unique_ptr member:
+#
+#     isymbols_ = impl.isymbols_ ? impl.isymbols_->Copy() : nullptr;
+#
+# std::unique_ptr has no operator= taking a raw pointer in any C++ standard, so
+# that line is ill-formed everywhere. It survived because nothing instantiates
+# FstImpl::operator=, and gcc <= 13 only checks a template member's body when it
+# is instantiated. gcc 14 resolves non-dependent operator expressions at template
+# definition time, so it rejects the line in every translation unit that so much
+# as includes <fst/fst.h>, with no instantiation involved. Verified: gcc 14.2
+# fails on a file whose entire content is '#include <fst/fst.h>'.
+#
+# That is why this only shows up now. build_itn_deps.sh builds OpenFST with
+# gcc-12 (its templates ICE on newer gcc at -O2) and upstream's own images build
+# the runtime with gcc-13, so neither compiler ever sees it. LocalAI's
+# backend/Dockerfile.golang installs gcc-14 and makes it the default via
+# update-alternatives, and fst_normalizer.cpp is the one translation unit here
+# that includes OpenFST, so it is the one that breaks.
+#
+# The fix is the same spelling FstImpl::SetInputSymbols already uses 80 lines
+# further down, and matches the copy constructor's deep-copy intent exactly. It
+# is applied to the installed prefix rather than to the OpenFST checkout because
+# the prefix is the only copy the cmake build compiles against; libfst.so is
+# already linked by this point and cannot contain the function, since no
+# compiler could ever have emitted it. Only these two lines are affected: gcc 14
+# reports exactly two errors over the whole OpenFST include closure, both here.
+#
+# Guarded on both sides so a pinned-version bump cannot silently no-op it: the
+# first check fails if neither the broken nor the fixed spelling is present, the
+# last fails if the broken one survives.
+patch-itn-headers:
+	@test -f $(ITN_FST_HEADER) || { \
+		echo "ERROR: $(ITN_FST_HEADER) missing; the ITN prefix is not installed." >&2; \
+		exit 1; }
+	@grep -q 'isymbols_ = impl.isymbols_' $(ITN_FST_HEADER) || \
+	 grep -q 'isymbols_.reset(impl.isymbols_' $(ITN_FST_HEADER) || { \
+		echo "ERROR: FstImpl::operator= in $(ITN_FST_HEADER) matches neither the" >&2; \
+		echo "       known-broken nor the patched form. OpenFST changed upstream;" >&2; \
+		echo "       re-check whether this patch is still needed before removing it." >&2; \
+		exit 1; }
+	sed -i -E 's|^([[:space:]]*)([io]symbols_) = (impl\.[io]symbols_ \? impl\.[io]symbols_->Copy\(\) : nullptr);$$|\1\2.reset(\3);|' $(ITN_FST_HEADER)
+	@if grep -q 'symbols_ = impl.[io]symbols_' $(ITN_FST_HEADER); then \
+		echo "ERROR: the FstImpl::operator= patch did not apply to $(ITN_FST_HEADER)." >&2; \
+		exit 1; \
+	fi
+
+itn: $(ITN_MARKER)
+
+$(SENTENCEPIECE_MARKER): | sources/NeMo-Speech.cpp
+	# Upstream's license copies use GNU install's -D flag, which BSD install
+	# does not support. Homebrew CMake 4 also rejects SentencePiece's old policy
+	# floor. Patch both incompatibilities before running the helper on Darwin.
+	@if [ "$(shell uname -s)" = Darwin ]; then \
+		cd sources/NeMo-Speech.cpp && \
+		mkdir -p .deps/sentencepiece/share/licenses/nemo-speech/third_party/sentencepiece && \
+		perl -pi \
+			-e 's/install -Dm0644/install -m 0644/g;' \
+			-e 's/-DCMAKE_BUILD_TYPE=Release /-DCMAKE_BUILD_TYPE=Release -DCMAKE_POLICY_VERSION_MINIMUM=3.5 /;' \
+			scripts/build_sentencepiece_static.sh; \
+	fi
+	cd sources/NeMo-Speech.cpp && JOBS=$(JOBS) scripts/build_sentencepiece_static.sh
+
+sentencepiece: $(SENTENCEPIECE_MARKER)
+
+# Only a WITH_NORM=ON build needs the ITN stack, and it must exist before cmake
+# configures, since the WITH_NORM cmake block find_library()s into the prefix
+# with REQUIRED.
+NEMO_RUNTIME_PREREQS=$(SENTENCEPIECE_MARKER)
+ifeq ($(WITH_NORM),ON)
+NEMO_RUNTIME_PREREQS+=$(ITN_MARKER)
+endif
+
+# Upstream sets CMAKE_LIBRARY_OUTPUT_DIRECTORY to ${CMAKE_BINARY_DIR}/bin, so the
+# shared objects land in build/bin rather than at the top of the build tree.
+#
+# patch-ggml is order-only: it is phony and therefore always runs, but an
+# order-only prerequisite does not mark this target out of date, so an
+# already-built tree is not relinked on every invocation.
+sources/NeMo-Speech.cpp/build/bin/libnemo_speech_asr_c.so: $(NEMO_RUNTIME_PREREQS) | patch-ggml
+	cd sources/NeMo-Speech.cpp && cmake -B build -G Ninja $(CMAKE_ARGS)
+	cd sources/NeMo-Speech.cpp && cmake --build build -j$(JOBS)
+
+# Stage the runtime next to the Go sources so purego.Dlopen finds it during
+# local development and so package.sh has a single directory to bundle from.
+#
+# ASR and NMT build a dedicated _c shared object that links the C++ implementation
+# in privately. TTS does not: upstream compiles its c_api.cpp straight into
+# libnemo_speech_tts and only aliases the nemo_speech_tts_c CMake target, so the
+# TTS C ABI ships without the _c suffix.
+stage-libs: sources/NeMo-Speech.cpp/build/bin/libnemo_speech_asr_c.so
+	# -a keeps the SOVERSION symlink a symlink instead of duplicating the payload.
+	cp -af sources/NeMo-Speech.cpp/build/bin/libnemo_speech_asr_c.* .
+	cp -af sources/NeMo-Speech.cpp/build/bin/libnemo_speech_tts.* .
+	cp -af sources/NeMo-Speech.cpp/build/bin/libnemo_speech_nmt_c.* .
+	# The _c libraries are thin ABI shims with a DT_NEEDED on the C++
+	# implementation DSO, so dlopen fails without these next to them. TTS needs
+	# no counterpart, its implementation and ABI live in the same object.
+	cp -af sources/NeMo-Speech.cpp/build/bin/libnemo_speech_asr.* .
+	cp -af sources/NeMo-Speech.cpp/build/bin/libnemo_speech_nmt.* .
+	# nemo_speech_text_normalization is STATIC but links sparrowhawk, fstfar and
+	# fst PUBLIC, so those become DT_NEEDED on libnemo_speech_asr.so. They live in
+	# a project-local prefix that nothing else on the system provides, so without
+	# staging them here the packaged backend cannot dlopen at all.
+	#
+	# Keyed on the prefix existing rather than on WITH_NORM, so this stages what
+	# the tree actually built. A WITH_NORM=ON build cannot reach here without the
+	# prefix (the library rule takes ITN_MARKER as a prerequisite), and if a
+	# library that needs Sparrowhawk somehow arrives unstaged, package.sh's
+	# closure guard fails the build rather than shipping it.
+	@if [ -d "$(ITN_LIB_DIR)" ]; then \
+		echo "cp -af $(ITN_LIB_DIR)/*.so* ."; \
+		cp -af $(ITN_LIB_DIR)/*.so* .; \
+	fi
+
+## Builds the native runtime and stops short of the Go binary. Everything it
+## touches lives under sources/, a clone pinned by NEMO_SPEECH_VERSION, so
+## nothing here can observe a change elsewhere in the LocalAI tree.
+## Dockerfile.golang calls this from a layer that copies in this directory and
+## nothing else, which keeps the multi-minute ggml/llama.cpp compile in the
+## registry layer cache across builds whose only change is on the Go side.
+## Without it that prebuild is skipped and a CUDA build recompiles all of
+## upstream on every Go-side edit. See .agents/ci-caching.md.
+engine: stage-libs
+
+nemo-speech-cpp-grpc: stage-libs
+	# CGO_ENABLED=0 matches whisper / parakeet-cpp / omnivoice-cpp: the runtime is
+	# reached through purego.Dlopen, not cgo, and a static binary is what lets
+	# run.sh route execution through the packaged lib/ld.so.
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o nemo-speech-cpp-grpc .
+
+# The dlopen tests need the staged shared objects on the loader path, the same
+# way parakeet-cpp sets it up. Depends on stage-libs so that path is not an
+# empty directory on a clean tree, which would fail the tests confusingly.
+#
+# NEMO_SPEECH_REQUIRE_LIBS turns a missing library from a skip into a failure.
+# The ABI specs are the only thing standing between this backend and silent
+# memory corruption, so a run that reaches them and quietly skips them is worse
+# than one that fails: it reports green having checked nothing.
+test: stage-libs
+	NEMO_SPEECH_REQUIRE_LIBS=1 LD_LIBRARY_PATH=$(CURDIR):$$LD_LIBRARY_PATH $(GOCMD) test ./... -count=1
+
+package: nemo-speech-cpp-grpc
+	bash package.sh
+
+# What backend/Dockerfile.golang invokes. It must leave both the binary and a
+# populated package/ behind, because the final image stage copies package/.
+build: package
+
+clean:
+	# Every .so here is staged output (nemo runtime plus, on a WITH_NORM build,
+	# the ITN stack), and the SOVERSION suffix means the payload is *.so.1, so
+	# the globs have to reach past the .so.
+	rm -f nemo-speech-cpp-grpc
+	rm -f *.so *.so.* *.dylib
+	rm -rf package
+	rm -rf sources/NeMo-Speech.cpp/build
+
+purge: clean
+	rm -rf sources

backend/go/nemo-speech-cpp/abi.go

@@ -0,0 +1,423 @@
+package main
+
+// purego binds by name at runtime and the config structs cross the ABI by
+// pointer, so neither a renamed symbol nor a mis-laid-out mirror struct is
+// visible to the compiler or the linker. Everything here is transcribed from
+// sources/NeMo-Speech.cpp/include/nemo_speech/{asr,diar,tts,nmt}.h, and
+// abi_test.go asserts it against the real shared objects.
+
+import (
+	"fmt"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+)
+
+var (
+	asrLib uintptr
+	ttsLib uintptr
+	nmtLib uintptr
+)
+
+// ---- ASR ----
+
+var (
+	ASRCreate                func(cfg unsafe.Pointer, out *uintptr) int32
+	ASRDestroy               func(recognizer uintptr)
+	ASRRecognizeF32          func(recognizer uintptr, options unsafe.Pointer, samples *float32, nSamples uint64, sampleRate int32, out *uintptr) int32
+	ASRStreamingRecognize    func(recognizer uintptr, options unsafe.Pointer, out *uintptr) int32
+	ASRStreamPushF32         func(stream uintptr, samples *float32, nSamples uint64, sampleRate int32) int32
+	ASRStreamForceEndpoint   func(stream uintptr) int32
+	ASRStreamFinish          func(stream uintptr) int32
+	ASRStreamNext            func(stream uintptr, out *uintptr) int32
+	ASRStreamClose           func(stream uintptr)
+	ASRRecognitionOptionsDef func() cASRRecognitionOptions
+
+	ASRResultIsFinal          func(result uintptr) bool
+	ASRResultAudioProcessed   func(result uintptr) float32
+	ASRResultAlternativeCount func(result uintptr) uint64
+	ASRResultTranscript       func(result uintptr, alt uint64) string
+	ASRResultConfidence       func(result uintptr, alt uint64) float32
+	ASRResultWordCount        func(result uintptr, alt uint64) uint64
+	ASRResultWordText         func(result uintptr, alt, i uint64) string
+	ASRResultWordStartTime    func(result uintptr, alt, i uint64) int32
+	ASRResultWordEndTime      func(result uintptr, alt, i uint64) int32
+	ASRResultWordConfidence   func(result uintptr, alt, i uint64) float32
+	ASRResultWordSpeakerTag   func(result uintptr, alt, i uint64) int32
+	ASRResultLanguageCount    func(result uintptr, alt uint64) uint64
+	ASRResultLanguageCode     func(result uintptr, alt, i uint64) string
+	ASRResultDestroy          func(result uintptr)
+
+	ASRLastError func() string
+	ASRVersion   func() string
+)
+
+// ---- Diarization (exported from the ASR library) ----
+
+var (
+	DiarCreate          func(cfg unsafe.Pointer, out *uintptr) int32
+	DiarDestroy         func(model uintptr)
+	DiarNumSpeakers     func(model uintptr) int32
+	DiarSecondsPerFrame func(model uintptr) float64
+	DiarStreamOpen      func(model uintptr, out *uintptr) int32
+	DiarStreamPushF32   func(stream uintptr, samples *float32, nSamples uint64, sampleRate int32) int32
+	DiarStreamFinish    func(stream uintptr) int32
+	DiarStreamClose     func(stream uintptr)
+	// cfg is the optional nemo_speech_diar_segmentation_config (NULL = library
+	// defaults). The two-call count-then-fill pattern is documented on the C
+	// declaration in diar.h.
+	DiarSegments func(stream uintptr, cfg unsafe.Pointer, out unsafe.Pointer, capacity uint64, count *uint64) int32
+)
+
+// ---- TTS ----
+
+var (
+	TTSCreate                  func(cfg unsafe.Pointer, out *uintptr) int32
+	TTSDestroy                 func(synthesizer uintptr)
+	TTSSampleRate              func(synthesizer uintptr) int32
+	TTSSpeakerCount            func(synthesizer uintptr) int32
+	TTSSpeakerName             func(synthesizer uintptr, i uint64) string
+	TTSSynthesizeText          func(synthesizer uintptr, options unsafe.Pointer, text string, callback uintptr, userData uintptr, statsOut unsafe.Pointer) int32
+	TTSRuntimeConfigDefault    func() cTTSRuntimeConfig
+	TTSSynthesisOptionsDefault func() cTTSSynthesisOptions
+	TTSLastError               func() string
+	TTSVersion                 func() string
+)
+
+// ---- NMT ----
+
+var (
+	NMTCreate         func(cfg unsafe.Pointer, out *uintptr) int32
+	NMTDestroy        func(translator uintptr)
+	NMTTranslate      func(translator uintptr, texts *uintptr, nTexts uint64, source, target string, out *uintptr) int32
+	NMTResultCount    func(result uintptr) uint64
+	NMTResultText     func(result uintptr, i uint64) string
+	NMTResultLanguage func(result uintptr, i uint64) string
+	NMTResultDestroy  func(result uintptr)
+	NMTLastError      func() string
+	NMTVersion        func() string
+)
+
+// ---- C struct mirrors ----
+//
+// Each mirrors a struct in include/nemo_speech/*.h field for field. The leading
+// Size field is the C `size_t size` the runtime validates against its own
+// sizeof, which is what makes a layout mismatch detectable at runtime instead
+// of silently corrupting memory. Blank fields are System V AMD64 / AAPCS64
+// padding: C inserts it implicitly, Go does not, so it has to be written out.
+// See abi_test.go, which pins both every total size and every field offset.
+
+type cASRBackendConfig struct {
+	Size uintptr
+	GPU  int32
+	_    [4]byte // trailing pad to the struct's 8-byte alignment
+}
+
+type cASRModelConfig struct {
+	Size uintptr
+	Path uintptr
+	Name uintptr
+}
+
+type cASRVADConfig struct {
+	Size          uintptr
+	ModelPath     uintptr
+	EnableMasking bool
+	_             [3]byte
+	Onset         float32
+	Offset        float32
+	_             [4]byte
+}
+
+type cASRPostprocConfig struct {
+	Size              uintptr
+	ProfanityListPath uintptr
+	ITNModelDir       uintptr
+	PNCModelPath      uintptr
+}
+
+type cASRDiarConfig struct {
+	Size               uintptr
+	ModelPath          uintptr
+	ChunkFrames        int32
+	RightContextFrames int32
+	LeftContextFrames  int32
+	FIFOFrames         int32
+	SpkcacheFrames     int32
+	UpdatePeriodFrames int32
+}
+
+type cASRRecognizerConfig struct {
+	Size        uintptr
+	Backend     uintptr
+	Model       uintptr
+	Streaming   uintptr
+	Decoder     uintptr
+	VAD         uintptr
+	Endpointing uintptr
+	Postproc    uintptr
+	Diar        uintptr
+	Batching    uintptr
+}
+
+type cASRRecognitionOptions struct {
+	Size                       uintptr
+	RequestID                  uintptr
+	LanguageCode               uintptr
+	InterimResults             bool
+	EnableWordTimeOffsets      bool
+	EnableAutomaticPunctuation bool
+	VerbatimTranscripts        bool
+	ProfanityFilter            bool
+	_                          [3]byte
+	StopHistoryEouMs           int32
+	_                          [4]byte
+	SpeechContexts             uintptr
+	SpeechContextCount         uintptr
+	MaxAlternatives            int32
+	EnableSpeakerDiarization   bool
+	_                          [3]byte
+	MaxSpeakerCount            int32
+	_                          [4]byte
+}
+
+// cDiarModelConfig mirrors nemo_speech_diar_model_config (diar.h). This is the
+// standalone Sortformer pipeline's own config and is NOT cASRDiarConfig, which
+// is the diarizer attached to a recognizer: this one carries gpu and preset,
+// that one does not.
+//
+// The six frame counts are sentinel-sensitive. src/asr/c_api.cpp applies each
+// one only when it is > 0, EXCEPT left_context_frames, which it applies when it
+// is >= 0. A zero-valued struct would therefore pin the left context to 0
+// rather than leave the preset's value alone, so loadDiarizer writes -1 into
+// all six.
+type cDiarModelConfig struct {
+	Size      uintptr
+	ModelPath uintptr
+	GPU       int32
+	_         [4]byte // pad to the alignment of the pointer that follows
+	Preset    uintptr
+	// Encoder-frame geometry overrides, applied on top of the preset.
+	ChunkFrames        int32
+	RightContextFrames int32
+	LeftContextFrames  int32
+	FIFOFrames         int32
+	SpkcacheFrames     int32
+	UpdatePeriodFrames int32
+}
+
+// cDiarSegmentationConfig mirrors nemo_speech_diar_segmentation_config
+// (diar.h): the NeMo ts_vad postprocessing applied when turning per-frame
+// speaker probabilities into segments.
+//
+// onset and offset are float, the four durations are double. That mixture is
+// the whole reason this mirror needs its offsets pinned: writing all six as
+// float32 or all six as float64 both produce a struct C would read shifted.
+type cDiarSegmentationConfig struct {
+	Size           uintptr
+	Onset          float32
+	Offset         float32
+	PadOnsetSec    float64
+	PadOffsetSec   float64
+	MinGapSec      float64
+	MinDurationSec float64
+}
+
+// cDiarSegment mirrors nemo_speech_diar_segment (diar.h), the element type
+// nemo_speech_diar_segments fills.
+//
+// It has no leading size field: unlike the config structs it travels from C to
+// Go, so there is no caller-declared size for the runtime to validate against.
+// The times are already SECONDS (double), not frame indices, so nothing here
+// needs the model's seconds-per-frame to be interpreted. Speaker is 1-based,
+// matching WordInfo.speaker_tag on the ASR surface.
+type cDiarSegment struct {
+	StartTime float64
+	EndTime   float64
+	Speaker   int32
+	_         [4]byte // trailing pad to the struct's 8-byte alignment
+}
+
+type cTTSModelConfig struct {
+	Size                   uintptr
+	MagpieModel            uintptr
+	CodecModel             uintptr
+	TokenizerModelDir      uintptr
+	TextNormalizerModelDir uintptr
+}
+
+// cTTSRuntimeConfig mirrors nemo_speech_tts_runtime_config. The four backend /
+// mode fields are C enums, which this toolchain lays out as int32.
+type cTTSRuntimeConfig struct {
+	Size                uintptr
+	Speaker             int32
+	Threads             int32
+	CodecThreads        int32
+	Seed                int32
+	Steps               int32
+	TopK                int32
+	ChunkFrames         int32
+	CodecQueueDepth     int32
+	CodecHistoryFrames  int32
+	CodecFutureFrames   int32
+	WindowMs            int32
+	Temperature         float32
+	OverrideTemperature bool
+	_                   [3]byte
+	CFGScale            float32
+	OverrideCFGScale    bool
+	UseCFG              bool
+	UseLocalTransformer bool
+	UseKVCache          bool
+	UseStatefulCodec    bool
+	CodecCPU            bool
+	FlushPartialChunk   bool
+	Verbose             bool
+	LTBackend           int32
+	SamplingBackend     int32
+	UMAMode             int32
+	LongformMode        int32
+	LTFP32              bool
+	_                   [7]byte
+}
+
+type cTTSSynthesizerConfig struct {
+	Size                uintptr
+	Model               uintptr
+	Runtime             uintptr
+	DefaultLanguageCode uintptr
+	DefaultVoiceName    uintptr
+}
+
+type cTTSSynthesisOptions struct {
+	Size                uintptr
+	RequestID           uintptr
+	LanguageCode        uintptr
+	Speaker             int32
+	Seed                int32
+	Steps               int32
+	TopK                int32
+	Temperature         float32
+	OverrideTemperature bool
+	_                   [3]byte
+	CFGScale            float32
+	OverrideCFGScale    bool
+	_                   [3]byte
+	VoiceName           uintptr
+	OutputSampleRate    int32
+	_                   [4]byte
+}
+
+type cNMTBackendConfig struct {
+	Size uintptr
+	GPU  int32
+	_    [4]byte
+}
+
+type cNMTModelConfig struct {
+	Size uintptr
+	Path uintptr
+	NCtx int32
+	_    [4]byte
+}
+
+type cNMTTranslatorConfig struct {
+	Size       uintptr
+	Backend    uintptr
+	Model      uintptr
+	Generation uintptr
+	Pool       uintptr
+}
+
+// symbol pairs a Go function pointer with its exported C name. Keeping the
+// name next to the var means `nm -D libnemo_speech_asr_c.so.1 | grep nemo_speech`
+// is enough to spot drift after a pin bump.
+type symbol struct {
+	fn   any
+	name string
+	lib  *uintptr
+}
+
+func symbols() []symbol {
+	return []symbol{
+		{&ASRCreate, "nemo_speech_asr_create", &asrLib},
+		{&ASRDestroy, "nemo_speech_asr_destroy", &asrLib},
+		{&ASRRecognizeF32, "nemo_speech_asr_recognize_f32", &asrLib},
+		{&ASRStreamingRecognize, "nemo_speech_asr_streaming_recognize", &asrLib},
+		{&ASRStreamPushF32, "nemo_speech_asr_stream_push_f32", &asrLib},
+		{&ASRStreamForceEndpoint, "nemo_speech_asr_stream_force_endpoint", &asrLib},
+		{&ASRStreamFinish, "nemo_speech_asr_stream_finish", &asrLib},
+		{&ASRStreamNext, "nemo_speech_asr_stream_next", &asrLib},
+		{&ASRStreamClose, "nemo_speech_asr_stream_close", &asrLib},
+		{&ASRRecognitionOptionsDef, "nemo_speech_asr_recognition_options_default", &asrLib},
+		{&ASRResultIsFinal, "nemo_speech_asr_result_is_final", &asrLib},
+		{&ASRResultAudioProcessed, "nemo_speech_asr_result_audio_processed", &asrLib},
+		{&ASRResultAlternativeCount, "nemo_speech_asr_result_alternative_count", &asrLib},
+		{&ASRResultTranscript, "nemo_speech_asr_result_transcript", &asrLib},
+		{&ASRResultConfidence, "nemo_speech_asr_result_confidence", &asrLib},
+		{&ASRResultWordCount, "nemo_speech_asr_result_word_count", &asrLib},
+		{&ASRResultWordText, "nemo_speech_asr_result_word_text", &asrLib},
+		{&ASRResultWordStartTime, "nemo_speech_asr_result_word_start_time", &asrLib},
+		{&ASRResultWordEndTime, "nemo_speech_asr_result_word_end_time", &asrLib},
+		{&ASRResultWordConfidence, "nemo_speech_asr_result_word_confidence", &asrLib},
+		{&ASRResultWordSpeakerTag, "nemo_speech_asr_result_word_speaker_tag", &asrLib},
+		{&ASRResultLanguageCount, "nemo_speech_asr_result_language_count", &asrLib},
+		{&ASRResultLanguageCode, "nemo_speech_asr_result_language_code", &asrLib},
+		{&ASRResultDestroy, "nemo_speech_asr_result_destroy", &asrLib},
+		{&ASRLastError, "nemo_speech_asr_last_error", &asrLib},
+		{&ASRVersion, "nemo_speech_asr_version", &asrLib},
+
+		{&DiarCreate, "nemo_speech_diar_create", &asrLib},
+		{&DiarDestroy, "nemo_speech_diar_destroy", &asrLib},
+		{&DiarNumSpeakers, "nemo_speech_diar_num_speakers", &asrLib},
+		{&DiarSecondsPerFrame, "nemo_speech_diar_seconds_per_frame", &asrLib},
+		{&DiarStreamOpen, "nemo_speech_diar_stream_open", &asrLib},
+		{&DiarStreamPushF32, "nemo_speech_diar_stream_push_f32", &asrLib},
+		{&DiarStreamFinish, "nemo_speech_diar_stream_finish", &asrLib},
+		{&DiarStreamClose, "nemo_speech_diar_stream_close", &asrLib},
+		{&DiarSegments, "nemo_speech_diar_segments", &asrLib},
+
+		{&TTSCreate, "nemo_speech_tts_create", &ttsLib},
+		{&TTSDestroy, "nemo_speech_tts_destroy", &ttsLib},
+		{&TTSSampleRate, "nemo_speech_tts_sample_rate", &ttsLib},
+		{&TTSSpeakerCount, "nemo_speech_tts_speaker_count", &ttsLib},
+		{&TTSSpeakerName, "nemo_speech_tts_speaker_name", &ttsLib},
+		{&TTSSynthesizeText, "nemo_speech_tts_synthesize_text", &ttsLib},
+		{&TTSRuntimeConfigDefault, "nemo_speech_tts_runtime_config_default", &ttsLib},
+		{&TTSSynthesisOptionsDefault, "nemo_speech_tts_synthesis_options_default", &ttsLib},
+		{&TTSLastError, "nemo_speech_tts_last_error", &ttsLib},
+		{&TTSVersion, "nemo_speech_tts_version", &ttsLib},
+
+		{&NMTCreate, "nemo_speech_nmt_create", &nmtLib},
+		{&NMTDestroy, "nemo_speech_nmt_destroy", &nmtLib},
+		{&NMTTranslate, "nemo_speech_nmt_translate", &nmtLib},
+		{&NMTResultCount, "nemo_speech_nmt_result_count", &nmtLib},
+		{&NMTResultText, "nemo_speech_nmt_result_text", &nmtLib},
+		{&NMTResultLanguage, "nemo_speech_nmt_result_language", &nmtLib},
+		{&NMTResultDestroy, "nemo_speech_nmt_result_destroy", &nmtLib},
+		{&NMTLastError, "nemo_speech_nmt_last_error", &nmtLib},
+		{&NMTVersion, "nemo_speech_nmt_version", &nmtLib},
+	}
+}
+
+// registerSymbols binds every entry point. purego panics on a missing symbol,
+// so this recovers and returns the offending name: after an upstream pin bump a
+// rename must fail loudly at startup, not at first inference.
+func registerSymbols() error {
+	for _, s := range symbols() {
+		if err := registerOne(s); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func registerOne(s symbol) (err error) {
+	defer func() {
+		if r := recover(); r != nil {
+			err = fmt.Errorf("nemo-speech-cpp: binding %q: %v", s.name, r)
+		}
+	}()
+	purego.RegisterLibFunc(s.fn, *s.lib, s.name)
+	return nil
+}

backend/go/nemo-speech-cpp/abi_test.go

@@ -0,0 +1,317 @@
+package main
+
+import (
+	"os"
+	"unsafe"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// requireLibs reports whether a missing shared library must fail the specs
+// instead of skipping them.
+//
+// librariesPresent stats bare filenames relative to the working directory,
+// while openLibraries resolves them through the loader search path, so the two
+// can legitimately disagree. Under `make test` that is harmless because the
+// stage-libs prerequisite puts the .so files in the working directory, but any
+// other invocation would skip every library-backed spec and still report a
+// green run. The Makefile sets NEMO_SPEECH_REQUIRE_LIBS=1 so no CI path can
+// pass on a silent skip; leaving it unset keeps the pure-Go specs runnable on a
+// checkout with no build.
+func requireLibs() bool {
+	return os.Getenv("NEMO_SPEECH_REQUIRE_LIBS") == "1"
+}
+
+// librariesPresent reports whether a local build is available to bind against.
+func librariesPresent() bool {
+	for _, n := range []string{
+		libraryName("NEMO_SPEECH_ASR_LIBRARY", "libnemo_speech_asr_c"),
+		libraryName("NEMO_SPEECH_TTS_LIBRARY", "libnemo_speech_tts"),
+		libraryName("NEMO_SPEECH_NMT_LIBRARY", "libnemo_speech_nmt_c"),
+	} {
+		if _, err := os.Stat(n); err != nil {
+			return false
+		}
+	}
+	return true
+}
+
+// layout is one expected number transcribed from the C headers.
+type layout struct {
+	what string
+	got  uintptr
+	want uintptr
+}
+
+// The `want` column is what a C compiler reports for the structs in
+// include/nemo_speech/{asr,diar,tts,nmt}.h under the System V AMD64 / AAPCS64 rules
+// both supported targets follow. Regenerate after an upstream pin bump with a
+// throwaway program over the installed headers:
+//
+//	printf('SIZE %%zu\n', sizeof(nemo_speech_asr_recognition_options));
+//	printf('OFF  %%zu\n', offsetof(nemo_speech_asr_recognition_options, max_speaker_count));
+//
+// Sizes alone are not enough: two padding mistakes can cancel out and leave the
+// total unchanged while every field between them reads from the wrong offset,
+// so each mirror pins its field offsets too.
+func structSizes() []layout {
+	return []layout{
+		{"cASRBackendConfig", unsafe.Sizeof(cASRBackendConfig{}), 16},
+		{"cASRModelConfig", unsafe.Sizeof(cASRModelConfig{}), 24},
+		{"cASRVADConfig", unsafe.Sizeof(cASRVADConfig{}), 32},
+		{"cASRPostprocConfig", unsafe.Sizeof(cASRPostprocConfig{}), 32},
+		{"cASRDiarConfig", unsafe.Sizeof(cASRDiarConfig{}), 40},
+		{"cASRRecognizerConfig", unsafe.Sizeof(cASRRecognizerConfig{}), 80},
+		{"cASRRecognitionOptions", unsafe.Sizeof(cASRRecognitionOptions{}), 72},
+		{"cDiarModelConfig", unsafe.Sizeof(cDiarModelConfig{}), 56},
+		{"cDiarSegmentationConfig", unsafe.Sizeof(cDiarSegmentationConfig{}), 48},
+		{"cDiarSegment", unsafe.Sizeof(cDiarSegment{}), 24},
+		{"cTTSModelConfig", unsafe.Sizeof(cTTSModelConfig{}), 40},
+		{"cTTSRuntimeConfig", unsafe.Sizeof(cTTSRuntimeConfig{}), 96},
+		{"cTTSSynthesizerConfig", unsafe.Sizeof(cTTSSynthesizerConfig{}), 40},
+		{"cTTSSynthesisOptions", unsafe.Sizeof(cTTSSynthesisOptions{}), 72},
+		{"cNMTBackendConfig", unsafe.Sizeof(cNMTBackendConfig{}), 16},
+		{"cNMTModelConfig", unsafe.Sizeof(cNMTModelConfig{}), 24},
+		{"cNMTTranslatorConfig", unsafe.Sizeof(cNMTTranslatorConfig{}), 40},
+	}
+}
+
+func structOffsets() []layout {
+	return []layout{
+		{"cASRBackendConfig.GPU", unsafe.Offsetof(cASRBackendConfig{}.GPU), 8},
+
+		{"cASRModelConfig.Path", unsafe.Offsetof(cASRModelConfig{}.Path), 8},
+		{"cASRModelConfig.Name", unsafe.Offsetof(cASRModelConfig{}.Name), 16},
+
+		{"cASRVADConfig.ModelPath", unsafe.Offsetof(cASRVADConfig{}.ModelPath), 8},
+		{"cASRVADConfig.EnableMasking", unsafe.Offsetof(cASRVADConfig{}.EnableMasking), 16},
+		{"cASRVADConfig.Onset", unsafe.Offsetof(cASRVADConfig{}.Onset), 20},
+		{"cASRVADConfig.Offset", unsafe.Offsetof(cASRVADConfig{}.Offset), 24},
+
+		{"cASRPostprocConfig.ProfanityListPath", unsafe.Offsetof(cASRPostprocConfig{}.ProfanityListPath), 8},
+		{"cASRPostprocConfig.ITNModelDir", unsafe.Offsetof(cASRPostprocConfig{}.ITNModelDir), 16},
+		{"cASRPostprocConfig.PNCModelPath", unsafe.Offsetof(cASRPostprocConfig{}.PNCModelPath), 24},
+
+		{"cASRDiarConfig.ModelPath", unsafe.Offsetof(cASRDiarConfig{}.ModelPath), 8},
+		{"cASRDiarConfig.ChunkFrames", unsafe.Offsetof(cASRDiarConfig{}.ChunkFrames), 16},
+		{"cASRDiarConfig.RightContextFrames", unsafe.Offsetof(cASRDiarConfig{}.RightContextFrames), 20},
+		{"cASRDiarConfig.LeftContextFrames", unsafe.Offsetof(cASRDiarConfig{}.LeftContextFrames), 24},
+		{"cASRDiarConfig.FIFOFrames", unsafe.Offsetof(cASRDiarConfig{}.FIFOFrames), 28},
+		{"cASRDiarConfig.SpkcacheFrames", unsafe.Offsetof(cASRDiarConfig{}.SpkcacheFrames), 32},
+		{"cASRDiarConfig.UpdatePeriodFrames", unsafe.Offsetof(cASRDiarConfig{}.UpdatePeriodFrames), 36},
+
+		{"cASRRecognizerConfig.Backend", unsafe.Offsetof(cASRRecognizerConfig{}.Backend), 8},
+		{"cASRRecognizerConfig.Model", unsafe.Offsetof(cASRRecognizerConfig{}.Model), 16},
+		{"cASRRecognizerConfig.Streaming", unsafe.Offsetof(cASRRecognizerConfig{}.Streaming), 24},
+		{"cASRRecognizerConfig.Decoder", unsafe.Offsetof(cASRRecognizerConfig{}.Decoder), 32},
+		{"cASRRecognizerConfig.VAD", unsafe.Offsetof(cASRRecognizerConfig{}.VAD), 40},
+		{"cASRRecognizerConfig.Endpointing", unsafe.Offsetof(cASRRecognizerConfig{}.Endpointing), 48},
+		{"cASRRecognizerConfig.Postproc", unsafe.Offsetof(cASRRecognizerConfig{}.Postproc), 56},
+		{"cASRRecognizerConfig.Diar", unsafe.Offsetof(cASRRecognizerConfig{}.Diar), 64},
+		{"cASRRecognizerConfig.Batching", unsafe.Offsetof(cASRRecognizerConfig{}.Batching), 72},
+
+		{"cASRRecognitionOptions.RequestID", unsafe.Offsetof(cASRRecognitionOptions{}.RequestID), 8},
+		{"cASRRecognitionOptions.LanguageCode", unsafe.Offsetof(cASRRecognitionOptions{}.LanguageCode), 16},
+		{"cASRRecognitionOptions.InterimResults", unsafe.Offsetof(cASRRecognitionOptions{}.InterimResults), 24},
+		{"cASRRecognitionOptions.EnableWordTimeOffsets", unsafe.Offsetof(cASRRecognitionOptions{}.EnableWordTimeOffsets), 25},
+		{"cASRRecognitionOptions.EnableAutomaticPunctuation", unsafe.Offsetof(cASRRecognitionOptions{}.EnableAutomaticPunctuation), 26},
+		{"cASRRecognitionOptions.VerbatimTranscripts", unsafe.Offsetof(cASRRecognitionOptions{}.VerbatimTranscripts), 27},
+		{"cASRRecognitionOptions.ProfanityFilter", unsafe.Offsetof(cASRRecognitionOptions{}.ProfanityFilter), 28},
+		{"cASRRecognitionOptions.StopHistoryEouMs", unsafe.Offsetof(cASRRecognitionOptions{}.StopHistoryEouMs), 32},
+		{"cASRRecognitionOptions.SpeechContexts", unsafe.Offsetof(cASRRecognitionOptions{}.SpeechContexts), 40},
+		{"cASRRecognitionOptions.SpeechContextCount", unsafe.Offsetof(cASRRecognitionOptions{}.SpeechContextCount), 48},
+		{"cASRRecognitionOptions.MaxAlternatives", unsafe.Offsetof(cASRRecognitionOptions{}.MaxAlternatives), 56},
+		{"cASRRecognitionOptions.EnableSpeakerDiarization", unsafe.Offsetof(cASRRecognitionOptions{}.EnableSpeakerDiarization), 60},
+		{"cASRRecognitionOptions.MaxSpeakerCount", unsafe.Offsetof(cASRRecognitionOptions{}.MaxSpeakerCount), 64},
+
+		{"cDiarModelConfig.ModelPath", unsafe.Offsetof(cDiarModelConfig{}.ModelPath), 8},
+		{"cDiarModelConfig.GPU", unsafe.Offsetof(cDiarModelConfig{}.GPU), 16},
+		{"cDiarModelConfig.Preset", unsafe.Offsetof(cDiarModelConfig{}.Preset), 24},
+		{"cDiarModelConfig.ChunkFrames", unsafe.Offsetof(cDiarModelConfig{}.ChunkFrames), 32},
+		{"cDiarModelConfig.RightContextFrames", unsafe.Offsetof(cDiarModelConfig{}.RightContextFrames), 36},
+		{"cDiarModelConfig.LeftContextFrames", unsafe.Offsetof(cDiarModelConfig{}.LeftContextFrames), 40},
+		{"cDiarModelConfig.FIFOFrames", unsafe.Offsetof(cDiarModelConfig{}.FIFOFrames), 44},
+		{"cDiarModelConfig.SpkcacheFrames", unsafe.Offsetof(cDiarModelConfig{}.SpkcacheFrames), 48},
+		{"cDiarModelConfig.UpdatePeriodFrames", unsafe.Offsetof(cDiarModelConfig{}.UpdatePeriodFrames), 52},
+
+		{"cDiarSegmentationConfig.Onset", unsafe.Offsetof(cDiarSegmentationConfig{}.Onset), 8},
+		{"cDiarSegmentationConfig.Offset", unsafe.Offsetof(cDiarSegmentationConfig{}.Offset), 12},
+		{"cDiarSegmentationConfig.PadOnsetSec", unsafe.Offsetof(cDiarSegmentationConfig{}.PadOnsetSec), 16},
+		{"cDiarSegmentationConfig.PadOffsetSec", unsafe.Offsetof(cDiarSegmentationConfig{}.PadOffsetSec), 24},
+		{"cDiarSegmentationConfig.MinGapSec", unsafe.Offsetof(cDiarSegmentationConfig{}.MinGapSec), 32},
+		{"cDiarSegmentationConfig.MinDurationSec", unsafe.Offsetof(cDiarSegmentationConfig{}.MinDurationSec), 40},
+
+		{"cDiarSegment.StartTime", unsafe.Offsetof(cDiarSegment{}.StartTime), 0},
+		{"cDiarSegment.EndTime", unsafe.Offsetof(cDiarSegment{}.EndTime), 8},
+		{"cDiarSegment.Speaker", unsafe.Offsetof(cDiarSegment{}.Speaker), 16},
+
+		{"cTTSModelConfig.MagpieModel", unsafe.Offsetof(cTTSModelConfig{}.MagpieModel), 8},
+		{"cTTSModelConfig.CodecModel", unsafe.Offsetof(cTTSModelConfig{}.CodecModel), 16},
+		{"cTTSModelConfig.TokenizerModelDir", unsafe.Offsetof(cTTSModelConfig{}.TokenizerModelDir), 24},
+		{"cTTSModelConfig.TextNormalizerModelDir", unsafe.Offsetof(cTTSModelConfig{}.TextNormalizerModelDir), 32},
+
+		{"cTTSRuntimeConfig.Speaker", unsafe.Offsetof(cTTSRuntimeConfig{}.Speaker), 8},
+		{"cTTSRuntimeConfig.Threads", unsafe.Offsetof(cTTSRuntimeConfig{}.Threads), 12},
+		{"cTTSRuntimeConfig.CodecThreads", unsafe.Offsetof(cTTSRuntimeConfig{}.CodecThreads), 16},
+		{"cTTSRuntimeConfig.Seed", unsafe.Offsetof(cTTSRuntimeConfig{}.Seed), 20},
+		{"cTTSRuntimeConfig.Steps", unsafe.Offsetof(cTTSRuntimeConfig{}.Steps), 24},
+		{"cTTSRuntimeConfig.TopK", unsafe.Offsetof(cTTSRuntimeConfig{}.TopK), 28},
+		{"cTTSRuntimeConfig.ChunkFrames", unsafe.Offsetof(cTTSRuntimeConfig{}.ChunkFrames), 32},
+		{"cTTSRuntimeConfig.CodecQueueDepth", unsafe.Offsetof(cTTSRuntimeConfig{}.CodecQueueDepth), 36},
+		{"cTTSRuntimeConfig.CodecHistoryFrames", unsafe.Offsetof(cTTSRuntimeConfig{}.CodecHistoryFrames), 40},
+		{"cTTSRuntimeConfig.CodecFutureFrames", unsafe.Offsetof(cTTSRuntimeConfig{}.CodecFutureFrames), 44},
+		{"cTTSRuntimeConfig.WindowMs", unsafe.Offsetof(cTTSRuntimeConfig{}.WindowMs), 48},
+		{"cTTSRuntimeConfig.Temperature", unsafe.Offsetof(cTTSRuntimeConfig{}.Temperature), 52},
+		{"cTTSRuntimeConfig.OverrideTemperature", unsafe.Offsetof(cTTSRuntimeConfig{}.OverrideTemperature), 56},
+		{"cTTSRuntimeConfig.CFGScale", unsafe.Offsetof(cTTSRuntimeConfig{}.CFGScale), 60},
+		{"cTTSRuntimeConfig.OverrideCFGScale", unsafe.Offsetof(cTTSRuntimeConfig{}.OverrideCFGScale), 64},
+		{"cTTSRuntimeConfig.UseCFG", unsafe.Offsetof(cTTSRuntimeConfig{}.UseCFG), 65},
+		{"cTTSRuntimeConfig.UseLocalTransformer", unsafe.Offsetof(cTTSRuntimeConfig{}.UseLocalTransformer), 66},
+		{"cTTSRuntimeConfig.UseKVCache", unsafe.Offsetof(cTTSRuntimeConfig{}.UseKVCache), 67},
+		{"cTTSRuntimeConfig.UseStatefulCodec", unsafe.Offsetof(cTTSRuntimeConfig{}.UseStatefulCodec), 68},
+		{"cTTSRuntimeConfig.CodecCPU", unsafe.Offsetof(cTTSRuntimeConfig{}.CodecCPU), 69},
+		{"cTTSRuntimeConfig.FlushPartialChunk", unsafe.Offsetof(cTTSRuntimeConfig{}.FlushPartialChunk), 70},
+		{"cTTSRuntimeConfig.Verbose", unsafe.Offsetof(cTTSRuntimeConfig{}.Verbose), 71},
+		{"cTTSRuntimeConfig.LTBackend", unsafe.Offsetof(cTTSRuntimeConfig{}.LTBackend), 72},
+		{"cTTSRuntimeConfig.SamplingBackend", unsafe.Offsetof(cTTSRuntimeConfig{}.SamplingBackend), 76},
+		{"cTTSRuntimeConfig.UMAMode", unsafe.Offsetof(cTTSRuntimeConfig{}.UMAMode), 80},
+		{"cTTSRuntimeConfig.LongformMode", unsafe.Offsetof(cTTSRuntimeConfig{}.LongformMode), 84},
+		{"cTTSRuntimeConfig.LTFP32", unsafe.Offsetof(cTTSRuntimeConfig{}.LTFP32), 88},
+
+		{"cTTSSynthesizerConfig.Model", unsafe.Offsetof(cTTSSynthesizerConfig{}.Model), 8},
+		{"cTTSSynthesizerConfig.Runtime", unsafe.Offsetof(cTTSSynthesizerConfig{}.Runtime), 16},
+		{"cTTSSynthesizerConfig.DefaultLanguageCode", unsafe.Offsetof(cTTSSynthesizerConfig{}.DefaultLanguageCode), 24},
+		{"cTTSSynthesizerConfig.DefaultVoiceName", unsafe.Offsetof(cTTSSynthesizerConfig{}.DefaultVoiceName), 32},
+
+		{"cTTSSynthesisOptions.RequestID", unsafe.Offsetof(cTTSSynthesisOptions{}.RequestID), 8},
+		{"cTTSSynthesisOptions.LanguageCode", unsafe.Offsetof(cTTSSynthesisOptions{}.LanguageCode), 16},
+		{"cTTSSynthesisOptions.Speaker", unsafe.Offsetof(cTTSSynthesisOptions{}.Speaker), 24},
+		{"cTTSSynthesisOptions.Seed", unsafe.Offsetof(cTTSSynthesisOptions{}.Seed), 28},
+		{"cTTSSynthesisOptions.Steps", unsafe.Offsetof(cTTSSynthesisOptions{}.Steps), 32},
+		{"cTTSSynthesisOptions.TopK", unsafe.Offsetof(cTTSSynthesisOptions{}.TopK), 36},
+		{"cTTSSynthesisOptions.Temperature", unsafe.Offsetof(cTTSSynthesisOptions{}.Temperature), 40},
+		{"cTTSSynthesisOptions.OverrideTemperature", unsafe.Offsetof(cTTSSynthesisOptions{}.OverrideTemperature), 44},
+		{"cTTSSynthesisOptions.CFGScale", unsafe.Offsetof(cTTSSynthesisOptions{}.CFGScale), 48},
+		{"cTTSSynthesisOptions.OverrideCFGScale", unsafe.Offsetof(cTTSSynthesisOptions{}.OverrideCFGScale), 52},
+		{"cTTSSynthesisOptions.VoiceName", unsafe.Offsetof(cTTSSynthesisOptions{}.VoiceName), 56},
+		{"cTTSSynthesisOptions.OutputSampleRate", unsafe.Offsetof(cTTSSynthesisOptions{}.OutputSampleRate), 64},
+
+		{"cNMTBackendConfig.GPU", unsafe.Offsetof(cNMTBackendConfig{}.GPU), 8},
+		{"cNMTModelConfig.Path", unsafe.Offsetof(cNMTModelConfig{}.Path), 8},
+		{"cNMTModelConfig.NCtx", unsafe.Offsetof(cNMTModelConfig{}.NCtx), 16},
+
+		{"cNMTTranslatorConfig.Backend", unsafe.Offsetof(cNMTTranslatorConfig{}.Backend), 8},
+		{"cNMTTranslatorConfig.Model", unsafe.Offsetof(cNMTTranslatorConfig{}.Model), 16},
+		{"cNMTTranslatorConfig.Generation", unsafe.Offsetof(cNMTTranslatorConfig{}.Generation), 24},
+		{"cNMTTranslatorConfig.Pool", unsafe.Offsetof(cNMTTranslatorConfig{}.Pool), 32},
+	}
+}
+
+var _ = Describe("C struct mirrors", func() {
+	// These need no shared object, so they run on any checkout and catch a
+	// transcription slip the moment it is introduced.
+	It("matches the C sizeof of every mirrored struct", func() {
+		for _, l := range structSizes() {
+			Expect(l.got).To(Equal(l.want), "%s: Go mirror is %d bytes, C says %d", l.what, l.got, l.want)
+		}
+	})
+
+	It("matches the C offset of every mirrored field", func() {
+		for _, l := range structOffsets() {
+			Expect(l.got).To(Equal(l.want), "%s: Go offset %d, C offset %d", l.what, l.got, l.want)
+		}
+	})
+})
+
+var _ = Describe("C ABI binding", func() {
+	BeforeEach(func() {
+		if !librariesPresent() {
+			if requireLibs() {
+				cwd, _ := os.Getwd()
+				Fail("NEMO_SPEECH_REQUIRE_LIBS=1 but the shared libraries are not in " + cwd +
+					": these specs are the ABI defence and must not be skipped." +
+					" Run make -C backend/go/nemo-speech-cpp stage-libs")
+			}
+			Skip("shared libraries not built, run make in backend/go/nemo-speech-cpp")
+		}
+		Expect(openLibraries()).To(Succeed())
+	})
+
+	It("resolves every bound symbol", func() {
+		Expect(symbols()).ToNot(BeEmpty())
+		for _, s := range symbols() {
+			Expect(registerOne(s)).To(Succeed())
+		}
+	})
+
+	// The library reports its own sizeof through the size field of each
+	// defaults struct. A Go mirror that disagrees means every field after the
+	// first divergence is read from the wrong offset, which no compiler or
+	// linker check would catch. The three structs below are the only ones with
+	// a defaults entry point, so they are the only ones the runtime can be
+	// asked about directly.
+	It("mirrors the C recognition-options struct layout", func() {
+		def := ASRRecognitionOptionsDef()
+		Expect(def.Size).To(Equal(unsafe.Sizeof(cASRRecognitionOptions{})),
+			"cASRRecognitionOptions does not match the C layout")
+	})
+
+	It("mirrors the C TTS runtime-config struct layout", func() {
+		def := TTSRuntimeConfigDefault()
+		Expect(def.Size).To(Equal(unsafe.Sizeof(cTTSRuntimeConfig{})),
+			"cTTSRuntimeConfig does not match the C layout")
+	})
+
+	It("mirrors the C TTS synthesis-options struct layout", func() {
+		def := TTSSynthesisOptionsDefault()
+		Expect(def.Size).To(Equal(unsafe.Sizeof(cTTSSynthesisOptions{})),
+			"cTTSSynthesisOptions does not match the C layout")
+	})
+
+	// A size match alone cannot see a field read from the wrong offset when two
+	// padding mistakes cancel out, and structOffsets checks the mirrors against
+	// numbers transcribed by the same hand that wrote them. This spec is the
+	// only layer independent of that transcription: it reads values back out of
+	// the running library, so a systematically wrong table cannot hide here.
+	//
+	// Deliberately narrow. An earlier version pinned roughly forty default
+	// values, which would make a legitimate pin bump (threads 4 to 8, or a
+	// flipped flush_partial_chunk) fail with a message that reads like a layout
+	// error. What survives is only the values that are contract, not tuning:
+	//
+	//   - max_alternatives is the single non-zero in an otherwise memset-zero
+	//     struct, and asr.h documents "<= 1 = 1-best (default)". It pins offset
+	//     56, deep in the tail past the bool run.
+	//   - The synthesis-options run of four -1 sentinels, each documented in
+	//     tts.h as "< 0 = synthesizer default", pins offsets 24 through 36, and
+	//     temperature witnesses that the run stops exactly at offset 40. A
+	//     mirror whose tail is shifted by one field spills a -1 into that zero.
+	//   - Two -1 sentinels at the ends of the runtime config's long int32 run
+	//     pin offset 20 and offset 40 without depending on any tunable.
+	//
+	// Sources: src/asr/c_api.cpp nemo_speech_asr_recognition_options_default,
+	// src/tts/magpietts/runtime.h MagpieRuntimeConfig, src/tts/c_api.cpp
+	// nemo_speech_tts_synthesis_options_default.
+	It("reads the documented default values back through the mirrors", func() {
+		asr := ASRRecognitionOptionsDef()
+		Expect(asr.MaxAlternatives).To(Equal(int32(1)))
+
+		rt := TTSRuntimeConfigDefault()
+		Expect(rt.Seed).To(Equal(int32(-1)))
+		Expect(rt.CodecHistoryFrames).To(Equal(int32(-1)))
+
+		opt := TTSSynthesisOptionsDefault()
+		Expect(opt.Speaker).To(Equal(int32(-1)))
+		Expect(opt.Seed).To(Equal(int32(-1)))
+		Expect(opt.Steps).To(Equal(int32(-1)))
+		Expect(opt.TopK).To(Equal(int32(-1)))
+		Expect(opt.Temperature).To(Equal(float32(0)))
+	})
+
+	It("reports a non-empty version from each library", func() {
+		Expect(ASRVersion()).ToNot(BeEmpty())
+		Expect(TTSVersion()).ToNot(BeEmpty())
+		Expect(NMTVersion()).ToNot(BeEmpty())
+	})
+})

backend/go/nemo-speech-cpp/asr.go

@@ -0,0 +1,374 @@
+package main
+
+import (
+	"context"
+	"runtime"
+	"strconv"
+	"strings"
+	"time"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// asrWord is one decoded word with its millisecond offsets and 1-based speaker
+// tag (0 when diarization was not requested).
+type asrWord struct {
+	Text    string
+	Start   int32
+	End     int32
+	Speaker int32
+}
+
+// pinPtr pins v for the lifetime of p and returns its address in the uintptr
+// form the config structs carry.
+//
+// The config structs mirror C, so their pointer members are uintptr, which the
+// collector does not trace. Everything reachable only through one of them is
+// therefore invisible to the GC while C is reading it, exactly as described on
+// cstr, and needs the same pin. runtime.KeepAlive would cover collection but
+// says nothing about relocation, and the guarantee wanted here is that the
+// address C holds stays the address of the object.
+func pinPtr[T any](p *runtime.Pinner, v *T) uintptr {
+	p.Pin(v)
+	// #nosec G103 -- v is pinned into p on the previous line, so its address is
+	// stable and traced for as long as p lives; every caller defers p.Unpin only
+	// after the create call that reads it. One-way, like cstr: nothing converts
+	// this uintptr back to a pointer.
+	return uintptr(unsafe.Pointer(v))
+}
+
+// asrDiarConfig builds the config for the diarizer attached to a recognizer.
+//
+// Extracted from loadASR for the same reason diarModelConfig was extracted from
+// loadDiarizer: the six frame counts are sentinel-sensitive and invisible to
+// every other check in the tree. src/asr/c_api.cpp:151-165 applies five of them
+// when they are > 0 but applies left_context_frames when it is >= 0, so a
+// dropped -1 does not fall back to the model's own streaming geometry, it pins
+// the left context to zero. The struct is the right shape either way, so the
+// layout assertions in abi_test.go cannot see it and only a spec on this builder
+// can.
+//
+// diarGeometryDefault is shared with the standalone diarizer rather than
+// restated: it is the same sentinel, from the same rule, in the same runtime.
+//
+// modelPath is a C pointer from cstr, not a Go string, and the caller owns its
+// release.
+func asrDiarConfig(modelPath uintptr) cASRDiarConfig {
+	return cASRDiarConfig{
+		Size:               unsafe.Sizeof(cASRDiarConfig{}),
+		ModelPath:          modelPath,
+		ChunkFrames:        diarGeometryDefault,
+		RightContextFrames: diarGeometryDefault,
+		LeftContextFrames:  diarGeometryDefault,
+		FIFOFrames:         diarGeometryDefault,
+		SpkcacheFrames:     diarGeometryDefault,
+		UpdatePeriodFrames: diarGeometryDefault,
+	}
+}
+
+// loadASR creates the recognizer, attaching VAD, PnC, ITN and diarization when
+// the corresponding options were set.
+//
+// Every field below is assigned by name against include/nemo_speech/asr.h. The
+// sub-configs are optional pointers: a nil one means "library defaults", which
+// is why each is populated only when its option was given rather than always
+// being attached with empty strings.
+//
+// Each struct's Size is load-bearing, not decoration. The runtime decides a
+// field is present with HAS_FIELD (src/asr/c_api.cpp), which tests the caller's
+// size against offsetof(field) + sizeof(field), so a config sent with Size 0
+// has every field ignored and the model silently loads with defaults.
+//
+// This must not take engineMu: Load is its only caller and already holds it.
+func (n *NemoSpeech) loadASR(modelFile string) error {
+	// nemo_speech_asr_create deep-copies every const char* into a std::string
+	// (src/asr/c_api.cpp to_config, via str_or_empty) and retains no pointer
+	// afterwards, so pinning for the duration of the create call is both
+	// necessary and sufficient.
+	var pinner runtime.Pinner
+	defer pinner.Unpin()
+
+	pathP, freePath := cstr(modelFile)
+	defer freePath()
+
+	model := cASRModelConfig{Size: unsafe.Sizeof(cASRModelConfig{}), Path: pathP}
+	backend := cASRBackendConfig{Size: unsafe.Sizeof(cASRBackendConfig{}), GPU: n.opts.gpu}
+
+	cfg := cASRRecognizerConfig{
+		Size:    unsafe.Sizeof(cASRRecognizerConfig{}),
+		Backend: pinPtr(&pinner, &backend),
+		Model:   pinPtr(&pinner, &model),
+	}
+
+	var vad cASRVADConfig
+	if n.opts.vadModel != "" {
+		p, free := cstr(n.opts.vadModel)
+		defer free()
+		vad = cASRVADConfig{Size: unsafe.Sizeof(cASRVADConfig{}), ModelPath: p}
+		cfg.VAD = pinPtr(&pinner, &vad)
+	}
+
+	var postproc cASRPostprocConfig
+	if n.opts.itnDir != "" || n.opts.pncModel != "" {
+		itnP, freeITN := cstr(n.opts.itnDir)
+		defer freeITN()
+		pncP, freePNC := cstr(n.opts.pncModel)
+		defer freePNC()
+		postproc = cASRPostprocConfig{
+			Size:         unsafe.Sizeof(cASRPostprocConfig{}),
+			ITNModelDir:  itnP,
+			PNCModelPath: pncP,
+		}
+		cfg.Postproc = pinPtr(&pinner, &postproc)
+	}
+
+	var diar cASRDiarConfig
+	if n.opts.diarModel != "" {
+		p, free := cstr(n.opts.diarModel)
+		defer free()
+		diar = asrDiarConfig(p)
+		cfg.Diar = pinPtr(&pinner, &diar)
+	}
+
+	xlog.Info("nemo-speech-cpp: creating recognizer",
+		"gpu", n.opts.gpu,
+		"vad", n.opts.vadModel != "",
+		"pnc", n.opts.pncModel != "",
+		"itn", n.opts.itnDir != "",
+		"diarization", n.opts.diarModel != "")
+
+	// #nosec G103 -- cfg is a local POD struct passed as a pointer for the
+	// duration of this call only; every uintptr member it carries is either a
+	// cstr allocation or a pinPtr address, all pinned above and released by the
+	// defers, and nemo_speech_asr_create deep-copies and retains nothing.
+	if st := ASRCreate(unsafe.Pointer(&cfg), &n.recognizer); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: asr create: %s", ASRLastError())
+	}
+	return nil
+}
+
+// recognizeF32 runs one offline decode and returns the result handle, which the
+// caller must destroy.
+//
+// The empty-input guard is here rather than at the call site because &pcm[0]
+// panics on a zero-length slice: Go never reaches the C side's own "empty
+// audio" rejection. A silent clip or a truncated upload decodes to zero
+// samples, which is ordinary input, not an exotic one.
+//
+// The caller must hold engineMu.
+func recognizeF32(recognizer uintptr, opts *cASRRecognitionOptions, pcm []float32, sampleRate int32) (uintptr, error) {
+	if len(pcm) == 0 {
+		return 0, status.Error(codes.InvalidArgument, "nemo-speech-cpp: empty audio")
+	}
+
+	var result uintptr
+	// #nosec G103 -- opts is the caller's live struct, borrowed for this call
+	// only; its LanguageCode is a cstr allocation the caller keeps pinned across
+	// it. &pcm[0] is guarded by the empty check above and the length handed over
+	// is exactly len(pcm), so the runtime cannot read past the slice.
+	if st := ASRRecognizeF32(recognizer, unsafe.Pointer(opts),
+		&pcm[0], uint64(len(pcm)), sampleRate, &result); st != 0 {
+		return 0, statusErrorf(st, "nemo-speech-cpp: recognize: %s", ASRLastError())
+	}
+	return result, nil
+}
+
+// msToNanos converts a runtime word offset to the wire unit. The runtime
+// reports milliseconds (src/asr/types.h); TranscriptSegment.start/end and
+// TranscriptWord.start/end are int64 nanoseconds, which core/backend reads
+// straight into a time.Duration.
+func msToNanos(ms int32) int64 {
+	return int64(ms) * int64(time.Millisecond)
+}
+
+// extractWords pulls the top alternative's words out of a result handle.
+func extractWords(result uintptr) []asrWord {
+	if ASRResultAlternativeCount(result) == 0 {
+		return nil
+	}
+	count := ASRResultWordCount(result, 0)
+	words := make([]asrWord, 0, count)
+	for i := uint64(0); i < count; i++ {
+		words = append(words, asrWord{
+			Text:    ASRResultWordText(result, 0, i),
+			Start:   ASRResultWordStartTime(result, 0, i),
+			End:     ASRResultWordEndTime(result, 0, i),
+			Speaker: ASRResultWordSpeakerTag(result, 0, i),
+		})
+	}
+	return words
+}
+
+// wordsRequested reports whether the caller asked for word-level timestamps.
+// The OpenAI transcription API gates word timings behind
+// timestamp_granularities[] containing "word" and defaults to segment level
+// otherwise; every backend here follows that contract (see
+// backend/go/parakeet-cpp).
+func wordsRequested(granularities []string) bool {
+	for _, g := range granularities {
+		if strings.EqualFold(strings.TrimSpace(g), "word") {
+			return true
+		}
+	}
+	return false
+}
+
+// wordsToSegments groups words into one segment per consecutive speaker run.
+// Without diarization every word carries speaker 0, so this collapses to a
+// single segment.
+//
+// The boundary is a CHANGE of speaker, not the first appearance of one: a
+// conversation that returns to an earlier speaker has to start a new turn
+// rather than reopen the old one.
+//
+// withWords additionally attaches the per-word timings that
+// core/backend/transcript.go turns into the response's word list. It is off by
+// default because the OpenAI contract asks for word timestamps explicitly, and
+// a long transcript pays for every word twice otherwise.
+func wordsToSegments(words []asrWord, withWords bool) []*pb.TranscriptSegment {
+	if len(words) == 0 {
+		return nil
+	}
+
+	var segs []*pb.TranscriptSegment
+	start := 0
+	flush := func(end int) {
+		run := words[start:end]
+		texts := make([]string, 0, len(run))
+		for _, w := range run {
+			texts = append(texts, w.Text)
+		}
+		seg := &pb.TranscriptSegment{
+			// #nosec G115 -- TranscriptSegment.Id is int32 on the wire, and segs
+			// holds one entry per speaker run over the words of a single decode
+			// result, which exhausts memory long before it reaches 2^31.
+			Id:    int32(len(segs)),
+			Text:  strings.Join(texts, " "),
+			Start: msToNanos(run[0].Start),
+			End:   msToNanos(run[len(run)-1].End),
+		}
+		// The speaker tag is 1-based with 0 meaning untagged, so an undiarized
+		// run must stay unlabelled rather than be attributed to a speaker "0".
+		if run[0].Speaker > 0 {
+			seg.Speaker = strconv.Itoa(int(run[0].Speaker))
+		}
+		if withWords {
+			seg.Words = wordsToProto(run)
+		}
+		segs = append(segs, seg)
+	}
+
+	for i := 1; i < len(words); i++ {
+		if words[i].Speaker != words[start].Speaker {
+			flush(i)
+			start = i
+		}
+	}
+	flush(len(words))
+	return segs
+}
+
+// AudioTranscription decodes the audio at req.Dst and returns one offline
+// transcription.
+//
+// The whole body runs inside withEngine, so the family check and the C calls
+// that trust the handle happen under a single acquisition of engineMu. Decoding
+// the audio is in there too: pkg/grpc/server.go already serialises RPCs on this
+// backend through base.SingleThread, so the lock costs no concurrency, and the
+// alternative (check, unlock, decode, relock) is the exact gap Free can land in.
+func (n *NemoSpeech) AudioTranscription(ctx context.Context, req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
+	var out *pb.TranscriptResult
+	if err := n.withEngine(familyASR, func() error {
+		r, err := n.transcribe(req)
+		out = r
+		return err
+	}); err != nil {
+		return pb.TranscriptResult{}, err
+	}
+	// transcribe returns a non-nil result whenever it returns a nil error, so
+	// this cannot fire today. It is a guard rather than a comment because the
+	// alternative to stating the invariant is a nil dereference in an RPC
+	// handler if a later edit ever adds a success path that forgets to set it.
+	if out == nil {
+		return pb.TranscriptResult{}, status.Error(codes.Internal,
+			"nemo-speech-cpp: transcription produced no result")
+	}
+
+	// Assembled field by field rather than dereferenced: the RPC signature
+	// returns the proto message by value, but the message embeds a mutex, so
+	// copying the struct is a copylocks violation. Every backend in this tree
+	// gets around it the same way, by only ever returning a composite literal.
+	return pb.TranscriptResult{
+		Text:     out.Text,
+		Segments: out.Segments,
+		Language: out.Language,
+		Duration: out.Duration,
+	}, nil
+}
+
+// transcribe is AudioTranscription's body. The caller must hold engineMu.
+func (n *NemoSpeech) transcribe(req *pb.TranscriptRequest) (*pb.TranscriptResult, error) {
+	if req.GetDst() == "" {
+		return nil, status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: TranscriptRequest.dst (audio path) is required")
+	}
+
+	pcm, sampleRate, err := decodeAudioMono16k(req.GetDst())
+	if err != nil {
+		return nil, status.Errorf(codes.InvalidArgument,
+			"nemo-speech-cpp: read audio: %v", err)
+	}
+	// Rejected here, before anything crosses the ABI, and not only inside
+	// recognizeF32: a silent or truncated upload decodes to zero samples, and
+	// there is no point building options and pinning strings for a request
+	// that cannot produce a transcript. recognizeF32 keeps its own guard as a
+	// precondition on the function.
+	if len(pcm) == 0 {
+		return nil, status.Error(codes.InvalidArgument, "nemo-speech-cpp: empty audio")
+	}
+
+	// A per-request language wins over the model-level default; both may be
+	// empty, which the runtime reads as auto/model default.
+	language := req.GetLanguage()
+	if language == "" {
+		language = n.opts.languageCode
+	}
+	langP, freeLang := cstr(language)
+	defer freeLang()
+
+	opts := ASRRecognitionOptionsDef()
+	opts.LanguageCode = langP
+	// Segments are built out of word offsets, so they are always asked for.
+	opts.EnableWordTimeOffsets = true
+	// Keyed on the recognizer owning a diar model, not on req.Diarize: asr.h
+	// documents that a request asking for diarization from a recognizer created
+	// without one fails with INVALID_ARGUMENT, and setting diar_model is already
+	// the operator's opt-in.
+	opts.EnableSpeakerDiarization = n.opts.diarModel != ""
+
+	result, err := recognizeF32(n.recognizer, &opts, pcm, sampleRate)
+	if err != nil {
+		return nil, err
+	}
+	defer ASRResultDestroy(result)
+
+	out := &pb.TranscriptResult{
+		Text: ASRResultTranscript(result, 0),
+		Segments: wordsToSegments(extractWords(result),
+			wordsRequested(req.GetTimestampGranularities())),
+	}
+	// Multilingual models report what they decided the audio was; monolingual
+	// ones report nothing, and an empty language is better than echoing back
+	// whatever the caller guessed.
+	if ASRResultLanguageCount(result, 0) > 0 {
+		out.Language = ASRResultLanguageCode(result, 0, 0)
+	}
+	if sampleRate > 0 {
+		out.Duration = float32(len(pcm)) / float32(sampleRate)
+	}
+	return out, nil
+}

backend/go/nemo-speech-cpp/asr_stream.go

@@ -0,0 +1,526 @@
+package main
+
+import (
+	"context"
+	"strings"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// streamChunkSamples is one push into a streaming session. At 16 kHz mono 1600
+// samples is 100 ms, short enough that the decoder is polled often enough to
+// see an endpoint promptly and short enough that a cancelled request stops
+// within one push.
+const streamChunkSamples = 1600
+
+// The rates nemo_speech_asr_stream_push_f32 will resample from (asr.h). Outside
+// this range the runtime has nothing to do with the audio, and 0 is NOT
+// "unknown": it means "these samples are already at the model rate".
+const (
+	minStreamSampleRate = 8000
+	maxStreamSampleRate = 96000
+	// TranscriptLiveConfig.sample_rate documents 0 as 16 kHz, which is a
+	// different meaning from the C API's 0, so it is resolved before the push.
+	defaultLiveSampleRate = 16000
+)
+
+// streamResult is one result lifted out of C memory. Everything is copied
+// before nemo_speech_asr_result_destroy runs, so a streamResult outlives the
+// handle it came from.
+type streamResult struct {
+	Text  string
+	Final bool
+	Words []asrWord
+}
+
+// asrSession is the streaming half of the ASR C API, narrowed to the four
+// entry points the two streaming RPCs use.
+//
+// It is an interface because there is no NeMo GGUF small enough to keep in the
+// tree, so the loops on top of it (chunking, the need-more-audio drain, the
+// live config/reset protocol) would otherwise have no test at all. The seam is
+// at the ABI, not at the model: a fake session scripts what the C API returns,
+// it does not pretend to transcribe anything.
+type asrSession interface {
+	// push buffers audio. It does not decode; next drives that.
+	push(pcm []float32, sampleRate int32) error
+	// finish flushes the decoder tail. The end-of-stream final then comes back
+	// from next.
+	finish() error
+	// next pulls one result. ok=false means the decoder needs more audio,
+	// which is a pause in the stream and not an error or an end.
+	next() (result streamResult, ok bool, err error)
+	close()
+}
+
+// sessionOpener creates a session for one language. n.openSession is the
+// C-backed implementation.
+type sessionOpener func(language string) (asrSession, error)
+
+// cSession is the real asrSession, over one nemo_speech_asr_stream.
+type cSession struct {
+	handle uintptr
+}
+
+func (s *cSession) push(pcm []float32, sampleRate int32) error {
+	// &pcm[0] panics on an empty slice, and an empty frame is ordinary input
+	// from a live caller: it is a keepalive, not audio.
+	if len(pcm) == 0 {
+		return nil
+	}
+	if st := ASRStreamPushF32(s.handle, &pcm[0], uint64(len(pcm)), sampleRate); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: stream push: %s", ASRLastError())
+	}
+	return nil
+}
+
+func (s *cSession) finish() error {
+	if st := ASRStreamFinish(s.handle); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: stream finish: %s", ASRLastError())
+	}
+	return nil
+}
+
+func (s *cSession) next() (streamResult, bool, error) {
+	var handle uintptr
+	if st := ASRStreamNext(s.handle, &handle); st != 0 {
+		return streamResult{}, false, statusErrorf(st, "nemo-speech-cpp: stream next: %s", ASRLastError())
+	}
+	// OK with a NULL handle is the documented "need more audio". Reading it as
+	// an error aborts every stream at the first gap; reading it as "keep
+	// pulling" spins forever.
+	if handle == 0 {
+		return streamResult{}, false, nil
+	}
+	// Destroyed here rather than by the caller: everything below is copied out
+	// of C memory into Go values, so nothing survives that would need it, and
+	// a caller that returned early would otherwise leak the result.
+	defer ASRResultDestroy(handle)
+
+	return streamResult{
+		Text:  ASRResultTranscript(handle, 0),
+		Final: ASRResultIsFinal(handle),
+		Words: extractWords(handle),
+	}, true, nil
+}
+
+func (s *cSession) close() { ASRStreamClose(s.handle) }
+
+// openSession starts a streaming recognition on the loaded recognizer.
+//
+// The caller must hold engineMu.
+//
+// nemo_speech_asr_streaming_recognize copies the options (src/asr/c_api.cpp
+// to_options) and keeps no pointer into them, so the language buffer only has
+// to stay pinned across this call, exactly as in loadASR.
+func (n *NemoSpeech) openSession(language string) (asrSession, error) {
+	// A per-request language wins over the model-level default; both may be
+	// empty, which the runtime reads as auto/model default.
+	if language == "" {
+		language = n.opts.languageCode
+	}
+	langP, freeLang := cstr(language)
+	defer freeLang()
+
+	opts := ASRRecognitionOptionsDef()
+	opts.LanguageCode = langP
+	// Segments and the live word list are built out of word offsets, so they
+	// are always asked for.
+	opts.EnableWordTimeOffsets = true
+	// Keyed on the recognizer owning a diar model rather than on the request:
+	// asr.h documents that asking a recognizer created without one for
+	// diarization fails with INVALID_ARGUMENT.
+	opts.EnableSpeakerDiarization = n.opts.diarModel != ""
+	// interim_results is left off deliberately. The runtime emits interims from
+	// next() regardless of it, and they are filtered here rather than
+	// forwarded: see streamPCM's emit for why the wire contract cannot carry
+	// them.
+
+	var handle uintptr
+	// #nosec G103 -- opts is a local POD struct borrowed for this call only, and
+	// its one uintptr member (LanguageCode) is the cstr allocation pinned by the
+	// deferred freeLang above. to_options copies the struct, so nothing here
+	// outlives the call.
+	if st := ASRStreamingRecognize(n.recognizer, unsafe.Pointer(&opts), &handle); st != 0 {
+		return nil, statusErrorf(st, "nemo-speech-cpp: streaming recognize: %s", ASRLastError())
+	}
+	xlog.Debug("nemo-speech-cpp: streaming session open", "language", language)
+	return &cSession{handle: handle}, nil
+}
+
+// chunkPCM slices pcm into fixed-size chunks, leaving the final chunk short
+// rather than padding it: silence padding would push audio the caller never
+// sent through the encoder and shift the tail word timings.
+func chunkPCM(pcm []float32, size int) [][]float32 {
+	if len(pcm) == 0 {
+		return nil
+	}
+	out := make([][]float32, 0, (len(pcm)+size-1)/size)
+	for off := 0; off < len(pcm); off += size {
+		out = append(out, pcm[off:min(off+size, len(pcm))])
+	}
+	return out
+}
+
+// drain pulls every result the session currently has, handing each to emit.
+// It returns when the session reports it needs more audio, which is the loop's
+// only terminating condition.
+func drain(sess asrSession, emit func(streamResult) error) error {
+	for {
+		r, ok, err := sess.next()
+		if err != nil {
+			return err
+		}
+		if !ok {
+			return nil
+		}
+		if err := emit(r); err != nil {
+			return err
+		}
+	}
+}
+
+// streamPCM drives one whole clip through an open session, emitting each
+// finalized utterance as a delta and closing with the assembled result.
+//
+// Only finals become deltas, and the reason is the wire contract:
+// TranscriptStreamResponse.delta is newly-FINALIZED text that consumers
+// CONCATENATE (core/http/endpoints/openai/transcription.go, and the realtime
+// semantic-VAD path). An interim is the decoder's running hypothesis for the
+// utterance in flight, so forwarding "he", "hell", "hello", "Hello." would
+// assemble to "hehellhelloHello." rather than to the transcript. That the
+// runtime also postprocesses finals only (build_result_ in
+// src/asr/recognizer.cpp runs ITN and strip_formatting on the final, so it
+// rewrites rather than extends the interim) means there is no diffing trick
+// that would rescue them either.
+//
+// The cost is that the first delta of an utterance arrives at its endpoint
+// rather than mid-word.
+func streamPCM(ctx context.Context, sess asrSession, pcm []float32, sampleRate int32, wantWords bool, results chan<- *pb.TranscriptStreamResponse) error {
+	if len(pcm) == 0 {
+		return status.Error(codes.InvalidArgument, "nemo-speech-cpp: empty audio")
+	}
+
+	var (
+		full     strings.Builder
+		segments []*pb.TranscriptSegment
+		// sawEndpoint records a final that arrived before the tail flush, i.e.
+		// a real endpoint rather than the end of the file.
+		sawEndpoint bool
+		flushing    bool
+		tailText    string
+	)
+
+	emit := func(r streamResult) error {
+		if !r.Final {
+			return nil
+		}
+		if flushing {
+			tailText += r.Text
+		} else {
+			sawEndpoint = true
+		}
+		if r.Text == "" {
+			return nil
+		}
+
+		// The separator is part of the delta, not added when assembling the
+		// final text, so concatenating the deltas reproduces FinalResult.Text
+		// exactly. Utterance transcripts carry no leading or trailing space of
+		// their own (the runner clears its buffer at each endpoint).
+		delta := r.Text
+		if full.Len() > 0 {
+			delta = " " + delta
+		}
+		full.WriteString(delta)
+
+		// One segment run per utterance, renumbered into the running sequence.
+		// wordsToSegments splits a run further on a speaker change, so a
+		// diarized utterance contributes one segment per turn.
+		segs := wordsToSegments(r.Words, wantWords)
+		if len(segs) == 0 {
+			// Word offsets were requested but a decoder head may still return
+			// none; a segment carrying just the text beats dropping it.
+			segs = []*pb.TranscriptSegment{{Text: r.Text}}
+		}
+		for _, s := range segs {
+			// #nosec G115 -- TranscriptSegment.Id is int32 on the wire, and
+			// segments holds one entry per speaker run per finalized utterance of
+			// a single request, which exhausts memory long before it reaches 2^31.
+			s.Id = int32(len(segments))
+			segments = append(segments, s)
+		}
+
+		results <- &pb.TranscriptStreamResponse{Delta: delta}
+		return nil
+	}
+
+	for _, chunk := range chunkPCM(pcm, streamChunkSamples) {
+		// The RPC body holds engineMu for the whole stream, so Free waits on
+		// it. Without this check a client that disconnected mid-file would pin
+		// the model against unload until the whole clip had been pushed.
+		if err := ctx.Err(); err != nil {
+			return status.Error(codes.Canceled, "nemo-speech-cpp: transcription cancelled")
+		}
+		if err := sess.push(chunk, sampleRate); err != nil {
+			return err
+		}
+		if err := drain(sess, emit); err != nil {
+			return err
+		}
+	}
+
+	flushing = true
+	if err := sess.finish(); err != nil {
+		return err
+	}
+	if err := drain(sess, emit); err != nil {
+		return err
+	}
+
+	final := &pb.TranscriptResult{
+		Text:     full.String(),
+		Segments: segments,
+		// The tail flush returns whatever the decoder was still holding.
+		// Nothing held back after at least one endpoint means the last
+		// endpoint consumed the audio, which is what "the clip ended on an
+		// utterance boundary" means here. Text coming back means it ended
+		// mid-utterance.
+		Eou: sawEndpoint && tailText == "",
+	}
+	if sampleRate > 0 {
+		final.Duration = float32(len(pcm)) / float32(sampleRate)
+	}
+	results <- &pb.TranscriptStreamResponse{FinalResult: final}
+	return nil
+}
+
+// runLive drives one bidirectional live session. The protocol is the one
+// documented on the RPC in backend.proto: a Config first, a ready ack once the
+// session is open, deltas as utterances finalize, and a terminal result when
+// the caller closes its send side.
+//
+// There is no context here on purpose. The gRPC host closes `in` when the
+// stream context is cancelled (pkg/grpc/server.go's recv pump), so ranging
+// over it is what stops this loop, and that is also what releases engineMu for
+// a waiting Free.
+func runLive(open sessionOpener, in <-chan *pb.TranscriptLiveRequest, out chan<- *pb.TranscriptLiveResponse) error {
+	first, ok := <-in
+	if !ok {
+		// The caller closed without sending anything. Nothing was opened, so
+		// there is nothing to report.
+		return nil
+	}
+	cfg := first.GetConfig()
+	if cfg == nil {
+		return status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: the first live message must carry a config")
+	}
+	rate, err := liveSampleRate(cfg)
+	if err != nil {
+		return err
+	}
+
+	sess, err := open(cfg.GetLanguage())
+	if err != nil {
+		return err
+	}
+	// A mid-stream config replaces sess, so this closes whichever session is
+	// current when the RPC unwinds.
+	defer func() { sess.close() }()
+
+	// Callers block on the first Recv waiting for this and degrade to
+	// non-live transcription when it does not arrive, so it goes out before
+	// any audio is read.
+	out <- &pb.TranscriptLiveResponse{Ready: true}
+
+	var (
+		full     strings.Builder
+		flushing bool
+	)
+	emit := func(r streamResult) error {
+		// Finals only, for the same reason as streamPCM: an interim is a
+		// hypothesis the final rewrites, and delta is newly-finalized text.
+		if !r.Final || (r.Text == "" && len(r.Words) == 0) {
+			return nil
+		}
+
+		// The separator goes INTO the delta, exactly as in streamPCM, because
+		// the live consumer is the one that actually concatenates: the realtime
+		// semantic-VAD path joins the accumulated deltas with the empty string
+		// and only clears them at a turn reset, never at an endpoint. Adding
+		// the space when assembling the terminal text instead would make the
+		// running caption read "one.two." while the committed transcript read
+		// "one. two.".
+		delta := r.Text
+		if delta != "" && full.Len() > 0 {
+			delta = " " + delta
+		}
+		full.WriteString(delta)
+
+		out <- &pb.TranscriptLiveResponse{
+			Delta: delta,
+			// A final that arrives while audio is still coming IS the model's
+			// endpoint: the decoder resets its utterance there and the next one
+			// starts fresh, which is the turn boundary the realtime detector
+			// waits on. The final that comes back from the tail flush is the
+			// end of the STREAM, not a user yielding a turn, so it carries no
+			// eou even though the send side has already closed.
+			Eou:   !flushing,
+			Words: wordsToProto(r.Words),
+		}
+		return nil
+	}
+
+	for req := range in {
+		switch payload := req.GetPayload().(type) {
+		case *pb.TranscriptLiveRequest_Config:
+			// A rate cannot change inside a stream (asr.h) and the decoder
+			// keeps utterance state, so a reconfigure has to be a fresh
+			// session rather than a reconfigured one.
+			newRate, err := liveSampleRate(payload.Config)
+			if err != nil {
+				return err
+			}
+			// Opened before the old one is closed so a failure here leaves a
+			// live session for the deferred close, not a dangling handle.
+			next, err := open(payload.Config.GetLanguage())
+			if err != nil {
+				return err
+			}
+			sess.close()
+			sess, rate = next, newRate
+			full.Reset()
+		case *pb.TranscriptLiveRequest_Audio:
+			pcm := payload.Audio.GetPcm()
+			if len(pcm) == 0 {
+				continue
+			}
+			if err := sess.push(pcm, rate); err != nil {
+				return err
+			}
+			if err := drain(sess, emit); err != nil {
+				return err
+			}
+		}
+	}
+
+	// Send side closed: flush the tail and emit the terminal result. Like the
+	// other backends' live path this carries Text only; per-utterance segments
+	// and the duration are the file path's concern.
+	flushing = true
+	if err := sess.finish(); err != nil {
+		return err
+	}
+	if err := drain(sess, emit); err != nil {
+		return err
+	}
+	// Not trimmed: the terminal text is the verbatim concatenation of the
+	// deltas, which is the invariant the concatenating consumers rely on. The
+	// first delta never carries the separator, so there is no leading space to
+	// trim off in the first place.
+	out <- &pb.TranscriptLiveResponse{
+		FinalResult: &pb.TranscriptResult{Text: full.String()},
+	}
+	return nil
+}
+
+// liveSampleRate resolves TranscriptLiveConfig.sample_rate to the rate the C
+// API is given. The proto's 0 means 16 kHz; the C API's 0 means "already at the
+// model rate", so the two cannot be forwarded to each other.
+func liveSampleRate(cfg *pb.TranscriptLiveConfig) (int32, error) {
+	rate := cfg.GetSampleRate()
+	if rate == 0 {
+		return defaultLiveSampleRate, nil
+	}
+	if rate < minStreamSampleRate || rate > maxStreamSampleRate {
+		return 0, status.Errorf(codes.InvalidArgument,
+			"nemo-speech-cpp: unsupported live sample_rate %d (accepted: 0 or %d-%d Hz)",
+			rate, minStreamSampleRate, maxStreamSampleRate)
+	}
+	return rate, nil
+}
+
+// wordsToProto converts decoded words to the wire form. TranscriptWord.start
+// and .end are int64 nanoseconds; the runtime reports milliseconds.
+func wordsToProto(words []asrWord) []*pb.TranscriptWord {
+	if len(words) == 0 {
+		return nil
+	}
+	out := make([]*pb.TranscriptWord, len(words))
+	for i, w := range words {
+		out[i] = &pb.TranscriptWord{
+			Text:  w.Text,
+			Start: msToNanos(w.Start),
+			End:   msToNanos(w.End),
+		}
+	}
+	return out
+}
+
+// AudioTranscriptionStream decodes the audio at req.Dst through the streaming
+// recognizer, emitting each finalized utterance as it lands.
+//
+// The body runs inside withEngine for the reason documented on withEngine, and
+// that holds engineMu for the whole stream: Free waits rather than destroying
+// the recognizer under a half-finished stream. streamPCM honours ctx so the
+// wait is bounded by the client's disconnect rather than by its silence.
+func (n *NemoSpeech) AudioTranscriptionStream(ctx context.Context, req *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
+	// The host ranges over this channel and only returns once it closes, so
+	// every path out of here, rejection included, has to close it.
+	defer close(results)
+
+	return n.withEngine(familyASR, func() error {
+		return n.transcribeStream(ctx, req, results)
+	})
+}
+
+// transcribeStream is AudioTranscriptionStream's body. The caller must hold
+// engineMu.
+func (n *NemoSpeech) transcribeStream(ctx context.Context, req *pb.TranscriptRequest, results chan<- *pb.TranscriptStreamResponse) error {
+	if req.GetDst() == "" {
+		return status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: TranscriptRequest.dst (audio path) is required")
+	}
+	// Checked before the decode so a client that has already gone away does
+	// not pay for an ffmpeg run, and so a cancellation is never reported as a
+	// broken file.
+	if err := ctx.Err(); err != nil {
+		return status.Error(codes.Canceled, "nemo-speech-cpp: transcription cancelled")
+	}
+
+	pcm, sampleRate, err := decodeAudioMono16k(req.GetDst())
+	if err != nil {
+		return status.Errorf(codes.InvalidArgument, "nemo-speech-cpp: read audio: %v", err)
+	}
+	// Before the session is opened, for the same reason as the offline path:
+	// there is no transcript to be had from zero samples, and opening a stream
+	// only to close it again asks the runtime to allocate decoder state for
+	// nothing.
+	if len(pcm) == 0 {
+		return status.Error(codes.InvalidArgument, "nemo-speech-cpp: empty audio")
+	}
+
+	sess, err := n.openSession(req.GetLanguage())
+	if err != nil {
+		return err
+	}
+	defer sess.close()
+
+	return streamPCM(ctx, sess, pcm, sampleRate,
+		wordsRequested(req.GetTimestampGranularities()), results)
+}
+
+// AudioTranscriptionLive serves the bidirectional live RPC over one streaming
+// session. See runLive for the protocol and withEngine for the locking.
+func (n *NemoSpeech) AudioTranscriptionLive(in <-chan *pb.TranscriptLiveRequest, out chan<- *pb.TranscriptLiveResponse) error {
+	defer close(out)
+
+	return n.withEngine(familyASR, func() error {
+		return runLive(n.openSession, in, out)
+	})
+}

backend/go/nemo-speech-cpp/asr_stream_test.go

@@ -0,0 +1,699 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"path/filepath"
+	"time"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// fakeSession is a scripted asrSession. It stands in for the streaming C API,
+// not for a model: no NeMo GGUF is small enough to keep in the tree, and the
+// need-more-audio drain is the easiest thing in this file to get subtly wrong
+// (a mishandled NULL either spins forever or drops every result).
+//
+// script is one batch of results per drain. next() hands back the current
+// batch one result at a time and then reports "need more audio" exactly once,
+// which advances to the next batch. That is precisely the C contract:
+// nemo_speech_asr_stream_next returns OK with a NULL handle when the decoder
+// has consumed the buffered audio, and the loop must resume after the next
+// push rather than treat it as the end of the stream.
+type fakeSession struct {
+	script [][]streamResult
+	batch  int
+	pos    int
+
+	pushed   [][]float32
+	rates    []int32
+	finished int
+	closed   int
+
+	pushErr   error
+	finishErr error
+	nextErr   error
+}
+
+func (f *fakeSession) push(pcm []float32, sampleRate int32) error {
+	if f.pushErr != nil {
+		return f.pushErr
+	}
+	f.pushed = append(f.pushed, pcm)
+	f.rates = append(f.rates, sampleRate)
+	return nil
+}
+
+func (f *fakeSession) finish() error {
+	if f.finishErr != nil {
+		return f.finishErr
+	}
+	f.finished++
+	return nil
+}
+
+func (f *fakeSession) next() (streamResult, bool, error) {
+	if f.nextErr != nil {
+		return streamResult{}, false, f.nextErr
+	}
+	if f.batch >= len(f.script) {
+		return streamResult{}, false, nil
+	}
+	if f.pos >= len(f.script[f.batch]) {
+		f.batch++
+		f.pos = 0
+		return streamResult{}, false, nil
+	}
+	r := f.script[f.batch][f.pos]
+	f.pos++
+	return r, true, nil
+}
+
+func (f *fakeSession) close() { f.closed++ }
+
+// samples returns the flat concatenation of everything pushed, so a spec can
+// assert the whole clip reached the engine without caring how it was sliced.
+func (f *fakeSession) samples() []float32 {
+	var out []float32
+	for _, c := range f.pushed {
+		out = append(out, c...)
+	}
+	return out
+}
+
+// collect drains a response channel into a slice. The channels are unbuffered
+// in the specs on purpose: a producer that stops honouring cancellation would
+// otherwise fill a buffer and look healthy.
+func collect[T any](ch chan T) chan []T {
+	done := make(chan []T, 1)
+	go func() {
+		var got []T
+		for v := range ch {
+			got = append(got, v)
+		}
+		done <- got
+	}()
+	return done
+}
+
+var _ = Describe("chunkPCM", func() {
+	It("splits into equal chunks when evenly divisible", func() {
+		chunks := chunkPCM(make([]float32, 400), 100)
+		Expect(chunks).To(HaveLen(4))
+		for _, c := range chunks {
+			Expect(c).To(HaveLen(100))
+		}
+	})
+
+	// Padding the tail with silence would push phantom audio through the
+	// encoder and shift the tail word timings, so the final chunk stays short.
+	It("makes the final chunk short rather than padding it", func() {
+		chunks := chunkPCM(make([]float32, 250), 100)
+		Expect(chunks).To(HaveLen(3))
+		Expect(chunks[2]).To(HaveLen(50))
+	})
+
+	It("returns one chunk when the input is shorter than the chunk size", func() {
+		chunks := chunkPCM(make([]float32, 10), 100)
+		Expect(chunks).To(HaveLen(1))
+		Expect(chunks[0]).To(HaveLen(10))
+	})
+
+	It("returns nothing for empty input", func() {
+		Expect(chunkPCM(nil, 100)).To(BeEmpty())
+		Expect(chunkPCM([]float32{}, 100)).To(BeEmpty())
+	})
+
+	// Every spec above works on all-zero audio, so none of them can tell a
+	// correct slicing from one that reorders or repeats windows. Audio fed out
+	// of order still decodes, it just decodes to nonsense.
+	It("preserves sample order across the chunk boundaries", func() {
+		pcm := []float32{1, 2, 3, 4, 5}
+		chunks := chunkPCM(pcm, 2)
+		Expect(chunks).To(HaveLen(3))
+		Expect(chunks[0]).To(Equal([]float32{1, 2}))
+		Expect(chunks[1]).To(Equal([]float32{3, 4}))
+		Expect(chunks[2]).To(Equal([]float32{5}))
+	})
+})
+
+var _ = Describe("drain", func() {
+	It("emits every result in a batch and stops on need-more-audio", func() {
+		sess := &fakeSession{script: [][]streamResult{
+			{{Text: "a"}, {Text: "b", Final: true}},
+			{{Text: "c"}},
+		}}
+		var got []string
+		Expect(drain(sess, func(r streamResult) error {
+			got = append(got, r.Text)
+			return nil
+		})).To(Succeed())
+		Expect(got).To(Equal([]string{"a", "b"}))
+	})
+
+	// The NULL handle is a pause, not an end: the next drain, after more audio
+	// has been pushed, must pick the stream back up.
+	It("resumes on the next drain after a need-more-audio pause", func() {
+		sess := &fakeSession{script: [][]streamResult{{{Text: "a"}}, {{Text: "b"}}}}
+		var got []string
+		emit := func(r streamResult) error { got = append(got, r.Text); return nil }
+		Expect(drain(sess, emit)).To(Succeed())
+		Expect(drain(sess, emit)).To(Succeed())
+		Expect(got).To(Equal([]string{"a", "b"}))
+	})
+
+	It("returns nothing and no error for a stream with no results ready", func() {
+		var got []string
+		Expect(drain(&fakeSession{}, func(r streamResult) error {
+			got = append(got, r.Text)
+			return nil
+		})).To(Succeed())
+		Expect(got).To(BeEmpty())
+	})
+
+	It("propagates a failure from the runtime", func() {
+		sess := &fakeSession{nextErr: errors.New("boom")}
+		Expect(drain(sess, func(streamResult) error { return nil })).To(MatchError(ContainSubstring("boom")))
+	})
+
+	It("stops pulling once emit fails", func() {
+		sess := &fakeSession{script: [][]streamResult{{{Text: "a"}, {Text: "b"}}}}
+		Expect(drain(sess, func(streamResult) error {
+			return errors.New("send failed")
+		})).To(MatchError(ContainSubstring("send failed")))
+		Expect(sess.pos).To(Equal(1))
+	})
+})
+
+var _ = Describe("streamPCM", func() {
+	streamWords := func(ctx context.Context, sess asrSession, pcm []float32, rate int32, wantWords bool) ([]*pb.TranscriptStreamResponse, error) {
+		GinkgoHelper()
+		results := make(chan *pb.TranscriptStreamResponse)
+		done := collect(results)
+		err := streamPCM(ctx, sess, pcm, rate, wantWords, results)
+		close(results)
+		return <-done, err
+	}
+	stream := func(ctx context.Context, sess asrSession, pcm []float32, rate int32) ([]*pb.TranscriptStreamResponse, error) {
+		GinkgoHelper()
+		return streamWords(ctx, sess, pcm, rate, false)
+	}
+
+	It("pushes the whole clip in chunks at the clip's own sample rate", func() {
+		sess := &fakeSession{}
+		pcm := make([]float32, streamChunkSamples*2+7)
+		_, err := stream(context.Background(), sess, pcm, 16000)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(sess.pushed).To(HaveLen(3))
+		Expect(sess.samples()).To(HaveLen(len(pcm)))
+		for _, r := range sess.rates {
+			Expect(r).To(Equal(int32(16000)))
+		}
+	})
+
+	It("finishes the stream once, after the last chunk", func() {
+		sess := &fakeSession{}
+		_, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(sess.finished).To(Equal(1))
+	})
+
+	// Interims are the decoder's running hypothesis for the utterance in
+	// flight. The wire contract is that delta is newly FINALIZED text and that
+	// concatenating the deltas reproduces the transcript, so forwarding an
+	// interim would duplicate every word it later re-sends inside the final.
+	It("emits a delta per final and nothing for interims", func() {
+		sess := &fakeSession{script: [][]streamResult{{
+			{Text: "hel"},
+			{Text: "hello"},
+			{Text: "Hello.", Final: true},
+		}}}
+		got, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).ToNot(HaveOccurred())
+
+		var deltas []string
+		for _, r := range got {
+			if r.GetDelta() != "" {
+				deltas = append(deltas, r.GetDelta())
+			}
+		}
+		Expect(deltas).To(Equal([]string{"Hello."}))
+	})
+
+	It("reproduces the final transcript by concatenating the deltas", func() {
+		sess := &fakeSession{script: [][]streamResult{{
+			{Text: "One.", Final: true},
+			{Text: "Two.", Final: true},
+		}}}
+		got, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).ToNot(HaveOccurred())
+
+		var joined string
+		var final *pb.TranscriptResult
+		for _, r := range got {
+			joined += r.GetDelta()
+			if r.GetFinalResult() != nil {
+				final = r.GetFinalResult()
+			}
+		}
+		Expect(final).ToNot(BeNil())
+		Expect(final.GetText()).To(Equal("One. Two."))
+		Expect(joined).To(Equal(final.GetText()))
+	})
+
+	It("sends the terminal final result last and only once", func() {
+		sess := &fakeSession{script: [][]streamResult{{{Text: "hi", Final: true}}}}
+		got, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).ToNot(BeEmpty())
+
+		var finals int
+		for _, r := range got {
+			if r.GetFinalResult() != nil {
+				finals++
+			}
+		}
+		Expect(finals).To(Equal(1))
+		Expect(got[len(got)-1].GetFinalResult()).ToNot(BeNil())
+	})
+
+	It("reports the clip duration in seconds", func() {
+		sess := &fakeSession{}
+		got, err := stream(context.Background(), sess, make([]float32, 8000), 16000)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got[len(got)-1].GetFinalResult().GetDuration()).To(BeNumerically("~", 0.5, 1e-6))
+	})
+
+	It("builds per-utterance segments with nanosecond timestamps", func() {
+		sess := &fakeSession{script: [][]streamResult{{
+			{Text: "one", Final: true, Words: []asrWord{{Text: "one", Start: 0, End: 500}}},
+			{Text: "two", Final: true, Words: []asrWord{{Text: "two", Start: 900, End: 1400}}},
+		}}}
+		got, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).ToNot(HaveOccurred())
+
+		segs := got[len(got)-1].GetFinalResult().GetSegments()
+		Expect(segs).To(HaveLen(2))
+		Expect(segs[0].GetId()).To(Equal(int32(0)))
+		Expect(segs[1].GetId()).To(Equal(int32(1)))
+		Expect(time.Duration(segs[1].GetStart())).To(Equal(900 * time.Millisecond))
+		Expect(time.Duration(segs[1].GetEnd())).To(Equal(1400 * time.Millisecond))
+	})
+
+	// core/backend/transcript.go builds the response's word list out of
+	// TranscriptSegment.Words, so leaving it unset makes
+	// timestamp_granularities: ["word"] come back empty.
+	It("attaches the word timings only when they were asked for", func() {
+		script := func() [][]streamResult {
+			return [][]streamResult{{{Text: "one", Final: true,
+				Words: []asrWord{{Text: "one", Start: 100, End: 500}}}}}
+		}
+
+		got, err := streamWords(context.Background(), &fakeSession{script: script()}, make([]float32, 10), 16000, true)
+		Expect(err).ToNot(HaveOccurred())
+		segs := got[len(got)-1].GetFinalResult().GetSegments()
+		Expect(segs[0].GetWords()).To(HaveLen(1))
+		Expect(segs[0].GetWords()[0].GetText()).To(Equal("one"))
+		Expect(time.Duration(segs[0].GetWords()[0].GetStart())).To(Equal(100 * time.Millisecond))
+
+		got, err = streamWords(context.Background(), &fakeSession{script: script()}, make([]float32, 10), 16000, false)
+		Expect(err).ToNot(HaveOccurred())
+		segs = got[len(got)-1].GetFinalResult().GetSegments()
+		Expect(segs[0].GetText()).To(Equal("one"))
+		Expect(segs[0].GetWords()).To(BeEmpty())
+	})
+
+	// The flush that nemo_speech_asr_stream_finish triggers returns whatever the
+	// decoder was still holding. Nothing held back means the last endpoint
+	// consumed the audio, which is exactly "the clip ended on an utterance
+	// boundary"; text coming back means it ended mid-utterance.
+	It("marks eou when the tail flush had nothing left to emit", func() {
+		sess := &fakeSession{script: [][]streamResult{
+			{{Text: "done.", Final: true}},
+			{{Text: "", Final: true}},
+		}}
+		got, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got[len(got)-1].GetFinalResult().GetEou()).To(BeTrue())
+	})
+
+	It("does not mark eou when the tail flush produced text", func() {
+		sess := &fakeSession{script: [][]streamResult{
+			{{Text: "done.", Final: true}},
+			{{Text: "and more", Final: true}},
+		}}
+		got, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got[len(got)-1].GetFinalResult().GetEou()).To(BeFalse())
+	})
+
+	// The RPC body runs inside withEngine, so it holds the engine mutex for the
+	// whole stream and Free waits on it. A loop that ignored cancellation would
+	// pin the model against unload for as long as a disconnected client's audio
+	// takes to push.
+	It("stops promptly when the request context is cancelled", func() {
+		ctx, cancel := context.WithCancel(context.Background())
+		cancel()
+		sess := &fakeSession{}
+		_, err := stream(ctx, sess, make([]float32, streamChunkSamples*4), 16000)
+		Expect(status.Code(err)).To(Equal(codes.Canceled))
+		Expect(sess.pushed).To(BeEmpty())
+	})
+
+	It("reports a push failure", func() {
+		sess := &fakeSession{pushErr: errors.New("push blew up")}
+		_, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).To(MatchError(ContainSubstring("push blew up")))
+	})
+
+	It("reports a finish failure", func() {
+		sess := &fakeSession{finishErr: errors.New("finish blew up")}
+		_, err := stream(context.Background(), sess, make([]float32, 10), 16000)
+		Expect(err).To(MatchError(ContainSubstring("finish blew up")))
+	})
+})
+
+var _ = Describe("runLive", func() {
+	// live drives runLive against a fake opener and returns everything the RPC
+	// wrote plus the sessions it opened.
+	live := func(reqs []*pb.TranscriptLiveRequest, script ...[][]streamResult) ([]*pb.TranscriptLiveResponse, []*fakeSession, error) {
+		GinkgoHelper()
+		var opened []*fakeSession
+		open := func(language string) (asrSession, error) {
+			s := &fakeSession{}
+			if len(opened) < len(script) {
+				s.script = script[len(opened)]
+			}
+			opened = append(opened, s)
+			return s, nil
+		}
+
+		in := make(chan *pb.TranscriptLiveRequest)
+		out := make(chan *pb.TranscriptLiveResponse)
+		done := collect(out)
+		go func() {
+			defer close(in)
+			for _, r := range reqs {
+				in <- r
+			}
+		}()
+		err := runLive(open, in, out)
+		close(out)
+		return <-done, opened, err
+	}
+
+	cfg := func(rate int32) *pb.TranscriptLiveRequest {
+		return &pb.TranscriptLiveRequest{Payload: &pb.TranscriptLiveRequest_Config{
+			Config: &pb.TranscriptLiveConfig{SampleRate: rate},
+		}}
+	}
+	audio := func(pcm ...float32) *pb.TranscriptLiveRequest {
+		return &pb.TranscriptLiveRequest{Payload: &pb.TranscriptLiveRequest_Audio{
+			Audio: &pb.TranscriptLiveAudio{Pcm: pcm},
+		}}
+	}
+
+	It("requires the first message to carry a config", func() {
+		_, opened, err := live([]*pb.TranscriptLiveRequest{audio(1, 2, 3)})
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(opened).To(BeEmpty())
+	})
+
+	It("returns without error when the caller closes without sending anything", func() {
+		got, opened, err := live(nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(BeEmpty())
+		Expect(opened).To(BeEmpty())
+	})
+
+	// Callers block on the first Recv waiting for this ack, and degrade to
+	// non-live transcription when it does not arrive.
+	It("acknowledges a successful open before any transcript", func() {
+		got, _, err := live([]*pb.TranscriptLiveRequest{cfg(0)})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).ToNot(BeEmpty())
+		Expect(got[0].GetReady()).To(BeTrue())
+	})
+
+	// The proto documents 0 as "16 kHz". The C API reads 0 as "these samples
+	// are already at the model rate" and skips resampling, so forwarding the
+	// zero through would silently mean something else.
+	It("resolves the default sample rate to 16 kHz before pushing", func() {
+		_, opened, err := live([]*pb.TranscriptLiveRequest{cfg(0), audio(1, 2, 3)})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(opened).To(HaveLen(1))
+		Expect(opened[0].rates).To(Equal([]int32{16000}))
+	})
+
+	It("pushes at the configured sample rate", func() {
+		_, opened, err := live([]*pb.TranscriptLiveRequest{cfg(8000), audio(1, 2, 3)})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(opened[0].rates).To(Equal([]int32{8000}))
+		Expect(opened[0].samples()).To(Equal([]float32{1, 2, 3}))
+	})
+
+	It("rejects a sample rate the runtime cannot resample", func() {
+		_, opened, err := live([]*pb.TranscriptLiveRequest{cfg(4000)})
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(opened).To(BeEmpty())
+	})
+
+	It("ignores an empty audio frame instead of pushing it", func() {
+		_, opened, err := live([]*pb.TranscriptLiveRequest{cfg(0), audio()})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(opened[0].pushed).To(BeEmpty())
+	})
+
+	It("streams a delta with its words and marks the utterance boundary", func() {
+		got, _, err := live(
+			[]*pb.TranscriptLiveRequest{cfg(0), audio(1)},
+			[][]streamResult{{
+				{Text: "partial"},
+				{Text: "Hello there.", Final: true, Words: []asrWord{
+					{Text: "Hello", Start: 100, End: 400},
+					{Text: "there", Start: 400, End: 900},
+				}},
+			}},
+		)
+		Expect(err).ToNot(HaveOccurred())
+
+		var deltas []*pb.TranscriptLiveResponse
+		for _, r := range got {
+			if r.GetDelta() != "" {
+				deltas = append(deltas, r)
+			}
+		}
+		Expect(deltas).To(HaveLen(1))
+		Expect(deltas[0].GetDelta()).To(Equal("Hello there."))
+		Expect(deltas[0].GetEou()).To(BeTrue())
+		Expect(deltas[0].GetWords()).To(HaveLen(2))
+		Expect(time.Duration(deltas[0].GetWords()[1].GetStart())).To(Equal(400 * time.Millisecond))
+		Expect(time.Duration(deltas[0].GetWords()[1].GetEnd())).To(Equal(900 * time.Millisecond))
+	})
+
+	It("finishes and closes the session when the caller closes the send side", func() {
+		got, opened, err := live(
+			[]*pb.TranscriptLiveRequest{cfg(0), audio(1)},
+			[][]streamResult{{{Text: "one.", Final: true}}, {{Text: "two.", Final: true}}},
+		)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(opened[0].finished).To(Equal(1))
+		Expect(opened[0].closed).To(Equal(1))
+		Expect(got[len(got)-1].GetFinalResult()).ToNot(BeNil())
+		Expect(got[len(got)-1].GetFinalResult().GetText()).To(Equal("one. two."))
+	})
+
+	// The live path is the one with a consumer that really concatenates: the
+	// realtime semantic-VAD path joins the accumulated deltas with the empty
+	// string and clears them only at a turn reset, never at an utterance
+	// boundary. A separator added when assembling the terminal text instead of
+	// inside the delta makes the running caption read "one.two." while the
+	// committed transcript reads "one. two.".
+	It("reproduces the final transcript by concatenating the deltas", func() {
+		got, _, err := live(
+			[]*pb.TranscriptLiveRequest{cfg(0), audio(1)},
+			[][]streamResult{{{Text: "one.", Final: true}}, {{Text: "two.", Final: true}}},
+		)
+		Expect(err).ToNot(HaveOccurred())
+
+		var joined string
+		var final *pb.TranscriptResult
+		for _, r := range got {
+			joined += r.GetDelta()
+			if r.GetFinalResult() != nil {
+				final = r.GetFinalResult()
+			}
+		}
+		Expect(final).ToNot(BeNil())
+		Expect(final.GetText()).To(Equal("one. two."))
+		Expect(joined).To(Equal(final.GetText()))
+	})
+
+	// Eou is the model's endpoint, which is a user yielding the turn. The final
+	// that comes back from the tail flush is the end of the stream: the send
+	// side has already closed, so reporting a turn boundary there tells the
+	// turn detector something that did not happen.
+	It("marks the endpoint finals but not the tail flush", func() {
+		got, _, err := live(
+			[]*pb.TranscriptLiveRequest{cfg(0), audio(1)},
+			[][]streamResult{{{Text: "one.", Final: true}}, {{Text: "two.", Final: true}}},
+		)
+		Expect(err).ToNot(HaveOccurred())
+
+		var eous []bool
+		for _, r := range got {
+			if r.GetDelta() != "" {
+				eous = append(eous, r.GetEou())
+			}
+		}
+		Expect(eous).To(Equal([]bool{true, false}))
+	})
+
+	// A rate cannot change inside a stream and the decoder keeps no state
+	// across a reset, so a second config has to be a fresh session, not a
+	// reconfigured one.
+	It("opens a fresh session on a mid-stream config and drops the old transcript", func() {
+		got, opened, err := live(
+			[]*pb.TranscriptLiveRequest{cfg(0), audio(1), cfg(0), audio(2)},
+			[][]streamResult{{{Text: "dropped.", Final: true}}},
+			[][]streamResult{{{Text: "kept.", Final: true}}},
+		)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(opened).To(HaveLen(2))
+		Expect(opened[0].closed).To(Equal(1))
+		Expect(got[len(got)-1].GetFinalResult().GetText()).To(Equal("kept."))
+	})
+
+	It("reports a push failure and still closes the session", func() {
+		var opened []*fakeSession
+		open := func(string) (asrSession, error) {
+			s := &fakeSession{pushErr: errors.New("push blew up")}
+			opened = append(opened, s)
+			return s, nil
+		}
+		in := make(chan *pb.TranscriptLiveRequest, 2)
+		in <- cfg(0)
+		in <- audio(1, 2)
+		close(in)
+		out := make(chan *pb.TranscriptLiveResponse, 8)
+
+		err := runLive(open, in, out)
+		Expect(err).To(MatchError(ContainSubstring("push blew up")))
+		Expect(opened[0].closed).To(Equal(1))
+	})
+
+	It("propagates a failure to open the session", func() {
+		open := func(string) (asrSession, error) { return nil, errors.New("no streaming here") }
+		in := make(chan *pb.TranscriptLiveRequest, 1)
+		in <- cfg(0)
+		close(in)
+		out := make(chan *pb.TranscriptLiveResponse, 8)
+		Expect(runLive(open, in, out)).To(MatchError(ContainSubstring("no streaming here")))
+	})
+})
+
+var _ = Describe("AudioTranscriptionStream", func() {
+	run := func(ctx context.Context, n *NemoSpeech, req *pb.TranscriptRequest) ([]*pb.TranscriptStreamResponse, error) {
+		GinkgoHelper()
+		results := make(chan *pb.TranscriptStreamResponse)
+		done := collect(results)
+		err := n.AudioTranscriptionStream(ctx, req, results)
+		return <-done, err
+	}
+
+	// The RPC owns the channel: the gRPC host ranges over it and only returns
+	// once it closes, so a rejection path that forgets to close hangs the call
+	// instead of failing it.
+	It("closes the results channel on every rejection path", func() {
+		for _, n := range []*NemoSpeech{{fam: familyTTS}, {fam: familyASR}, {}} {
+			_, err := run(context.Background(), n, &pb.TranscriptRequest{})
+			Expect(err).To(HaveOccurred())
+			Expect(n.engineMu.TryLock()).To(BeTrue())
+			n.engineMu.Unlock()
+		}
+	})
+
+	It("refuses a model loaded as another family", func() {
+		n := &NemoSpeech{fam: familyTTS}
+		_, err := run(context.Background(), n, &pb.TranscriptRequest{Dst: "x.wav"})
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(err.Error()).To(ContainSubstring("tts"))
+	})
+
+	It("requires a destination path", func() {
+		n := &NemoSpeech{fam: familyASR}
+		_, err := run(context.Background(), n, &pb.TranscriptRequest{})
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	// Cancellation is checked before the decode so a client that has already
+	// gone away does not pay for an ffmpeg run, and so the check cannot be
+	// mistaken for the decode failing.
+	It("returns cancelled without touching the audio", func() {
+		ctx, cancel := context.WithCancel(context.Background())
+		cancel()
+		n := &NemoSpeech{fam: familyASR}
+		_, err := run(ctx, n, &pb.TranscriptRequest{
+			Dst: filepath.Join(GinkgoT().TempDir(), "absent.wav"),
+		})
+		Expect(status.Code(err)).To(Equal(codes.Canceled))
+	})
+
+	It("reports an audio file it cannot read", func() {
+		n := &NemoSpeech{fam: familyASR}
+		_, err := run(context.Background(), n, &pb.TranscriptRequest{
+			Dst: filepath.Join(GinkgoT().TempDir(), "absent.wav"),
+		})
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	// Same ordering constraint as the offline path: a clip that decodes to no
+	// samples has to be refused before a session is opened, which is also
+	// before any bound entry point is called. Nothing is loaded here, so a
+	// guard placed after the open would panic instead of failing.
+	It("refuses a decodable clip that carries no samples, before opening a session", func() {
+		path := filepath.Join(GinkgoT().TempDir(), "silence.wav")
+		writeMono16kWAV(path, 0)
+
+		n := &NemoSpeech{fam: familyASR}
+		var err error
+		Expect(func() {
+			_, err = run(context.Background(), n, &pb.TranscriptRequest{Dst: path})
+		}).ToNot(Panic())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("empty audio"))
+	})
+})
+
+var _ = Describe("AudioTranscriptionLive", func() {
+	It("refuses a model loaded as another family and closes the output", func() {
+		n := &NemoSpeech{fam: familyNMT}
+		in := make(chan *pb.TranscriptLiveRequest)
+		close(in)
+		out := make(chan *pb.TranscriptLiveResponse)
+		done := collect(out)
+
+		err := n.AudioTranscriptionLive(in, out)
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(<-done).To(BeEmpty())
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+
+	It("refuses an unloaded model", func() {
+		n := &NemoSpeech{}
+		in := make(chan *pb.TranscriptLiveRequest)
+		close(in)
+		out := make(chan *pb.TranscriptLiveResponse)
+		done := collect(out)
+
+		err := n.AudioTranscriptionLive(in, out)
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(<-done).To(BeEmpty())
+	})
+})

backend/go/nemo-speech-cpp/asr_test.go

@@ -0,0 +1,367 @@
+package main
+
+import (
+	"context"
+	"math"
+	"os"
+	"path/filepath"
+	"time"
+	"unsafe"
+
+	"github.com/go-audio/audio"
+	"github.com/go-audio/wav"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// writeMono16kWAV writes `frames` samples of 16 kHz mono 16-bit silence.
+// That is already AudioToWav's target format, so the decode path copies the
+// file through instead of shelling out to ffmpeg, which the test host may not
+// have.
+func writeMono16kWAV(path string, frames int) {
+	GinkgoHelper()
+	f, err := os.Create(path)
+	Expect(err).ToNot(HaveOccurred())
+	enc := wav.NewEncoder(f, 16000, 16, 1, 1)
+	Expect(enc.Write(&audio.IntBuffer{
+		Format:         &audio.Format{NumChannels: 1, SampleRate: 16000},
+		SourceBitDepth: 16,
+		Data:           make([]int, frames),
+	})).To(Succeed())
+	Expect(enc.Close()).To(Succeed())
+	Expect(f.Close()).To(Succeed())
+}
+
+var _ = Describe("wordsToSegments", func() {
+	It("groups words into one segment per speaker run", func() {
+		words := []asrWord{
+			{Text: "hello", Start: 0, End: 400, Speaker: 1},
+			{Text: "there", Start: 400, End: 800, Speaker: 1},
+			{Text: "hi", Start: 900, End: 1200, Speaker: 2},
+		}
+		segs := wordsToSegments(words, false)
+		Expect(segs).To(HaveLen(2))
+		Expect(segs[0].Text).To(Equal("hello there"))
+		Expect(segs[1].Text).To(Equal("hi"))
+	})
+
+	// A run is bounded by a CHANGE of speaker, not by the speaker id being new.
+	// Grouping that keyed on the id itself (a map, or a comparison against the
+	// first word) would merge the two A turns into one segment spanning B, and
+	// the three-word spec above cannot see that because it never returns to an
+	// earlier speaker.
+	It("starts a new segment when an earlier speaker takes another turn", func() {
+		words := []asrWord{
+			{Text: "one", Start: 0, End: 100, Speaker: 1},
+			{Text: "two", Start: 100, End: 200, Speaker: 2},
+			{Text: "three", Start: 200, End: 300, Speaker: 1},
+		}
+		segs := wordsToSegments(words, false)
+		Expect(segs).To(HaveLen(3))
+		Expect(segs[0].Text).To(Equal("one"))
+		Expect(segs[1].Text).To(Equal("two"))
+		Expect(segs[2].Text).To(Equal("three"))
+	})
+
+	// TranscriptSegment.start/end are int64 nanoseconds, not seconds:
+	// core/backend/transcript.go reads them straight into a time.Duration. The
+	// runtime reports word offsets in milliseconds (src/asr/types.h:46).
+	It("converts millisecond word times to nanoseconds", func() {
+		words := []asrWord{{Text: "a", Start: 1500, End: 2250, Speaker: 0}}
+		segs := wordsToSegments(words, false)
+		Expect(segs).To(HaveLen(1))
+		Expect(time.Duration(segs[0].Start)).To(Equal(1500 * time.Millisecond))
+		Expect(time.Duration(segs[0].End)).To(Equal(2250 * time.Millisecond))
+	})
+
+	It("spans a segment from its first word's start to its last word's end", func() {
+		words := []asrWord{
+			{Text: "a", Start: 100, End: 200, Speaker: 0},
+			{Text: "b", Start: 500, End: 900, Speaker: 0},
+		}
+		segs := wordsToSegments(words, false)
+		Expect(segs).To(HaveLen(1))
+		Expect(time.Duration(segs[0].Start)).To(Equal(100 * time.Millisecond))
+		Expect(time.Duration(segs[0].End)).To(Equal(900 * time.Millisecond))
+	})
+
+	It("produces a single segment when no speaker tags are present", func() {
+		words := []asrWord{
+			{Text: "a", Start: 0, End: 100, Speaker: 0},
+			{Text: "b", Start: 100, End: 200, Speaker: 0},
+		}
+		segs := wordsToSegments(words, false)
+		Expect(segs).To(HaveLen(1))
+		Expect(segs[0].Text).To(Equal("a b"))
+	})
+
+	It("returns no segments for no words", func() {
+		Expect(wordsToSegments(nil, false)).To(BeEmpty())
+		Expect(wordsToSegments([]asrWord{}, false)).To(BeEmpty())
+	})
+
+	It("numbers the segments from zero in order", func() {
+		words := []asrWord{
+			{Text: "a", Speaker: 1},
+			{Text: "b", Speaker: 2},
+			{Text: "c", Speaker: 3},
+		}
+		segs := wordsToSegments(words, false)
+		Expect(segs).To(HaveLen(3))
+		for i, s := range segs {
+			Expect(s.Id).To(Equal(int32(i)))
+		}
+	})
+
+	// TranscriptSegment.Words is what core/backend/transcript.go turns into the
+	// response's word list, so an unset one makes timestamp_granularities:
+	// ["word"] come back empty however good the timings were.
+	It("attaches the per-word timings only when they were asked for", func() {
+		words := []asrWord{
+			{Text: "a", Start: 0, End: 100},
+			{Text: "b", Start: 100, End: 250},
+		}
+		with := wordsToSegments(words, true)
+		Expect(with[0].Words).To(HaveLen(2))
+		Expect(with[0].Words[1].Text).To(Equal("b"))
+		Expect(time.Duration(with[0].Words[1].Start)).To(Equal(100 * time.Millisecond))
+		Expect(time.Duration(with[0].Words[1].End)).To(Equal(250 * time.Millisecond))
+
+		Expect(wordsToSegments(words, false)[0].Words).To(BeEmpty())
+	})
+
+	// A speaker change splits the run, and each segment must carry only its own
+	// words rather than the whole utterance's.
+	It("gives each speaker run only its own words", func() {
+		segs := wordsToSegments([]asrWord{
+			{Text: "a", Speaker: 1},
+			{Text: "b", Speaker: 2},
+		}, true)
+		Expect(segs).To(HaveLen(2))
+		Expect(segs[0].Words).To(HaveLen(1))
+		Expect(segs[0].Words[0].Text).To(Equal("a"))
+		Expect(segs[1].Words[0].Text).To(Equal("b"))
+	})
+
+	// The C ABI documents the speaker tag as 1-based with 0 meaning "untagged",
+	// so a run of untagged words must not come back attributed to a speaker
+	// literally named "0".
+	It("labels a diarized run and leaves an untagged one unlabelled", func() {
+		Expect(wordsToSegments([]asrWord{{Text: "a", Speaker: 2}}, false)[0].Speaker).To(Equal("2"))
+		Expect(wordsToSegments([]asrWord{{Text: "a", Speaker: 0}}, false)[0].Speaker).To(BeEmpty())
+	})
+})
+
+var _ = Describe("wordsRequested", func() {
+	It("recognises the OpenAI word granularity in any casing or padding", func() {
+		Expect(wordsRequested([]string{"word"})).To(BeTrue())
+		Expect(wordsRequested([]string{"segment", " Word "})).To(BeTrue())
+	})
+
+	It("defaults to segment level", func() {
+		Expect(wordsRequested(nil)).To(BeFalse())
+		Expect(wordsRequested([]string{"segment"})).To(BeFalse())
+	})
+})
+
+var _ = Describe("recognizeF32", func() {
+	// &pcm[0] panics on a zero-length slice, and a silent or empty upload is
+	// ordinary input rather than an exotic one. The C side rejects empty audio
+	// too, but Go never gets that far.
+	It("refuses empty audio instead of indexing an empty slice", func() {
+		// A zero options struct is enough: the guard has to fire before the
+		// options are ever handed across the ABI, and building real ones would
+		// need the library bound, which this spec deliberately does not.
+		opts := cASRRecognitionOptions{}
+		for _, pcm := range [][]float32{nil, {}} {
+			var (
+				handle uintptr
+				err    error
+			)
+			Expect(func() { handle, err = recognizeF32(0, &opts, pcm, 16000) }).ToNot(Panic())
+			Expect(err).To(HaveOccurred())
+			Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+			Expect(err.Error()).To(ContainSubstring("empty audio"))
+			Expect(handle).To(BeZero())
+		}
+	})
+})
+
+var _ = Describe("AudioTranscription", func() {
+	// The gate has to fire before anything expensive: a model loaded as TTS
+	// cannot transcribe whatever the request says, and reading the audio first
+	// would report a file problem for a configuration one.
+	It("refuses a model loaded as another family, before it reads the audio", func() {
+		n := &NemoSpeech{fam: familyTTS}
+		_, err := n.AudioTranscription(context.Background(), &pb.TranscriptRequest{
+			Dst: filepath.Join(GinkgoT().TempDir(), "does-not-exist.wav"),
+		})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(err.Error()).To(ContainSubstring("tts"))
+	})
+
+	It("refuses an unloaded model", func() {
+		n := &NemoSpeech{}
+		_, err := n.AudioTranscription(context.Background(), &pb.TranscriptRequest{Dst: "ignored.wav"})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+	})
+
+	// A lock leaked on a rejection path deadlocks the next request rather than
+	// failing it, which is far harder to diagnose than the failure itself.
+	It("releases the engine lock on every rejection path", func() {
+		n := &NemoSpeech{fam: familyTTS}
+		_, err := n.AudioTranscription(context.Background(), &pb.TranscriptRequest{Dst: "x.wav"})
+		Expect(err).To(HaveOccurred())
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+
+	It("reports an audio file it cannot read", func() {
+		n := &NemoSpeech{fam: familyASR}
+		_, err := n.AudioTranscription(context.Background(), &pb.TranscriptRequest{
+			Dst: filepath.Join(GinkgoT().TempDir(), "absent.wav"),
+		})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+
+	It("requires a destination path", func() {
+		n := &NemoSpeech{fam: familyASR}
+		_, err := n.AudioTranscription(context.Background(), &pb.TranscriptRequest{})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	// The whole rejection path end to end, on the input that actually reaches
+	// it: a silent or truncated upload decodes to zero samples, and the guard
+	// has to fire between the decode and the ABI. Nothing is loaded here (no
+	// recognizer, and the specs that bind the library may not have run), so
+	// this also pins the ORDER: a guard placed after the options are built
+	// calls a nil-bound entry point and panics rather than failing.
+	It("refuses a decodable clip that carries no samples", func() {
+		path := filepath.Join(GinkgoT().TempDir(), "silence.wav")
+		writeMono16kWAV(path, 0)
+
+		n := &NemoSpeech{fam: familyASR, recognizer: 0}
+		var err error
+		Expect(func() {
+			_, err = n.AudioTranscription(context.Background(), &pb.TranscriptRequest{Dst: path})
+		}).ToNot(Panic())
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("empty audio"))
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+})
+
+var _ = Describe("sampleRateOf", func() {
+	// 0 is not "unknown" to this runtime: nemo_speech_asr_recognize_f32 and
+	// nemo_speech_asr_stream_push_f32 both read a 0 rate as "these samples are
+	// already at the model rate" and skip resampling. Falling back to it for an
+	// undecodable header would silently pitch-shift the audio instead of
+	// failing, so an unknown rate has to be an error.
+	It("rejects a buffer whose format the decoder did not fill in", func() {
+		_, err := sampleRateOf(&audio.IntBuffer{})
+		Expect(err).To(HaveOccurred())
+	})
+
+	It("rejects a non-positive sample rate", func() {
+		_, err := sampleRateOf(&audio.IntBuffer{Format: &audio.Format{SampleRate: 0, NumChannels: 1}})
+		Expect(err).To(HaveOccurred())
+	})
+
+	// The WAV header carries the sample rate as an unsigned 32-bit field, which
+	// go-audio widens to int. Anything above the int32 range therefore passes a
+	// "> 0" test and then narrows to a NEGATIVE rate, which the runtime would take
+	// as a resampling ratio rather than reject. The failure is silent, so the
+	// bound is asserted rather than left to the caller.
+	//
+	// Written as a conversion plus one rather than as the constant MaxInt32+1:
+	// the untyped form does not fit an int on a 32-bit build and would not
+	// compile there, while this wraps to a negative rate the same guard rejects.
+	It("rejects a rate that would not survive the narrowing to int32", func() {
+		_, err := sampleRateOf(&audio.IntBuffer{
+			Format: &audio.Format{SampleRate: int(math.MaxInt32) + 1, NumChannels: 1},
+		})
+		Expect(err).To(HaveOccurred())
+	})
+
+	It("returns the decoded rate", func() {
+		rate, err := sampleRateOf(&audio.IntBuffer{Format: &audio.Format{SampleRate: 22050, NumChannels: 1}})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(rate).To(Equal(int32(22050)))
+	})
+})
+
+var _ = Describe("decodeAudioMono16k", func() {
+	It("decodes a 16 kHz mono WAV to float32 samples at its own rate", func() {
+		path := filepath.Join(GinkgoT().TempDir(), "silence.wav")
+		writeMono16kWAV(path, 800)
+
+		pcm, rate, err := decodeAudioMono16k(path)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(rate).To(Equal(int32(16000)))
+		Expect(pcm).To(HaveLen(800))
+	})
+
+	// A zero-frame WAV is what a truncated upload decodes to, and it is the
+	// input recognizeF32's guard exists for.
+	It("decodes a WAV with no frames to an empty slice", func() {
+		path := filepath.Join(GinkgoT().TempDir(), "empty.wav")
+		writeMono16kWAV(path, 0)
+
+		pcm, _, err := decodeAudioMono16k(path)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(pcm).To(BeEmpty())
+	})
+
+	It("reports a file that does not exist", func() {
+		_, _, err := decodeAudioMono16k(filepath.Join(GinkgoT().TempDir(), "nope.wav"))
+		Expect(err).To(HaveOccurred())
+	})
+})
+
+// The six frame counts on the recognizer-attached diarizer are
+// sentinel-sensitive and invisible to every other check in the tree.
+// src/asr/c_api.cpp:151-165 applies five of them when they are > 0 but applies
+// left_context_frames when it is >= 0, so a dropped -1 does not fall back to
+// the model's own streaming geometry, it pins the left context to zero. The
+// struct is the right shape either way, so abi_test.go's layout assertions
+// cannot see it.
+var _ = Describe("asrDiarConfig", func() {
+	It("keeps the model path it was given", func() {
+		Expect(asrDiarConfig(42).ModelPath).To(Equal(uintptr(42)))
+	})
+
+	// A config sent with the wrong size has every field past it ignored by
+	// HAS_FIELD, and the diarizer attaches with defaults instead of failing.
+	It("declares the size the runtime validates against", func() {
+		Expect(asrDiarConfig(42).Size).To(Equal(unsafe.Sizeof(cASRDiarConfig{})))
+	})
+
+	It("leaves every frame count at the sentinel that means default", func() {
+		cfg := asrDiarConfig(42)
+		Expect(cfg.ChunkFrames).To(Equal(diarGeometryDefault))
+		Expect(cfg.RightContextFrames).To(Equal(diarGeometryDefault))
+		Expect(cfg.LeftContextFrames).To(Equal(diarGeometryDefault))
+		Expect(cfg.FIFOFrames).To(Equal(diarGeometryDefault))
+		Expect(cfg.SpkcacheFrames).To(Equal(diarGeometryDefault))
+		Expect(cfg.UpdatePeriodFrames).To(Equal(diarGeometryDefault))
+	})
+
+	// Stated separately from the field-by-field assertions above: the whole
+	// group is only "unset" to the runtime while the sentinel stays negative,
+	// and zero is a value it would apply to the left context.
+	It("uses a negative sentinel, not zero", func() {
+		Expect(diarGeometryDefault).To(BeNumerically("<", 0))
+	})
+})

backend/go/nemo-speech-cpp/audio.go

@@ -0,0 +1,86 @@
+package main
+
+import (
+	"errors"
+	"math"
+	"os"
+	"path/filepath"
+
+	"github.com/go-audio/audio"
+	"github.com/go-audio/wav"
+	"github.com/mudler/LocalAI/pkg/utils"
+)
+
+// decodeAudioMono16k converts an arbitrary audio file to 16 kHz mono PCM and
+// returns the float32 samples together with the rate they are actually at.
+//
+// pkg/utils exposes the ffmpeg normalisation (AudioToWav) but no decode, so
+// every Go ASR backend pairs it with go-audio itself. This mirrors
+// backend/go/parakeet-cpp rather than adding a shared helper: the backends
+// differ in what they need back (parakeet wants a duration, this one wants the
+// sample rate to hand to the runtime), so a shared signature would be a
+// lowest-common-denominator of both.
+func decodeAudioMono16k(path string) ([]float32, int32, error) {
+	dir, err := os.MkdirTemp("", "nemo-speech")
+	if err != nil {
+		return nil, 0, err
+	}
+	defer func() { _ = os.RemoveAll(dir) }()
+
+	// A WAV already at 16 kHz mono 16-bit is hardlinked or copied through
+	// without spawning ffmpeg, so the common case costs nothing.
+	converted := filepath.Join(dir, "converted.wav")
+	if err := utils.AudioToWav(path, converted); err != nil {
+		return nil, 0, err
+	}
+
+	// #nosec G304 -- converted is filepath.Join of a directory this function just
+	// created with os.MkdirTemp and a constant basename. The request-controlled
+	// path is the INPUT to AudioToWav and never reaches this open.
+	fh, err := os.Open(converted)
+	if err != nil {
+		return nil, 0, err
+	}
+	defer func() { _ = fh.Close() }()
+
+	buf, err := wav.NewDecoder(fh).FullPCMBuffer()
+	if err != nil {
+		return nil, 0, err
+	}
+
+	// The rate is read back from the decoded file rather than assumed to be
+	// 16000. AudioToWav always lands there today, but the runtime resamples
+	// anything from 8 to 96 kHz off this number, so a wrong one would not fail,
+	// it would silently pitch-shift the audio and quietly degrade the transcript.
+	rate, err := sampleRateOf(buf)
+	if err != nil {
+		return nil, 0, err
+	}
+	return buf.AsFloat32Buffer().Data, rate, nil
+}
+
+// sampleRateOf reads the decoded rate back off the buffer.
+//
+// It is an error rather than a zero fallback because 0 is not "unknown" to this
+// runtime: nemo_speech_asr_recognize_f32 and nemo_speech_asr_stream_push_f32
+// both read a 0 rate as "these samples are already at the model rate" and skip
+// resampling (include/nemo_speech/asr.h). Handing 0 over for a header the
+// decoder could not read would not fail, it would silently pitch-shift the
+// audio and quietly degrade the transcript, which is the same failure the
+// caller comment warns about for a wrong rate.
+//
+// The upper bound is what makes the narrowing to int32 safe rather than merely
+// unlikely. go-audio reads the WAV header's sample rate as an unsigned 32-bit
+// field into an int, so on a 64-bit build a header claiming more than 2^31-1
+// survives the "> 0" test and then narrows to a NEGATIVE rate, which the runtime
+// would take as a resampling ratio. Nothing this backend decodes can reach that
+// today (AudioToWav either passes through a WAV it has confirmed is exactly
+// 16 kHz or runs ffmpeg with -ar 16000), but that is a property of a helper in
+// another package, and this function exists precisely because the rate is read
+// back rather than assumed.
+func sampleRateOf(buf *audio.IntBuffer) (int32, error) {
+	if buf.Format == nil || buf.Format.SampleRate <= 0 || buf.Format.SampleRate > math.MaxInt32 {
+		return 0, errors.New("nemo-speech-cpp: decoded audio has no usable sample rate")
+	}
+	return int32(buf.Format.SampleRate), nil
+}

backend/go/nemo-speech-cpp/diar.go

@@ -0,0 +1,502 @@
+package main
+
+import (
+	"strconv"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// diarSegmentsMaxAttempts bounds the count-then-fill retry.
+//
+// On a finished stream the count is stable and one attempt is always enough.
+// The bound exists because the RPC holds engineMu for its whole body, so a
+// runtime whose count kept growing would not merely spin, it would block the
+// unload behind it.
+const diarSegmentsMaxAttempts = 4
+
+// maxDiarSegments caps the buffer collectSegments will allocate from a count
+// the C side reported.
+//
+// make() panics rather than erroring on a length it cannot satisfy, and a
+// panic in an RPC handler takes the backend process down, so an uninitialised
+// or corrupted size_t coming back across the ABI would kill the model rather
+// than fail the request. The ceiling turns that into a diagnosable error.
+//
+// It is set far above anything real: a segment spans at least one 80 ms frame,
+// so 2^22 segments is upwards of 93 hours of audio, and the buffer itself
+// would already be 100 MB at 24 bytes each.
+const maxDiarSegments = 1 << 22
+
+// diarSegmenter is the result half of the diarization C API: the two-call
+// protocol nemo_speech_diar_segments documents.
+//
+// The two calls are the same C function with a different `out`, but they are
+// separate methods here because their contracts differ. countSegments passes
+// out=NULL, which the runtime answers by writing *count and returning OK
+// without touching a buffer. fillSegments passes a real buffer and gets
+// INVALID_ARGUMENT if it is too short, having written *count first, which is
+// what makes a growth retry possible at all.
+type diarSegmenter interface {
+	// countSegments is the size query. It never fails for lack of a buffer.
+	countSegments() (uint64, error)
+	// fillSegments fills buf and returns the count the runtime reported. That
+	// count is meaningful even alongside an error: on a short buffer the
+	// runtime writes it before rejecting the call.
+	fillSegments(buf []cDiarSegment) (uint64, error)
+}
+
+// diarStream is one diarization job over the C API, narrowed to what the RPC
+// uses.
+//
+// It is an interface for the same reason asrSession is: no Sortformer GGUF is
+// small enough to keep in the tree, so without a seam at the ABI the loop on
+// top of it (the empty guard, chunking, finish-before-query, the growth retry)
+// would have no test at all. A fake here scripts what C returns; it does not
+// pretend to diarize anything.
+type diarStream interface {
+	diarSegmenter
+	push(pcm []float32, sampleRate int32) error
+	finish() error
+	close()
+}
+
+// diarStreamOpener creates a job. n.openDiarStream is the C-backed one.
+//
+// The segmentation config is handed over at open time rather than per query
+// because it belongs to the whole job: every segments call on one stream must
+// use the same postprocessing or the segment ids would not be comparable
+// between calls.
+type diarStreamOpener func(cfg *cDiarSegmentationConfig) (diarStream, error)
+
+// cDiarStream is the real diarStream, over one nemo_speech_diar_stream.
+type cDiarStream struct {
+	handle uintptr
+	cfg    *cDiarSegmentationConfig
+}
+
+// cfgPtr hands the segmentation config to C, or NULL when the request asked
+// for no postprocessing. NULL is not the same as a zeroed struct in spirit
+// even though src/asr/c_api.cpp treats them alike today: diar.h documents NULL
+// as "library defaults", so it is the one form that cannot be invalidated by a
+// future field whose sentinel is not zero.
+func (s *cDiarStream) cfgPtr() unsafe.Pointer {
+	if s.cfg == nil {
+		return nil
+	}
+	// #nosec G103 -- a plain *T to unsafe.Pointer conversion of a non-nil,
+	// GC-traced field. cDiarSegmentationConfig is pure scalars (no uintptr
+	// members to pin) and the stream owns it for its whole life, so the only
+	// requirement is that it outlive the DiarSegments call, which it does.
+	return unsafe.Pointer(s.cfg)
+}
+
+func (s *cDiarStream) push(pcm []float32, sampleRate int32) error {
+	// &pcm[0] panics on an empty slice before the C side ever sees the call.
+	if len(pcm) == 0 {
+		return nil
+	}
+	if st := DiarStreamPushF32(s.handle, &pcm[0], uint64(len(pcm)), sampleRate); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: diarization push: %s", ASRLastError())
+	}
+	return nil
+}
+
+func (s *cDiarStream) finish() error {
+	if st := DiarStreamFinish(s.handle); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: diarization finish: %s", ASRLastError())
+	}
+	return nil
+}
+
+func (s *cDiarStream) close() { DiarStreamClose(s.handle) }
+
+func (s *cDiarStream) countSegments() (uint64, error) {
+	var count uint64
+	// out=NULL and capacity=0: the size query. The runtime reads capacity only
+	// once it has a buffer to check it against.
+	if st := DiarSegments(s.handle, s.cfgPtr(), nil, 0, &count); st != 0 {
+		return 0, statusErrorf(st,
+			"nemo-speech-cpp: diarization segment count: %s", ASRLastError())
+	}
+	return count, nil
+}
+
+func (s *cDiarStream) fillSegments(buf []cDiarSegment) (uint64, error) {
+	if len(buf) == 0 {
+		// A NULL out would silently turn this into a second size query, and the
+		// caller would read it as "filled nothing" rather than "asked nothing".
+		return 0, status.Error(codes.Internal,
+			"nemo-speech-cpp: diarization segment fill needs a buffer")
+	}
+	var count uint64
+	// #nosec G103 -- &buf[0] is guarded by the empty check above, and the
+	// capacity handed over is exactly len(buf), so the runtime cannot write past
+	// the caller's allocation. collectSegments sizes buf under maxDiarSegments
+	// and rejects a reported count larger than it rather than slicing to it.
+	st := DiarSegments(s.handle, s.cfgPtr(), unsafe.Pointer(&buf[0]), uint64(len(buf)), &count)
+	if st != 0 {
+		// count is returned alongside the error on purpose: a too-small buffer
+		// is rejected only after the runtime has written the size it wanted.
+		return count, statusErrorf(st,
+			"nemo-speech-cpp: diarization segments: %s", ASRLastError())
+	}
+	return count, nil
+}
+
+// diarGeometryDefault is the sentinel that means "keep the preset's value" for
+// every one of nemo_speech_diar_model_config's six frame counts.
+//
+// It has to be negative, not zero, and that is not a style choice.
+// src/asr/c_api.cpp:497-512 applies five of the six overrides when they are
+// > 0 but applies left_context_frames when it is >= 0, so a zero-valued config
+// reads as "unset" for five fields and as an explicit left context of zero for
+// the sixth. That silently changes the model's streaming geometry, and no
+// layout assertion can see it because the struct is the right shape either way.
+const diarGeometryDefault int32 = -1
+
+// diarModelConfig builds the create-time config for the standalone diarizer.
+//
+// Extracted from loadDiarizer purely so the sentinels above can be asserted:
+// they are invisible to every other check in the tree, including the layout
+// assertions, so a spec pinning them is the only thing standing between a
+// dropped -1 and a quietly mis-configured model.
+//
+// modelPath is a C pointer from cstr, not a Go string, and the caller owns its
+// release. preset is deliberately left NULL, which diar.h reads as "streaming".
+// The "offline" preset is a different accuracy/latency tradeoff for long files
+// and is worth exposing, but not on an unverified guess: no Sortformer GGUF
+// exists here to measure the difference on.
+func diarModelConfig(modelPath uintptr, gpu int32) cDiarModelConfig {
+	return cDiarModelConfig{
+		Size:               unsafe.Sizeof(cDiarModelConfig{}),
+		ModelPath:          modelPath,
+		GPU:                gpu,
+		ChunkFrames:        diarGeometryDefault,
+		RightContextFrames: diarGeometryDefault,
+		LeftContextFrames:  diarGeometryDefault,
+		FIFOFrames:         diarGeometryDefault,
+		SpkcacheFrames:     diarGeometryDefault,
+		UpdatePeriodFrames: diarGeometryDefault,
+	}
+}
+
+// loadDiarizer creates the standalone Sortformer diarizer.
+//
+// This must not take engineMu: Load is its only caller and already holds it.
+func (n *NemoSpeech) loadDiarizer(modelFile string) error {
+	pathP, freePath := cstr(modelFile)
+	defer freePath()
+
+	cfg := diarModelConfig(pathP, n.opts.gpu)
+
+	xlog.Info("nemo-speech-cpp: creating diarizer", "gpu", n.opts.gpu)
+
+	// #nosec G103 -- cfg is a local POD struct borrowed for this call only. Its
+	// only uintptr member is ModelPath, the cstr allocation pinned by the
+	// deferred freePath above (Preset is deliberately NULL), and
+	// nemo_speech_diar_create deep-copies the path and retains nothing.
+	if st := DiarCreate(unsafe.Pointer(&cfg), &n.diarizer); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: diarizer create: %s", ASRLastError())
+	}
+	return nil
+}
+
+// openDiarStream starts a diarization job on the loaded model.
+//
+// The caller must hold engineMu.
+func (n *NemoSpeech) openDiarStream(cfg *cDiarSegmentationConfig) (diarStream, error) {
+	var handle uintptr
+	if st := DiarStreamOpen(n.diarizer, &handle); st != 0 {
+		return nil, statusErrorf(st,
+			"nemo-speech-cpp: diarization stream open: %s", ASRLastError())
+	}
+	return &cDiarStream{handle: handle, cfg: cfg}, nil
+}
+
+// sizeofDiarSegmentationConfig is the size the runtime validates the config
+// against. It is a function so the specs can assert the value the config
+// actually carries rather than restate the number.
+func sizeofDiarSegmentationConfig() uintptr {
+	return unsafe.Sizeof(cDiarSegmentationConfig{})
+}
+
+// segmentationConfig maps the request's postprocessing knobs onto
+// nemo_speech_diar_segmentation_config, or returns nil when none were set.
+//
+// Only two of DiarizeRequest's tuning fields have a real equivalent here, and
+// both are exact rather than approximate: NeMo's ts_vad postprocessing is the
+// same algorithm the proto's wording describes.
+//
+//   - min_duration_on  ("discard segments shorter than this") is min_duration_sec
+//     ("drop segments shorter than this"), which c_api.cpp assigns to
+//     DiarSegmentationCfg.min_duration_on.
+//   - min_duration_off ("merge gaps shorter than this") is min_gap_sec ("fill
+//     silence gaps shorter than this"), assigned to min_duration_off.
+//
+// The names cross over between the proto and the C header, which is exactly the
+// kind of transposition a layout assertion cannot see, so each mapping is
+// pinned by its own spec.
+//
+// Nothing is written for a non-positive value: the runtime tests every field
+// with > 0 and keeps its default otherwise, so a zero here means "unset" on
+// both sides.
+func segmentationConfig(req *pb.DiarizeRequest) *cDiarSegmentationConfig {
+	cfg := cDiarSegmentationConfig{Size: sizeofDiarSegmentationConfig()}
+
+	var set bool
+	if v := req.GetMinDurationOn(); v > 0 {
+		cfg.MinDurationSec = float64(v)
+		set = true
+	}
+	if v := req.GetMinDurationOff(); v > 0 {
+		cfg.MinGapSec = float64(v)
+		set = true
+	}
+	if !set {
+		return nil
+	}
+	return &cfg
+}
+
+// unsupportedRequestFields names the DiarizeRequest fields this backend cannot
+// honour, so they are logged rather than silently dropped.
+//
+// Each is a deliberate omission, not a gap waiting to be filled:
+//
+//   - num_speakers, min_speakers, max_speakers: Sortformer is end-to-end and
+//     its speaker capacity is fixed by the checkpoint (v2: 4).
+//     nemo_speech_diar_num_speakers reports that capacity, it does not set it,
+//     and there is no config field for a target count.
+//   - clustering_threshold: there is no clustering stage. The nearest knob is
+//     the onset/offset probability hysteresis, which is a different quantity on
+//     a different scale, so mapping one onto the other would invent an
+//     equivalence the header does not have.
+//   - include_text: this pipeline carries no ASR at all (diar.h: "no ASR
+//     involved"). Word-level speaker tags on a transcript are the ASR surface's
+//     job, through diar_model plus enable_speaker_diarization.
+//   - threads: neither nemo_speech_diar_model_config nor the segmentation
+//     config has a thread count.
+func unsupportedRequestFields(req *pb.DiarizeRequest) []string {
+	var out []string
+	if req.GetNumSpeakers() != 0 {
+		out = append(out, "num_speakers")
+	}
+	if req.GetMinSpeakers() != 0 {
+		out = append(out, "min_speakers")
+	}
+	if req.GetMaxSpeakers() != 0 {
+		out = append(out, "max_speakers")
+	}
+	if req.GetClusteringThreshold() != 0 {
+		out = append(out, "clustering_threshold")
+	}
+	if req.GetIncludeText() {
+		out = append(out, "include_text")
+	}
+	if req.GetThreads() != 0 {
+		out = append(out, "threads")
+	}
+	return out
+}
+
+// collectSegments runs the count-then-fill protocol and returns the segments.
+//
+// The growth retry is not defensive padding. nemo_speech_diar_segments writes
+// *count and only then rejects a buffer that is too small, so the size a
+// rejected call reports is the size to retry with; without the retry a stream
+// that gained a segment between the two calls would fail the whole request.
+// Truncating to the first count instead would be worse still, dropping turns
+// with nothing to show for it.
+func collectSegments(s diarSegmenter) ([]cDiarSegment, error) {
+	want, err := s.countSegments()
+	if err != nil {
+		return nil, err
+	}
+
+	for range diarSegmentsMaxAttempts {
+		if want == 0 {
+			// No segments means no fill: the fill call needs a non-empty buffer
+			// to be distinguishable from a second size query.
+			return nil, nil
+		}
+		if want > maxDiarSegments {
+			return nil, status.Errorf(codes.Internal,
+				"nemo-speech-cpp: diarization reported %d segments, above the %d ceiling", want, maxDiarSegments)
+		}
+
+		buf := make([]cDiarSegment, want)
+		got, fillErr := s.fillSegments(buf)
+		if fillErr == nil {
+			if got > want {
+				// The runtime cannot report this on success (it rejects a short
+				// buffer instead), so it means the ABI is not what this code
+				// thinks it is. Slicing to it would read past the allocation.
+				return nil, status.Errorf(codes.Internal,
+					"nemo-speech-cpp: diarization returned %d segments for a %d-segment buffer", got, want)
+			}
+			return buf[:got], nil
+		}
+		// A count that did not grow means the call failed for some other
+		// reason, and retrying the same size would just fail the same way.
+		if got <= want {
+			return nil, fillErr
+		}
+		want = got
+	}
+
+	return nil, status.Error(codes.Internal,
+		"nemo-speech-cpp: diarization segment count kept growing, giving up")
+}
+
+// toDiarizeSegments converts the runtime's segments to the wire form.
+//
+// No unit conversion happens here, and that is the point: nemo_speech_diar_segment
+// carries start_time and end_time in SECONDS already (diar.h), and
+// DiarizeSegment.start/end are seconds too. The frame indices the model works
+// in never reach this layer, so nemo_speech_diar_seconds_per_frame is not
+// involved. The narrowing to float32 is the proto's choice of type; at 80 ms
+// resolution it is lossless for any clip short enough to hold in memory.
+//
+// The speaker label is the runtime's 1-based tag rendered as a decimal string,
+// which is what wordsToSegments emits for the ASR path. The same speaker has to
+// read the same way whether the caller diarized a file or transcribed it.
+func toDiarizeSegments(in []cDiarSegment) []*pb.DiarizeSegment {
+	if len(in) == 0 {
+		return nil
+	}
+	out := make([]*pb.DiarizeSegment, 0, len(in))
+	for i, s := range in {
+		out = append(out, &pb.DiarizeSegment{
+			Id:      int32(i),
+			Start:   float32(s.StartTime),
+			End:     float32(s.EndTime),
+			Speaker: strconv.Itoa(int(s.Speaker)),
+		})
+	}
+	return out
+}
+
+// distinctSpeakers counts the speaker labels present in the segments.
+//
+// This is what DiarizeResponse.num_speakers is documented to hold, and it is
+// NOT nemo_speech_diar_num_speakers: that reports the checkpoint's capacity
+// (four for Sortformer v2), so a two-person interview would come back claiming
+// four speakers.
+func distinctSpeakers(segs []*pb.DiarizeSegment) int32 {
+	seen := make(map[string]struct{}, len(segs))
+	for _, s := range segs {
+		seen[s.GetSpeaker()] = struct{}{}
+	}
+	// #nosec G115 -- seen holds at most one entry per segment, and collectSegments
+	// refuses any count above maxDiarSegments (2^22), so this is orders of
+	// magnitude below the int32 the proto field is.
+	return int32(len(seen))
+}
+
+// diarizePCM drives one whole clip through a diarization job.
+//
+// The caller must hold engineMu.
+func diarizePCM(open diarStreamOpener, pcm []float32, sampleRate int32, cfg *cDiarSegmentationConfig) (*pb.DiarizeResponse, error) {
+	// Before the stream is opened, not inside the push: a silent or truncated
+	// upload decodes to zero samples, &pcm[0] panics on that, and there is no
+	// diarization to be had from it anyway.
+	if len(pcm) == 0 {
+		return nil, status.Error(codes.InvalidArgument, "nemo-speech-cpp: empty audio")
+	}
+
+	stream, err := open(cfg)
+	if err != nil {
+		return nil, err
+	}
+	defer stream.close()
+
+	// Chunked rather than pushed whole so the runtime advances as it goes
+	// instead of buffering the entire clip before the first chunk boundary.
+	for _, chunk := range chunkPCM(pcm, streamChunkSamples) {
+		if err := stream.push(chunk, sampleRate); err != nil {
+			return nil, err
+		}
+	}
+	// Before the query, always: finish is what labels the audio tail, so
+	// segmenting first drops the last turn of every clip.
+	if err := stream.finish(); err != nil {
+		return nil, err
+	}
+
+	raw, err := collectSegments(stream)
+	if err != nil {
+		return nil, err
+	}
+
+	segs := toDiarizeSegments(raw)
+	out := &pb.DiarizeResponse{
+		Segments:    segs,
+		NumSpeakers: distinctSpeakers(segs),
+	}
+	// 0 is the proto's "unknown" and the C API's "already at the model rate",
+	// so a rate that means the latter must not be divided by.
+	if sampleRate > 0 {
+		out.Duration = float32(len(pcm)) / float32(sampleRate)
+	}
+	// Language and the per-segment text stay empty: there is no ASR in this
+	// pipeline to fill them, and the proto documents both as optional.
+	return out, nil
+}
+
+// Diarize labels who spoke when in the audio at req.Dst.
+//
+// The whole body runs inside withEngine, so the family check and the C calls
+// that trust the handle happen under a single acquisition of engineMu. The
+// audio decode is in there too, for the reason documented on
+// AudioTranscription: the backend already serialises RPCs, so the wider hold
+// costs nothing, and the narrower one is the gap Free can land in.
+func (n *NemoSpeech) Diarize(req *pb.DiarizeRequest) (pb.DiarizeResponse, error) {
+	var out *pb.DiarizeResponse
+	if err := n.withEngine(familyDiarization, func() error {
+		r, err := n.diarize(req)
+		out = r
+		return err
+	}); err != nil {
+		return pb.DiarizeResponse{}, err
+	}
+	if out == nil {
+		return pb.DiarizeResponse{}, status.Error(codes.Internal,
+			"nemo-speech-cpp: diarization produced no result")
+	}
+
+	// Assembled field by field rather than dereferenced: the RPC returns the
+	// message by value and the message embeds a mutex, so copying the struct is
+	// a copylocks violation.
+	return pb.DiarizeResponse{
+		Segments:    out.Segments,
+		NumSpeakers: out.NumSpeakers,
+		Duration:    out.Duration,
+		Language:    out.Language,
+	}, nil
+}
+
+// diarize is Diarize's body. The caller must hold engineMu.
+func (n *NemoSpeech) diarize(req *pb.DiarizeRequest) (*pb.DiarizeResponse, error) {
+	if req.GetDst() == "" {
+		return nil, status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: DiarizeRequest.dst (audio path) is required")
+	}
+	// Logged rather than rejected: a client that asks for a speaker count still
+	// wants the diarization it can have, and a request that names a field this
+	// backend drops should say so somewhere the operator can find it.
+	if dropped := unsupportedRequestFields(req); len(dropped) > 0 {
+		xlog.Warn("nemo-speech-cpp: ignoring diarization request fields this model has no equivalent for",
+			"fields", dropped)
+	}
+
+	pcm, sampleRate, err := decodeAudioMono16k(req.GetDst())
+	if err != nil {
+		return nil, status.Errorf(codes.InvalidArgument, "nemo-speech-cpp: read audio: %v", err)
+	}
+
+	return diarizePCM(n.openDiarStream, pcm, sampleRate, segmentationConfig(req))
+}

backend/go/nemo-speech-cpp/diar_test.go

@@ -0,0 +1,540 @@
+package main
+
+import (
+	"errors"
+	"path/filepath"
+	"unsafe"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// fakeDiarStream scripts what the C API returns for one diarization job.
+//
+// There is no Sortformer GGUF in the tree, so this is the only way the loop on
+// top of the ABI (the empty guard, chunking, the count-then-fill protocol, the
+// buffer growth retry) gets tested at all. It fakes the C contract, not the
+// model: segs is whatever nemo_speech_diar_segments would have produced.
+type fakeDiarStream struct {
+	segs []cDiarSegment
+
+	// countErr and fillErrs script failures. fillErrs is consumed one entry per
+	// fillSegments call so a growth retry can be scripted.
+	countErr error
+	fillErrs []error
+	// queryCount, when non-zero, is what the size query reports instead of
+	// len(segs), so a runtime that under-reported can be scripted.
+	queryCount uint64
+	// growTo, when non-zero, is the count reported by the FIRST fillSegments
+	// call, standing in for a runtime whose segment list outgrew the size query.
+	growTo uint64
+
+	pushed   [][]float32
+	rates    []int32
+	finished int
+	closed   int
+	counts   int
+	fills    int
+	// opened records the segmentation config the opener was handed.
+	cfg *cDiarSegmentationConfig
+}
+
+func (f *fakeDiarStream) push(pcm []float32, rate int32) error {
+	f.pushed = append(f.pushed, pcm)
+	f.rates = append(f.rates, rate)
+	return nil
+}
+
+func (f *fakeDiarStream) finish() error {
+	f.finished++
+	return nil
+}
+
+func (f *fakeDiarStream) close() { f.closed++ }
+
+func (f *fakeDiarStream) countSegments() (uint64, error) {
+	f.counts++
+	if f.countErr != nil {
+		return 0, f.countErr
+	}
+	if f.queryCount > 0 {
+		return f.queryCount, nil
+	}
+	return uint64(len(f.segs)), nil
+}
+
+func (f *fakeDiarStream) fillSegments(buf []cDiarSegment) (uint64, error) {
+	f.fills++
+	var err error
+	if len(f.fillErrs) > 0 {
+		err, f.fillErrs = f.fillErrs[0], f.fillErrs[1:]
+	}
+	if f.fills == 1 && f.growTo > 0 {
+		// The runtime writes *count before it rejects a short buffer, so a
+		// growth failure still reports the count the caller needs.
+		return f.growTo, err
+	}
+	n := copy(buf, f.segs)
+	return uint64(n), err
+}
+
+// allPushed flattens what the fake received, so a spec can assert the audio
+// arrived intact regardless of how it was chunked.
+func (f *fakeDiarStream) allPushed() []float32 {
+	var out []float32
+	for _, c := range f.pushed {
+		out = append(out, c...)
+	}
+	return out
+}
+
+func (f *fakeDiarStream) opener() diarStreamOpener {
+	return func(cfg *cDiarSegmentationConfig) (diarStream, error) {
+		f.cfg = cfg
+		return f, nil
+	}
+}
+
+var _ = Describe("Diarize", func() {
+	It("refuses when the loaded model is not a diarization model", func() {
+		n := &NemoSpeech{fam: familyASR}
+		_, err := n.Diarize(&pb.DiarizeRequest{Dst: "/tmp/whatever.wav"})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+	})
+
+	It("refuses on a model that was never loaded", func() {
+		n := &NemoSpeech{}
+		_, err := n.Diarize(&pb.DiarizeRequest{Dst: "/tmp/whatever.wav"})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+	})
+
+	// The family gate has to run before anything reads the request, or a
+	// misrouted request would be reported as a bad path rather than as a model
+	// that cannot diarize.
+	It("reports a missing audio path on a diarization model", func() {
+		n := &NemoSpeech{fam: familyDiarization}
+		_, err := n.Diarize(&pb.DiarizeRequest{})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("dst"))
+	})
+
+	It("reports audio it cannot read", func() {
+		n := &NemoSpeech{fam: familyDiarization}
+		missing := filepath.Join(GinkgoT().TempDir(), "absent.wav")
+		_, err := n.Diarize(&pb.DiarizeRequest{Dst: missing})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("read audio"))
+	})
+
+	// A rejection must leave the mutex free, or the next request deadlocks
+	// rather than fails.
+	It("releases the engine lock on every rejection path", func() {
+		n := &NemoSpeech{fam: familyDiarization}
+		_, err := n.Diarize(&pb.DiarizeRequest{})
+		Expect(err).To(HaveOccurred())
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+})
+
+var _ = Describe("diarizePCM", func() {
+	// Task 7 found that a purego-bound entry point reached with zero samples
+	// panics on &pcm[0], so a silent clip must be rejected before the stream is
+	// ever opened, not inside the push.
+	It("rejects empty audio without opening a stream", func() {
+		opened := false
+		open := func(*cDiarSegmentationConfig) (diarStream, error) {
+			opened = true
+			return &fakeDiarStream{}, nil
+		}
+		_, err := diarizePCM(open, nil, 16000, nil)
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("empty audio"))
+		Expect(opened).To(BeFalse())
+	})
+
+	It("pushes the whole clip, finishes, and closes the stream", func() {
+		pcm := make([]float32, streamChunkSamples*2+7)
+		for i := range pcm {
+			pcm[i] = float32(i)
+		}
+		f := &fakeDiarStream{}
+		_, err := diarizePCM(f.opener(), pcm, 16000, nil)
+		Expect(err).ToNot(HaveOccurred())
+
+		Expect(f.allPushed()).To(Equal(pcm))
+		Expect(f.pushed).To(HaveLen(3), "the clip must be chunked, not pushed whole")
+		Expect(f.rates).To(HaveEach(int32(16000)))
+		Expect(f.finished).To(Equal(1))
+		Expect(f.closed).To(Equal(1))
+	})
+
+	// Segments must come from a finished stream: the tail of the audio is only
+	// labelled by finish, so asking first silently drops the last turn.
+	It("finishes the stream before it asks for segments", func() {
+		f := &fakeDiarStream{segs: []cDiarSegment{{StartTime: 0, EndTime: 1, Speaker: 1}}}
+		f.fillErrs = nil
+		var finishedAtCount int
+		wrapped := func(cfg *cDiarSegmentationConfig) (diarStream, error) {
+			f.cfg = cfg
+			return &countObserver{fakeDiarStream: f, seen: &finishedAtCount}, nil
+		}
+		_, err := diarizePCM(wrapped, []float32{1, 2, 3}, 16000, nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(finishedAtCount).To(Equal(1), "the size query ran before finish")
+	})
+
+	It("converts the runtime's seconds straight through and numbers the segments", func() {
+		f := &fakeDiarStream{segs: []cDiarSegment{
+			{StartTime: 0, EndTime: 0.8, Speaker: 1},
+			{StartTime: 0.8, EndTime: 2.0, Speaker: 2},
+		}}
+		res, err := diarizePCM(f.opener(), []float32{1, 2, 3}, 16000, nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(res.Segments).To(HaveLen(2))
+
+		Expect(res.Segments[0].GetId()).To(Equal(int32(0)))
+		Expect(res.Segments[0].GetStart()).To(BeNumerically("~", 0.0, 1e-6))
+		Expect(res.Segments[0].GetEnd()).To(BeNumerically("~", 0.8, 1e-6))
+		Expect(res.Segments[0].GetSpeaker()).To(Equal("1"))
+
+		Expect(res.Segments[1].GetId()).To(Equal(int32(1)))
+		Expect(res.Segments[1].GetStart()).To(BeNumerically("~", 0.8, 1e-6))
+		Expect(res.Segments[1].GetEnd()).To(BeNumerically("~", 2.0, 1e-6))
+		Expect(res.Segments[1].GetSpeaker()).To(Equal("2"))
+	})
+
+	It("reports the clip duration in seconds", func() {
+		f := &fakeDiarStream{}
+		res, err := diarizePCM(f.opener(), make([]float32, 32000), 16000, nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(res.GetDuration()).To(BeNumerically("~", 2.0, 1e-6))
+	})
+
+	// 0 is the proto's documented "unknown", and it is also the C API's "these
+	// samples are already at the model rate", so a rate that cannot be trusted
+	// must not be turned into a duration.
+	It("reports no duration when the sample rate is unknown", func() {
+		f := &fakeDiarStream{}
+		res, err := diarizePCM(f.opener(), make([]float32, 32000), 0, nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(res.GetDuration()).To(BeZero())
+	})
+
+	It("hands the segmentation config to the opener", func() {
+		f := &fakeDiarStream{}
+		cfg := &cDiarSegmentationConfig{MinDurationSec: 0.5}
+		_, err := diarizePCM(f.opener(), []float32{1}, 16000, cfg)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(f.cfg).To(BeIdenticalTo(cfg))
+	})
+
+	It("closes the stream when the segment query fails", func() {
+		f := &fakeDiarStream{countErr: errors.New("boom")}
+		_, err := diarizePCM(f.opener(), []float32{1}, 16000, nil)
+		Expect(err).To(MatchError(ContainSubstring("boom")))
+		Expect(f.closed).To(Equal(1))
+	})
+
+	// The pipeline carries no ASR, so text and language stay empty whatever the
+	// caller asked for.
+	It("leaves the transcript fields empty", func() {
+		f := &fakeDiarStream{segs: []cDiarSegment{{StartTime: 0, EndTime: 1, Speaker: 1}}}
+		res, err := diarizePCM(f.opener(), []float32{1}, 16000, nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(res.GetLanguage()).To(BeEmpty())
+		Expect(res.Segments[0].GetText()).To(BeEmpty())
+	})
+})
+
+// countObserver records how many size queries had run by the time finish was
+// called, so the ordering can be asserted without reaching into diarizePCM.
+type countObserver struct {
+	*fakeDiarStream
+	seen *int
+}
+
+func (c *countObserver) finish() error {
+	*c.seen = c.counts + 1 // finish must run before the first query
+	return c.fakeDiarStream.finish()
+}
+
+var _ = Describe("distinctSpeakers", func() {
+	It("counts labels, not segments", func() {
+		segs := []*pb.DiarizeSegment{
+			{Speaker: "1"}, {Speaker: "2"}, {Speaker: "1"}, {Speaker: "2"}, {Speaker: "1"},
+		}
+		Expect(distinctSpeakers(segs)).To(Equal(int32(2)))
+	})
+
+	It("counts a single-speaker recording as one", func() {
+		segs := []*pb.DiarizeSegment{{Speaker: "1"}, {Speaker: "1"}, {Speaker: "1"}}
+		Expect(distinctSpeakers(segs)).To(Equal(int32(1)))
+	})
+
+	// Four segments over three labels, not three over three: with the segment
+	// count and the label count equal, a `return len(segs)` would satisfy this
+	// spec and it would assert nothing.
+	It("counts every distinct label once", func() {
+		segs := []*pb.DiarizeSegment{{Speaker: "1"}, {Speaker: "2"}, {Speaker: "3"}, {Speaker: "2"}}
+		Expect(distinctSpeakers(segs)).To(Equal(int32(3)))
+	})
+
+	It("is zero with no segments", func() {
+		Expect(distinctSpeakers(nil)).To(BeZero())
+	})
+
+	// The response field is documented as the count of distinct labels in
+	// `segments`, which is not the model's capacity: Sortformer v2 can label
+	// four speakers whatever the clip actually contains.
+	It("reports what the segments contain, not the model capacity", func() {
+		f := &fakeDiarStream{segs: []cDiarSegment{
+			{StartTime: 0, EndTime: 1, Speaker: 1},
+			{StartTime: 1, EndTime: 2, Speaker: 2},
+			{StartTime: 2, EndTime: 3, Speaker: 1},
+		}}
+		res, err := diarizePCM(f.opener(), []float32{1}, 16000, nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(res.GetNumSpeakers()).To(Equal(int32(2)))
+	})
+
+	It("reports no speakers when the runtime found no segments", func() {
+		f := &fakeDiarStream{}
+		res, err := diarizePCM(f.opener(), []float32{1}, 16000, nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(res.Segments).To(BeEmpty())
+		Expect(res.GetNumSpeakers()).To(BeZero())
+	})
+})
+
+var _ = Describe("collectSegments", func() {
+	It("skips the fill entirely when there is nothing to collect", func() {
+		f := &fakeDiarStream{}
+		segs, err := collectSegments(f)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(segs).To(BeEmpty())
+		Expect(f.counts).To(Equal(1))
+		Expect(f.fills).To(BeZero(), "a zero count must not be followed by a fill")
+	})
+
+	It("sizes the buffer from the query and fills it", func() {
+		f := &fakeDiarStream{segs: []cDiarSegment{
+			{StartTime: 0, EndTime: 1, Speaker: 1},
+			{StartTime: 1, EndTime: 2, Speaker: 2},
+		}}
+		segs, err := collectSegments(f)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(segs).To(HaveLen(2))
+		Expect(segs[1].Speaker).To(Equal(int32(2)))
+		Expect(f.counts).To(Equal(1))
+		Expect(f.fills).To(Equal(1))
+	})
+
+	// nemo_speech_diar_segments writes *count and only then rejects a buffer
+	// that is too small, so the rejected call still reports the size to retry
+	// with. Truncating instead would silently drop turns.
+	It("grows the buffer and retries when the count outran the query", func() {
+		f := &fakeDiarStream{
+			segs: []cDiarSegment{
+				{StartTime: 0, EndTime: 1, Speaker: 1},
+				{StartTime: 1, EndTime: 2, Speaker: 2},
+				{StartTime: 2, EndTime: 3, Speaker: 1},
+			},
+			// The query saw two, the fill found three and rejected the buffer.
+			queryCount: 2,
+			growTo:     3,
+			fillErrs:   []error{errors.New("capacity too small (need 3)")},
+		}
+		segs, err := collectSegments(f)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(segs).To(HaveLen(3))
+		Expect(f.fills).To(Equal(2))
+	})
+
+	It("propagates a failure that is not about capacity", func() {
+		f := &fakeDiarStream{
+			segs:     []cDiarSegment{{StartTime: 0, EndTime: 1, Speaker: 1}},
+			fillErrs: []error{errors.New("boom")},
+		}
+		_, err := collectSegments(f)
+		Expect(err).To(MatchError(ContainSubstring("boom")))
+		Expect(f.fills).To(Equal(1), "a non-capacity failure must not be retried")
+	})
+
+	// make() panics on a length it cannot satisfy, and a panic in an RPC
+	// handler kills the backend process. A count that could only come from an
+	// uninitialised or corrupted size_t must fail the request instead.
+	It("refuses an implausible count rather than trying to allocate it", func() {
+		f := &fakeDiarStream{queryCount: maxDiarSegments + 1}
+		_, err := collectSegments(f)
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+		Expect(err.Error()).To(ContainSubstring("ceiling"))
+		Expect(f.fills).To(BeZero(), "nothing must be allocated or filled for a bad count")
+	})
+
+	It("still accepts a count right at the ceiling", func() {
+		// Only the guard is under test, so the fill is scripted to report zero
+		// rather than actually materialising a hundred megabytes of segments.
+		f := &fakeDiarStream{queryCount: maxDiarSegments}
+		segs, err := collectSegments(f)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(segs).To(BeEmpty())
+		Expect(f.fills).To(Equal(1))
+	})
+
+	It("propagates a failed size query", func() {
+		f := &fakeDiarStream{countErr: errors.New("no stream")}
+		_, err := collectSegments(f)
+		Expect(err).To(MatchError(ContainSubstring("no stream")))
+		Expect(f.fills).To(BeZero())
+	})
+
+	// A runtime whose count grew on every attempt would otherwise loop forever
+	// holding engineMu, which blocks the unload too.
+	It("gives up rather than retrying forever", func() {
+		f := &fakeDiarStream{segs: []cDiarSegment{{StartTime: 0, EndTime: 1, Speaker: 1}}}
+		g := &alwaysGrowing{fakeDiarStream: f}
+		_, err := collectSegments(g)
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+		Expect(f.fills).To(Equal(diarSegmentsMaxAttempts))
+	})
+})
+
+// alwaysGrowing reports a bigger count on every fill, which is the pathological
+// case the attempt bound exists for.
+type alwaysGrowing struct {
+	*fakeDiarStream
+	n uint64
+}
+
+func (a *alwaysGrowing) fillSegments([]cDiarSegment) (uint64, error) {
+	a.n += 10
+	a.fills++
+	return a.n, errors.New("capacity too small")
+}
+
+// The six frame counts are the one part of the create config that no other
+// check in the tree can see. The layout assertions pin the struct's shape, and
+// a wrong VALUE keeps that shape exactly, so without these specs deleting a
+// sentinel is invisible: c_api.cpp applies left_context_frames at >= 0, so a
+// dropped -1 there silently pins the model's left context to zero.
+var _ = Describe("diarModelConfig", func() {
+	It("declares its own size so the runtime accepts the fields", func() {
+		Expect(diarModelConfig(0, -1).Size).To(Equal(unsafe.Sizeof(cDiarModelConfig{})))
+	})
+
+	It("carries the model path and the configured device", func() {
+		cfg := diarModelConfig(0xDEADBEEF, 2)
+		Expect(cfg.ModelPath).To(Equal(uintptr(0xDEADBEEF)))
+		Expect(cfg.GPU).To(Equal(int32(2)))
+	})
+
+	It("passes the CPU sentinel through untouched", func() {
+		Expect(diarModelConfig(0, -1).GPU).To(Equal(int32(-1)))
+	})
+
+	// Asserted field by field rather than as a whole struct so a failure names
+	// the sentinel that went missing.
+	It("leaves every frame-geometry override at the negative sentinel", func() {
+		cfg := diarModelConfig(0, -1)
+		Expect(cfg.ChunkFrames).To(Equal(int32(-1)), "chunk_frames")
+		Expect(cfg.RightContextFrames).To(Equal(int32(-1)), "right_context_frames")
+		Expect(cfg.FIFOFrames).To(Equal(int32(-1)), "fifo_frames")
+		Expect(cfg.SpkcacheFrames).To(Equal(int32(-1)), "spkcache_frames")
+		Expect(cfg.UpdatePeriodFrames).To(Equal(int32(-1)), "update_period_frames")
+
+		// Called out on its own because it is the only one of the six the
+		// runtime applies at >= 0: zero here is a valid explicit left context,
+		// not "unset", so this is the field a dropped sentinel actually breaks.
+		Expect(cfg.LeftContextFrames).To(Equal(int32(-1)), "left_context_frames")
+		Expect(cfg.LeftContextFrames).To(BeNumerically("<", 0),
+			"left_context_frames is applied at >= 0, so a non-negative value pins the geometry")
+	})
+
+	// The preset selects the streaming geometry wholesale, so it has to stay
+	// NULL until there is a model to verify a different one against.
+	It("leaves the preset unset", func() {
+		Expect(diarModelConfig(0, -1).Preset).To(BeZero())
+	})
+})
+
+var _ = Describe("segmentationConfig", func() {
+	// A request that set nothing must stay NULL on the C side: diar.h documents
+	// NULL as "library defaults", and those defaults are NeMo's callhome-tuned
+	// values for this checkpoint rather than zeros.
+	It("is absent when the request asked for no postprocessing", func() {
+		Expect(segmentationConfig(&pb.DiarizeRequest{})).To(BeNil())
+	})
+
+	It("maps min_duration_on onto the minimum segment duration", func() {
+		cfg := segmentationConfig(&pb.DiarizeRequest{MinDurationOn: 0.4})
+		Expect(cfg).ToNot(BeNil())
+		Expect(cfg.MinDurationSec).To(BeNumerically("~", 0.4, 1e-6))
+		Expect(cfg.MinGapSec).To(BeZero())
+	})
+
+	It("maps min_duration_off onto the gap fill", func() {
+		cfg := segmentationConfig(&pb.DiarizeRequest{MinDurationOff: 0.25})
+		Expect(cfg).ToNot(BeNil())
+		Expect(cfg.MinGapSec).To(BeNumerically("~", 0.25, 1e-6))
+		Expect(cfg.MinDurationSec).To(BeZero())
+	})
+
+	It("declares its own size so the runtime accepts the fields", func() {
+		cfg := segmentationConfig(&pb.DiarizeRequest{MinDurationOn: 0.4})
+		Expect(cfg.Size).To(Equal(sizeofDiarSegmentationConfig()))
+	})
+
+	// The onset/offset hysteresis is not a clustering threshold and Sortformer
+	// has no clustering stage at all, so mapping one onto the other would be an
+	// invented equivalence. It has to stay unset.
+	It("ignores fields this pipeline has no equivalent for", func() {
+		Expect(segmentationConfig(&pb.DiarizeRequest{
+			NumSpeakers:         2,
+			MinSpeakers:         1,
+			MaxSpeakers:         4,
+			ClusteringThreshold: 0.7,
+			IncludeText:         true,
+			Threads:             8,
+		})).To(BeNil())
+	})
+
+	It("ignores non-positive values, which the runtime reads as unset", func() {
+		Expect(segmentationConfig(&pb.DiarizeRequest{MinDurationOn: -1, MinDurationOff: 0})).To(BeNil())
+	})
+})
+
+var _ = Describe("unsupportedRequestFields", func() {
+	It("is empty for a request this backend can honour in full", func() {
+		Expect(unsupportedRequestFields(&pb.DiarizeRequest{
+			Dst:            "/tmp/a.wav",
+			MinDurationOn:  0.4,
+			MinDurationOff: 0.2,
+		})).To(BeEmpty())
+	})
+
+	It("names every field it had to drop", func() {
+		Expect(unsupportedRequestFields(&pb.DiarizeRequest{
+			NumSpeakers:         2,
+			MinSpeakers:         1,
+			MaxSpeakers:         4,
+			ClusteringThreshold: 0.7,
+			IncludeText:         true,
+			Threads:             8,
+		})).To(ConsistOf(
+			"num_speakers", "min_speakers", "max_speakers",
+			"clustering_threshold", "include_text", "threads",
+		))
+	})
+})

backend/go/nemo-speech-cpp/family.go

@@ -0,0 +1,116 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+
+	gguf "github.com/gpustack/gguf-parser-go"
+)
+
+// auxOnlyArchitectures are converted NeMo components that attach to a primary
+// model but are never loadable on their own. Pointing a model config at one is
+// a configuration mistake worth naming explicitly.
+var auxOnlyArchitectures = map[string]string{
+	"nemo-nano-codec": "a TTS codec, set it with the codec_model option on a magpietts model",
+	"vad":             "a VAD model, set it with the vad_model option on an asr model",
+	"pnc":             "a punctuation model, set it with the pnc_model option on an asr model",
+}
+
+// familyFor maps a GGUF general.architecture value to a model family.
+//
+// Unknown architectures resolve to NMT rather than an error: NMT GGUFs come
+// from llama.cpp's converter and carry an ordinary LLM architecture, so there
+// is no NeMo-specific string to match. The user selected this backend
+// explicitly, which is the signal that the model is meant for it.
+func familyFor(arch string) (family, error) {
+	if reason, ok := auxOnlyArchitectures[arch]; ok {
+		return familyUnknown, fmt.Errorf(
+			"nemo-speech-cpp: %q is %s, not a model that can be loaded directly", arch, reason)
+	}
+	switch arch {
+	case "asr":
+		return familyASR, nil
+	case "sortformer":
+		return familyDiarization, nil
+	case "magpietts":
+		return familyTTS, nil
+	}
+	return familyNMT, nil
+}
+
+// ggufArchitecture reads general.architecture from a GGUF file.
+func ggufArchitecture(path string) (string, error) {
+	f, err := gguf.ParseGGUFFile(path, gguf.UseMMap(), gguf.SkipLargeMetadata())
+	if err != nil {
+		return "", fmt.Errorf("nemo-speech-cpp: parse gguf %q: %w", path, err)
+	}
+	kv, found := f.Header.MetadataKV.Index([]string{"general.architecture"})
+	if found == 0 {
+		return "", fmt.Errorf("nemo-speech-cpp: %q has no general.architecture key", path)
+	}
+	arch := kv["general.architecture"]
+	// ValueString panics on a mistyped key, and a hand-written or half-converted
+	// GGUF is exactly where that happens. This function is the load-time guard;
+	// it reports, it does not take the process down.
+	if arch.ValueType != gguf.GGUFMetadataValueTypeString {
+		return "", fmt.Errorf(
+			"nemo-speech-cpp: %q has a non-string general.architecture (type %v)", path, arch.ValueType)
+	}
+	return arch.ValueString(), nil
+}
+
+// discoverTTSAssets fills in codecModel and tokenizerDir when they were not set
+// explicitly, by scanning the primary GGUF's own directory.
+//
+// A missing asset is a hard error rather than a warning: the runtime would
+// otherwise load and emit garbage audio, which surfaces far from the cause.
+func discoverTTSAssets(primaryGGUF string, o *loadOptions) error {
+	dir := filepath.Dir(primaryGGUF)
+
+	if o.codecModel == "" {
+		entries, err := os.ReadDir(dir)
+		if err != nil {
+			return fmt.Errorf("nemo-speech-cpp: scan %q for a codec model: %w", dir, err)
+		}
+		for _, e := range entries {
+			if e.IsDir() {
+				continue
+			}
+			name := e.Name()
+			candidate := filepath.Join(dir, name)
+			// Skip the primary model itself: a file called nanocodec-magpie.gguf
+			// would otherwise be selected as its own codec. Compare basenames,
+			// because candidate is Cleaned by filepath.Join while primaryGGUF
+			// arrives as the caller wrote it, so "/models//magpie.gguf" would
+			// slip past a whole-path equality.
+			if name == filepath.Base(primaryGGUF) {
+				continue
+			}
+			if strings.Contains(strings.ToLower(name), "nanocodec") ||
+				strings.Contains(strings.ToLower(name), "nano-codec") {
+				o.codecModel = candidate
+				break
+			}
+		}
+	}
+	if o.codecModel == "" {
+		return fmt.Errorf(
+			"nemo-speech-cpp: no NanoCodec GGUF found next to %q, set the codec_model option",
+			primaryGGUF)
+	}
+
+	if o.tokenizerDir == "" {
+		candidate := filepath.Join(dir, "extracted")
+		if st, err := os.Stat(candidate); err == nil && st.IsDir() {
+			o.tokenizerDir = candidate
+		}
+	}
+	if o.tokenizerDir == "" {
+		return fmt.Errorf(
+			"nemo-speech-cpp: no tokenizer directory found next to %q, set the tokenizer_dir option",
+			primaryGGUF)
+	}
+	return nil
+}

backend/go/nemo-speech-cpp/family_test.go

@@ -0,0 +1,168 @@
+package main
+
+import (
+	"encoding/binary"
+	"os"
+	"path/filepath"
+
+	gguf "github.com/gpustack/gguf-parser-go"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("familyFor", func() {
+	It("maps the NeMo architectures to their families", func() {
+		for arch, want := range map[string]family{
+			"asr":        familyASR,
+			"sortformer": familyDiarization,
+			"magpietts":  familyTTS,
+		} {
+			got, err := familyFor(arch)
+			Expect(err).ToNot(HaveOccurred(), "arch %q", arch)
+			Expect(got).To(Equal(want), "arch %q", arch)
+		}
+	})
+
+	It("treats an unknown architecture as NMT", func() {
+		// NMT GGUFs are produced by llama.cpp's converter, so they carry an LLM
+		// architecture such as qwen3 rather than a NeMo-specific string.
+		got, err := familyFor("qwen3")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal(familyNMT))
+	})
+
+	It("rejects an auxiliary-only architecture as a primary model", func() {
+		for _, arch := range []string{"nemo-nano-codec", "vad", "pnc"} {
+			_, err := familyFor(arch)
+			Expect(err).To(HaveOccurred(), "arch %q", arch)
+			Expect(err.Error()).To(ContainSubstring(arch))
+		}
+	})
+})
+
+// ggufWithArchValue builds a minimal GGUF v3 carrying general.architecture as
+// its single metadata entry, with the caller's value type and encoded value.
+func ggufWithArchValue(valueType gguf.GGUFMetadataValueType, value []byte) []byte {
+	const key = "general.architecture"
+
+	var b []byte
+	b = append(b, 'G', 'G', 'U', 'F')
+	b = binary.LittleEndian.AppendUint32(b, 3) // version
+	b = binary.LittleEndian.AppendUint64(b, 0) // tensor count
+	b = binary.LittleEndian.AppendUint64(b, 1) // metadata kv count
+	b = binary.LittleEndian.AppendUint64(b, uint64(len(key)))
+	b = append(b, key...)
+	b = binary.LittleEndian.AppendUint32(b, uint32(valueType))
+	return append(b, value...)
+}
+
+// writeGGUFWithUint32Arch writes a minimal GGUF v3 whose single metadata entry
+// is general.architecture typed UINT32 rather than STRING. Handwritten and
+// half-converted files really do carry mistyped keys, and the parser hands them
+// back rather than rejecting them.
+func writeGGUFWithUint32Arch(path string) {
+	b := ggufWithArchValue(gguf.GGUFMetadataValueTypeUint32, binary.LittleEndian.AppendUint32(nil, 7))
+	ExpectWithOffset(1, os.WriteFile(path, b, 0o600)).To(Succeed())
+}
+
+// writeGGUFWithArch writes a minimal GGUF v3 that parses cleanly and reports
+// arch as its general.architecture. It is the only way to reach the code past
+// ggufArchitecture in a test, since there are no real NeMo GGUFs to point at.
+func writeGGUFWithArch(path, arch string) {
+	v := binary.LittleEndian.AppendUint64(nil, uint64(len(arch)))
+	v = append(v, arch...)
+	ExpectWithOffset(1, os.WriteFile(path, ggufWithArchValue(gguf.GGUFMetadataValueTypeString, v), 0o600)).To(Succeed())
+}
+
+var _ = Describe("ggufArchitecture", func() {
+	It("returns an error rather than panicking on a file that is not a GGUF", func() {
+		p := filepath.Join(GinkgoT().TempDir(), "not-a-model.gguf")
+		Expect(os.WriteFile(p, []byte("definitely not a gguf header"), 0o600)).To(Succeed())
+
+		_, err := ggufArchitecture(p)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring(p))
+	})
+
+	It("returns an error rather than panicking when general.architecture is not a string", func() {
+		p := filepath.Join(GinkgoT().TempDir(), "mistyped-arch.gguf")
+		writeGGUFWithUint32Arch(p)
+
+		arch, err := ggufArchitecture(p)
+		Expect(err).To(HaveOccurred())
+		Expect(arch).To(BeEmpty())
+		Expect(err.Error()).To(ContainSubstring("general.architecture"))
+	})
+})
+
+var _ = Describe("discoverTTSAssets", func() {
+	var dir string
+
+	BeforeEach(func() {
+		dir = GinkgoT().TempDir()
+	})
+
+	write := func(name string) string {
+		p := filepath.Join(dir, name)
+		Expect(os.WriteFile(p, []byte("x"), 0o600)).To(Succeed())
+		return p
+	}
+
+	It("finds a sibling nanocodec gguf and extracted dir", func() {
+		primary := write("magpie.f16.gguf")
+		codec := write("nemo-nano-codec-22khz.f16.gguf")
+		Expect(os.Mkdir(filepath.Join(dir, "extracted"), 0o755)).To(Succeed())
+
+		o := loadOptions{}
+		Expect(discoverTTSAssets(primary, &o)).To(Succeed())
+		Expect(o.codecModel).To(Equal(codec))
+		Expect(o.tokenizerDir).To(Equal(filepath.Join(dir, "extracted")))
+	})
+
+	It("never selects the primary gguf as its own codec", func() {
+		// A file named so it would match a naive *.gguf scan.
+		primary := write("nanocodec-magpie.gguf")
+		Expect(os.Mkdir(filepath.Join(dir, "extracted"), 0o755)).To(Succeed())
+
+		o := loadOptions{}
+		err := discoverTTSAssets(primary, &o)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("codec_model"))
+	})
+
+	It("never selects the primary gguf as its own codec through an uncleaned path", func() {
+		// LocalAI joins the model directory and the model name itself, so a
+		// trailing separator on ModelPath produces a doubled slash here. The
+		// self-codec guard has to survive that.
+		write("nanocodec-magpie.gguf")
+		primary := dir + "//nanocodec-magpie.gguf"
+		Expect(os.Mkdir(filepath.Join(dir, "extracted"), 0o755)).To(Succeed())
+
+		o := loadOptions{}
+		err := discoverTTSAssets(primary, &o)
+		Expect(err).To(HaveOccurred())
+		Expect(o.codecModel).To(BeEmpty())
+		Expect(err.Error()).To(ContainSubstring("codec_model"))
+	})
+
+	It("does not overwrite explicitly configured paths", func() {
+		primary := write("magpie.f16.gguf")
+		write("nemo-nano-codec.gguf")
+		Expect(os.Mkdir(filepath.Join(dir, "extracted"), 0o755)).To(Succeed())
+
+		o := loadOptions{codecModel: "/explicit/codec.gguf", tokenizerDir: "/explicit/tok"}
+		Expect(discoverTTSAssets(primary, &o)).To(Succeed())
+		Expect(o.codecModel).To(Equal("/explicit/codec.gguf"))
+		Expect(o.tokenizerDir).To(Equal("/explicit/tok"))
+	})
+
+	It("names the missing option key when the tokenizer dir cannot be found", func() {
+		primary := write("magpie.f16.gguf")
+		write("nemo-nano-codec.gguf")
+
+		o := loadOptions{}
+		err := discoverTTSAssets(primary, &o)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("tokenizer_dir"))
+	})
+})

backend/go/nemo-speech-cpp/main.go

@@ -0,0 +1,76 @@
+package main
+
+// Started internally by LocalAI, one gRPC server per loaded model.
+//
+// Binds NVIDIA NeMo-Speech.cpp through purego. The runtime splits its C ABI
+// across three shared objects: asr (which also exports the diarization
+// symbols), tts, and nmt. Library names can be overridden with
+// NEMO_SPEECH_ASR_LIBRARY / _TTS_LIBRARY / _NMT_LIBRARY, mirroring the
+// PARAKEET_LIBRARY convention in the sibling backends.
+//
+// The naming is asymmetric on purpose: upstream links a dedicated
+// libnemo_speech_asr_c / libnemo_speech_nmt_c around a private C++ core, but
+// compiles the TTS c_api straight into libnemo_speech_tts and only aliases the
+// nemo_speech_tts_c CMake target, so there is no libnemo_speech_tts_c on disk.
+import (
+	"flag"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/ebitengine/purego"
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var addr = flag.String("addr", "localhost:50051", "the address to connect to")
+
+// libSuffix is the platform's shared-object extension.
+func libSuffix() string {
+	if runtime.GOOS == "darwin" {
+		return ".dylib"
+	}
+	return ".so"
+}
+
+// libraryName resolves an override env var, falling back to the platform name.
+func libraryName(envVar, base string) string {
+	if v := os.Getenv(envVar); v != "" {
+		return v
+	}
+	return base + libSuffix()
+}
+
+func main() {
+	flag.Parse()
+
+	if err := openLibraries(); err != nil {
+		panic(err)
+	}
+
+	if err := grpc.StartServer(*addr, &NemoSpeech{}); err != nil {
+		panic(err)
+	}
+}
+
+// openLibraries dlopens the three C ABI shared objects. All three are opened
+// eagerly so a packaging mistake fails at startup with a clear message rather
+// than at first inference of one particular family.
+func openLibraries() error {
+	for _, l := range []struct {
+		env  string
+		base string
+		dst  *uintptr
+	}{
+		{"NEMO_SPEECH_ASR_LIBRARY", "libnemo_speech_asr_c", &asrLib},
+		{"NEMO_SPEECH_TTS_LIBRARY", "libnemo_speech_tts", &ttsLib},
+		{"NEMO_SPEECH_NMT_LIBRARY", "libnemo_speech_nmt_c", &nmtLib},
+	} {
+		name := libraryName(l.env, l.base)
+		h, err := purego.Dlopen(name, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+		if err != nil {
+			return fmt.Errorf("nemo-speech-cpp: dlopen %q: %w", name, err)
+		}
+		*l.dst = h
+	}
+	return registerSymbols()
+}

backend/go/nemo-speech-cpp/nemospeech.go

@@ -0,0 +1,273 @@
+package main
+
+import (
+	"errors"
+	"fmt"
+	"runtime"
+	"sync"
+	"unsafe"
+
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// family is the model family selected at load time from the GGUF architecture.
+type family int
+
+const (
+	familyUnknown family = iota
+	familyASR
+	familyDiarization
+	familyTTS
+	familyNMT
+)
+
+func (f family) String() string {
+	switch f {
+	case familyASR:
+		return "asr"
+	case familyDiarization:
+		return "diarization"
+	case familyTTS:
+		return "tts"
+	case familyNMT:
+		return "nmt"
+	}
+	return "unknown"
+}
+
+// NemoSpeech is one loaded model. Exactly one of the handles is non-zero,
+// matching fam.
+type NemoSpeech struct {
+	base.SingleThread
+
+	fam  family
+	opts loadOptions
+
+	// engineMu guards fam and the handles, and serializes calls into the C
+	// runtime for this model. Its participants today are withEngine and Free;
+	// the per-family RPCs in Tasks 6 to 9 join it by routing through withEngine.
+	engineMu sync.Mutex
+
+	// synth and nmt are shortened rather than spelled out: synthesizer and
+	// translator are the names of the two RPC-side interfaces those handles are
+	// wrapped in (tts.go, nmt.go), and a field sharing a name with an interface in
+	// the same package makes every construction site read as a conversion.
+	recognizer uintptr
+	diarizer   uintptr
+	synth      uintptr
+	nmt        uintptr
+}
+
+// cstr allocates a NUL-terminated C string and returns its pointer plus a
+// release function. The empty string maps to a null pointer because the C API
+// treats NULL and "" as equivalent for every optional field.
+//
+// The address leaves the Go type system as a uintptr, which the collector does
+// not trace, so the bytes are pinned for as long as C may read them. Pinning is
+// the only mechanism with a documented guarantee here: the config structs hold
+// raw addresses, and an unpinned Go allocation is free to be collected (and, in
+// principle, moved) the moment its last traced reference dies.
+//
+// The returned pointer is for C only, and the direction is one-way. Converting
+// it back to an unsafe.Pointer to read the bytes from Go is checked by checkptr
+// (which -race turns on) and kills the process with
+//
+//	fatal error: checkptr: pointer arithmetic result points to invalid allocation
+//
+// as soon as the address lands inside a Go allocation, which is exactly what
+// this produces. C reading it is fine because C is not instrumented; Go reading
+// it back is not.
+//
+// The caller MUST defer the release function immediately, in the same statement
+// that takes the pointer. Dropping it leaks the pin, which the runtime reports
+// at the next collection as:
+//
+//	runtime.Pinner: found leaking pinned pointer; forgot to call Unpin()?
+//
+// That is loud and wrong-looking on purpose: the alternative failure mode is C
+// reading freed memory, which shows up as rare corruption with no trace back
+// to here.
+func cstr(s string) (uintptr, func()) {
+	if s == "" {
+		return 0, func() {}
+	}
+	b := append([]byte(s), 0)
+	pin := new(runtime.Pinner)
+	pin.Pin(&b[0])
+	// #nosec G103 -- b is non-empty (s != "" above) and &b[0] is pinned on the
+	// previous line, so the address C receives cannot be collected or moved
+	// until the returned release runs. One-way by construction: the doc comment
+	// above forbids converting this uintptr back, which is what keeps checkptr
+	// (and therefore -race) out of it.
+	return uintptr(unsafe.Pointer(&b[0])), func() {
+		if pin == nil {
+			return
+		}
+		pin.Unpin()
+		pin = nil
+	}
+}
+
+// There is deliberately no inverse of cstr in this package. Every C entry point
+// that returns a string is bound in abi.go with a Go `string` return, which
+// purego converts from the char* itself, so a hand-rolled reader would have no
+// production caller and would exist only as an unsafe helper waiting to be
+// pointed at the wrong kind of address. Reach for purego's conversion instead;
+// if a future symbol genuinely needs the raw char* (to tell NULL from ""), bind
+// it as uintptr at that call site, where the ownership can be reasoned about.
+
+// requireFamily gates an RPC on the family selected at load time. Returning
+// Unimplemented rather than a nil dereference means a misconfigured model YAML
+// produces a message a user can act on.
+//
+// Callers must already hold engineMu: Free writes n.fam under it, so an
+// unlocked read here is a data race. Use withEngine rather than calling this
+// directly.
+func (n *NemoSpeech) requireFamily(want family) error {
+	if n.fam != want {
+		return status.Errorf(codes.Unimplemented,
+			"nemo-speech-cpp: this model was loaded as %s, not %s", n.fam, want)
+	}
+	return nil
+}
+
+// withEngine runs fn holding engineMu, having first checked the family.
+//
+// Every RPC must go through this rather than calling requireFamily on its own.
+// pkg/grpc/server.go takes the backend lock around each RPC but calls Free
+// without it, so a teardown can land mid-request. Checking the family and then
+// making the C calls that trust it under two separate acquisitions leaves a
+// window in which Free destroys the handle, and the request goes on to use a
+// zeroed one.
+func (n *NemoSpeech) withEngine(want family, fn func() error) error {
+	n.engineMu.Lock()
+	defer n.engineMu.Unlock()
+
+	if err := n.requireFamily(want); err != nil {
+		return err
+	}
+	return fn()
+}
+
+func (n *NemoSpeech) Load(opts *pb.ModelOptions) error {
+	modelFile := opts.GetModelFile()
+	if modelFile == "" {
+		return errors.New("nemo-speech-cpp: ModelFile is required")
+	}
+
+	// Free writes fam and the handles under engineMu and runs without the
+	// backend lock that serialises the RPCs (pkg/grpc/server.go), so the
+	// load-side writes to those same fields need the same protection: without
+	// it this is the write-side half of the race withEngine closed on the read
+	// side. n.opts is in here too, since the loaders read it.
+	//
+	// The loaders called below must NOT take engineMu themselves; sync.Mutex is
+	// not reentrant and this is why.
+	n.engineMu.Lock()
+	defer n.engineMu.Unlock()
+
+	n.opts = parseOptions(opts.GetOptions(), opts.GetModelPath())
+
+	arch, err := ggufArchitecture(modelFile)
+	if err != nil {
+		return err
+	}
+	fam, err := familyFor(arch)
+	if err != nil {
+		return err
+	}
+	xlog.Info("nemo-speech-cpp: loading model", "arch", arch, "family", fam.String())
+
+	// fam is committed only once the family-specific loader has succeeded.
+	// requireFamily is the gate every RPC goes through, so a half-loaded model
+	// that kept its family would route requests at a handle that was never
+	// created.
+	switch fam {
+	case familyASR:
+		err = n.loadASR(modelFile)
+	case familyDiarization:
+		err = n.loadDiarizer(modelFile)
+	case familyTTS:
+		if err = discoverTTSAssets(modelFile, &n.opts); err == nil {
+			err = n.loadTTS(modelFile)
+		}
+	case familyNMT:
+		err = n.loadNMT(modelFile)
+	default:
+		err = fmt.Errorf("nemo-speech-cpp: unhandled family for architecture %q", arch)
+	}
+	if err != nil {
+		return err
+	}
+	n.fam = fam
+	return nil
+}
+
+// Free destroys the runtime handle created at load time.
+//
+// base.SingleThread.Free is a no-op that derived backends are expected to
+// override, and every family here owns C memory that only its own destroy
+// entry point can release, so without this an unloaded model leaks a whole
+// acoustic model. Clearing fam as well means an RPC that races the unload is
+// refused by the gate rather than handed a dangling handle, but that only holds
+// for callers that took engineMu, which today means callers that went through
+// withEngine.
+func (n *NemoSpeech) Free() error {
+	n.engineMu.Lock()
+	defer n.engineMu.Unlock()
+
+	// Guarded on the handle, not on fam: a load that failed part way through
+	// leaves fam unset, and the destroy functions are nil pointers until
+	// openLibraries has bound them.
+	// Each is tested independently rather than switched on: the one-handle
+	// invariant is an invariant, and if it ever broke, a switch would silently
+	// leak the others.
+	if n.recognizer != 0 {
+		ASRDestroy(n.recognizer)
+		n.recognizer = 0
+	}
+	if n.diarizer != 0 {
+		DiarDestroy(n.diarizer)
+		n.diarizer = 0
+	}
+	if n.synth != 0 {
+		TTSDestroy(n.synth)
+		n.synth = 0
+	}
+	if n.nmt != 0 {
+		NMTDestroy(n.nmt)
+		n.nmt = 0
+	}
+	n.fam = familyUnknown
+	return nil
+}
+
+// The loaders are one per family: loadASR in asr.go, loadDiarizer in diar.go,
+// loadTTS in tts.go and loadNMT in nmt.go. Each populates its config structs
+// from n.opts and stores the handle in the matching field.
+//
+// Locking protocol, in both directions:
+//
+//   - Every RPC must hold engineMu across its family check AND its C calls,
+//     which means wrapping its body in withEngine. Free runs without the
+//     backend lock (pkg/grpc/server.go:1019), so anything that checks the
+//     family and then releases the lock before calling C can have the handle
+//     destroyed underneath it. asr.go's AudioTranscription is the worked
+//     example: even the audio decode sits inside the closure, because the
+//     backend already serialises RPCs through base.SingleThread and so the
+//     wider hold costs nothing.
+//   - A loader must NOT take engineMu. Load holds it across the whole switch,
+//     and sync.Mutex is not reentrant, so locking in a loader deadlocks.
+//
+// One consequence the streaming RPCs have to plan around: a stream whose body
+// is wrapped in withEngine holds engineMu for the WHOLE stream, so Free blocks
+// until the stream ends rather than tearing the handle out from under it. That
+// is the behaviour we want (a half-closed stream over a destroyed recognizer
+// has no good outcome), but it means an unload waits on a client that has
+// stopped sending, so a streaming loop must have its own way out: honour the
+// request context and stop on it, rather than blocking forever on the next
+// chunk.

backend/go/nemo-speech-cpp/nemospeech_suite_test.go

@@ -0,0 +1,13 @@
+package main
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestNemoSpeech(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "nemo-speech-cpp Backend Suite")
+}

backend/go/nemo-speech-cpp/nemospeech_test.go

@@ -0,0 +1,260 @@
+package main
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"runtime"
+	"sync"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+var _ = Describe("requireFamily", func() {
+	It("accepts the loaded family", func() {
+		n := &NemoSpeech{fam: familyASR}
+		Expect(n.requireFamily(familyASR)).To(Succeed())
+	})
+
+	It("rejects a mismatched family with Unimplemented and names both", func() {
+		n := &NemoSpeech{fam: familyTTS}
+		err := n.requireFamily(familyASR)
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(err.Error()).To(ContainSubstring("tts"))
+		Expect(err.Error()).To(ContainSubstring("asr"))
+	})
+
+	It("rejects an unloaded model", func() {
+		n := &NemoSpeech{}
+		err := n.requireFamily(familyASR)
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+	})
+
+	It("rejects every family when the model is unloaded", func() {
+		n := &NemoSpeech{}
+		for _, f := range []family{familyASR, familyDiarization, familyTTS, familyNMT} {
+			Expect(n.requireFamily(f)).To(HaveOccurred(), "family %s must be gated on an unloaded model", f)
+		}
+	})
+})
+
+// The brief's round-trip spec (cstr then a reader) cannot exist: cstr pins a Go
+// allocation, and converting a uintptr back into a pointer to Go memory is a
+// checkptr violation that aborts the process under -race. So cstr is asserted
+// on what is observable without dereferencing its result.
+var _ = Describe("cstr", func() {
+	It("returns a non-null pointer for a non-empty string", func() {
+		p, free := cstr("hello")
+		defer free()
+		Expect(p).ToNot(BeZero())
+	})
+
+	It("returns a null pointer for the empty string", func() {
+		// The C API documents NULL and "" as equivalent for optional fields, and
+		// passing NULL avoids allocating for every unset option.
+		p, free := cstr("")
+		defer free()
+		Expect(p).To(BeZero())
+	})
+
+	// The pin has to hold for the whole create call, which spans at least one
+	// safepoint. A collection must therefore neither move nor invalidate the
+	// address that C was handed.
+	It("keeps the pointer stable across a garbage collection", func() {
+		p, free := cstr("/models/nemo/parakeet.gguf")
+		defer free()
+		before := p
+		runtime.GC()
+		runtime.GC()
+		Expect(p).To(Equal(before))
+	})
+
+	It("survives releasing more than once", func() {
+		_, free := cstr("twice")
+		free()
+		Expect(free).ToNot(Panic())
+	})
+
+	// A dropped release leaks the pin, and the runtime turns that into a process
+	// abort at some later collection. Nothing can catch it, so this only pins the
+	// contract in prose: release in the same statement that takes the pointer.
+	It("releases without panicking when used as documented", func() {
+		Expect(func() {
+			p, free := cstr("released")
+			defer free()
+			_ = p
+		}).ToNot(Panic())
+	})
+})
+
+// pkg/grpc/server.go:1019 calls Free without taking the backend lock every
+// other RPC holds, so a teardown really can land while a request is in flight.
+// The family check and the C calls that trust it therefore have to happen under
+// engineMu together, or Free can destroy the handle in the gap between them.
+var _ = Describe("engine locking", func() {
+	It("serialises a teardown against an in-flight request", func() {
+		n := &NemoSpeech{fam: familyASR}
+
+		var wg sync.WaitGroup
+		wg.Add(2)
+		go func() {
+			defer GinkgoRecover()
+			defer wg.Done()
+			for i := 0; i < 2000; i++ {
+				// Errors are expected once the teardown wins the race; what must
+				// not happen is an unsynchronised read of the family.
+				_ = n.withEngine(familyASR, func() error { return nil })
+			}
+		}()
+		go func() {
+			defer GinkgoRecover()
+			defer wg.Done()
+			for i := 0; i < 2000; i++ {
+				Expect(n.Free()).To(Succeed())
+			}
+		}()
+		wg.Wait()
+	})
+
+	It("refuses the body when the family does not match, and still unlocks", func() {
+		n := &NemoSpeech{fam: familyTTS}
+		called := false
+		err := n.withEngine(familyASR, func() error {
+			called = true
+			return nil
+		})
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(called).To(BeFalse())
+
+		// A lock leaked on the rejection path would deadlock the next request
+		// rather than fail it, so prove the mutex is free afterwards.
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+
+	It("propagates the body's error and still unlocks", func() {
+		n := &NemoSpeech{fam: familyASR}
+		boom := errors.New("boom")
+		Expect(n.withEngine(familyASR, func() error { return boom })).To(MatchError(boom))
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+})
+
+var _ = Describe("Free", func() {
+	// The destroy entry points are nil function values until openLibraries has
+	// bound them, so an unloaded model must not reach them. LocalAI frees every
+	// backend it shuts down, including one whose Load failed.
+	It("is a no-op on a model that was never loaded", func() {
+		n := &NemoSpeech{}
+		Expect(n.Free()).To(Succeed())
+	})
+
+	It("is idempotent", func() {
+		n := &NemoSpeech{}
+		Expect(n.Free()).To(Succeed())
+		Expect(n.Free()).To(Succeed())
+	})
+
+	It("does not reach the runtime for a load that failed part way through", func() {
+		n := &NemoSpeech{}
+		path := filepath.Join(GinkgoT().TempDir(), "broken.gguf")
+		Expect(os.WriteFile(path, []byte("broken"), 0o600)).To(Succeed())
+		Expect(n.Load(&pb.ModelOptions{ModelFile: path})).ToNot(Succeed())
+		Expect(n.Free()).To(Succeed())
+	})
+})
+
+var _ = Describe("Load", func() {
+	It("rejects an empty model file", func() {
+		n := &NemoSpeech{}
+		err := n.Load(&pb.ModelOptions{})
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("ModelFile"))
+	})
+
+	It("reports a model file that does not exist", func() {
+		n := &NemoSpeech{}
+		missing := filepath.Join(GinkgoT().TempDir(), "absent.gguf")
+		err := n.Load(&pb.ModelOptions{ModelFile: missing})
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("absent.gguf"))
+	})
+
+	It("reports a file that is not a GGUF", func() {
+		n := &NemoSpeech{}
+		path := filepath.Join(GinkgoT().TempDir(), "notagguf.gguf")
+		Expect(os.WriteFile(path, []byte("this is not a gguf file at all"), 0o600)).To(Succeed())
+		err := n.Load(&pb.ModelOptions{ModelFile: path})
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("nemo-speech-cpp"))
+	})
+
+	// A failed load must not leave a family selected, or the RPC gate would wave
+	// requests through to a nil handle.
+	It("leaves no family selected when the load fails", func() {
+		n := &NemoSpeech{}
+		path := filepath.Join(GinkgoT().TempDir(), "broken.gguf")
+		Expect(os.WriteFile(path, []byte("broken"), 0o600)).To(Succeed())
+		Expect(n.Load(&pb.ModelOptions{ModelFile: path})).ToNot(Succeed())
+		Expect(n.fam).To(Equal(familyUnknown))
+		Expect(n.requireFamily(familyASR)).To(HaveOccurred())
+	})
+
+	// The load path picks a family and only then runs that family's loader, so
+	// there is a window where the family is known and the load still fails.
+	// Committing n.fam before the loader runs would leave the RPC gate open on a
+	// handle that was never created, and pkg/grpc/server.go keeps serving the
+	// instance after a failed LoadModel, so the next request really would reach
+	// it. TTS is the only family whose loader can fail before touching C.
+	It("does not select the family until that family's loader has succeeded", func() {
+		dir := GinkgoT().TempDir()
+		path := filepath.Join(dir, "magpie.f16.gguf")
+		writeGGUFWithArch(path, "magpietts")
+
+		// Self-guard: if the handwritten GGUF ever stops parsing, Load would fail
+		// at ggufArchitecture instead, before a family is ever chosen, and the
+		// assertions below would pass without exercising the ordering at all.
+		Expect(ggufArchitecture(path)).To(Equal("magpietts"))
+
+		// No sibling codec in the directory, so discoverTTSAssets fails after
+		// familyFor has already resolved familyTTS.
+		n := &NemoSpeech{}
+		err := n.Load(&pb.ModelOptions{ModelFile: path})
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("codec_model"))
+
+		Expect(n.fam).To(Equal(familyUnknown))
+		Expect(n.requireFamily(familyTTS)).To(HaveOccurred())
+	})
+
+	It("closes the family gate again after a free", func() {
+		n := &NemoSpeech{fam: familyASR}
+		Expect(n.Free()).To(Succeed())
+		Expect(n.fam).To(Equal(familyUnknown))
+		Expect(n.requireFamily(familyASR)).To(HaveOccurred())
+	})
+
+	It("parses the model options before it touches the model file", func() {
+		// The options are what tell a TTS load where its codec lives, so they have
+		// to be in place before any family-specific loader runs.
+		n := &NemoSpeech{}
+		path := filepath.Join(GinkgoT().TempDir(), "broken.gguf")
+		Expect(os.WriteFile(path, []byte("broken"), 0o600)).To(Succeed())
+		Expect(n.Load(&pb.ModelOptions{
+			ModelFile: path,
+			ModelPath: "/models",
+			Options:   []string{"gpu:2", "codec_model:codec.gguf"},
+		})).ToNot(Succeed())
+		Expect(n.opts.gpu).To(Equal(int32(2)))
+		Expect(n.opts.codecModel).To(Equal("/models/codec.gguf"))
+	})
+})

backend/go/nemo-speech-cpp/nmt.go

@@ -0,0 +1,368 @@
+package main
+
+import (
+	"regexp"
+	"runtime"
+	"strings"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// pairDirective matches a leading "[src->tgt] " override.
+//
+// Each side is an unbounded run of two-letter segments, not one or two of them.
+// Either side may also be omitted, which keeps the model-level default for it:
+// resolve_tag accepts a READY pair tag in one field with the other empty
+// (src/nmt/langpairs.cc:167-172), so "[->en-de]" names a pair for one request.
+//
+// The two rules together are what force the unbounded run. A regional code on
+// its own is only two segments (pt-br, zh-cn, es-us) and would parse under a
+// stricter pattern; it is the SINGLE-FIELD form of a regional pair that runs to
+// three (en-zh-cn, en-zh-tw, en-es-us, en-pt-br, pt-br-en, zh-tw-en). And the
+// failure is not a mis-split: a pattern too short to cover the tag does not
+// match the directive at all, so the whole bracket survives into the text and
+// is handed to the model as something to translate.
+//
+// The codes are not normalised or validated here. normalize_language_code
+// lowercases and folds BCP-47 down to a supported base, and is_supported has the
+// authoritative table; duplicating either would be a second source of truth that
+// drifts on the next pin bump.
+var pairDirective = regexp.MustCompile(`^\[\s*([a-zA-Z]{2}(?:-[a-zA-Z]{2})*)?\s*->\s*([a-zA-Z]{2}(?:-[a-zA-Z]{2})*)?\s*\]\s*`)
+
+// translator is the NMT half of the C API, narrowed to what Predict uses.
+//
+// It is an interface for the same reason synthesizer and diarStream are: no
+// Riva-Translate GGUF is small enough to keep in the tree, so the layer above
+// the ABI (pair resolution, validation, the text array, the single-chunk stream)
+// would otherwise have no test at all. A fake here scripts what the C API
+// returns; it does not pretend to translate anything.
+type translator interface {
+	// translate returns one translation per input text, in order.
+	translate(texts []string, source, target string) ([]string, error)
+}
+
+// cTranslator is the real translator, over one nemo_speech_nmt_translator.
+type cTranslator struct {
+	handle uintptr
+}
+
+// nmtTexts builds the `const char* const* texts` argument and returns it with
+// the release the caller MUST defer.
+//
+// Two levels need pinning, not one. cstr pins each string's bytes, but the array
+// carrying their addresses is a separate Go allocation holding uintptrs: the
+// collector neither traces through it nor is obliged to leave it where it is,
+// and C dereferences it for the whole call. Pinning only the strings would leave
+// the array itself free to move out from under the runtime.
+//
+// An empty element is refused rather than passed on. cstr maps "" to NULL and
+// src/nmt/c_api.cpp maps a NULL element back to "" (str_or_empty), so a blank
+// text would come back as a confident translation of nothing rather than an
+// error.
+func nmtTexts(texts []string) ([]uintptr, func(), error) {
+	pin := new(runtime.Pinner)
+	// The pin is released first so that the array stops being pinned before the
+	// strings it points at do.
+	frees := []func(){pin.Unpin}
+	release := func() {
+		for _, f := range frees {
+			f()
+		}
+	}
+
+	if len(texts) == 0 {
+		return nil, release, status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: nothing to translate")
+	}
+
+	ptrs := make([]uintptr, len(texts))
+	for i, t := range texts {
+		if t == "" {
+			return nil, release, status.Error(codes.InvalidArgument,
+				"nemo-speech-cpp: nothing to translate")
+		}
+		p, free := cstr(t)
+		frees = append(frees, free)
+		ptrs[i] = p
+	}
+	pin.Pin(&ptrs[0])
+	return ptrs, release, nil
+}
+
+func (t *cTranslator) translate(texts []string, source, target string) ([]string, error) {
+	ptrs, release, err := nmtTexts(texts)
+	if err != nil {
+		release()
+		return nil, err
+	}
+	defer release()
+
+	// source and target cross as Go strings: purego NUL-terminates and copies
+	// them itself for the duration of the call, and c_api.cpp deep-copies both
+	// into std::string before doing anything with them.
+	var result uintptr
+	if st := NMTTranslate(t.handle, &ptrs[0], uint64(len(ptrs)), source, target, &result); st != 0 {
+		// An unsupported language pair arrives here as INVALID_ARGUMENT
+		// (src/nmt/translator.cpp throws std::invalid_argument, which
+		// src/nmt/c_api.cpp's guard maps to it), which statusErrorf turns into
+		// the caller-facing code rather than Internal.
+		return nil, statusErrorf(st, "nemo-speech-cpp: translate: %s", NMTLastError())
+	}
+	defer NMTResultDestroy(result)
+
+	count := NMTResultCount(result)
+	out := make([]string, 0, count)
+	for i := uint64(0); i < count; i++ {
+		out = append(out, NMTResultText(result, i))
+	}
+	return out, nil
+}
+
+// nmtTranslatorConfig builds the create-time config.
+//
+// Extracted from loadNMT so its four adjacent pointer fields can be asserted
+// against distinct sentinels. Backend, Model, Generation and Pool are all
+// uintptr and all sit next to each other, so transposing two of them changes
+// neither the struct's size nor any field's offset: the layout assertions in
+// abi_test.go are blind to it, and what it produces at runtime is the backend
+// config being read as the model config.
+//
+// Generation and Pool stay NULL, which nmt.h documents as "library defaults":
+// max_new_tokens (256) and contexts (1) are create-time settings this backend
+// has no option to fill them from, and PredictOptions carries no per-request
+// equivalent that a create-time config could honour anyway.
+//
+// backend and model are pinned addresses, not Go pointers, and the caller owns
+// the pins.
+func nmtTranslatorConfig(backend, model uintptr) cNMTTranslatorConfig {
+	return cNMTTranslatorConfig{
+		Size:    unsafe.Sizeof(cNMTTranslatorConfig{}),
+		Backend: backend,
+		Model:   model,
+	}
+}
+
+// loadNMT creates the Riva-Translate translator.
+//
+// This must not take engineMu: Load is its only caller and already holds it.
+func (n *NemoSpeech) loadNMT(modelFile string) error {
+	// nemo_speech_nmt_create deep-copies the path into a std::string
+	// (src/nmt/c_api.cpp to_config, via str_or_empty) and retains no pointer
+	// afterwards, so pinning for the duration of the create call is both
+	// necessary and sufficient.
+	var pinner runtime.Pinner
+	defer pinner.Unpin()
+
+	pathP, freePath := cstr(modelFile)
+	defer freePath()
+
+	// NCtx is left at 0, which to_config reads as "keep the default" (it applies
+	// the field only when > 0) and which the runtime resolves to 1024 tokens.
+	// That is sized for the sentence-length input Riva-Translate is built for,
+	// and raising it costs one n_ctx-sized KV cache per pooled context, so it
+	// wants a deliberate option rather than a guess made here.
+	model := cNMTModelConfig{Size: unsafe.Sizeof(cNMTModelConfig{}), Path: pathP}
+	// BackendConfig.gpu defaults to 0 in C++ (device 0), not to CPU, and
+	// to_config assigns it unconditionally, so the option's own -1 default is
+	// what keeps an unconfigured model on the CPU.
+	backend := cNMTBackendConfig{Size: unsafe.Sizeof(cNMTBackendConfig{}), GPU: n.opts.gpu}
+
+	cfg := nmtTranslatorConfig(pinPtr(&pinner, &backend), pinPtr(&pinner, &model))
+
+	xlog.Info("nemo-speech-cpp: creating translator",
+		"gpu", n.opts.gpu,
+		"source_language", n.opts.sourceLanguage,
+		"target_language", n.opts.targetLanguage)
+
+	// #nosec G103 -- cfg is a local POD struct borrowed for this call only. Its
+	// Backend and Model members are pinPtr addresses held by the pinner unpinned
+	// on return, Model.Path is the cstr allocation freed by the defer above, and
+	// nemo_speech_nmt_create deep-copies everything it reads.
+	if st := NMTCreate(unsafe.Pointer(&cfg), &n.nmt); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: nmt create: %s", NMTLastError())
+	}
+	return nil
+}
+
+// languagePair resolves the languages for one request and returns the text to
+// translate.
+//
+// nemo_speech_nmt_translate takes explicit source and target languages and has
+// no free-form generation entry point at all, so there is no prompt in the LLM
+// sense to carry an instruction. The pair therefore comes from the model
+// options, and a leading "[src->tgt]" directive is the only per-request control
+// Predict can offer.
+func (n *NemoSpeech) languagePair(prompt string) (source, target, text string) {
+	source, target = n.opts.sourceLanguage, n.opts.targetLanguage
+
+	m := pairDirective.FindStringSubmatch(prompt)
+	if m == nil {
+		return source, target, strings.TrimSpace(prompt)
+	}
+	// An omitted side keeps the model-level default rather than blanking it.
+	if m[1] != "" {
+		source = m[1]
+	}
+	if m[2] != "" {
+		target = m[2]
+	}
+	// The directive must not survive into the text: the runtime wraps it in a
+	// chat template (src/nmt/langpairs.cc build_prompt), so anything left here is
+	// translated along with the sentence.
+	return source, target, strings.TrimSpace(prompt[len(m[0]):])
+}
+
+// unsupportedPredictFields names the PredictOptions fields a caller may have set
+// that this C API has no way to honour, so they are logged rather than silently
+// dropped.
+//
+// The list is deliberately narrow. Everything nemo_speech_nmt_translate accepts
+// is in its five arguments: a translator, the texts, and two language codes.
+// Everything else in PredictOptions is therefore unsupported, and naming all of
+// it would log on every single request, because LocalAI fills the sampling
+// defaults in from the model config whether or not the user asked for them.
+//
+// So the sampling and decoding knobs (temperature, top_p, top_k, min_p, seed,
+// tokens, repeat/frequency/presence penalties, mirostat, tfz, typical_p,
+// stop_prompts, prompt caching, rope scaling, n_draft, logit_bias) are ignored
+// silently: there is no field for any of them on either side of the ABI.
+// max_new_tokens and n_ctx exist but are CREATE-time settings on the translator,
+// not per-request ones, so PredictOptions.Tokens has nowhere to go either.
+//
+// What is named here is the structural asks: requests that only make sense
+// against a general language model, where honouring them partially would be
+// worse than saying nothing at all.
+func unsupportedPredictFields(opts *pb.PredictOptions) []string {
+	var out []string
+	if opts.GetGrammar() != "" {
+		out = append(out, "grammar")
+	}
+	if opts.GetTools() != "" {
+		out = append(out, "tools")
+	}
+	if len(opts.GetImages()) > 0 {
+		out = append(out, "images")
+	}
+	if len(opts.GetVideos()) > 0 {
+		out = append(out, "videos")
+	}
+	if len(opts.GetAudios()) > 0 {
+		out = append(out, "audios")
+	}
+	if opts.GetNegativePrompt() != "" {
+		out = append(out, "negative_prompt")
+	}
+	if opts.GetLogprobs() > 0 {
+		out = append(out, "logprobs")
+	}
+	return out
+}
+
+// translateText runs one translation and returns it.
+//
+// The two rejections happen before anything crosses the ABI. An empty text would
+// otherwise reach the runtime as a NULL element (see nmtTexts), and a missing
+// target would come back as "unsupported language pair:  -> ", which names
+// neither the option the operator has to set nor the request that failed.
+func translateText(t translator, source, target, text string) (string, error) {
+	if text == "" {
+		return "", status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: PredictOptions.prompt is required, it is the text to translate")
+	}
+	if target == "" {
+		return "", status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: no target language: set the target_language model option, "+
+				"or prefix the prompt with a [src->tgt] directive")
+	}
+
+	out, err := t.translate([]string{text}, source, target)
+	if err != nil {
+		return "", err
+	}
+	// One text in, one translation out. A call that returned OK with none is a
+	// runtime bug, and the empty string it would hand back reaches the user as a
+	// successful but blank completion with nothing anywhere to say why.
+	if len(out) == 0 {
+		return "", status.Error(codes.Internal, "nemo-speech-cpp: translation produced no result")
+	}
+	return out[0], nil
+}
+
+// streamTranslation runs one translation and puts the whole of it on out as a
+// single chunk.
+//
+// That is a limit of the C API and not a shortcut taken here.
+// nemo_speech_nmt_translate has no token callback and no incremental result: it
+// returns once the decode has finished, with the completed text. There is
+// nothing finer to stream, and splitting the finished string into fake chunks
+// would imitate progress that never happened.
+//
+// out is not closed here. PredictStream owns it, and closing it in one of two
+// places depending on how far the request got is how a stream ends up
+// half-closed.
+func streamTranslation(t translator, source, target, text string, out chan<- string) error {
+	translated, err := translateText(t, source, target, text)
+	if err != nil {
+		return err
+	}
+	out <- translated
+	return nil
+}
+
+// resolveRequest is the shared front half of both RPCs: it names what it is
+// dropping and works out the pair and the text.
+func (n *NemoSpeech) resolveRequest(opts *pb.PredictOptions) (source, target, text string) {
+	// Logged rather than rejected, for the reason the diarization path logs its
+	// own dropped fields: a caller that asked for something extra still wants the
+	// translation it can have, and a request naming a field this backend drops
+	// should say so where an operator can find it.
+	if dropped := unsupportedPredictFields(opts); len(dropped) > 0 {
+		xlog.Warn("nemo-speech-cpp: ignoring request fields this model has no equivalent for",
+			"fields", dropped)
+	}
+	return n.languagePair(opts.GetPrompt())
+}
+
+// Predict translates PredictOptions.Prompt.
+//
+// The whole body runs inside withEngine, so the family check and the C calls
+// that trust the handle happen under a single acquisition of engineMu. See the
+// handoff notes at the bottom of nemospeech.go: Free runs without the backend
+// lock, so anything that checks the family and then releases the lock before
+// calling C can have the handle destroyed underneath it.
+func (n *NemoSpeech) Predict(opts *pb.PredictOptions) (string, error) {
+	var out string
+	if err := n.withEngine(familyNMT, func() error {
+		source, target, text := n.resolveRequest(opts)
+		s, err := translateText(&cTranslator{handle: n.nmt}, source, target, text)
+		out = s
+		return err
+	}); err != nil {
+		return "", err
+	}
+	return out, nil
+}
+
+// PredictStream translates PredictOptions.Prompt and emits the result on
+// results.
+//
+// results is closed on EVERY path, including the family rejection and a
+// validation failure, and the close is deferred outside withEngine so that a
+// rejected family still closes it. This is the LEGACY streaming contract, which
+// is the opposite of PredictStreamRich's: pkg/grpc/server.go:529 calls this and
+// then blocks on a drain goroutine that only finishes when the channel closes,
+// so a channel left open does not fail the request, it hangs the RPC and, with
+// the backend lock still held, every request queued behind it. The rich variant
+// is the one whose channel the host closes; this one is not.
+func (n *NemoSpeech) PredictStream(opts *pb.PredictOptions, results chan string) error {
+	defer close(results)
+
+	return n.withEngine(familyNMT, func() error {
+		source, target, text := n.resolveRequest(opts)
+		return streamTranslation(&cTranslator{handle: n.nmt}, source, target, text, results)
+	})
+}

backend/go/nemo-speech-cpp/nmt_test.go

@@ -0,0 +1,416 @@
+package main
+
+import (
+	"errors"
+	"unsafe"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// fakeTranslator scripts the C API's answer and records what it was asked, so
+// the layer above the ABI (pair resolution, validation, the single-element text
+// array) has a test at all. No Riva-Translate GGUF is small enough to keep in
+// the tree, and this pretends to translate nothing.
+type fakeTranslator struct {
+	texts          []string
+	source, target string
+	calls          int
+
+	out []string
+	err error
+}
+
+func (f *fakeTranslator) translate(texts []string, source, target string) ([]string, error) {
+	f.calls++
+	f.texts = texts
+	f.source = source
+	f.target = target
+	return f.out, f.err
+}
+
+// collectStrings drains ch until it closes and hands back everything it saw.
+// The host does the same, so a channel this backend forgets to close hangs the
+// RPC rather than failing it.
+func collectStrings(ch chan string) chan []string {
+	done := make(chan []string, 1)
+	go func() {
+		var got []string
+		for s := range ch {
+			got = append(got, s)
+		}
+		done <- got
+	}()
+	return done
+}
+
+var _ = Describe("languagePair", func() {
+	It("uses the configured pair and returns the prompt unchanged", func() {
+		n := &NemoSpeech{opts: loadOptions{sourceLanguage: "en", targetLanguage: "de"}}
+		src, tgt, text := n.languagePair("hello world")
+		Expect(src).To(Equal("en"))
+		Expect(tgt).To(Equal("de"))
+		Expect(text).To(Equal("hello world"))
+	})
+
+	// nemo_speech_nmt_translate takes explicit languages and has no prompt path,
+	// so an inline directive is the only way a caller can pick a pair per request.
+	It("honours an inline pair directive and strips it from the text", func() {
+		n := &NemoSpeech{opts: loadOptions{sourceLanguage: "en", targetLanguage: "de"}}
+		src, tgt, text := n.languagePair("[en->fr] hello world")
+		Expect(src).To(Equal("en"))
+		Expect(tgt).To(Equal("fr"))
+		Expect(text).To(Equal("hello world"))
+	})
+
+	// The directive has to be gone from what reaches the model: the runtime
+	// wraps the text in a chat template (src/nmt/langpairs.cc build_prompt), so
+	// a leftover "[en->fr]" would be translated along with the sentence.
+	It("leaves no trace of the directive in the translated text", func() {
+		n := &NemoSpeech{opts: loadOptions{targetLanguage: "de"}}
+		_, _, text := n.languagePair("[en->fr] hello world")
+		Expect(text).ToNot(ContainSubstring("["))
+		Expect(text).ToNot(ContainSubstring("->"))
+		Expect(text).ToNot(ContainSubstring("fr"))
+		Expect(text).To(Equal("hello world"))
+	})
+
+	It("leaves an unparseable directive in the text", func() {
+		n := &NemoSpeech{opts: loadOptions{sourceLanguage: "en", targetLanguage: "de"}}
+		src, tgt, text := n.languagePair("[not a directive] hi")
+		Expect(src).To(Equal("en"))
+		Expect(tgt).To(Equal("de"))
+		Expect(text).To(Equal("[not a directive] hi"))
+	})
+
+	It("trims surrounding whitespace from the text", func() {
+		n := &NemoSpeech{opts: loadOptions{sourceLanguage: "en", targetLanguage: "de"}}
+		_, _, text := n.languagePair("  hello  ")
+		Expect(text).To(Equal("hello"))
+	})
+
+	// The model's own tags carry region subtags (src/nmt/langpairs.cc: en-zh-cn,
+	// pt-br, es-us), so a directive that only accepted bare two-letter codes
+	// could not name half the pairs the runtime supports.
+	It("accepts a regional code on either side", func() {
+		n := &NemoSpeech{}
+		src, tgt, text := n.languagePair("[pt-br->en] ola")
+		Expect(src).To(Equal("pt-br"))
+		Expect(tgt).To(Equal("en"))
+		Expect(text).To(Equal("ola"))
+
+		src, tgt, _ = n.languagePair("[en->zh-cn] hi")
+		Expect(src).To(Equal("en"))
+		Expect(tgt).To(Equal("zh-cn"))
+	})
+
+	// resolve_tag accepts a ready pair tag in one field with the other empty, so
+	// a directive that names only one side must keep the configured value for the
+	// other rather than blanking it.
+	It("keeps the configured code for a side the directive omits", func() {
+		n := &NemoSpeech{opts: loadOptions{sourceLanguage: "en", targetLanguage: "de"}}
+		src, tgt, text := n.languagePair("[->fr] hello")
+		Expect(src).To(Equal("en"))
+		Expect(tgt).To(Equal("fr"))
+		Expect(text).To(Equal("hello"))
+
+		src, tgt, _ = n.languagePair("[fr->] hello")
+		Expect(src).To(Equal("fr"))
+		Expect(tgt).To(Equal("de"))
+	})
+
+	// resolve_tag (src/nmt/langpairs.cc:167-172) accepts a READY pair tag in one
+	// field with the other empty, and the model's own tags run to three segments
+	// (en-zh-cn, en-zh-tw, en-es-us, en-pt-br). That single-field three-segment
+	// form is the case a two-segment pattern cannot express: it does not merely
+	// mis-split the tag, it fails to match the directive at all, so the whole
+	// bracket survives into the text and is handed to the model as something to
+	// translate.
+	//
+	// Two-segment codes like pt-br and zh-cn are NOT this case; they parse either
+	// way.
+	It("accepts a three-segment pair tag given in one side of the directive", func() {
+		n := &NemoSpeech{opts: loadOptions{targetLanguage: "de"}}
+		src, tgt, text := n.languagePair("[->en-zh-cn] hi")
+		Expect(src).To(BeEmpty())
+		Expect(tgt).To(Equal("en-zh-cn"))
+		Expect(text).To(Equal("hi"))
+
+		src, tgt, text = n.languagePair("[pt-br-en->] hola")
+		Expect(src).To(Equal("pt-br-en"))
+		Expect(tgt).To(Equal("de"))
+		Expect(text).To(Equal("hola"))
+	})
+
+	It("does not treat a bracketed sentence as a directive", func() {
+		n := &NemoSpeech{opts: loadOptions{targetLanguage: "de"}}
+		_, _, text := n.languagePair("[see figure 1] the cat sat")
+		Expect(text).To(Equal("[see figure 1] the cat sat"))
+	})
+})
+
+var _ = Describe("nmtTranslatorConfig", func() {
+	// Backend, Model, Generation and Pool are four adjacent same-typed pointers.
+	// Transposing two of them changes neither the struct's size nor any field's
+	// offset, so the layout assertions in abi_test.go cannot see it, and the
+	// failure it produces is the runtime reading the backend config as the model
+	// config. Distinct sentinels are the only thing that catches it.
+	It("wires each pointer into its own field", func() {
+		cfg := nmtTranslatorConfig(0xB, 0xD)
+		Expect(cfg.Backend).To(Equal(uintptr(0xB)))
+		Expect(cfg.Model).To(Equal(uintptr(0xD)))
+	})
+
+	// NULL is what nmt.h documents as "library defaults" for a subsystem config,
+	// and this backend has no option to fill either of them from.
+	It("leaves the generation and pool configs null", func() {
+		cfg := nmtTranslatorConfig(0xB, 0xD)
+		Expect(cfg.Generation).To(BeZero())
+		Expect(cfg.Pool).To(BeZero())
+	})
+
+	// The runtime decides a field is present with HAS_FIELD, which tests the
+	// caller's size against offsetof + sizeof (src/nmt/c_api.cpp), so a config
+	// sent with Size 0 has every field ignored and the model loads from a path
+	// it was never given.
+	It("declares its own size", func() {
+		Expect(nmtTranslatorConfig(0xB, 0xD).Size).To(Equal(unsafe.Sizeof(cNMTTranslatorConfig{})))
+	})
+})
+
+var _ = Describe("nmtTexts", func() {
+	It("produces one non-null pointer per text", func() {
+		ptrs, release, err := nmtTexts([]string{"one", "two", "three"})
+		Expect(err).ToNot(HaveOccurred())
+		defer release()
+
+		Expect(ptrs).To(HaveLen(3))
+		for i, p := range ptrs {
+			Expect(p).ToNot(BeZero(), "texts[%d] must not be NULL", i)
+		}
+		// Distinct addresses: one buffer reused for every element would make the
+		// runtime translate the last text three times.
+		Expect(ptrs[0]).ToNot(Equal(ptrs[1]))
+		Expect(ptrs[1]).ToNot(Equal(ptrs[2]))
+	})
+
+	// cstr maps "" to NULL and src/nmt/c_api.cpp maps a NULL element back to "",
+	// so a blank text would be answered with a translation of nothing instead of
+	// an error.
+	It("refuses an empty element", func() {
+		_, release, err := nmtTexts([]string{"one", ""})
+		Expect(release).ToNot(BeNil())
+		release()
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	It("refuses an empty batch", func() {
+		_, release, err := nmtTexts(nil)
+		Expect(release).ToNot(BeNil())
+		release()
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	It("survives releasing more than once", func() {
+		_, release, err := nmtTexts([]string{"once"})
+		Expect(err).ToNot(HaveOccurred())
+		release()
+		Expect(release).ToNot(Panic())
+	})
+})
+
+var _ = Describe("translateText", func() {
+	It("passes the resolved pair and the text through to the runtime", func() {
+		f := &fakeTranslator{out: []string{"hallo welt"}}
+		got, err := translateText(f, "en", "de", "hello world")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("hallo welt"))
+		Expect(f.texts).To(Equal([]string{"hello world"}))
+		Expect(f.source).To(Equal("en"))
+		Expect(f.target).To(Equal("de"))
+	})
+
+	// A single-pair model is configured with target_language alone, and
+	// resolve_tag accepts a ready tag in one field with the other empty.
+	It("allows an empty source language", func() {
+		f := &fakeTranslator{out: []string{"ciao"}}
+		_, err := translateText(f, "", "en-it", "hi")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(f.source).To(BeEmpty())
+	})
+
+	It("rejects a missing target language and names the option to set", func() {
+		f := &fakeTranslator{}
+		_, err := translateText(f, "en", "", "hello")
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("target_language"))
+		Expect(f.calls).To(BeZero())
+	})
+
+	It("rejects an empty text without calling the runtime", func() {
+		f := &fakeTranslator{}
+		_, err := translateText(f, "en", "de", "")
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(f.calls).To(BeZero())
+	})
+
+	It("propagates a runtime failure", func() {
+		boom := errors.New("boom")
+		_, err := translateText(&fakeTranslator{err: boom}, "en", "de", "hello")
+		Expect(err).To(MatchError(boom))
+	})
+
+	// A call that returned OK with no translations is a runtime bug, and the
+	// empty string it would hand back reaches the user as a successful but blank
+	// completion with nothing anywhere to say why.
+	It("refuses a result that carries no translation", func() {
+		_, err := translateText(&fakeTranslator{}, "en", "de", "hello")
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+	})
+
+	It("takes the first translation when the runtime returns several", func() {
+		f := &fakeTranslator{out: []string{"first", "second"}}
+		got, err := translateText(f, "en", "de", "hello")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("first"))
+	})
+})
+
+var _ = Describe("Predict", func() {
+	It("refuses a model loaded as another family", func() {
+		n := &NemoSpeech{fam: familyASR}
+		out, err := n.Predict(&pb.PredictOptions{Prompt: "hello"})
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(out).To(BeEmpty())
+
+		// A lock leaked on the rejection path deadlocks the next request rather
+		// than failing it.
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+
+	It("refuses an unloaded model", func() {
+		n := &NemoSpeech{}
+		_, err := n.Predict(&pb.PredictOptions{Prompt: "hello"})
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+	})
+
+	// The validation has to happen before anything crosses the ABI: nothing is
+	// loaded here, so a guard placed after the C call would panic on a nil
+	// function value instead of failing the request.
+	It("rejects an empty prompt before it reaches the runtime", func() {
+		n := &NemoSpeech{fam: familyNMT, opts: loadOptions{targetLanguage: "de"}}
+		var err error
+		Expect(func() {
+			_, err = n.Predict(&pb.PredictOptions{})
+		}).ToNot(Panic())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	It("rejects a request with no target language, before it reaches the runtime", func() {
+		n := &NemoSpeech{fam: familyNMT}
+		var err error
+		Expect(func() {
+			_, err = n.Predict(&pb.PredictOptions{Prompt: "hello"})
+		}).ToNot(Panic())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("target_language"))
+	})
+})
+
+var _ = Describe("PredictStream", func() {
+	// pkg/grpc/server.go drains this channel from a goroutine and then blocks on
+	// that goroutine finishing, so a channel left open does not fail the request,
+	// it hangs the RPC and every request queued behind the backend lock.
+	It("closes the channel when the family does not match", func() {
+		n := &NemoSpeech{fam: familyTTS}
+		ch := make(chan string)
+		done := collectStrings(ch)
+
+		err := n.PredictStream(&pb.PredictOptions{Prompt: "hello"}, ch)
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		Expect(<-done).To(BeEmpty())
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+
+	It("closes the channel when the request is rejected", func() {
+		n := &NemoSpeech{fam: familyNMT}
+		ch := make(chan string)
+		done := collectStrings(ch)
+
+		err := n.PredictStream(&pb.PredictOptions{Prompt: "hello"}, ch)
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(<-done).To(BeEmpty())
+	})
+
+	It("closes the channel on an unloaded model", func() {
+		n := &NemoSpeech{}
+		ch := make(chan string)
+		done := collectStrings(ch)
+
+		Expect(n.PredictStream(&pb.PredictOptions{Prompt: "hi"}, ch)).ToNot(Succeed())
+		Expect(<-done).To(BeEmpty())
+	})
+
+	// The C API has no token callback, so the whole translation is one chunk.
+	// The seam is the only place that can be asserted without a model.
+	It("emits the whole translation as a single chunk", func() {
+		f := &fakeTranslator{out: []string{"hallo welt"}}
+		ch := make(chan string)
+		done := collectStrings(ch)
+
+		Expect(streamTranslation(f, "en", "de", "hello world", ch)).To(Succeed())
+		close(ch)
+		Expect(<-done).To(Equal([]string{"hallo welt"}))
+	})
+
+	It("emits nothing when the translation fails", func() {
+		f := &fakeTranslator{err: errors.New("boom")}
+		ch := make(chan string)
+		done := collectStrings(ch)
+
+		Expect(streamTranslation(f, "en", "de", "hello", ch)).ToNot(Succeed())
+		close(ch)
+		Expect(<-done).To(BeEmpty())
+	})
+})
+
+var _ = Describe("unsupportedPredictFields", func() {
+	It("names nothing for a plain translation request", func() {
+		Expect(unsupportedPredictFields(&pb.PredictOptions{Prompt: "hello"})).To(BeEmpty())
+	})
+
+	// The sampling knobs are deliberately absent from this list: LocalAI fills
+	// them in from the model config on every request, so warning about them
+	// would log on every translation and say nothing.
+	It("stays quiet about sampling parameters the runtime has no field for", func() {
+		Expect(unsupportedPredictFields(&pb.PredictOptions{
+			Prompt:      "hello",
+			Temperature: 0.7,
+			TopP:        0.9,
+			TopK:        40,
+			Seed:        42,
+			Tokens:      256,
+		})).To(BeEmpty())
+	})
+
+	It("names the asks the C API cannot serve at all", func() {
+		got := unsupportedPredictFields(&pb.PredictOptions{
+			Prompt:         "hello",
+			Grammar:        "root ::= x",
+			Tools:          `[{"type":"function"}]`,
+			Images:         []string{"a.png"},
+			Videos:         []string{"a.mp4"},
+			Audios:         []string{"a.wav"},
+			NegativePrompt: "no",
+			Logprobs:       3,
+		})
+		Expect(got).To(ConsistOf("grammar", "tools", "images", "videos", "audios",
+			"negative_prompt", "logprobs"))
+	})
+})

backend/go/nemo-speech-cpp/options.go

@@ -0,0 +1,100 @@
+package main
+
+import (
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"github.com/mudler/xlog"
+)
+
+// loadOptions holds the parsed model-level options. Path fields are resolved
+// against ModelOptions.ModelPath at parse time so every consumer sees an
+// absolute path.
+type loadOptions struct {
+	// ASR
+	vadModel     string
+	pncModel     string
+	diarModel    string
+	itnDir       string
+	languageCode string
+
+	// TTS
+	codecModel   string
+	tokenizerDir string
+	tnDir        string
+
+	// NMT
+	sourceLanguage string
+	targetLanguage string
+
+	// gpu is the device index passed to the runtime's backend config.
+	// -1 selects CPU, matching the C API's own sentinel.
+	gpu int32
+}
+
+// splitOption splits on the FIRST colon so values may themselves contain one.
+func splitOption(o string) (key, value string, ok bool) {
+	i := strings.Index(o, ":")
+	if i < 0 {
+		return "", "", false
+	}
+	return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true
+}
+
+// resolve makes a relative asset path absolute against the models directory.
+// Empty stays empty so callers can test for "unset".
+func resolve(base, p string) string {
+	if p == "" || filepath.IsAbs(p) {
+		return p
+	}
+	return filepath.Join(base, p)
+}
+
+// parseOptions reads the backend "key:value" option slice. Unknown keys are
+// ignored rather than rejected, so a config written for a newer backend still
+// loads on an older one.
+func parseOptions(opts []string, modelPath string) loadOptions {
+	o := loadOptions{gpu: -1}
+	for _, oo := range opts {
+		key, value, ok := splitOption(oo)
+		if !ok {
+			continue
+		}
+		switch key {
+		case "vad_model":
+			o.vadModel = resolve(modelPath, value)
+		case "pnc_model":
+			o.pncModel = resolve(modelPath, value)
+		case "diar_model":
+			o.diarModel = resolve(modelPath, value)
+		case "itn_dir":
+			o.itnDir = resolve(modelPath, value)
+		case "language_code":
+			o.languageCode = value
+		case "codec_model":
+			o.codecModel = resolve(modelPath, value)
+		case "tokenizer_dir":
+			o.tokenizerDir = resolve(modelPath, value)
+		case "tn_dir":
+			o.tnDir = resolve(modelPath, value)
+		case "source_language":
+			o.sourceLanguage = value
+		case "target_language":
+			o.targetLanguage = value
+		case "gpu":
+			// An unknown key is ignored for forward compatibility, but a known key
+			// with an unparseable value is a typo, and this one fails expensively:
+			// the model still loads and still produces correct output, just on CPU
+			// and far slower, with nothing anywhere to say why.
+			n, err := strconv.ParseInt(value, 10, 32)
+			if err != nil {
+				xlog.Warn("nemo-speech-cpp: ignoring unparseable option value, falling back to CPU",
+					"key", key, "value", value)
+				continue
+			}
+			o.gpu = int32(n)
+		}
+	}
+	return o
+}

backend/go/nemo-speech-cpp/options_test.go

@@ -0,0 +1,70 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("parseOptions", func() {
+	It("parses every known key", func() {
+		o := parseOptions([]string{
+			"vad_model:silero.gguf",
+			"pnc_model:pnc.gguf",
+			"diar_model:sortformer.gguf",
+			"itn_dir:tn_configs",
+			"language_code:es-ES",
+			"codec_model:nanocodec.gguf",
+			"tokenizer_dir:extracted",
+			"tn_dir:tn",
+			"source_language:en",
+			"target_language:de",
+		}, "/models")
+
+		Expect(o.vadModel).To(Equal("/models/silero.gguf"))
+		Expect(o.pncModel).To(Equal("/models/pnc.gguf"))
+		Expect(o.diarModel).To(Equal("/models/sortformer.gguf"))
+		Expect(o.itnDir).To(Equal("/models/tn_configs"))
+		Expect(o.languageCode).To(Equal("es-ES"))
+		Expect(o.codecModel).To(Equal("/models/nanocodec.gguf"))
+		Expect(o.tokenizerDir).To(Equal("/models/extracted"))
+		Expect(o.tnDir).To(Equal("/models/tn"))
+		Expect(o.sourceLanguage).To(Equal("en"))
+		Expect(o.targetLanguage).To(Equal("de"))
+	})
+
+	It("leaves absolute paths untouched", func() {
+		o := parseOptions([]string{"vad_model:/abs/silero.gguf"}, "/models")
+		Expect(o.vadModel).To(Equal("/abs/silero.gguf"))
+	})
+
+	It("ignores unknown keys and entries without a separator", func() {
+		o := parseOptions([]string{"nonsense", "unknown_key:value"}, "/models")
+		Expect(o).To(Equal(loadOptions{gpu: -1}))
+	})
+
+	It("trims whitespace around keys and values", func() {
+		o := parseOptions([]string{"  language_code : en-US  "}, "/models")
+		Expect(o.languageCode).To(Equal("en-US"))
+	})
+
+	It("keeps a value containing a colon intact", func() {
+		// URIs must survive the split on the FIRST colon.
+		o := parseOptions([]string{"tokenizer_dir:/a/b:c"}, "/models")
+		Expect(o.tokenizerDir).To(Equal("/a/b:c"))
+	})
+
+	It("leaves an empty value empty so callers can detect unset", func() {
+		o := parseOptions([]string{"vad_model:"}, "/models")
+		Expect(o.vadModel).To(BeEmpty())
+	})
+
+	It("defaults gpu to -1 meaning CPU", func() {
+		o := parseOptions(nil, "/models")
+		Expect(o.gpu).To(Equal(int32(-1)))
+	})
+
+	It("parses an explicit gpu index", func() {
+		o := parseOptions([]string{"gpu:0"}, "/models")
+		Expect(o.gpu).To(Equal(int32(0)))
+	})
+})

backend/go/nemo-speech-cpp/package.sh

@@ -0,0 +1,161 @@
+#!/bin/bash
+#
+# Bundle the nemo-speech-cpp-grpc binary, the five nemo_speech shared objects,
+# the text-normalization stack on a WITH_NORM build, the core runtime libs
+# (libc/libstdc++/libgomp + ld.so) and the GPU runtime for the active BUILD_TYPE
+# so the package is self-contained. Mirrors backend/go/whisper/package.sh;
+# run.sh routes the (CGO_ENABLED=0) binary through lib/ld.so so the packaged
+# libc is used instead of the host's.
+#
+# Five, not three: ASR and NMT each ship a thin _c ABI shim plus the
+# implementation DSO it depends on, while TTS ships one object with no _c
+# suffix at all.
+
+set -e
+
+CURDIR=$(dirname "$(realpath "$0")")
+REPO_ROOT="${CURDIR}/../../.."
+
+mkdir -p "$CURDIR/package/lib"
+
+cp -avf "$CURDIR/nemo-speech-cpp-grpc" "$CURDIR/package/"
+cp -avf "$CURDIR/run.sh" "$CURDIR/package/"
+
+# The runtime ships three C ABI shared objects, not one. All three are
+# required: main.go dlopens them eagerly, so a package missing any of them
+# fails at startup. ASR and NMT expose the ABI through a dedicated _c library;
+# TTS compiles its c_api into libnemo_speech_tts itself and has no _c variant,
+# hence the asymmetric list. purego.Dlopen resolves them via the
+# NEMO_SPEECH_*_LIBRARY paths that run.sh points at lib/.
+#
+# libnemo_speech_asr and libnemo_speech_nmt are in the list because the matching
+# _c shims carry a DT_NEEDED on them: dlopen of the shim fails without the
+# implementation DSO alongside it.
+for lib in libnemo_speech_asr_c libnemo_speech_asr libnemo_speech_tts libnemo_speech_nmt_c libnemo_speech_nmt; do
+	cp -avf "$CURDIR"/${lib}.so* "$CURDIR/package/lib/" 2>/dev/null || true
+	cp -avf "$CURDIR"/${lib}*.dylib "$CURDIR/package/lib/" 2>/dev/null || true
+	if ! ls "$CURDIR"/package/lib/${lib}.* >/dev/null 2>&1; then
+		echo "ERROR: ${lib} shared library not found in $CURDIR, run 'make' first" >&2
+		exit 1
+	fi
+done
+
+# Text normalization (WITH_NORM=ON, Linux only) links Sparrowhawk and OpenFST
+# into libnemo_speech_asr.so. Those live in a project-local prefix that the
+# Makefile stages here, so anything staged that is not a nemo_speech object is
+# part of that stack. Absent on a WITH_NORM=OFF build, which is why this is a
+# glob that tolerates no matches rather than a required list.
+shopt -s nullglob
+for so in "$CURDIR"/*.so "$CURDIR"/*.so.* "$CURDIR"/*.dylib; do
+	case "$(basename "$so")" in
+		libnemo_speech_*) continue ;;
+	esac
+	cp -avf "$so" "$CURDIR/package/lib/"
+done
+shopt -u nullglob
+
+# Detect architecture and copy the core runtime libs the shared objects link
+# against, plus the matching dynamic loader as lib/ld.so.
+source "$CURDIR/../../../scripts/build/package-system-libs.sh" "$CURDIR/package/lib" ""
+
+# Dependency-closure guard.
+#
+# The lists above are maintained by hand, and the WITH_NORM build in particular
+# pulls in transitive dependencies nobody enumerated: Sparrowhawk drags in
+# protobuf, re2 and absl, none of which package-system-libs.sh provides. Rather
+# than hard-code that set, walk the DT_NEEDED entries of everything staged and
+# copy whatever is still unresolved. On a WITH_NORM=OFF build the closure is
+# already complete, so this copies nothing.
+#
+# Skipped deliberately: the core runtime set that package-system-libs.sh owns,
+# and the GPU stack that package-gpu-libs.sh owns.
+shopt -s nullglob
+staged_libs=("$CURDIR"/package/lib/*.so*)
+shopt -u nullglob
+
+if [ "$(uname)" != "Darwin" ] && [ "${#staged_libs[@]}" -gt 0 ]; then
+	# No silent skip. If the closure cannot be checked, the package cannot be
+	# shown to be complete, and shipping an unverified one is the failure this
+	# guard exists to prevent.
+	if command -v readelf >/dev/null 2>&1; then
+		read_needed() { readelf -d "$1" 2>/dev/null | sed -n 's/.*(NEEDED).*\[\(.*\)\]/\1/p'; }
+	elif command -v objdump >/dev/null 2>&1; then
+		read_needed() { objdump -p "$1" 2>/dev/null | awk '$1 == "NEEDED" { print $2 }'; }
+	else
+		echo "ERROR: neither readelf nor objdump is available, so the dependency" >&2
+		echo "       closure of ${#staged_libs[@]} staged libraries cannot be verified." >&2
+		echo "       Install binutils in the build image; refusing to ship an" >&2
+		echo "       unverified package." >&2
+		exit 1
+	fi
+
+	is_provided() {
+		case "$1" in
+			ld-linux*|libc.so.6|libstdc++.so.6|libgcc_s.so.1|libm.so.6|libgomp.so.1) return 0 ;;
+			libdl.so.2|librt.so.1|libpthread.so.0) return 0 ;;
+			libcuda*|libcudart*|libcublas*|libcublasLt*|libnvrtc*|libnvidia*) return 0 ;;
+			libamdhip*|libhsa*|librocm*|libze_*|libOpenCL*|libvulkan*) return 0 ;;
+		esac
+		[ -e "$CURDIR/package/lib/$1" ]
+	}
+
+	# Walk until the staged set stops growing. The glob below expands once per
+	# pass, so each pass advances the closure by exactly one dependency level;
+	# a copied library can itself pull in new dependencies.
+	#
+	# CLOSURE_MAX_PASSES is a runaway guard, not a depth limit. Exhausting it
+	# means the walk never converged and the package is therefore incomplete,
+	# which has to fail the build: a fixed pass count that just falls out of the
+	# loop would silently ship a package missing its deepest libraries, and
+	# libnemo_speech_asr -> sparrowhawk -> protobuf -> absl already runs several
+	# levels deep.
+	CLOSURE_MAX_PASSES="${CLOSURE_MAX_PASSES:-64}"
+	converged=0
+	for (( pass=1; pass<=CLOSURE_MAX_PASSES; pass++ )); do
+		missing=0
+		for so in "$CURDIR"/package/lib/*.so*; do
+			[ -f "$so" ] || continue
+			for need in $(read_needed "$so"); do
+				# Written as an if rather than "is_provided && continue" so a
+				# false return cannot trip set -e via the AND-list exit status.
+				if is_provided "$need"; then
+					continue
+				fi
+				# Resolve against the staging dir first, then the system loader.
+				src="$(LD_LIBRARY_PATH="$CURDIR:$CURDIR/package/lib:${LD_LIBRARY_PATH:-}" \
+					ldd "$so" 2>/dev/null | awk -v n="$need" '$1 == n { print $3 }' | head -1)"
+				if [ -z "$src" ] || [ ! -e "$src" ]; then
+					echo "ERROR: $(basename "$so") needs $need and it could not be resolved." >&2
+					echo "       The packaged backend would fail to dlopen at runtime." >&2
+					exit 1
+				fi
+				cp -aLvf "$src" "$CURDIR/package/lib/$need"
+				missing=1
+			done
+		done
+		if [ "$missing" -eq 0 ]; then
+			converged=1
+			break
+		fi
+	done
+
+	if [ "$converged" -ne 1 ]; then
+		echo "ERROR: the dependency closure was still growing after" >&2
+		echo "       $CLOSURE_MAX_PASSES passes, so the package is incomplete and" >&2
+		echo "       would fail to dlopen at runtime. Refusing to ship it." >&2
+		exit 1
+	fi
+fi
+
+# Package GPU libraries (CUDA/ROCm/Intel/Vulkan loader + ICDs + drivers)
+# based on BUILD_TYPE so the backend can reach the GPU without the runtime
+# base image shipping those drivers.
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah "$CURDIR/package/" "$CURDIR/package/lib/"

backend/go/nemo-speech-cpp/run.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath "$0")")
+
+# The runtime splits its C ABI across three shared objects, so each gets its
+# own override variable. main.go reads exactly these names.
+if [ "$(uname)" = "Darwin" ]; then
+	export DYLD_LIBRARY_PATH="$CURDIR/lib:"$CURDIR":${DYLD_LIBRARY_PATH:-}"
+	export NEMO_SPEECH_ASR_LIBRARY="$CURDIR/lib/libnemo_speech_asr_c.dylib"
+	export NEMO_SPEECH_TTS_LIBRARY="$CURDIR/lib/libnemo_speech_tts.dylib"
+	export NEMO_SPEECH_NMT_LIBRARY="$CURDIR/lib/libnemo_speech_nmt_c.dylib"
+else
+	export LD_LIBRARY_PATH="$CURDIR/lib:"$CURDIR":${LD_LIBRARY_PATH:-}"
+	export NEMO_SPEECH_ASR_LIBRARY="$CURDIR/lib/libnemo_speech_asr_c.so"
+	export NEMO_SPEECH_TTS_LIBRARY="$CURDIR/lib/libnemo_speech_tts.so"
+	export NEMO_SPEECH_NMT_LIBRARY="$CURDIR/lib/libnemo_speech_nmt_c.so"
+fi
+
+# If a self-contained ld.so was packaged, route through it so the
+# packaged libc / libstdc++ are used instead of the host's (matches the
+# whisper backend's runtime layout). Linux only.
+if [ -f "$CURDIR/lib/ld.so" ]; then
+	echo "Using lib/ld.so"
+	exec "$CURDIR/lib/ld.so" "$CURDIR/nemo-speech-cpp-grpc" "$@"
+fi
+
+exec "$CURDIR/nemo-speech-cpp-grpc" "$@"

backend/go/nemo-speech-cpp/status.go

@@ -0,0 +1,102 @@
+package main
+
+import (
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// The status values every C entry point in this backend returns.
+//
+// There is no single C enum to mirror. asr.h:43-49, tts.h:38-44 and nmt.h:40-45
+// each declare their own, and diar.h has none of its own at all: it includes
+// asr.h and types every diarization function as nemo_speech_asr_status
+// (diar.h:18). The names the three surfaces share carry the same numbers:
+//
+//	value  asr.h                              tts.h                              nmt.h
+//	0      NEMO_SPEECH_ASR_OK                 NEMO_SPEECH_TTS_OK                 NEMO_SPEECH_NMT_OK
+//	1      NEMO_SPEECH_ASR_ERROR_INVALID_...  NEMO_SPEECH_TTS_ERROR_INVALID_...  NEMO_SPEECH_NMT_ERROR_INVALID_...
+//	2      NEMO_SPEECH_ASR_ERROR_OUT_OF_MEM.  NEMO_SPEECH_TTS_ERROR_OUT_OF_MEM.  NEMO_SPEECH_NMT_ERROR_OUT_OF_MEM.
+//	3      NEMO_SPEECH_ASR_ERROR_RUNTIME      NEMO_SPEECH_TTS_ERROR_RUNTIME      NEMO_SPEECH_NMT_ERROR_RUNTIME
+//	4      NEMO_SPEECH_ASR_ERROR_CANCELLED    NEMO_SPEECH_TTS_ERROR_CANCELLED    (not declared)
+//
+// The one divergence is 4, and it is an absence rather than a disagreement. ASR
+// and TTS both drive a consumer callback that can ask for the work to stop, and
+// cancellation is what they report when it does; nemo_speech_nmt_translate takes
+// no callback and returns only when the decode has finished, so the NMT surface
+// has no cancellation to name. That is why one table can serve all three: 4 is
+// not some other NMT status that would be mislabelled, it is a value the NMT
+// surface never produces.
+//
+// Recheck this table after an upstream pin bump. A status added to one header
+// and not the others is exactly the shape of change that would break the single
+// mapping, and nothing in the build or the linker can see it: purego binds by
+// name, and the return value is a bare int32 on the Go side.
+const (
+	statusOK              int32 = 0
+	statusInvalidArgument int32 = 1
+	statusOutOfMemory     int32 = 2
+	statusRuntime         int32 = 3
+	statusCancelled       int32 = 4
+)
+
+// statusCode maps a C status onto the gRPC code the caller should be told.
+//
+// What the mapping is really carrying is whose mistake the failure was.
+// INVALID_ARGUMENT is what every guard in src/{asr,tts,nmt}/c_api.cpp returns
+// for a std::invalid_argument from the runtime, and the things that throw it are
+// requests: an unknown voice_name (src/tts/synthesizer.cpp), an unsupported
+// language pair (src/nmt/translator.cpp), an out-of-range sample rate. Reporting
+// those as Internal turns a 400 into a 500 and sends the user hunting for a
+// broken model or a broken backend instead of fixing the request.
+//
+// OUT_OF_MEMORY is a resource limit rather than a defect, which is what
+// ResourceExhausted means, and it is the one failure a client can sensibly
+// retry later or retry smaller. CANCELLED is the consumer having stopped
+// listening, which is not a failure of this backend at all: the streaming sinks
+// return false once their client is gone (see ttsDeliverPCM), and the runtime
+// turns that into status 4.
+//
+// RUNTIME, and anything a future pin adds that this table has not been taught,
+// stay Internal. An unrecognised status is precisely the case where the backend
+// does not know whose fault it was, and Internal is the honest answer.
+func statusCode(st int32) codes.Code {
+	switch st {
+	case statusOK:
+		return codes.OK
+	case statusInvalidArgument:
+		return codes.InvalidArgument
+	case statusOutOfMemory:
+		return codes.ResourceExhausted
+	case statusCancelled:
+		return codes.Canceled
+	case statusRuntime:
+		// Named rather than folded into the default so this switch reads as the
+		// whole enum. A status the table has never heard of is a different thing
+		// from a runtime error even though both answer Internal, and a reader
+		// checking the mapping against the headers should not have to work out
+		// which arm RUNTIME lands in.
+		return codes.Internal
+	default:
+		return codes.Internal
+	}
+}
+
+// statusErrorf builds the gRPC error for a failed C call.
+//
+// Every C call site in this backend goes through this rather than through
+// status.Errorf directly, and that is the whole point of it existing: the
+// mapping used to be written out at exactly one of sixteen call sites, so the
+// same backend answered an unsupported language pair with InvalidArgument and an
+// unknown TTS voice, which is the same class of caller mistake against the same
+// process, with Internal.
+//
+// The OK guard is not defensive noise. status.Errorf(codes.OK, ...) returns a
+// nil error, so a call site that built its error without first checking the
+// status would report a hard C failure as a successful request with no
+// diagnostic anywhere. Returning Internal instead keeps that mistake loud.
+func statusErrorf(st int32, format string, args ...any) error {
+	if st == statusOK {
+		return status.Errorf(codes.Internal, format, args...)
+	}
+	return status.Errorf(statusCode(st), format, args...)
+}

backend/go/nemo-speech-cpp/status_test.go

@@ -0,0 +1,109 @@
+package main
+
+import (
+	"os"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+var _ = Describe("C status mapping", func() {
+	// The whole enum, so a status that quietly moves to a different code is
+	// visible here rather than in a bug report about an HTTP 500. The names on
+	// the left are transcribed from asr.h:43-49, tts.h:38-44 and nmt.h:40-45;
+	// see the table in status.go for how the three surfaces line up.
+	DescribeTable("maps each declared C status onto a gRPC code",
+		func(st int32, want codes.Code) {
+			Expect(statusCode(st)).To(Equal(want))
+		},
+		Entry("OK", statusOK, codes.OK),
+		Entry("INVALID_ARGUMENT", statusInvalidArgument, codes.InvalidArgument),
+		Entry("OUT_OF_MEMORY", statusOutOfMemory, codes.ResourceExhausted),
+		Entry("RUNTIME", statusRuntime, codes.Internal),
+		Entry("CANCELLED", statusCancelled, codes.Canceled),
+	)
+
+	// A pin bump that adds a status this table has never been taught must not
+	// guess. Internal is the honest answer when the backend does not know whose
+	// mistake the failure was.
+	DescribeTable("reports an unknown status as Internal",
+		func(st int32) {
+			Expect(statusCode(st)).To(Equal(codes.Internal))
+		},
+		Entry("one past the last declared value", int32(5)),
+		Entry("far past it", int32(99)),
+		Entry("negative", int32(-1)),
+	)
+
+	It("carries the mapped code and the formatted message into the error", func() {
+		err := statusErrorf(statusInvalidArgument, "nemo-speech-cpp: %s: %d", "synthesize", 7)
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(err.Error()).To(ContainSubstring("nemo-speech-cpp: synthesize: 7"))
+	})
+
+	// status.Errorf(codes.OK, ...) returns nil, so a call site that built its
+	// error without checking the status first would turn a hard C failure into a
+	// silent success with no diagnostic anywhere.
+	It("never returns nil, not even for OK", func() {
+		err := statusErrorf(statusOK, "nemo-speech-cpp: should not happen")
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+	})
+})
+
+// These drive real C statuses out of the real shared objects, one per family,
+// rather than asserting the Go mapping against itself.
+//
+// A NULL handle is the one failure every surface can be provoked into without a
+// model: nemo_speech_asr_recognize_f32 and nemo_speech_nmt_translate check the
+// handle up front, nemo_speech_tts_synthesize_text does the same, and the
+// diarization stream entry points throw std::invalid_argument for a dead stream,
+// which src/asr/c_api.cpp's guard maps to the same status. All four are the
+// caller's mistake, and the point of the exercise is that all four now come back
+// as InvalidArgument instead of Internal.
+var _ = Describe("C status mapping at the call sites", func() {
+	BeforeEach(func() {
+		if !librariesPresent() {
+			if requireLibs() {
+				cwd, _ := os.Getwd()
+				Fail("NEMO_SPEECH_REQUIRE_LIBS=1 but the shared libraries are not in " + cwd +
+					": these specs are the ABI defence and must not be skipped." +
+					" Run make -C backend/go/nemo-speech-cpp stage-libs")
+			}
+			Skip("shared libraries not built, run make in backend/go/nemo-speech-cpp")
+		}
+		Expect(openLibraries()).To(Succeed())
+	})
+
+	It("reports an ASR INVALID_ARGUMENT as InvalidArgument", func() {
+		opts := ASRRecognitionOptionsDef()
+		// Non-empty PCM on purpose: recognizeF32 rejects an empty slice itself,
+		// which would prove nothing about what the C side returned.
+		_, err := recognizeF32(0, &opts, []float32{0, 0, 0}, 16000)
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	It("reports a diarization INVALID_ARGUMENT as InvalidArgument", func() {
+		err := (&cDiarStream{}).finish()
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	It("reports a TTS INVALID_ARGUMENT as InvalidArgument", func() {
+		s := &cSynthesizer{}
+		err := s.synthesize(&pb.TTSRequest{Text: "hello"}, "en", func([]byte) bool { return true })
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+
+	It("reports an NMT INVALID_ARGUMENT as InvalidArgument", func() {
+		_, err := (&cTranslator{}).translate([]string{"hello"}, "en", "de")
+		Expect(err).To(HaveOccurred())
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+	})
+})

backend/go/nemo-speech-cpp/tts.go

@@ -0,0 +1,600 @@
+package main
+
+import (
+	"bytes"
+	"math"
+	"os"
+	"runtime"
+	"strconv"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	laudio "github.com/mudler/LocalAI/pkg/audio"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// The backend-preference enum from include/nemo_speech/tts.h. The C type is an
+// enum, which this toolchain lays out as int32, so the values are written here
+// rather than inferred.
+const (
+	ttsBackendAuto int32 = 0
+	ttsBackendCPU  int32 = 1
+)
+
+// maxWAVDataBytes is the largest PCM payload a RIFF WAV can describe.
+//
+// Both size fields in the header are uint32, so a longer payload would not
+// merely be unusual, it would wrap and produce a file whose header disagrees
+// with its contents. At 22.05 kHz mono 16-bit that ceiling is about 27 hours of
+// speech, so nothing real is being refused.
+//
+// Typed int64 rather than left untyped so the comparison below is the same one
+// on every architecture: an untyped constant this large does not fit in a
+// 32-bit int and would not compile there at all.
+const maxWAVDataBytes int64 = math.MaxUint32 - laudio.WAVHeaderSize
+
+// wavStreamingSize is the placeholder both size fields carry while the total
+// length is still unknown. Players read it as "stream until the socket closes".
+const wavStreamingSize = 0xFFFFFFFF
+
+// ttsSink receives one PCM chunk, already copied into Go memory.
+//
+// It returns false to cancel the synthesis in progress: that is the C
+// callback's only way to stop work early, and the runtime turns it into
+// NEMO_SPEECH_TTS_ERROR_CANCELLED.
+type ttsSink func(pcm []byte) bool
+
+// ttsSinkTable maps the user_data value handed to C back to the Go sink the
+// chunk belongs to.
+//
+// A single "current sink" pointer would be enough for one model, since every
+// RPC holds that model's engineMu for its whole body. It is not enough for the
+// process: engineMu is per-NemoSpeech, one backend process can hold several
+// loaded models, and the callback below is shared by all of them, so two TTS
+// models synthesizing at once would overwrite each other's sink. The id is what
+// keeps them apart.
+//
+// The id is an integer and never a Go pointer. user_data crosses into C as a
+// void*, which the collector does not trace, so a Go pointer parked there would
+// have exactly the lifetime problem cstr documents.
+type ttsSinkTable struct {
+	mu    sync.Mutex
+	next  uintptr
+	sinks map[uintptr]ttsSink
+}
+
+var ttsSinks = &ttsSinkTable{sinks: map[uintptr]ttsSink{}}
+
+// register adds sink and returns its id together with the release the caller
+// MUST defer. Ids start at 1 so a zeroed or stale user_data cannot resolve to
+// somebody else's sink.
+func (t *ttsSinkTable) register(sink ttsSink) (uintptr, func()) {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+
+	t.next++
+	id := t.next
+	t.sinks[id] = sink
+	return id, func() {
+		t.mu.Lock()
+		defer t.mu.Unlock()
+		delete(t.sinks, id)
+	}
+}
+
+// lookup returns the sink for id, or nil once it has been released.
+func (t *ttsSinkTable) lookup(id uintptr) ttsSink {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+
+	return t.sinks[id]
+}
+
+var (
+	ttsCallbackOnce sync.Once
+	ttsCallbackFn   uintptr
+)
+
+// ttsPCMCallback returns the C function pointer the runtime drives PCM through,
+// compiling it on first use.
+//
+// Exactly one is ever created per process, and that is a hard requirement
+// rather than a tidiness argument. purego.NewCallback writes into a fixed table
+// of maxCB = 2000 entries (purego/syscall_sysv.go) and never releases an entry,
+// so a callback compiled per request panics the whole backend process with
+// "purego: the maximum number of callbacks has been reached" on the 2001st
+// synthesis. Per model load is not safe either: a server that swaps models
+// reaches the same ceiling, just later and even less predictably. Routing every
+// synthesis through one callback plus a user_data id is what keeps the count at
+// one for the life of the process.
+func ttsPCMCallback() uintptr {
+	ttsCallbackOnce.Do(func() { ttsCallbackFn = purego.NewCallback(ttsDeliverPCM) })
+	return ttsCallbackFn
+}
+
+// ttsDeliverPCM is the body of that callback: nemo_speech_tts_pcm_callback,
+// which the runtime invokes on its own thread for each chunk it produces.
+//
+// The bytes are copied rather than aliased. The pointer addresses a std::string
+// the runtime owns and reuses for the next chunk (src/tts/c_api.cpp
+// make_callback), so a slice over it would be rewritten under the consumer as
+// soon as this returns.
+func ttsDeliverPCM(pcm unsafe.Pointer, nBytes uint64, userData uintptr) bool {
+	// The table's lock is released before the sink runs, which matters because
+	// TTSStream's sink blocks on a channel send until its client drains it.
+	// Holding the lock across that would stall every other model's callback
+	// behind one slow consumer.
+	sink := ttsSinks.lookup(userData)
+	if sink == nil {
+		// The request that registered this sink has already returned, so there
+		// is nowhere to put the audio. false cancels rather than letting the
+		// runtime synthesize to completion into a consumer that stopped
+		// listening.
+		return false
+	}
+	// c_api.cpp filters empty chunks before calling us, so this is belt and
+	// braces: unsafe.Slice on a null pointer is what it protects against.
+	if pcm == nil || nBytes == 0 {
+		return true
+	}
+
+	buf := make([]byte, nBytes)
+	// #nosec G103 -- pcm and nBytes are the C-owned buffer and its length from
+	// one callback invocation, both null/zero-checked above. The slice is read
+	// only, its length is the length the runtime declared for that buffer, and it
+	// is copied into Go memory here and never retained past this return.
+	copy(buf, unsafe.Slice((*byte)(pcm), nBytes))
+	return sink(buf)
+}
+
+// synthesizer is the TTS half of the C API, narrowed to what the two RPCs use.
+//
+// It is an interface for the same reason asrSession and diarStream are: no
+// MagpieTTS GGUF is small enough to keep in the tree, so the logic layered on
+// top of the ABI (validation, the WAV framing, chunk ordering) would otherwise
+// have no test at all. The seam is at the ABI, not at the model: a fake here
+// scripts what the C API emits, it does not pretend to synthesize anything.
+type synthesizer interface {
+	// sampleRate is the rate the PCM chunks arrive at.
+	sampleRate() int32
+	// synthesize maps req onto the runtime's per-request options and runs one
+	// synthesis, handing each chunk to sink as it is produced.
+	synthesize(req *pb.TTSRequest, defaultLanguage string, sink ttsSink) error
+}
+
+// cSynthesizer is the real synthesizer, over one nemo_speech_tts_synthesizer.
+type cSynthesizer struct {
+	handle uintptr
+}
+
+func (s *cSynthesizer) sampleRate() int32 { return TTSSampleRate(s.handle) }
+
+func (s *cSynthesizer) synthesize(req *pb.TTSRequest, defaultLanguage string, sink ttsSink) error {
+	// Started from the runtime's own defaults, not from a zero struct: every
+	// numeric field here is sentinel-sensitive (speaker/seed < 0, steps/top_k
+	// <= 0 all mean "use the synthesizer's value"), and a zeroed struct would
+	// read as speaker 0, seed 0 and zero decoding steps.
+	opts := TTSSynthesisOptionsDefault()
+
+	// A per-request language wins over the model-level default; both may be
+	// empty, which the runtime resolves to the synthesizer's own default.
+	language := req.GetLanguage()
+	if language == "" {
+		language = defaultLanguage
+	}
+	langP, freeLang := cstr(language)
+	defer freeLang()
+	opts.LanguageCode = langP
+
+	speaker, voiceName := resolveSpeaker(req.GetVoice())
+	opts.Speaker = speaker
+	voiceP, freeVoice := cstr(voiceName)
+	defer freeVoice()
+	opts.VoiceName = voiceP
+
+	applySynthesisParams(&opts, req.GetParams())
+
+	id, release := ttsSinks.register(sink)
+	defer release()
+
+	// stats_out is NULL: nemo_speech_tts_synthesis_stats is 300-odd bytes of
+	// timing detail with nowhere to go on either RPC, and the C API documents
+	// NULL as the way to decline it.
+	// #nosec G103 -- opts is a local POD struct borrowed for this call only. Its
+	// two uintptr members (LanguageCode, VoiceName) are cstr allocations pinned
+	// by the defers above, and this entry point is synchronous, so it returns
+	// before those pins are released even though the callbacks run off-thread.
+	st := TTSSynthesizeText(s.handle, unsafe.Pointer(&opts), req.GetText(), ttsPCMCallback(), id, nil)
+	if st != 0 {
+		// An unknown voice_name arrives here as INVALID_ARGUMENT
+		// (src/tts/synthesizer.cpp throws std::invalid_argument, which
+		// src/tts/c_api.cpp's guard maps to it), and a consumer that stopped
+		// reading arrives as CANCELLED. Neither is this backend's failure, so
+		// neither goes out as Internal.
+		return statusErrorf(st, "nemo-speech-cpp: synthesize: %s", TTSLastError())
+	}
+	return nil
+}
+
+// resolveSpeaker splits a request's voice into the two fields the C API has for
+// it: a speaker index and a voice name.
+//
+// nemo_speech_tts_synthesis_options.voice_name is documented as ignored
+// whenever speaker >= 0, and src/tts/synthesizer.cpp only calls resolve_speaker
+// when options.speaker is negative, so the two are alternatives and never a
+// pair. A named voice must therefore leave the index at -1 or the name is
+// silently dropped.
+//
+// The numeric split cannot change what the runtime picks: resolve_speaker parses
+// a numeric voice_name itself, so anything this function passes through as a
+// name and that happens to be a number lands on the same speaker anyway. What it
+// must not do is let a NEGATIVE number through as an index. "-1" is not a
+// speaker, it is the sentinel for "use the default", and treating it as an index
+// would turn a request naming an invalid voice into one that quietly synthesizes
+// in the default voice instead of being rejected.
+func resolveSpeaker(voice string) (int32, string) {
+	if voice == "" {
+		return -1, ""
+	}
+	if idx, err := strconv.ParseInt(voice, 10, 32); err == nil && idx >= 0 {
+		return int32(idx), ""
+	}
+	return -1, voice
+}
+
+// applySynthesisParams maps TTSRequest.params onto the runtime's per-request
+// options.
+//
+// Only the five knobs nemo_speech_tts_synthesis_options actually has are read.
+// An unset or unparseable value leaves the field alone rather than resetting it:
+// the struct arrives carrying the runtime's defaults, and params is documented
+// as "unset leaves the backend's configured defaults".
+//
+// The sentinels are the reason each write is guarded rather than unconditional.
+// src/tts/magpietts/runtime.cpp takes the request's seed only when it is >= 0
+// and its steps and top_k only when they are > 0, so writing a parsed 0 or a
+// negative would not merely be ignored, it would erase the option's meaning for
+// a caller who passed "0" expecting something.
+//
+// temperature and cfg_scale each need their override flag set as well. The
+// runtime reads the float only when the flag is true and otherwise falls back to
+// the synthesizer's config, so a temperature written without its flag is
+// silently discarded.
+func applySynthesisParams(o *cTTSSynthesisOptions, params map[string]string) {
+	if len(params) == 0 {
+		return
+	}
+	if v, ok := parseInt32Param(params["seed"]); ok && v >= 0 {
+		o.Seed = v
+	}
+	if v, ok := parseInt32Param(params["steps"]); ok && v > 0 {
+		o.Steps = v
+	}
+	if v, ok := parseInt32Param(params["top_k"]); ok && v > 0 {
+		o.TopK = v
+	}
+	if v, ok := parseFloat32Param(params["temperature"]); ok {
+		o.Temperature = v
+		o.OverrideTemperature = true
+	}
+	if v, ok := parseFloat32Param(params["cfg_scale"]); ok {
+		o.CFGScale = v
+		o.OverrideCFGScale = true
+	}
+}
+
+// parseInt32Param reads one params entry. ok is false for an absent or
+// unparseable value, which the caller reads as "leave the default".
+func parseInt32Param(v string) (int32, bool) {
+	if v == "" {
+		return 0, false
+	}
+	n, err := strconv.ParseInt(v, 10, 32)
+	if err != nil {
+		xlog.Warn("nemo-speech-cpp: ignoring unparseable TTS parameter", "value", v)
+		return 0, false
+	}
+	return int32(n), true
+}
+
+func parseFloat32Param(v string) (float32, bool) {
+	if v == "" {
+		return 0, false
+	}
+	f, err := strconv.ParseFloat(v, 32)
+	if err != nil {
+		xlog.Warn("nemo-speech-cpp: ignoring unparseable TTS parameter", "value", v)
+		return 0, false
+	}
+	return float32(f), true
+}
+
+// ttsModelConfig builds the create-time model config.
+//
+// Extracted from loadTTS and asserted field by field because three adjacent
+// members of nemo_speech_tts_model_config are same-typed paths. Swapping two of
+// them changes neither the struct's size nor any field's offset, so the layout
+// assertions in abi_test.go cannot see it, and the failure it produces is the
+// runtime loading the codec as the acoustic model.
+//
+// Every argument is a C pointer from cstr, not a Go string, and the caller owns
+// the releases. tnDir may be null: text_normalizer_model_dir is optional and an
+// empty one leaves the text unchanged.
+func ttsModelConfig(magpieModel, codecModel, tokenizerDir, tnDir uintptr) cTTSModelConfig {
+	return cTTSModelConfig{
+		Size:                   unsafe.Sizeof(cTTSModelConfig{}),
+		MagpieModel:            magpieModel,
+		CodecModel:             codecModel,
+		TokenizerModelDir:      tokenizerDir,
+		TextNormalizerModelDir: tnDir,
+	}
+}
+
+// ttsRuntimeBackend maps the backend's gpu option onto the TTS runtime's
+// backend preference.
+//
+// nemo_speech_tts_runtime_config has no device index at all, only a three-way
+// AUTO/CPU/CUDA preference, so a gpu option naming a particular device cannot be
+// honoured and AUTO is the honest answer for it. A negative gpu is different: it
+// is the option's documented "CPU" across this whole backend (asr.h: "-1 = CPU")
+// and it is also the default, so it has to pin the preference rather than leave
+// the runtime free to pick CUDA.
+func ttsRuntimeBackend(gpu int32) int32 {
+	if gpu < 0 {
+		return ttsBackendCPU
+	}
+	return ttsBackendAuto
+}
+
+// loadTTS creates the MagpieTTS synthesizer.
+//
+// It runs after discoverTTSAssets, so codecModel and tokenizerDir are already
+// resolved and non-empty; tnDir stays optional.
+//
+// This must not take engineMu: Load is its only caller and already holds it.
+func (n *NemoSpeech) loadTTS(modelFile string) error {
+	// nemo_speech_tts_create deep-copies every const char* into a std::string
+	// (src/tts/c_api.cpp, via str_or_empty) and keeps no pointer afterwards, so
+	// pinning across the create call is both necessary and sufficient.
+	var pinner runtime.Pinner
+	defer pinner.Unpin()
+
+	magpieP, freeMagpie := cstr(modelFile)
+	defer freeMagpie()
+	codecP, freeCodec := cstr(n.opts.codecModel)
+	defer freeCodec()
+	tokenizerP, freeTokenizer := cstr(n.opts.tokenizerDir)
+	defer freeTokenizer()
+	tnP, freeTN := cstr(n.opts.tnDir)
+	defer freeTN()
+
+	model := ttsModelConfig(magpieP, codecP, tokenizerP, tnP)
+
+	rt := TTSRuntimeConfigDefault()
+	backend := ttsRuntimeBackend(n.opts.gpu)
+	rt.LTBackend = backend
+	rt.SamplingBackend = backend
+	// The codec is a separate graph with its own placement, so a CPU-only
+	// request has to say so here too or it would still try to run on the GPU.
+	rt.CodecCPU = backend == ttsBackendCPU
+
+	langP, freeLang := cstr(n.opts.languageCode)
+	defer freeLang()
+
+	cfg := cTTSSynthesizerConfig{
+		Size:                unsafe.Sizeof(cTTSSynthesizerConfig{}),
+		Model:               pinPtr(&pinner, &model),
+		Runtime:             pinPtr(&pinner, &rt),
+		DefaultLanguageCode: langP,
+	}
+
+	xlog.Info("nemo-speech-cpp: creating synthesizer",
+		"gpu", n.opts.gpu,
+		"codec", n.opts.codecModel,
+		"tokenizer", n.opts.tokenizerDir,
+		"text_normalizer", n.opts.tnDir != "")
+
+	// Compiled before the handle exists so that a full callback table fails the
+	// load, where the operator can see it, rather than the first synthesis.
+	ttsPCMCallback()
+
+	// #nosec G103 -- cfg is a local POD struct borrowed for this call only. Model
+	// and Runtime are pinPtr addresses held by the pinner unpinned on return, the
+	// paths they carry are cstr allocations freed by the defers above, and
+	// nemo_speech_tts_create deep-copies every string it reads.
+	if st := TTSCreate(unsafe.Pointer(&cfg), &n.synth); st != 0 {
+		return statusErrorf(st, "nemo-speech-cpp: tts create: %s", TTSLastError())
+	}
+	return nil
+}
+
+// validateTTSRequest rejects what the runtime would reject, before anything
+// crosses the ABI, and names the fields this backend drops.
+//
+// Empty text is checked here rather than left to the C side for the error code:
+// src/tts/synthesizer.cpp throws "text is required", which arrives as a status
+// this layer would otherwise report as Internal, and an empty prompt is a client
+// mistake, not a backend failure.
+//
+// instructions is logged rather than rejected, for the reason the diarization
+// path logs its own dropped fields: a caller that asked for an expressive style
+// still wants the audio it can have, and a request naming something this backend
+// silently ignores should say so where an operator can find it. There is nothing
+// to map it onto, because MagpieTTS conditions on a speaker, not on a prose
+// style description: nemo_speech_tts_synthesis_options has speaker and
+// voice_name and no free-text field at all.
+func validateTTSRequest(req *pb.TTSRequest) error {
+	if req.GetText() == "" {
+		return status.Error(codes.InvalidArgument, "nemo-speech-cpp: TTSRequest.text is required")
+	}
+	if req.GetInstructions() != "" {
+		xlog.Warn("nemo-speech-cpp: ignoring TTSRequest.instructions, this model has no equivalent")
+	}
+	return nil
+}
+
+// outputSampleRate reads the rate the synthesizer emits at.
+//
+// A non-positive rate is refused rather than passed on. nemo_speech_tts_sample_rate
+// answers 0 for a null handle, and a WAV header carrying 0 is not a slightly
+// wrong file, it is one no player can decode and one whose duration is
+// undefined.
+func outputSampleRate(s synthesizer) (uint32, error) {
+	rate := s.sampleRate()
+	if rate <= 0 {
+		return 0, status.Error(codes.Internal,
+			"nemo-speech-cpp: the synthesizer reported no sample rate")
+	}
+	return uint32(rate), nil
+}
+
+// wavFile frames PCM as a complete WAV: a header with real sizes, then the
+// samples.
+//
+// pcm is little-endian signed 16-bit mono, which is what the runtime's callback
+// delivers, and is exactly what pkg/audio's header describes, so nothing is
+// converted on the way through.
+func wavFile(pcm []byte, sampleRate uint32) ([]byte, error) {
+	if int64(len(pcm)) > maxWAVDataBytes {
+		return nil, status.Errorf(codes.Internal,
+			"nemo-speech-cpp: synthesis produced %d bytes, more than a WAV header can describe", len(pcm))
+	}
+
+	var buf bytes.Buffer
+	// #nosec G115 -- len(pcm) is checked against maxWAVDataBytes (MaxUint32 minus
+	// the header) immediately above, so the narrowing to uint32 cannot wrap.
+	h := laudio.NewWAVHeaderWithRate(uint32(len(pcm)), sampleRate)
+	if err := h.Write(&buf); err != nil {
+		return nil, status.Errorf(codes.Internal, "nemo-speech-cpp: write WAV header: %v", err)
+	}
+	buf.Write(pcm)
+	return buf.Bytes(), nil
+}
+
+// streamingWAVHeader is the first chunk of a streamed synthesis: the same header
+// with both sizes left unknown, since the total length is not known until the
+// synthesis ends.
+//
+// NewWAVHeaderWithRate derives ChunkSize from the payload length, so the RIFF
+// size has to be overwritten as well: 36 + 0xFFFFFFFF wraps to 35, which is a
+// smaller number than the header itself.
+func streamingWAVHeader(sampleRate uint32) []byte {
+	h := laudio.NewWAVHeaderWithRate(wavStreamingSize, sampleRate)
+	h.ChunkSize = wavStreamingSize
+
+	var buf bytes.Buffer
+	// Write only fails on the writer, and bytes.Buffer does not fail.
+	_ = h.Write(&buf)
+	return buf.Bytes()
+}
+
+// synthesizeWAV runs one synthesis and writes the whole result to dst.
+func synthesizeWAV(s synthesizer, req *pb.TTSRequest, defaultLanguage string) error {
+	if err := validateTTSRequest(req); err != nil {
+		return err
+	}
+	if req.GetDst() == "" {
+		return status.Error(codes.InvalidArgument,
+			"nemo-speech-cpp: TTSRequest.dst (output path) is required")
+	}
+
+	// Read before the synthesis rather than after: it is what the header is
+	// built from, and failing on a bad handle here costs nothing, where failing
+	// after costs the whole synthesis.
+	rate, err := outputSampleRate(s)
+	if err != nil {
+		return err
+	}
+
+	var pcm []byte
+	err = s.synthesize(req, defaultLanguage, func(chunk []byte) bool {
+		pcm = append(pcm, chunk...)
+		return true
+	})
+	if err != nil {
+		return err
+	}
+	// A synthesis that returned OK having emitted nothing is a runtime bug, but
+	// the file it would produce is a valid empty WAV, which reaches the user as
+	// silence with no error anywhere.
+	if len(pcm) == 0 {
+		return status.Error(codes.Internal, "nemo-speech-cpp: synthesis produced no audio")
+	}
+
+	out, err := wavFile(pcm, rate)
+	if err != nil {
+		return err
+	}
+	if err := os.WriteFile(req.GetDst(), out, 0o600); err != nil {
+		return status.Errorf(codes.Internal, "nemo-speech-cpp: write %q: %v", req.GetDst(), err)
+	}
+	return nil
+}
+
+// streamWAV runs one synthesis and emits a WAV header followed by each PCM
+// chunk as the runtime produces it.
+//
+// The header is the backend's job, not the caller's: pkg/grpc/server.go only
+// ever sets Reply.Audio on this path and never Reply.Message, and
+// core/backend/tts.go's own header branch is keyed on Message, so a backend that
+// emitted bare PCM would stream something no client could decode. sherpa-onnx
+// and magpie-tts-cpp both do the same.
+//
+// out is not closed here. TTSStream owns it, and closing it in one of two places
+// depending on how far the request got is how a stream ends up half-closed.
+func streamWAV(s synthesizer, req *pb.TTSRequest, defaultLanguage string, out chan<- []byte) error {
+	if err := validateTTSRequest(req); err != nil {
+		return err
+	}
+
+	rate, err := outputSampleRate(s)
+	if err != nil {
+		return err
+	}
+	out <- streamingWAVHeader(rate)
+
+	return s.synthesize(req, defaultLanguage, func(chunk []byte) bool {
+		out <- chunk
+		return true
+	})
+}
+
+// TTS synthesizes req.Text and writes a WAV to req.Dst.
+//
+// The whole body runs inside withEngine, so the family check and the C calls
+// that trust the handle happen under a single acquisition of engineMu. See the
+// handoff notes at the bottom of nemospeech.go: Free runs without the backend
+// lock, so anything that checks the family and then releases the lock before
+// calling C can have the handle destroyed underneath it.
+func (n *NemoSpeech) TTS(req *pb.TTSRequest) error {
+	return n.withEngine(familyTTS, func() error {
+		return synthesizeWAV(&cSynthesizer{handle: n.synth}, req, n.opts.languageCode)
+	})
+}
+
+// TTSStream synthesizes req.Text and emits the audio on results as it is
+// produced.
+//
+// results is closed on EVERY path, including the family rejection and a
+// validation failure, and the close is deferred outside withEngine so that a
+// rejected family still closes it. pkg/grpc/server.go drains this channel from a
+// goroutine and then blocks on that goroutine finishing, so a channel left open
+// does not fail the request, it hangs the RPC and, because the backend lock is
+// still held, every request behind it.
+//
+// Holding engineMu for the whole stream is deliberate and is the consequence
+// documented on the locking protocol: an unload waits for the stream to end
+// rather than destroying the synthesizer underneath it. There is no unbounded
+// wait here, because unlike the ASR streams this one is driven by the runtime
+// and ends when the text does, not when a client decides to stop sending.
+func (n *NemoSpeech) TTSStream(req *pb.TTSRequest, results chan []byte) error {
+	defer close(results)
+
+	return n.withEngine(familyTTS, func() error {
+		return streamWAV(&cSynthesizer{handle: n.synth}, req, n.opts.languageCode, results)
+	})
+}

backend/go/nemo-speech-cpp/tts_test.go

@@ -0,0 +1,727 @@
+package main
+
+import (
+	"encoding/binary"
+	"errors"
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"os"
+	"path/filepath"
+	"sync"
+	"unsafe"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+
+	laudio "github.com/mudler/LocalAI/pkg/audio"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// puregoCallbackTableSize is the hard ceiling purego compiles callbacks into:
+// maxCB in purego/syscall_sysv.go, which panics rather than growing once it is
+// full and never releases an entry. Read off the module source for v0.10.0
+// rather than assumed, because the whole point of the specs below is that
+// exceeding it kills the process.
+const puregoCallbackTableSize = 2000
+
+// fakeSynthesizer scripts what the TTS C API emits for one synthesis.
+//
+// There is no MagpieTTS GGUF in the tree, so this is the only way the logic on
+// top of the ABI (validation, WAV framing, chunk ordering, channel closure)
+// gets tested at all. It fakes the C contract, not the model: chunks is
+// whatever nemo_speech_tts_synthesize_text would have handed the callback.
+type fakeSynthesizer struct {
+	rate   int32
+	chunks [][]byte
+	err    error
+
+	calls     int
+	gotReq    *pb.TTSRequest
+	gotLang   string
+	cancelled bool
+}
+
+func (f *fakeSynthesizer) sampleRate() int32 { return f.rate }
+
+func (f *fakeSynthesizer) synthesize(req *pb.TTSRequest, defaultLanguage string, sink ttsSink) error {
+	f.calls++
+	f.gotReq = req
+	f.gotLang = defaultLanguage
+	for _, c := range f.chunks {
+		if !sink(c) {
+			f.cancelled = true
+			break
+		}
+	}
+	return f.err
+}
+
+var _ = Describe("resolveSpeaker", func() {
+	It("passes a numeric voice through as a speaker index", func() {
+		idx, name := resolveSpeaker("3")
+		Expect(idx).To(Equal(int32(3)))
+		Expect(name).To(BeEmpty())
+	})
+
+	It("passes a named voice through as a name with no index", func() {
+		// voice_name is ignored whenever speaker >= 0 (tts.h, and
+		// synthesizer.cpp only calls resolve_speaker for a negative speaker), so
+		// a named voice must leave the index negative or the name is dropped.
+		idx, name := resolveSpeaker("Aria")
+		Expect(idx).To(Equal(int32(-1)))
+		Expect(name).To(Equal("Aria"))
+	})
+
+	It("leaves both unset for an empty voice so the synthesizer default wins", func() {
+		idx, name := resolveSpeaker("")
+		Expect(idx).To(Equal(int32(-1)))
+		Expect(name).To(BeEmpty())
+	})
+
+	// A negative number is the C API's sentinel for "use the default", not a
+	// speaker. Passing it through as an index would turn a request naming an
+	// invalid voice into one that quietly synthesizes in the default voice.
+	// Handed on as a name instead, resolve_speaker rejects it.
+	It("does not let a negative number become a speaker index", func() {
+		idx, name := resolveSpeaker("-1")
+		Expect(idx).To(Equal(int32(-1)))
+		Expect(name).To(Equal("-1"))
+	})
+
+	It("treats a non-numeric voice that merely starts with digits as a name", func() {
+		idx, name := resolveSpeaker("3-alpha")
+		Expect(idx).To(Equal(int32(-1)))
+		Expect(name).To(Equal("3-alpha"))
+	})
+
+	It("keeps speaker 0 addressable", func() {
+		// 0 is a real speaker index, and the only sentinel here is < 0.
+		idx, name := resolveSpeaker("0")
+		Expect(idx).To(BeZero())
+		Expect(name).To(BeEmpty())
+	})
+})
+
+var _ = Describe("applySynthesisParams", func() {
+	// The struct the runtime hands out: speaker/seed/steps/top_k all -1, the
+	// overrides off. Written literally rather than taken from
+	// TTSSynthesisOptionsDefault so the specs run without the shared libraries.
+	defaults := func() cTTSSynthesisOptions {
+		return cTTSSynthesisOptions{
+			Size:    unsafe.Sizeof(cTTSSynthesisOptions{}),
+			Speaker: -1,
+			Seed:    -1,
+			Steps:   -1,
+			TopK:    -1,
+		}
+	}
+
+	It("leaves every default alone for an absent params map", func() {
+		o := defaults()
+		applySynthesisParams(&o, nil)
+		Expect(o).To(Equal(defaults()))
+	})
+
+	It("leaves every default alone for an empty params map", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{})
+		Expect(o).To(Equal(defaults()))
+	})
+
+	It("maps the five knobs the C options struct actually has", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{
+			"seed":        "42",
+			"steps":       "12",
+			"top_k":       "80",
+			"temperature": "0.7",
+			"cfg_scale":   "1.5",
+		})
+		Expect(o.Seed).To(Equal(int32(42)))
+		Expect(o.Steps).To(Equal(int32(12)))
+		Expect(o.TopK).To(Equal(int32(80)))
+		Expect(o.Temperature).To(BeNumerically("~", 0.7, 1e-6))
+		Expect(o.CFGScale).To(BeNumerically("~", 1.5, 1e-6))
+	})
+
+	// magpietts/runtime.cpp reads options.temperature only when
+	// override_temperature is true and otherwise falls back to the
+	// synthesizer's config, so a temperature written without its flag is
+	// silently discarded and the request looks like it was honoured.
+	It("sets the override flag with the temperature", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{"temperature": "0.4"})
+		Expect(o.OverrideTemperature).To(BeTrue())
+		Expect(o.OverrideCFGScale).To(BeFalse(), "cfg_scale was not asked for")
+	})
+
+	It("sets the override flag with the cfg scale", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{"cfg_scale": "2"})
+		Expect(o.OverrideCFGScale).To(BeTrue())
+		Expect(o.OverrideTemperature).To(BeFalse(), "temperature was not asked for")
+	})
+
+	It("keeps the defaults when a value cannot be parsed", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{
+			"seed":        "many",
+			"steps":       "",
+			"top_k":       "8.5",
+			"temperature": "warm",
+			"cfg_scale":   "-",
+		})
+		Expect(o).To(Equal(defaults()))
+	})
+
+	// The runtime takes a request's seed only when it is >= 0 and its steps and
+	// top_k only when they are > 0. Writing a parsed 0 or a negative would not
+	// be ignored downstream, it would erase the sentinel that means "use the
+	// synthesizer's value".
+	It("refuses values that would erase a sentinel", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{
+			"seed":  "-5",
+			"steps": "0",
+			"top_k": "0",
+		})
+		Expect(o.Seed).To(Equal(int32(-1)))
+		Expect(o.Steps).To(Equal(int32(-1)))
+		Expect(o.TopK).To(Equal(int32(-1)))
+	})
+
+	It("keeps seed 0, which is a real seed", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{"seed": "0"})
+		Expect(o.Seed).To(BeZero())
+	})
+
+	// TTSRequest carries fields with no equivalent in
+	// nemo_speech_tts_synthesis_options. They must not be smuggled in through a
+	// param name that happens to match.
+	It("ignores params the C options struct has no field for", func() {
+		o := defaults()
+		applySynthesisParams(&o, map[string]string{
+			"top_p":              "0.9",
+			"repetition_penalty": "1.1",
+			"speed":              "1.2",
+			"instructions":       "cheerful",
+		})
+		Expect(o).To(Equal(defaults()))
+	})
+})
+
+// The PCM callback is the one resource in this backend with a hard, silent,
+// process-wide ceiling: purego compiles each into a fixed table of 2000 entries
+// and never releases one, so a callback built per request takes the whole
+// backend process down with a panic after 2000 syntheses. Nothing about a
+// handful of manual calls shows that.
+var _ = Describe("ttsPCMCallback", func() {
+	It("compiles a usable callback", func() {
+		Expect(ttsPCMCallback()).ToNot(BeZero())
+	})
+
+	It("compiles exactly one callback however many times it is asked", func() {
+		first := ttsPCMCallback()
+
+		// One more than the table holds: a callback compiled per call panics
+		// with "purego: the maximum number of callbacks has been reached"
+		// before this loop ends, which is precisely the production failure.
+		for i := 0; i <= puregoCallbackTableSize; i++ {
+			Expect(ttsPCMCallback()).To(Equal(first),
+				"call %d returned a different callback, so a new one was compiled", i)
+		}
+	})
+
+	// A source-level assertion, deliberately, because the failure it guards
+	// against is invisible from inside the process: the way a per-request
+	// callback gets reintroduced is by someone calling purego.NewCallback at the
+	// synthesis site instead of going through ttsPCMCallback, and no in-process
+	// spec can reach that call without a MagpieTTS GGUF to synthesize with.
+	// Funnelling every compile through one accessor is what the whole design
+	// rests on, so the single call site is the invariant worth pinning.
+	It("compiles callbacks from exactly one place in the TTS path", func() {
+		fset := token.NewFileSet()
+		file, err := parser.ParseFile(fset, "tts.go", nil, 0)
+		Expect(err).ToNot(HaveOccurred())
+
+		// Counted over the syntax tree rather than by grepping the text: the
+		// doc comment on ttsPCMCallback names purego.NewCallback too, and a
+		// spec that cannot tell an explanation from a call would be pinning the
+		// prose.
+		var sites []string
+		ast.Inspect(file, func(n ast.Node) bool {
+			call, ok := n.(*ast.CallExpr)
+			if !ok {
+				return true
+			}
+			sel, ok := call.Fun.(*ast.SelectorExpr)
+			if !ok || sel.Sel.Name != "NewCallback" {
+				return true
+			}
+			if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "purego" {
+				sites = append(sites, fset.Position(call.Pos()).String())
+			}
+			return true
+		})
+		Expect(sites).To(HaveLen(1),
+			"every callback must be compiled through ttsPCMCallback, which memoises it")
+	})
+})
+
+var _ = Describe("the PCM sink table", func() {
+	It("routes a chunk to the sink registered for that id", func() {
+		var got []byte
+		id, release := ttsSinks.register(func(pcm []byte) bool {
+			got = pcm
+			return true
+		})
+		defer release()
+
+		src := []byte{1, 2, 3, 4}
+		Expect(ttsDeliverPCM(unsafe.Pointer(&src[0]), uint64(len(src)), id)).To(BeTrue())
+		Expect(got).To(Equal([]byte{1, 2, 3, 4}))
+	})
+
+	// The pointer addresses a std::string the runtime reuses for the next
+	// chunk, so a slice over it would be rewritten under the consumer.
+	It("copies the chunk out of the runtime's buffer", func() {
+		var got []byte
+		id, release := ttsSinks.register(func(pcm []byte) bool {
+			got = pcm
+			return true
+		})
+		defer release()
+
+		src := []byte{9, 8, 7}
+		Expect(ttsDeliverPCM(unsafe.Pointer(&src[0]), uint64(len(src)), id)).To(BeTrue())
+		src[0], src[1], src[2] = 0, 0, 0
+		Expect(got).To(Equal([]byte{9, 8, 7}))
+	})
+
+	It("gives each registration its own id", func() {
+		idA, releaseA := ttsSinks.register(func([]byte) bool { return true })
+		defer releaseA()
+		idB, releaseB := ttsSinks.register(func([]byte) bool { return true })
+		defer releaseB()
+
+		Expect(idA).ToNot(Equal(idB))
+		Expect(idA).ToNot(BeZero(), "id 0 is what a zeroed user_data would carry")
+		Expect(idB).ToNot(BeZero())
+	})
+
+	// Two models synthesizing at once share one callback, and engineMu is
+	// per-model, so nothing serialises them against each other.
+	It("keeps concurrent sinks apart", func() {
+		var mu sync.Mutex
+		got := map[uintptr][]byte{}
+
+		var wg sync.WaitGroup
+		for i := range 16 {
+			wg.Add(1)
+			go func() {
+				defer GinkgoRecover()
+				defer wg.Done()
+
+				src := []byte{byte(i)}
+				var mine []byte
+				id, release := ttsSinks.register(func(pcm []byte) bool {
+					mine = pcm
+					return true
+				})
+				defer release()
+
+				Expect(ttsDeliverPCM(unsafe.Pointer(&src[0]), 1, id)).To(BeTrue())
+				mu.Lock()
+				defer mu.Unlock()
+				got[id] = mine
+			}()
+		}
+		wg.Wait()
+
+		Expect(got).To(HaveLen(16))
+		for id, pcm := range got {
+			Expect(pcm).To(HaveLen(1), "sink %d received the wrong chunk", id)
+		}
+	})
+
+	// A released id means the request has returned. Answering true would leave
+	// the runtime synthesizing into nothing while the RPC that owns the lock
+	// waits for it.
+	It("cancels the synthesis when the sink is gone", func() {
+		id, release := ttsSinks.register(func([]byte) bool { return true })
+		release()
+
+		src := []byte{1}
+		Expect(ttsDeliverPCM(unsafe.Pointer(&src[0]), 1, id)).To(BeFalse())
+	})
+
+	It("cancels for a user_data that was never registered", func() {
+		src := []byte{1}
+		Expect(ttsDeliverPCM(unsafe.Pointer(&src[0]), 1, 0)).To(BeFalse())
+	})
+
+	It("accepts an empty chunk without touching the pointer", func() {
+		id, release := ttsSinks.register(func([]byte) bool {
+			Fail("an empty chunk must not reach the sink")
+			return true
+		})
+		defer release()
+
+		Expect(ttsDeliverPCM(nil, 0, id)).To(BeTrue())
+	})
+
+	It("passes the sink's cancellation back to the runtime", func() {
+		id, release := ttsSinks.register(func([]byte) bool { return false })
+		defer release()
+
+		src := []byte{1}
+		Expect(ttsDeliverPCM(unsafe.Pointer(&src[0]), 1, id)).To(BeFalse())
+	})
+})
+
+var _ = Describe("ttsModelConfig", func() {
+	// Three adjacent same-typed path fields: swapping two changes neither the
+	// struct size nor any offset, so abi_test.go's layout assertions cannot see
+	// it and the runtime would load the codec as the acoustic model.
+	It("assigns each path to its own field", func() {
+		cfg := ttsModelConfig(1, 2, 3, 4)
+		Expect(cfg.MagpieModel).To(Equal(uintptr(1)))
+		Expect(cfg.CodecModel).To(Equal(uintptr(2)))
+		Expect(cfg.TokenizerModelDir).To(Equal(uintptr(3)))
+		Expect(cfg.TextNormalizerModelDir).To(Equal(uintptr(4)))
+	})
+
+	// A config sent with the wrong size has every field past it ignored by
+	// HAS_FIELD, and the model loads with defaults instead of failing.
+	It("declares the size the runtime validates against", func() {
+		Expect(ttsModelConfig(1, 2, 3, 4).Size).To(Equal(unsafe.Sizeof(cTTSModelConfig{})))
+	})
+
+	It("leaves an unset text normalizer null", func() {
+		Expect(ttsModelConfig(1, 2, 3, 0).TextNormalizerModelDir).To(BeZero())
+	})
+})
+
+var _ = Describe("ttsRuntimeBackend", func() {
+	// -1 is this backend's documented "CPU" everywhere (asr.h: "-1 = CPU") and
+	// it is also the default, so it has to pin the preference rather than leave
+	// the runtime free to pick CUDA.
+	It("pins CPU for a negative gpu option", func() {
+		Expect(ttsRuntimeBackend(-1)).To(Equal(ttsBackendCPU))
+	})
+
+	// nemo_speech_tts_runtime_config has no device index at all, so a request
+	// for a particular device cannot be honoured and AUTO is the honest answer.
+	It("leaves the choice to the runtime when a device was named", func() {
+		Expect(ttsRuntimeBackend(0)).To(Equal(ttsBackendAuto))
+		Expect(ttsRuntimeBackend(3)).To(Equal(ttsBackendAuto))
+	})
+})
+
+var _ = Describe("WAV framing", func() {
+	// 16-bit mono little-endian, the format the runtime's callback delivers.
+	pcm := []byte{0x01, 0x00, 0xff, 0x7f, 0x00, 0x80}
+
+	It("writes a header the audio helpers can read back", func() {
+		out, err := wavFile(pcm, 22050)
+		Expect(err).ToNot(HaveOccurred())
+
+		body, rate := laudio.ParseWAV(out)
+		Expect(rate).To(Equal(22050))
+		Expect(body).To(Equal(pcm))
+	})
+
+	It("describes the payload it actually carries", func() {
+		out, err := wavFile(pcm, 22050)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(out).To(HaveLen(laudio.WAVHeaderSize + len(pcm)))
+
+		Expect(string(out[0:4])).To(Equal("RIFF"))
+		Expect(string(out[8:12])).To(Equal("WAVE"))
+		Expect(binary.LittleEndian.Uint32(out[4:8])).To(Equal(uint32(36 + len(pcm))))
+		Expect(binary.LittleEndian.Uint32(out[40:44])).To(Equal(uint32(len(pcm))))
+		Expect(binary.LittleEndian.Uint16(out[22:24])).To(Equal(uint16(1)), "mono")
+		Expect(binary.LittleEndian.Uint16(out[34:36])).To(Equal(uint16(16)), "16-bit")
+		Expect(binary.LittleEndian.Uint32(out[24:28])).To(Equal(uint32(22050)))
+		// byte rate = sample rate * block align, and a wrong one plays back at
+		// the wrong speed in players that trust it.
+		Expect(binary.LittleEndian.Uint32(out[28:32])).To(Equal(uint32(22050 * 2)))
+	})
+
+	It("carries whatever rate the synthesizer reported", func() {
+		out, err := wavFile(pcm, 44100)
+		Expect(err).ToNot(HaveOccurred())
+		_, rate := laudio.ParseWAV(out)
+		Expect(rate).To(Equal(44100))
+	})
+
+	Describe("the streaming header", func() {
+		It("is a complete header on its own", func() {
+			h := streamingWAVHeader(22050)
+			Expect(h).To(HaveLen(laudio.WAVHeaderSize))
+			Expect(string(h[0:4])).To(Equal("RIFF"))
+			Expect(string(h[8:12])).To(Equal("WAVE"))
+			Expect(binary.LittleEndian.Uint32(h[24:28])).To(Equal(uint32(22050)))
+		})
+
+		// NewWAVHeaderWithRate derives ChunkSize from the payload length, so
+		// leaving it alone would write 36 + 0xFFFFFFFF, which wraps to 35: a
+		// RIFF size smaller than the header itself.
+		It("leaves both sizes unknown rather than wrapping", func() {
+			h := streamingWAVHeader(22050)
+			Expect(binary.LittleEndian.Uint32(h[4:8])).To(Equal(uint32(0xFFFFFFFF)))
+			Expect(binary.LittleEndian.Uint32(h[40:44])).To(Equal(uint32(0xFFFFFFFF)))
+		})
+	})
+})
+
+var _ = Describe("synthesizeWAV", func() {
+	var dst string
+
+	BeforeEach(func() {
+		dst = filepath.Join(GinkgoT().TempDir(), "out.wav")
+	})
+
+	It("writes one WAV holding every chunk in order", func() {
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}, {2, 0}, {3, 0}}}
+		Expect(synthesizeWAV(s, &pb.TTSRequest{Text: "hello", Dst: dst}, "")).To(Succeed())
+
+		out, err := os.ReadFile(dst)
+		Expect(err).ToNot(HaveOccurred())
+		body, rate := laudio.ParseWAV(out)
+		Expect(rate).To(Equal(22050))
+		Expect(body).To(Equal([]byte{1, 0, 2, 0, 3, 0}))
+	})
+
+	It("hands the request and the model default language to the runtime", func() {
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}}}
+		req := &pb.TTSRequest{Text: "hello", Dst: dst, Voice: "Aria"}
+		Expect(synthesizeWAV(s, req, "it-IT")).To(Succeed())
+		Expect(s.gotReq).To(Equal(req))
+		Expect(s.gotLang).To(Equal("it-IT"))
+	})
+
+	It("rejects an empty text before it reaches the runtime", func() {
+		s := &fakeSynthesizer{rate: 22050}
+		err := synthesizeWAV(s, &pb.TTSRequest{Dst: dst}, "")
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(s.calls).To(BeZero())
+	})
+
+	// instructions has no equivalent in nemo_speech_tts_synthesis_options, which
+	// conditions on a speaker rather than a prose style. Dropping it must not
+	// fail the request: the caller still wants the audio it can have.
+	It("synthesizes anyway for a request carrying instructions it cannot honour", func() {
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}}}
+		instructions := "speak cheerfully"
+		Expect(synthesizeWAV(s, &pb.TTSRequest{
+			Text:         "hello",
+			Dst:          dst,
+			Instructions: &instructions,
+		}, "")).To(Succeed())
+		Expect(dst).To(BeAnExistingFile())
+	})
+
+	It("rejects a request with no destination", func() {
+		s := &fakeSynthesizer{rate: 22050}
+		err := synthesizeWAV(s, &pb.TTSRequest{Text: "hello"}, "")
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(s.calls).To(BeZero())
+	})
+
+	// A zero rate is what a null handle reports. The file it would produce is
+	// undecodable, and the synthesis that produced it would be wasted.
+	It("refuses to write a file at an unusable sample rate", func() {
+		s := &fakeSynthesizer{rate: 0, chunks: [][]byte{{1, 0}}}
+		err := synthesizeWAV(s, &pb.TTSRequest{Text: "hello", Dst: dst}, "")
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+		Expect(s.calls).To(BeZero())
+		Expect(dst).ToNot(BeAnExistingFile())
+	})
+
+	It("propagates a synthesis failure and writes nothing", func() {
+		boom := errors.New("boom")
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}}, err: boom}
+		Expect(synthesizeWAV(s, &pb.TTSRequest{Text: "hello", Dst: dst}, "")).To(MatchError(boom))
+		Expect(dst).ToNot(BeAnExistingFile())
+	})
+
+	// An empty WAV is a valid file, so this would otherwise reach the user as
+	// silence with no error anywhere.
+	It("fails rather than write a silent file when nothing was produced", func() {
+		s := &fakeSynthesizer{rate: 22050}
+		err := synthesizeWAV(s, &pb.TTSRequest{Text: "hello", Dst: dst}, "")
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+		Expect(dst).ToNot(BeAnExistingFile())
+	})
+
+	It("reports a destination it cannot write", func() {
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}}}
+		bad := filepath.Join(GinkgoT().TempDir(), "no-such-dir", "out.wav")
+		err := synthesizeWAV(s, &pb.TTSRequest{Text: "hello", Dst: bad}, "")
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+		Expect(err.Error()).To(ContainSubstring("out.wav"))
+	})
+})
+
+var _ = Describe("streamWAV", func() {
+	// drain collects everything streamWAV emits. The channel is buffered
+	// because streamWAV sends inline, so an unbuffered one would deadlock the
+	// spec rather than fail it.
+	drain := func(s synthesizer, req *pb.TTSRequest) ([][]byte, error) {
+		out := make(chan []byte, 16)
+		err := streamWAV(s, req, "", out)
+		close(out)
+
+		var got [][]byte
+		for c := range out {
+			got = append(got, c)
+		}
+		return got, err
+	}
+
+	It("emits the header first, then each chunk as it arrives", func() {
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}, {2, 0}}}
+		got, err := drain(s, &pb.TTSRequest{Text: "hello"})
+		Expect(err).ToNot(HaveOccurred())
+
+		Expect(got).To(HaveLen(3))
+		Expect(got[0]).To(Equal(streamingWAVHeader(22050)))
+		Expect(got[1]).To(Equal([]byte{1, 0}))
+		Expect(got[2]).To(Equal([]byte{2, 0}))
+	})
+
+	// pkg/grpc/server.go only ever sets Reply.Audio, so core/backend's own
+	// header branch (keyed on Reply.Message) never runs and a backend that
+	// emitted bare PCM would stream something no client could decode.
+	It("owns the header rather than leaving it to the caller", func() {
+		s := &fakeSynthesizer{rate: 44100, chunks: [][]byte{{1, 0}}}
+		got, err := drain(s, &pb.TTSRequest{Text: "hello"})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(string(got[0][0:4])).To(Equal("RIFF"))
+		Expect(binary.LittleEndian.Uint32(got[0][24:28])).To(Equal(uint32(44100)))
+	})
+
+	It("rejects an empty text before emitting anything", func() {
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}}}
+		got, err := drain(s, &pb.TTSRequest{})
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(got).To(BeEmpty())
+		Expect(s.calls).To(BeZero())
+	})
+
+	It("emits no header at an unusable sample rate", func() {
+		s := &fakeSynthesizer{rate: 0, chunks: [][]byte{{1, 0}}}
+		got, err := drain(s, &pb.TTSRequest{Text: "hello"})
+		Expect(status.Code(err)).To(Equal(codes.Internal))
+		Expect(got).To(BeEmpty())
+	})
+
+	It("propagates a synthesis failure after the chunks it did emit", func() {
+		boom := errors.New("boom")
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}}, err: boom}
+		got, err := drain(s, &pb.TTSRequest{Text: "hello"})
+		Expect(err).To(MatchError(boom))
+		Expect(got).To(HaveLen(2))
+	})
+
+	// streamWAV must not close the channel: TTSStream owns it, and closing in
+	// one of two places depending on how far the request got is how a stream
+	// ends up double-closed.
+	It("leaves the channel open for its caller to close", func() {
+		s := &fakeSynthesizer{rate: 22050, chunks: [][]byte{{1, 0}}}
+		out := make(chan []byte, 4)
+		Expect(streamWAV(s, &pb.TTSRequest{Text: "hello"}, "", out)).To(Succeed())
+		Expect(func() { close(out) }).ToNot(Panic())
+	})
+})
+
+var _ = Describe("the TTS RPCs", func() {
+	It("refuses TTS on a model loaded as another family", func() {
+		n := &NemoSpeech{fam: familyASR}
+		err := n.TTS(&pb.TTSRequest{Text: "hello", Dst: "/tmp/out.wav"})
+		Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+	})
+
+	It("refuses TTS on an unloaded model", func() {
+		n := &NemoSpeech{}
+		Expect(status.Code(n.TTS(&pb.TTSRequest{Text: "hello", Dst: "/tmp/out.wav"}))).
+			To(Equal(codes.Unimplemented))
+	})
+
+	It("releases the engine lock after a refusal", func() {
+		n := &NemoSpeech{fam: familyASR}
+		Expect(n.TTS(&pb.TTSRequest{Text: "hello", Dst: "/tmp/out.wav"})).ToNot(Succeed())
+		Expect(n.engineMu.TryLock()).To(BeTrue())
+		n.engineMu.Unlock()
+	})
+
+	// pkg/grpc/server.go drains this channel from a goroutine and then blocks
+	// on that goroutine finishing, so a channel left open does not fail the
+	// request, it hangs the RPC with the backend lock still held. Every exit
+	// path has to close it.
+	Describe("TTSStream channel closure", func() {
+		// streamed runs TTSStream the way the server does and returns once the
+		// channel has been closed, so a spec that hangs is a real hang.
+		streamed := func(n *NemoSpeech, req *pb.TTSRequest) ([][]byte, error) {
+			ch := make(chan []byte, 16)
+			done := make(chan [][]byte, 1)
+			go func() {
+				defer GinkgoRecover()
+				var got [][]byte
+				for c := range ch {
+					got = append(got, c)
+				}
+				done <- got
+			}()
+
+			err := n.TTSStream(req, ch)
+			var got [][]byte
+			Eventually(done).Should(Receive(&got))
+			return got, err
+		}
+
+		It("closes the channel when the family does not match", func() {
+			n := &NemoSpeech{fam: familyASR}
+			got, err := streamed(n, &pb.TTSRequest{Text: "hello"})
+			Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+			Expect(got).To(BeEmpty())
+		})
+
+		It("closes the channel when the model was never loaded", func() {
+			n := &NemoSpeech{}
+			_, err := streamed(n, &pb.TTSRequest{Text: "hello"})
+			Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+		})
+
+		// familyTTS with a zero handle: validation has to reject this before
+		// anything reaches the C entry points, which are nil function values
+		// until openLibraries has bound them.
+		It("closes the channel when the request is rejected", func() {
+			n := &NemoSpeech{fam: familyTTS}
+			got, err := streamed(n, &pb.TTSRequest{})
+			Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+			Expect(got).To(BeEmpty())
+		})
+
+		It("releases the engine lock afterwards", func() {
+			n := &NemoSpeech{fam: familyTTS}
+			_, err := streamed(n, &pb.TTSRequest{})
+			Expect(err).To(HaveOccurred())
+			Expect(n.engineMu.TryLock()).To(BeTrue())
+			n.engineMu.Unlock()
+		})
+	})
+
+	// The same guard on the offline path: a rejected request must not reach a
+	// nil C function through a zero handle.
+	It("rejects an invalid TTS request without touching the runtime", func() {
+		n := &NemoSpeech{fam: familyTTS}
+		Expect(status.Code(n.TTS(&pb.TTSRequest{Dst: "/tmp/out.wav"}))).To(Equal(codes.InvalidArgument))
+		Expect(status.Code(n.TTS(&pb.TTSRequest{Text: "hello"}))).To(Equal(codes.InvalidArgument))
+	})
+})

backend/go/omnivoice-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # omnivoice.cpp version
 OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=4f33af825d66e6ef1cb185e87b4589cacf747291
+OMNIVOICE_VERSION?=040c8b344d8c670ce1475194751d119b5ef82c78
 SO_TARGET?=libgomnivoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF