chore(dllm): bump dllm.cpp pin to P5 head

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.agents/dllm-backend.md

@@ -0,0 +1,138 @@
+# Working on the dllm Backend
+
+`mudler/dllm.cpp` is a standalone C++/ggml engine for DiffusionGemma
+block-diffusion models. LocalAI wraps it with a **pure-Go** backend at
+`backend/go/dllm/` that dlopens `libdllm.so` via purego (ebitengine/purego) -
+NOT cgo, and NOT a C++ grpc-server fork. The Go side owns chat templating
+(gemma4 renderer) and output parsing (gemma4 streaming parser) and implements
+the rich gRPC interface (`PredictRich`/`PredictStreamRich`, ChatDelta replies).
+
+> NOTE: github.com/mudler/dllm.cpp is still **private** (publishing is
+> planned). Until then the Makefile's anonymous clone fails; use the local-dev
+> symlink shortcut documented at the top of `backend/go/dllm/Makefile`
+> (symlink an out-of-tree `build/libdllm.so` into the backend dir and skip the
+> clone), or a git credential helper with repo access.
+
+## Pin
+
+`backend/go/dllm/Makefile` pins `DLLM_VERSION?=<sha>` at the top
+(whisper / parakeet-cpp / ds4 convention). The bump-deps bot
+(`.github/workflows/bump_deps.yaml`) tracks `mudler/dllm.cpp` `main` and
+rewrites that variable. After a manual bump: `make -C backend/go/dllm purge &&
+make -C backend/go/dllm` (the clone is keyed on the directory existing, not
+the sha).
+
+## C-ABI and the serialization contract
+
+The binding covers the 9-symbol flat C-ABI from dllm.cpp's
+`include/dllm_capi.h` (ABI v1; `main.go` hard-fails on a version mismatch):
+`abi_version, load, free, last_error, free_string, tokenize_json, generate,
+generate_stream, cancel`. Contract points the Go wiring encodes (`capi.go`
+header comment has the full list):
+
+- **One ctx = one concurrent generate/tokenize.** A per-model worker
+  goroutine (`Dllm.jobs` in `dllm.go`) owns ALL C calls, making the
+  serialization structural instead of lock discipline.
+- **`dllm_capi_cancel` is the ONE exception**: it only flips an atomic and may
+  be called from any goroutine mid-generate, so `Dllm.Cancel` bypasses the
+  worker queue. The flag resets at the start of each generate, so a watchdog
+  racing a new generate must re-issue cancel.
+- **`last_error` is a borrowed pointer** and must only be read AFTER the
+  failing call returned (never while a generate is in flight on the same ctx).
+- **Free vs in-flight requests**: requests hold `genMu.RLock` for their full
+  duration; `Free` takes the write lock, so it only runs when nothing is in
+  flight, then drains and closes the worker. Post-Free requests get a clean
+  "model not loaded" error.
+- `tokenize_json`/`generate` return malloc'd `char*` (bound as `uintptr`,
+  copied, then `dllm_capi_free_string`d); opts/params JSON must be a FLAT
+  object of scalars (`buildOptsJSON` rejects anything else).
+
+## Wire shape
+
+| RPC | Implementation |
+|---|---|
+| LoadModel | `dllm_capi_load` (params: `n_gpu_layers`, `n_threads`, `ctx_len`); `Options[]` parsed into per-request gen opts (`eb_*`, `blocks`, `kv_cache`) by `parseModelGenOpts` |
+| PredictRich | render (if templated) → `dllm_capi_generate` → parse → ONE Reply with aggregated ChatDeltas + legacy `Message` bytes |
+| PredictStreamRich | `dllm_capi_generate_stream`; per committed diffusion block → UTF-8 holdback → parser.Feed → one Reply per non-empty delta batch (channel closed by the CALLER, per `pkg/grpc/interface.go`) |
+| Predict / PredictStream | Legacy paths, delegate to the rich pair (legacy stream INVERTS channel ownership: the impl closes) |
+| TokenizeString | `dllm_capi_tokenize_json` (C side prepends BOS per `vocab.add_bos`) |
+| Cancel | `dllm_capi_cancel`, exposed as the `grpc.Cancellable` capability (`pkg/grpc/interface.go`): the gRPC server arms it via `context.AfterFunc` on the Predict/PredictStream context, so client disconnects/timeouts abort the in-flight generate - llama.cpp `IsCancelled()` parity for Go backends |
+
+`n_threads` and `ctx_len` are accepted-but-ignored by the engine at the
+current pin (the context bound comes from GGUF `n_ctx_train`); they are sent
+for forward compatibility.
+
+## Renderer / parser (the templated chat path)
+
+With `use_tokenizer_template` + raw Messages, the backend owns templating and
+parsing (the ds4 precedent, but in Go):
+
+- `gemma4_renderer.go` - `RenderGemma4(msgs, toolsJSON, enableThinking,
+  addGenerationPrompt)`. The file embeds the FULL `tokenizer.chat_template`
+  jinja (17466 bytes, md5 `8c34cf93c7a7815b3fdb300a009c4c17`) extracted
+  verbatim from `diffusiongemma-26B-A4B-it-BF16.gguf` via gguf-py - e.g.
+  `python scripts/dump_gguf.py model.gguf | grep -A400 chat_template` in the
+  dllm.cpp checkout - as a numbered comment block; every Go rule cites its
+  "tpl L<n>" line. Re-verify the md5 before blaming the renderer for a
+  mismatch with a new GGUF. **BOS exception**: the template emits
+  `{{- bos_token -}}` but the renderer deliberately does NOT - dllm.cpp's
+  `run_generate` tokenizes with `prepend_bos = vocab.add_bos` (true for
+  gemma4), so a literal `<bos>` would double it.
+- `gemma4_parser.go` - streaming state machine turning raw model text
+  (fragments can split anywhere, including mid-marker) into ChatDeltas:
+  thought channels → `reasoning_content`, `<|tool_call>call:name{...}` →
+  ToolCallDelta, `<turn|>` → done. Marker grammar cross-checked against vLLM
+  PR #45163's gemma4 tool/reasoning parsers. Malformed payloads are re-emitted
+  raw as content, never dropped.
+- Thinking is **opt-in** for this family (`Metadata["enable_thinking"]`,
+  default OFF - the inverse of ds4): the template gates every thinking branch
+  on `enable_thinking`, and the no-thinking render pre-closes an empty thought
+  channel, so the parser always starts in content state.
+- **UTF-8 boundary holdback** (`splitValidUTF8` in `dllm.go`): per-block
+  detokenization can split a multi-byte character across block boundaries, and
+  grpc-go refuses to marshal invalid UTF-8 in proto3 strings. An incomplete
+  trailing sequence (at most 3 bytes) is carried into the next block; genuinely
+  undecodable bytes become U+FFFD.
+
+Without `use_tokenizer_template`, the prompt passes through verbatim and the
+output is NOT gemma4-parsed (plain content, like any non-autoparsing backend).
+
+## Tests
+
+| Layer | Gate | What |
+|---|---|---|
+| `backend/go/dllm/*_test.go` (renderer/parser/wiring) | none - run in plain `go test ./backend/go/dllm/...` | Ginkgo specs over a fake `generator` seam; canonical renderer fixtures from transformers' `test_modeling_diffusion_gemma.py`, parser tables from the vLLM gemma4 parsers |
+| `backend/go/dllm/dllm_test.go` C-ABI smoke | `DLLM_TEST_LIBRARY` + `DLLM_TEST_TINY_MODEL` (dllm.cpp's `tests/fixtures/tiny_with_vocab.gguf`); Skips when unset | Drives the real `libdllm.so`: ABI check, load, tokenize `[2,18]`, deterministic generate, cancel (incl. mid-stream `Dllm.Cancel` aborting a deliberately slow `eb_max_steps:256` run in ~10ms) |
+| `tests/e2e-backends/dllm_test.go` | `BACKEND_TEST_DLLM=1` + `BACKEND_BINARY` (packaged run.sh) + `BACKEND_TEST_MODEL_FILE` (tiny fixture) | Templated chat round trip (Messages + UseTokenizerTemplate) over the real gRPC binary, non-streaming + streaming; plus client-context cancellation mid-stream (proves the `Cancellable` server plumbing end to end) |
+| Real-model e2e | `BACKEND_TEST_DLLM_REAL_MODEL_FILE` (26B BF16, ~50 GB) + `BACKEND_TEST_DLLM_REAL_GPU_LAYERS` | CUDA-13-class hardware only |
+
+Tool-call e2e is deliberately absent from the tiny-model spec: the fixture has
+random weights and cannot be coaxed into emitting tool markup; the unit tables
+carry that coverage.
+
+## Build matrix
+
+`cpu-dllm` (amd64 + arm64), `cuda13-dllm` (amd64), and
+`cuda13-nvidia-l4t-arm64-dllm` (arm64 CUDA: Jetson / DGX Spark GB10), via
+`.github/backend-matrix.yml`. No darwin/Metal. CUDA builds forward
+`-DDLLM_CUDA=ON` (dllm.cpp gates ggml's CUDA behind its own flag - a bare
+`-DGGML_CUDA=ON` is overridden by the cache FORCE). `libdllm.so` is
+self-contained (ggml statically absorbed, PIC), so `package.sh` only ships
+the binary, `run.sh` and that one .so (the parakeet-cpp-style stub layout;
+no ldd walk yet).
+
+## Known limitations
+
+- **Cancel granularity**: the C-ABI cancel flag is per-ctx and resets on
+  every generate entry, so a Cancel racing a NEW generate can be lost, and
+  with requests queued on the worker it aborts whichever generate is
+  currently running (acceptable: the server de-registers the hook on normal
+  completion, one process serves one model).
+- **Throughput**: ~0.15 tok/s on the 26B at default settings (GB10) - every
+  denoise step recomputes the full prompt+canvas. The upstream prefix-KV
+  cache (dllm.cpp P3) is the fix; `kv_cache:on` errors until it lands
+  (`auto`/`off` are accepted no-ops).
+- **Repo privacy**: see the note at the top - CI clone of dllm.cpp needs the
+  repo published (or credentials) before the backend images can build.
+- Engine spec/validation references: dllm.cpp `docs/validation.md` and
+  LocalAI `docs/superpowers/specs/2026-06-10-dllm-cpp-design.md`.

.github/backend-matrix.yml

@@ -703,19 +703,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-locate-anything-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -781,19 +768,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-omnivoice-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1569,19 +1543,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-locate-anything-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1608,19 +1569,6 @@ include:
     backend: "rfdetr-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-locate-anything-cpp'
-    base-image: "ubuntu:24.04"
-    ubuntu-version: '2404'
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1660,6 +1608,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-dllm'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "dllm"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1699,6 +1660,19 @@ include:
     backend: "parakeet-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-dllm'
+    base-image: "ubuntu:24.04"
+    ubuntu-version: '2404'
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "dllm"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1725,19 +1699,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-omnivoice-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1777,19 +1738,6 @@ include:
     backend: "qwen3-tts-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-omnivoice-cpp'
-    base-image: "ubuntu:24.04"
-    ubuntu-version: '2404'
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2884,74 +2832,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  # locate-anything-cpp
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-locate-anything-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'sycl_f32'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f32-locate-anything-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'sycl_f16'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f16-locate-anything-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-locate-anything-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-locate-anything-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3045,19 +2925,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2204'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-arm64-locate-anything-cpp'
-    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "locate-anything-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2204'
   # whisper
   - build-type: ''
     cuda-major-version: ""
@@ -3304,6 +3171,35 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # dllm
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-dllm'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "dllm"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-dllm'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "dllm"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3522,35 +3418,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  # omnivoice-cpp
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-omnivoice-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-omnivoice-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3564,19 +3431,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f32'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f32-omnivoice-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3590,19 +3444,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f16'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f16-omnivoice-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3617,20 +3458,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-omnivoice-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3645,20 +3472,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-omnivoice-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -3672,19 +3485,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2204'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-arm64-omnivoice-cpp'
-    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2204'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3698,19 +3498,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'hipblas'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-rocm-hipblas-omnivoice-cpp'
-    base-image: "rocm/dev-ubuntu-24.04:6.4.4"
-    runs-on: 'ubuntu-latest'
-    skip-drivers: 'false'
-    backend: "omnivoice-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   # vibevoice-cpp
   - build-type: ''
     cuda-major-version: ""
@@ -4541,10 +4328,6 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-qwen3-tts-cpp"
     build-type: "metal"
     lang: "go"
-  - backend: "omnivoice-cpp"
-    tag-suffix: "-metal-darwin-arm64-omnivoice-cpp"
-    build-type: "metal"
-    lang: "go"
   - backend: "vibevoice-cpp"
     tag-suffix: "-metal-darwin-arm64-vibevoice-cpp"
     build-type: "metal"
@@ -4613,10 +4396,6 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-silero-vad"
     build-type: "metal"
     lang: "go"
-  - backend: "sherpa-onnx"
-    tag-suffix: "-metal-darwin-arm64-sherpa-onnx"
-    build-type: "metal"
-    lang: "go"
   - backend: "local-store"
     tag-suffix: "-metal-darwin-arm64-local-store"
     build-type: "metal"
@@ -4624,9 +4403,3 @@ includeDarwin:
   - backend: "llama-cpp-quantization"
     tag-suffix: "-metal-darwin-arm64-llama-cpp-quantization"
     build-type: "mps"
-  - backend: "speaker-recognition"
-    tag-suffix: "-metal-darwin-arm64-speaker-recognition"
-    build-type: "mps"
-  - backend: "ds4"
-    tag-suffix: "-metal-darwin-arm64-ds4"
-    lang: "go"

.github/workflows/bump_deps.yaml

@@ -38,6 +38,10 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
+          - repository: "mudler/dllm.cpp"
+            variable: "DLLM_VERSION"
+            branch: "main"
+            file: "backend/go/dllm/Makefile"
           - repository: "leejet/stable-diffusion.cpp"
             variable: "STABLEDIFFUSION_GGML_VERSION"
             branch: "master"
@@ -62,18 +66,10 @@ jobs:
             variable: "RFDETR_VERSION"
             branch: "main"
             file: "backend/go/rfdetr-cpp/Makefile"
-          - repository: "mudler/locate-anything.cpp"
-            variable: "LOCATEANYTHING_VERSION"
-            branch: "master"
-            file: "backend/go/locate-anything-cpp/Makefile"
           - repository: "predict-woo/qwen3-tts.cpp"
             variable: "QWEN3TTS_CPP_VERSION"
             branch: "main"
             file: "backend/go/qwen3-tts-cpp/Makefile"
-          - repository: "ServeurpersoCom/omnivoice.cpp"
-            variable: "OMNIVOICE_VERSION"
-            branch: "master"
-            file: "backend/go/omnivoice-cpp/Makefile"
           - repository: "localai-org/vibevoice.cpp"
             variable: "VIBEVOICE_CPP_VERSION"
             branch: "master"

.github/workflows/test-extra.yml

@@ -38,7 +38,6 @@ jobs:
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
       qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
       rfdetr-cpp: ${{ steps.detect.outputs.rfdetr-cpp }}
-      locate-anything-cpp: ${{ steps.detect.outputs.locate-anything-cpp }}
       vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
       localvqe: ${{ steps.detect.outputs.localvqe }}
       voxtral: ${{ steps.detect.outputs.voxtral }}
@@ -564,7 +563,7 @@ jobs:
       - name: Run e2e-backends smoke
         env:
           BACKEND_IMAGE: quay.io/go-skynet/local-ai-backends:master-cpu-llama-cpp
-          BACKEND_TEST_CAPS: health,load,predict,stream,logprobs,logit_bias,tokenize
+          BACKEND_TEST_CAPS: health,load,predict,stream,logprobs,logit_bias
         run: |
           make test-extra-backend
   # Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked LLM.
@@ -902,45 +901,6 @@ jobs:
       - name: Test rfdetr-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/rfdetr-cpp test
-  # Per-backend e2e for locate-anything-cpp: builds the .so + Go binary and
-  # runs `make -C backend/go/locate-anything-cpp test`. test.sh fetches the
-  # locate-anything-q8_0 GGUF (~6.3 GB, NVIDIA LocateAnything-3B) from the
-  # published mudler/locate-anything.cpp-gguf HF repo + a COCO image, then the
-  # Go wire test loads the model and runs an open-vocabulary Detect, asserting
-  # at least one labeled box. Heavier than the other Go backends (it is a 3B),
-  # so it is gated to changes under backend/go/locate-anything-cpp/.
-  tests-locate-anything-cpp:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.locate-anything-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl libopenblas-dev
-      - name: Setup Go
-        uses: actions/setup-go@v5
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Build locate-anything-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/locate-anything-cpp
-      - name: Test locate-anything-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/locate-anything-cpp test
   # Per-backend smoke for vibevoice-cpp: builds the .so + Go binary and
   # runs `make -C backend/go/vibevoice-cpp test`. test.sh auto-downloads
   # the published mudler/vibevoice.cpp-models bundle (TTS Q8_0 + ASR Q4_K

AGENTS.md

@@ -26,6 +26,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 | [.agents/vllm-backend.md](.agents/vllm-backend.md) | Working on the vLLM / vLLM-omni backends — native parsers, ChatDelta, CPU build, libnuma packaging, backend hooks |
 | [.agents/sglang-backend.md](.agents/sglang-backend.md) | Working on the SGLang backend — `engine_args` validation against ServerArgs, speculative-decoding (EAGLE/EAGLE3/DFLASH/MTP) recipes, parser handling |
 | [.agents/ds4-backend.md](.agents/ds4-backend.md) | Working on the ds4 backend - DSML state machine, thinking modes, KV cache, Metal+CUDA matrix |
+| [.agents/dllm-backend.md](.agents/dllm-backend.md) | Working on the dllm backend (DiffusionGemma block-diffusion) - purego C-ABI binding, per-ctx serialization contract, gemma4 renderer/parser, gated test layers |
 | [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
 | [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
 | [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |

Dockerfile

@@ -108,7 +108,6 @@ RUN <<EOT bash
         apt-get update && \
         apt-get install -y --no-install-recommends \
             cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            cuda-nvrtc-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/dllm backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -566,7 +566,6 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/speaker-recognition
 	$(MAKE) -C backend/rust/kokoros kokoros-grpc
 	$(MAKE) -C backend/go/rfdetr-cpp
-	$(MAKE) -C backend/go/locate-anything-cpp
 
 test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/transformers test
@@ -594,7 +593,6 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/speaker-recognition test
 	$(MAKE) -C backend/rust/kokoros test
 	$(MAKE) -C backend/go/rfdetr-cpp test
-	$(MAKE) -C backend/go/locate-anything-cpp test
 
 ##
 ## End-to-end gRPC tests that exercise a built backend container image.
@@ -1173,10 +1171,12 @@ BACKEND_STABLEDIFFUSION_GGML = stablediffusion-ggml|golang|.|--progress=plain|tr
 BACKEND_WHISPER = whisper|golang|.|false|true
 BACKEND_CRISPASR = crispasr|golang|.|false|true
 BACKEND_PARAKEET_CPP = parakeet-cpp|golang|.|false|true
+# dllm is mudler/dllm.cpp, the DiffusionGemma block-diffusion engine,
+# wrapped by the purego backend at backend/go/dllm.
+BACKEND_DLLM = dllm|golang|.|false|true
 BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
-BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
 BACKEND_LOCALVQE = localvqe|golang|.|false|true
 BACKEND_OPUS = opus|golang|.|false|true
@@ -1263,6 +1263,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_STABLEDIFFUSION_GGML)))
 $(eval $(call generate-docker-build-target,$(BACKEND_WHISPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_CRISPASR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PARAKEET_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_DLLM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VOXTRAL)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OPUS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_RERANKERS)))
@@ -1295,7 +1296,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX)))
@@ -1313,7 +1313,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy
 
 ########################################################
 ### Mock Backend for E2E Tests

README.md

@@ -165,10 +165,6 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
 
 ## Latest News
 
-- **June 2026**: New [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (a tiny Go client for the Realtime API with a full talk-back voice loop and tool calling), plus [streaming of the realtime LLM / TTS / transcription pipeline stages](https://github.com/mudler/LocalAI/pull/10176) and [configurable WebRTC ICE candidates](https://github.com/mudler/LocalAI/pull/10231).
-- **June 2026**: Big speech push: the [parakeet.cpp](https://github.com/mudler/parakeet.cpp) ASR engine gains [NeMo-faithful segment timestamps](https://github.com/mudler/LocalAI/pull/10207), a [multilingual streaming Nemotron-3.5 model](https://github.com/mudler/LocalAI/pull/10199), [dynamic batching for concurrent transcription](https://github.com/mudler/LocalAI/pull/10112) and [CUDA graphs](https://github.com/mudler/LocalAI/pull/10273); the new [CrispASR backend](https://github.com/mudler/LocalAI/pull/10099) adds multi-architecture ASR + TTS, and [60 Piper TTS voices across 42 languages](https://github.com/mudler/LocalAI/pull/10296) land in the gallery (plus [per-request TTS instructions and params](https://github.com/mudler/LocalAI/pull/10172)).
-- **June 2026**: New backends and models: [locate-anything.cpp](https://github.com/mudler/LocalAI/pull/10264) for open-vocabulary object detection via ggml, [Ideogram4 image generation](https://github.com/mudler/LocalAI/pull/10201) in stablediffusion-ggml, [llama.cpp video input](https://github.com/mudler/LocalAI/pull/10216), and the [Gemma 4 QAT family with MTP speculative-decoding pairs](https://github.com/mudler/LocalAI/pull/10215). Plus an [interactive CLI chat mode](https://github.com/mudler/LocalAI/pull/10226) and [RAG source citations in agent responses](https://github.com/mudler/LocalAI/pull/10228).
-- **June 2026**: Distributed mode hardening: [prefix-cache-aware routing](https://github.com/mudler/LocalAI/pull/10071), a [production-ready request router with auto-sized embedding/rerank batches](https://github.com/mudler/LocalAI/pull/10104), [ds4 layer-split distributed inference](https://github.com/mudler/LocalAI/pull/10098), [NATS JWT auth + TLS/mTLS](https://github.com/mudler/LocalAI/pull/10159), and [resumable file uploads](https://github.com/mudler/LocalAI/pull/10109).
 - **May 2026**: **LocalAI 4.3.0** - `llama.cpp` [prompt cache on by default](https://github.com/mudler/LocalAI/pull/9925) (repeated system prompts collapse from minutes to seconds), [keyless cosign signing of backend OCI images](https://github.com/mudler/LocalAI/pull/9823), [per-API-key + per-user usage attribution](https://github.com/mudler/LocalAI/pull/9920), Distributed v3 with [per-request replica routing](https://github.com/mudler/LocalAI/pull/9968). [Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.3.0)
 - **May 2026**: **LocalAI 4.2.0** - LocalAI sees and hears: [voice recognition](https://github.com/mudler/LocalAI/pull/9500), [face recognition + antispoofing liveness](https://github.com/mudler/LocalAI/pull/9480), speaker diarization. Plus [drop-in Ollama API](https://github.com/mudler/LocalAI/pull/9284), [video generation](https://github.com/mudler/LocalAI/pull/9420), redesigned UI with i18n + admin-configurable branding, vLLM at feature parity with llama.cpp, and 11 new backends. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.2.0)
 - **April 2026**: **LocalAI 4.1.0** - LocalAI becomes a control tower: distributed cluster mode with VRAM-aware smart routing + autoscaling, multi-user platform with OIDC and API keys, per-user quotas with predictive analytics, in-UI fine-tuning with TRL (auto-export to GGUF), on-the-fly quantization backend, visual pipeline editor. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.1.0)
@@ -221,7 +217,7 @@ See the full [Backend & Model Compatibility Table](https://localai.io/model-comp
 - [Integrations & community projects](https://localai.io/docs/integrations/)
 - [Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)
 - [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
-- [Examples](https://github.com/mudler/LocalAI-examples) — including the [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (Go client for the Realtime API with tool calling)
+- [Examples](https://github.com/mudler/LocalAI-examples)
 
 ## Team
 

backend/Dockerfile.golang

@@ -206,16 +206,6 @@ RUN if [ "${BACKEND}" = "opus" ]; then \
     apt-get clean && rm -rf /var/lib/apt/lists/*; \
 fi
 
-# CrispASR's piper TTS backend dlopens libespeak-ng at runtime to phonemize
-# non-English text (the MIT-clean path; English uses a built-in G2P). Install
-# the espeak-ng runtime + its libpcaudio/libsonic deps + voice data so
-# package.sh can bundle them into the FROM scratch image.
-RUN if [ "${BACKEND}" = "crispasr" ]; then \
-    apt-get update && apt-get install -y --no-install-recommends \
-        espeak-ng-data libespeak-ng1 libpcaudio0 libsonic0 && \
-    apt-get clean && rm -rf /var/lib/apt/lists/*; \
-fi
-
 COPY . /LocalAI
 
 RUN git config --global --add safe.directory /LocalAI

backend/Dockerfile.python

@@ -126,7 +126,6 @@ RUN <<EOT bash
         apt-get update && \
         apt-get install -y --no-install-recommends \
             cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            cuda-nvrtc-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=d881f2a05e8ff6bec001315a36b794b4aa310173
+# Upstream pin lives below as DS4_VERSION?=8384adf0f9fa0f3bb342dd925372de778b95b263
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=d881f2a05e8ff6bec001315a36b794b4aa310173
+DS4_VERSION?=8384adf0f9fa0f3bb342dd925372de778b95b263
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=4c6595503fe45d5a39f88d194e270f64c7424677
+LLAMA_VERSION?=039e20a2db9e87b2477c76cc04905f3e1acad77f
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/grpc-server.cpp

@@ -3486,7 +3486,7 @@ public:
         if (body.count("prompt") != 0) {
             const bool add_special = json_value(body, "add_special", false);
 
-            llama_tokens tokens = tokenize_mixed(ctx_server.impl->vocab, body.at("prompt"), add_special, true);
+            llama_tokens tokens = tokenize_mixed(ctx_server.impl->vocab, body.at("content"), add_special, true);
 
 
             for (const auto& token : tokens) {

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=d745bda4386ae0f9d1d2f23fff8ec95d76428221
+CRISPASR_VERSION?=c29f6653a516a3001d923944dad8892072cc7334
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/crispasr/gocrispasr.go

@@ -11,7 +11,6 @@ import (
 
 	"github.com/go-audio/audio"
 	"github.com/go-audio/wav"
-	gguf "github.com/gpustack/gguf-parser-go"
 	"github.com/mudler/LocalAI/pkg/grpc/base"
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 	"github.com/mudler/LocalAI/pkg/utils"
@@ -38,39 +37,6 @@ var (
 
 type CrispASR struct {
 	base.SingleThread
-	// sampleRate is the output rate (Hz) of the loaded TTS engine's PCM, used to
-	// write a correct WAV header. Most CrispASR TTS backends emit 24 kHz, but
-	// piper returns its model's native rate (16 kHz for x_low/low voices,
-	// 22.05 kHz for medium/high), so it is read from the GGUF metadata at Load.
-	sampleRate int
-}
-
-// defaultTTSSampleRate is the output rate assumed for CrispASR TTS engines that
-// don't advertise one in GGUF metadata (vibevoice/orpheus/chatterbox/qwen3-tts
-// all emit 24 kHz). piper is the exception and carries piper.sample_rate.
-const defaultTTSSampleRate = 24000
-
-// piperSampleRate reads the piper.sample_rate metadata key from a GGUF model.
-// CrispASR's piper backend returns PCM at the model's native rate without
-// resampling, so the WAV header must match it. Returns ok=false for non-piper
-// models (key absent) or an unreadable file, letting the caller fall back to
-// defaultTTSSampleRate.
-func piperSampleRate(modelPath string) (int, bool) {
-	// Only scalar architecture keys are read, so skip the large array metadata
-	// (phoneme map) and mmap the header - same rationale as pkg/vram's reader.
-	f, err := gguf.ParseGGUFFile(modelPath, gguf.UseMMap(), gguf.SkipLargeMetadata())
-	if err != nil {
-		return 0, false
-	}
-	kv, ok := f.Header.MetadataKV.Get("piper.sample_rate")
-	if !ok || kv.ValueType != gguf.GGUFMetadataValueTypeUint32 {
-		return 0, false
-	}
-	rate := int(kv.ValueUint32())
-	if rate <= 0 {
-		return 0, false
-	}
-	return rate, true
 }
 
 // splitOption splits a "prefix:value" model option into its key and value,
@@ -137,14 +103,6 @@ func (w *CrispASR) Load(opts *pb.ModelOptions) error {
 		return fmt.Errorf("Failed to load CrispASR transcription model")
 	}
 
-	// Determine the TTS output sample rate for the WAV header. piper voices
-	// carry their native rate in GGUF metadata and CrispASR does not resample;
-	// every other engine emits the 24 kHz default.
-	w.sampleRate = defaultTTSSampleRate
-	if rate, ok := piperSampleRate(opts.ModelFile); ok {
-		w.sampleRate = rate
-	}
-
 	// Load the companion file (codec/tokenizer/s3gen) after the session is open.
 	// rc==0 means success or "not applicable" for the active backend; only a
 	// negative code is fatal.
@@ -432,7 +390,7 @@ func (w *CrispASR) synthesize(text string) ([]float32, error) {
 	}
 	defer CppTTSFree(ptr)
 	src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) //nolint:govet // ptr addresses C-allocated PCM returned across the purego boundary; copied out immediately below, before tts_free.
-	out := make([]float32, int(n))                               // copy out of C memory before free
+	out := make([]float32, int(n)) // copy out of C memory before free
 	copy(out, src)
 	return out, nil
 }
@@ -459,7 +417,7 @@ func (w *CrispASR) TTS(req *pb.TTSRequest) error {
 	if err != nil {
 		return err
 	}
-	return writeWAV(req.Dst, pcm, w.sampleRate)
+	return writeWAV24k(req.Dst, pcm)
 }
 
 // TTSStream is the streaming counterpart to TTS. CrispASR has no progressive
@@ -489,7 +447,7 @@ func (w *CrispASR) TTSStream(req *pb.TTSRequest, results chan []byte) error {
 	}
 	defer func() { _ = os.Remove(dst) }()
 
-	if err := writeWAV(dst, pcm, w.sampleRate); err != nil {
+	if err := writeWAV24k(dst, pcm); err != nil {
 		return err
 	}
 
@@ -501,14 +459,14 @@ func (w *CrispASR) TTSStream(req *pb.TTSRequest, results chan []byte) error {
 	return nil
 }
 
-// writeWAV writes pcm as a sampleRate Hz, mono, 16-bit PCM WAV at dst.
-func writeWAV(dst string, pcm []float32, sampleRate int) error {
+// writeWAV24k writes pcm as a 24000 Hz, mono, 16-bit PCM WAV at dst.
+func writeWAV24k(dst string, pcm []float32) error {
 	f, err := os.Create(dst)
 	if err != nil {
 		return fmt.Errorf("crispasr: create %q: %w", dst, err)
 	}
 
-	enc := wav.NewEncoder(f, sampleRate, 16, 1, 1)
+	enc := wav.NewEncoder(f, 24000, 16, 1, 1)
 	ints := make([]int, len(pcm))
 	for i, s := range pcm {
 		if s > 1 {
@@ -519,7 +477,7 @@ func writeWAV(dst string, pcm []float32, sampleRate int) error {
 		ints[i] = int(s * 32767)
 	}
 	buf := &audio.IntBuffer{
-		Format:         &audio.Format{NumChannels: 1, SampleRate: sampleRate},
+		Format:         &audio.Format{NumChannels: 1, SampleRate: 24000},
 		Data:           ints,
 		SourceBitDepth: 16,
 	}

backend/go/crispasr/gocrispasr_samplerate_test.go

@@ -1,164 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"encoding/binary"
-	"os"
-	"path/filepath"
-
-	"github.com/go-audio/wav"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-// GGUF metadata value type tags (subset) from the GGUF spec.
-const (
-	ggufTypeUint32 uint32 = 4
-	ggufTypeString uint32 = 8
-)
-
-type ggufKV struct {
-	key   string
-	vtype uint32
-	val   any
-}
-
-// writeMinimalGGUF emits a valid, tensor-less GGUF file carrying only the given
-// metadata key-values. Enough for the header-only parse path piperSampleRate
-// uses; avoids pulling a real multi-MB voice into the test.
-func writeMinimalGGUF(path string, kvs []ggufKV) error {
-	var b bytes.Buffer
-	b.WriteString("GGUF")                                // magic
-	_ = binary.Write(&b, binary.LittleEndian, uint32(3)) // version
-	_ = binary.Write(&b, binary.LittleEndian, uint64(0)) // tensor count
-	_ = binary.Write(&b, binary.LittleEndian, uint64(len(kvs)))
-	for _, kv := range kvs {
-		_ = binary.Write(&b, binary.LittleEndian, uint64(len(kv.key)))
-		b.WriteString(kv.key)
-		_ = binary.Write(&b, binary.LittleEndian, kv.vtype)
-		switch v := kv.val.(type) {
-		case uint32:
-			_ = binary.Write(&b, binary.LittleEndian, v)
-		case string:
-			_ = binary.Write(&b, binary.LittleEndian, uint64(len(v)))
-			b.WriteString(v)
-		}
-	}
-	return os.WriteFile(path, b.Bytes(), 0o644)
-}
-
-// wavSampleRate decodes the WAV header at path and returns its sample rate.
-func wavSampleRate(path string) (int, error) {
-	f, err := os.Open(path)
-	if err != nil {
-		return 0, err
-	}
-	defer func() { _ = f.Close() }()
-	dec := wav.NewDecoder(f)
-	dec.ReadInfo()
-	return int(dec.SampleRate), nil
-}
-
-var _ = Describe("piper sample rate", func() {
-	Context("piperSampleRate", func() {
-		It("reads piper.sample_rate from a piper GGUF (medium = 22050)", func() {
-			p := filepath.Join(GinkgoT().TempDir(), "voice.gguf")
-			Expect(writeMinimalGGUF(p, []ggufKV{
-				{key: "general.architecture", vtype: ggufTypeString, val: "piper"},
-				{key: "piper.sample_rate", vtype: ggufTypeUint32, val: uint32(22050)},
-			})).To(Succeed())
-
-			rate, ok := piperSampleRate(p)
-			Expect(ok).To(BeTrue(), "piper.sample_rate should be found")
-			Expect(rate).To(Equal(22050))
-		})
-
-		It("reads the low-quality rate (16000)", func() {
-			p := filepath.Join(GinkgoT().TempDir(), "voice.gguf")
-			Expect(writeMinimalGGUF(p, []ggufKV{
-				{key: "piper.sample_rate", vtype: ggufTypeUint32, val: uint32(16000)},
-			})).To(Succeed())
-
-			rate, ok := piperSampleRate(p)
-			Expect(ok).To(BeTrue())
-			Expect(rate).To(Equal(16000))
-		})
-
-		It("returns ok=false for a non-piper GGUF (no piper.sample_rate key)", func() {
-			p := filepath.Join(GinkgoT().TempDir(), "other.gguf")
-			Expect(writeMinimalGGUF(p, []ggufKV{
-				{key: "general.architecture", vtype: ggufTypeString, val: "vibevoice"},
-			})).To(Succeed())
-
-			_, ok := piperSampleRate(p)
-			Expect(ok).To(BeFalse())
-		})
-
-		It("returns ok=false for an unreadable/non-GGUF file", func() {
-			p := filepath.Join(GinkgoT().TempDir(), "garbage.gguf")
-			Expect(os.WriteFile(p, []byte("not a gguf"), 0o644)).To(Succeed())
-
-			_, ok := piperSampleRate(p)
-			Expect(ok).To(BeFalse())
-		})
-	})
-
-	// End-to-end through the built .so. Gated on CRISPASR_PIPER_MODEL_PATH (a
-	// real piper voice GGUF) like the other model-backed specs; never runs in
-	// default CI. Proves CrispASR's piper backend output rate flows into the
-	// WAV header instead of the hardcoded 24 kHz default.
-	Context("piper TTS end-to-end", func() {
-		It("writes the WAV at the model's native piper.sample_rate", func() {
-			model := os.Getenv("CRISPASR_PIPER_MODEL_PATH")
-			if model == "" {
-				Skip("set CRISPASR_PIPER_MODEL_PATH to run the piper e2e spec")
-			}
-			ensureLibLoaded()
-
-			expected, ok := piperSampleRate(model)
-			Expect(ok).To(BeTrue(), "model should carry piper.sample_rate metadata")
-
-			w := &CrispASR{}
-			Expect(w.Load(&pb.ModelOptions{
-				ModelFile: model,
-				Options:   []string{"backend:piper"},
-				Threads:   4,
-			})).To(Succeed())
-
-			dst := filepath.Join(GinkgoT().TempDir(), "piper.wav")
-			Expect(w.TTS(&pb.TTSRequest{Text: "Hello from CrispASR piper.", Dst: dst})).To(Succeed())
-
-			info, err := os.Stat(dst)
-			Expect(err).ToNot(HaveOccurred())
-			Expect(info.Size()).To(BeNumerically(">", 1024), "expected a non-trivial WAV")
-
-			rate, err := wavSampleRate(dst)
-			Expect(err).ToNot(HaveOccurred())
-			Expect(rate).To(Equal(expected),
-				"WAV header rate must equal the model's native piper.sample_rate, not the 24k default")
-		})
-	})
-
-	Context("writeWAV", func() {
-		It("writes the WAV header at the given sample rate (22050 for piper, not the 24k default)", func() {
-			dst := filepath.Join(GinkgoT().TempDir(), "out.wav")
-			pcm := make([]float32, 220) // 10 ms of silence is enough for a header
-			Expect(writeWAV(dst, pcm, 22050)).To(Succeed())
-
-			rate, err := wavSampleRate(dst)
-			Expect(err).ToNot(HaveOccurred())
-			Expect(rate).To(Equal(22050))
-		})
-
-		It("writes a 16000 Hz header for low-quality piper voices", func() {
-			dst := filepath.Join(GinkgoT().TempDir(), "out.wav")
-			pcm := make([]float32, 160)
-			Expect(writeWAV(dst, pcm, 16000)).To(Succeed())
-
-			rate, err := wavSampleRate(dst)
-			Expect(err).ToNot(HaveOccurred())
-			Expect(rate).To(Equal(16000))
-		})
-	})
-})

backend/go/crispasr/package.sh

@@ -51,32 +51,6 @@ else
     exit 1
 fi
 
-# Bundle espeak-ng (+ its libpcaudio/libsonic runtime deps) and its voice data so
-# the piper TTS backend can phonemize non-English text. CrispASR dlopens
-# libespeak-ng.so.1 at runtime (the MIT-clean path); the dlopen succeeds loading
-# libespeak-ng but FAILS if libpcaudio/libsonic are absent, so all three .so are
-# required. run.sh points CRISPASR_ESPEAK_DATA_PATH at the bundled data dir.
-# Best-effort: only copied when present, so a local dev build without espeak-ng
-# installed still packages the rest (English voices keep working).
-ESPEAK_LIBDIR=""
-for d in /usr/lib/x86_64-linux-gnu /usr/lib/aarch64-linux-gnu; do
-    if [ -f "$d/libespeak-ng.so.1" ]; then
-        ESPEAK_LIBDIR="$d"
-        break
-    fi
-done
-if [ -n "$ESPEAK_LIBDIR" ]; then
-    echo "Bundling espeak-ng from $ESPEAK_LIBDIR ..."
-    cp -arfLv "$ESPEAK_LIBDIR/libespeak-ng.so.1" $CURDIR/package/lib/
-    cp -arfLv "$ESPEAK_LIBDIR/libpcaudio.so.0" $CURDIR/package/lib/
-    cp -arfLv "$ESPEAK_LIBDIR/libsonic.so.0" $CURDIR/package/lib/
-    if [ -d "$ESPEAK_LIBDIR/espeak-ng-data" ]; then
-        cp -arfLv "$ESPEAK_LIBDIR/espeak-ng-data" $CURDIR/package/
-    fi
-else
-    echo "espeak-ng not found; non-English piper voices will not phonemize"
-fi
-
 # Package GPU libraries based on BUILD_TYPE
 # The GPU library packaging script will detect BUILD_TYPE and copy appropriate GPU libraries
 GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"

backend/go/crispasr/run.sh

@@ -41,11 +41,6 @@ fi
 export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
 export CRISPASR_LIBRARY=$LIBRARY
 
-# Point piper's espeak-ng phonemizer at the bundled voice data. The variable
-# names the directory CONTAINING espeak-ng-data (package.sh drops it next to
-# this script). Harmless when espeak-ng wasn't bundled.
-export CRISPASR_ESPEAK_DATA_PATH=$CURDIR
-
 # If there is a lib/ld.so, use it
 if [ -f $CURDIR/lib/ld.so ]; then
 	echo "Using lib/ld.so"

backend/go/dllm/.gitignore

@@ -0,0 +1,10 @@
+.cache/
+sources/
+build/
+package/
+dllm-grpc
+# build artifacts staged in-tree by the Makefile (cp from sources/) or
+# symlinked for local dev; the real sources live in dllm.cpp upstream.
+*.so
+*.so.*
+compile_commands.json

backend/go/dllm/Makefile

@@ -0,0 +1,101 @@
+# dllm backend Makefile.
+#
+# Upstream pin lives below as DLLM_VERSION?=<sha> so .github/bump_deps.sh
+# can find and update it - matches the whisper.cpp / parakeet-cpp / ds4
+# convention.
+#
+# Local dev shortcut: if you already have an out-of-tree dllm.cpp build,
+# you can symlink the .so into this directory and skip the clone/cmake
+# steps entirely, e.g.:
+#
+#   ln -sf /path/to/dllm.cpp/build/libdllm.so .
+#   go build -o dllm-grpc .
+#
+# That's what the gated C-ABI binding smoke uses (DLLM_TEST_LIBRARY). The
+# default target below does the proper clone-at-pin + cmake build so CI
+# doesn't need a side-checkout.
+#
+# NOTE: github.com/mudler/dllm.cpp is still private (publishing is planned);
+# until then the anonymous clone below fails. Use the symlink shortcut above
+# with a local checkout, or a git credential helper with access to the repo.
+
+# The pin below is the P5 performance-parity head (device-resident
+# self-conditioning, full-GPU placement at ngl >= n_layer, graph reuse,
+# device-side EB reductions: ~8x per-step on GB10, see dllm.cpp
+# docs/validation.md section 10). C-ABI unchanged (still version 1). It
+# also carries the multimodal entry points (dllm_capi_generate_mm /
+# dllm_capi_generate_stream_mm) the image-input path probes for; older
+# libs still load, but image requests then fail with "library predates
+# the multimodal entry points".
+DLLM_VERSION?=320b57756efc3460169b8ea9e8c782867198f2a5
+DLLM_REPO?=https://github.com/mudler/dllm.cpp
+
+GOCMD?=go
+GO_TAGS?=
+JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+BUILD_TYPE?=
+NATIVE?=false
+
+# libdllm.so is self-contained: dllm.cpp's CMakeLists statically absorbs ggml
+# (BUILD_SHARED_LIBS=OFF + PIC) into the shared lib, so dlopen needs no
+# libggml*.so alongside it, only system libs (libstdc++/libgomp/libc) the
+# runtime image already provides. Tests/CLI are upstream-only concerns.
+CMAKE_ARGS?=-DCMAKE_BUILD_TYPE=Release -DDLLM_BUILD_TESTS=OFF
+
+ifeq ($(NATIVE),false)
+	CMAKE_ARGS+=-DGGML_NATIVE=OFF
+endif
+
+# Same arch set the sibling ggml backends (acestep/vibevoice/qwen3-tts) bake
+# for their cublas images; override for a native build.
+CUDA_ARCHITECTURES?=75-virtual;80-virtual;86-real;89-real
+
+# dllm.cpp gates CUDA behind DLLM_CUDA (set(GGML_CUDA ... CACHE FORCE)), so
+# forward that instead of a bare -DGGML_CUDA=ON.
+ifeq ($(BUILD_TYPE),cublas)
+	CMAKE_ARGS+=-DDLLM_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="$(CUDA_ARCHITECTURES)"
+endif
+
+.PHONY: dllm-grpc package build clean purge test all
+
+all: dllm-grpc
+
+# Clone the upstream dllm.cpp source at the pinned commit (ggml comes in as
+# a submodule). Directory acts as the target so make only re-clones when
+# missing. After a DLLM_VERSION bump, run 'make purge && make' to refetch.
+sources/dllm.cpp:
+	mkdir -p sources/dllm.cpp
+	cd sources/dllm.cpp && \
+	git init -q && \
+	git remote add origin $(DLLM_REPO) && \
+	git fetch --depth 1 origin $(DLLM_VERSION) && \
+	git checkout FETCH_HEAD && \
+	git submodule update --init --recursive --depth 1 --single-branch
+
+# Build the shared lib out-of-tree, then stage it next to the Go sources so
+# purego.Dlopen("libdllm.so") and the packaging step both pick it up.
+libdllm.so: sources/dllm.cpp
+	cmake -B sources/dllm.cpp/build -S sources/dllm.cpp $(CMAKE_ARGS)
+	cmake --build sources/dllm.cpp/build --config Release -j$(JOBS)
+	cp -fv sources/dllm.cpp/build/libdllm.so ./
+
+dllm-grpc: libdllm.so main.go capi.go
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o dllm-grpc .
+
+package: dllm-grpc
+	bash package.sh
+
+build: package
+
+# Test target. The C-ABI binding smoke is gated on DLLM_TEST_LIBRARY +
+# DLLM_TEST_TINY_MODEL; without them the gated specs auto-skip and only the
+# pure-Go helper specs run.
+test:
+	LD_LIBRARY_PATH=$(CURDIR):$$LD_LIBRARY_PATH $(GOCMD) test ./... -count=1
+
+clean: purge
+	rm -rf libdllm.so* package dllm-grpc
+
+purge:
+	rm -rf sources/dllm.cpp

backend/go/dllm/capi.go

@@ -0,0 +1,326 @@
+package main
+
+// Typed Go wrappers over dllm.cpp's flat C-ABI (include/dllm_capi.h, ABI v1).
+//
+// Contract highlights the wrappers encode (see the header + src/capi.cpp):
+//   - tokenize_json/generate return malloc'd char* the CALLER owns: bound as
+//     uintptr, copied with goStringFromCPtr, released via dllm_capi_free_string.
+//   - last_error returns a BORROWED pointer (valid until the next call on the
+//     same ctx): bound as a plain string (purego copies), never freed, and only
+//     read AFTER the failing call has returned - reading it while a generate is
+//     in flight on the same ctx violates the per-ctx serialization contract.
+//   - All entry points except dllm_capi_cancel must be externally serialized
+//     per ctx (one ctx = one concurrent generate/tokenize). Cancel only flips
+//     an atomic and may be called from any goroutine mid-generate.
+//   - No C++ exception crosses the boundary; failures land in last_error.
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"sync"
+	"sync/atomic"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+)
+
+// dllmABIVersion is the DLLM_CAPI_ABI_VERSION this binding was written
+// against; main.go refuses to start against a libdllm.so reporting another.
+const dllmABIVersion = 1
+
+// purego-bound entry points from libdllm.so. Names match dllm_capi.h
+// exactly; loadCAPI (main.go) fills these in at boot.
+var (
+	cppAbiVersion func() int32
+	cppLoad       func(ggufPath, paramsJSON string) uintptr
+	cppFree       func(ctx uintptr)
+	cppLastError  func(ctx uintptr) string // borrowed pointer: purego copies, do NOT free
+	cppFreeString func(s uintptr)
+	// malloc'd char* returns, hence uintptr (see loadCAPI's doc comment).
+	cppTokenizeJSON func(ctx uintptr, text string) uintptr
+	cppGenerate     func(ctx uintptr, prompt, optsJSON string) uintptr
+	// on_block/on_step are C function pointers produced by purego.NewCallback;
+	// userData carries the streamCallStates registry key.
+	cppGenerateStream func(ctx uintptr, prompt, optsJSON string, onBlock, onStep, userData uintptr) int32
+	cppCancel         func(ctx uintptr)
+)
+
+// Optional multimodal entry points (dllm_capi.h's P4 surface). The ABI
+// version stays 1: presence is detected by PROBING the symbols with Dlsym at
+// boot (loadCAPI, mirroring the parakeet-cpp optional-symbol pattern). nil
+// means the loaded libdllm.so predates the mm surface; the wrappers below
+// then fail with errMMUnsupported instead of crashing on a nil call.
+var (
+	cppGenerateMM       func(ctx uintptr, prompt, imagesJSON, optsJSON string) uintptr
+	cppGenerateStreamMM func(ctx uintptr, prompt, imagesJSON, optsJSON string, onBlock, onStep, userData uintptr) int32
+)
+
+// mmImageMarker is the literal placeholder dllm_capi_generate_mm expands to
+// <boi> + soft-token placeholders + <eoi> (dllm_capi.h placeholder contract;
+// capi.cpp MM_MARKER). The prompt must carry exactly one marker per
+// images_json entry, in image order.
+const mmImageMarker = "<image>"
+
+// errMMUnsupported is returned for image-bearing requests against an old
+// text-only libdllm.so (the Dlsym probe found no mm symbols).
+var errMMUnsupported = errors.New(
+	"dllm: image input requires libdllm.so with the multimodal entry points (dllm_capi_generate_mm), but the loaded library predates them - rebuild/upgrade the dllm backend to use images")
+
+// cMMSupported reports whether the loaded libdllm.so carries the multimodal
+// generate pair. Both symbols ship together (same dllm.cpp commit), but the
+// guard requires both anyway so a half-present surface can never dispatch.
+func cMMSupported() bool {
+	return cppGenerateMM != nil && cppGenerateStreamMM != nil
+}
+
+// cAbiVersion returns the library's DLLM_CAPI_ABI_VERSION.
+func cAbiVersion() int32 {
+	return cppAbiVersion()
+}
+
+// cLoad opens the GGUF at path with the flat params JSON (e.g.
+// {"n_gpu_layers":99}). Returns 0 on failure; per the header contract there
+// is no ctx to carry the reason, the C side logs it to stderr (and
+// cLastError(0) only yields the static NULL-ctx message).
+func cLoad(path, paramsJSON string) uintptr {
+	return cppLoad(path, paramsJSON)
+}
+
+// cFree releases a ctx; safe on 0 (delete nullptr).
+func cFree(h uintptr) {
+	cppFree(h)
+}
+
+// cLastError returns the ctx's last error message (or the static NULL-ctx
+// message for h==0). The C pointer is borrowed and only valid until the next
+// call on the same ctx; purego's string return copies it immediately, so the
+// returned Go string is safe to keep. Must not be called while another call
+// on the same ctx is in flight.
+func cLastError(h uintptr) string {
+	return cppLastError(h)
+}
+
+// lastErrorOr is cLastError with a fallback for the empty-message case, so
+// wrapped errors never end in ": ".
+func lastErrorOr(h uintptr, fallback string) string {
+	if msg := cLastError(h); msg != "" {
+		return msg
+	}
+	return fallback
+}
+
+// cTokenizeJSON tokenizes text (the C side prepends bos per vocab.add_bos)
+// and returns the token ids as a JSON array string, e.g. "[2,18]".
+func cTokenizeJSON(h uintptr, text string) (string, error) {
+	ret := cppTokenizeJSON(h, text)
+	if ret == 0 {
+		return "", fmt.Errorf("dllm: tokenize failed: %s", lastErrorOr(h, "unknown error"))
+	}
+	out := goStringFromCPtr(ret)
+	cppFreeString(ret)
+	return out, nil
+}
+
+// cGenerate runs a blocking generation and returns the detokenized text.
+// optsJSON must be a FLAT JSON object of scalars (use buildOptsJSON); the C
+// parser rejects nested objects/arrays. NULL return -> last_error (read only
+// after the call returned, per the serialization contract); a cancelled call
+// surfaces as the "cancelled" message.
+func cGenerate(h uintptr, prompt, optsJSON string) (string, error) {
+	ret := cppGenerate(h, prompt, optsJSON)
+	if ret == 0 {
+		return "", fmt.Errorf("dllm: generate failed: %s", lastErrorOr(h, "unknown error"))
+	}
+	out := goStringFromCPtr(ret)
+	cppFreeString(ret)
+	return out, nil
+}
+
+// cGenerateMM is cGenerate's multimodal counterpart. imagesJSON is the flat
+// JSON array of image entries (data: base64 URIs here; the C side also takes
+// file paths) and the prompt must carry one mmImageMarker per entry - the
+// engine enforces the 1:1 match and reports mismatches through last_error.
+func cGenerateMM(h uintptr, prompt, imagesJSON, optsJSON string) (string, error) {
+	if !cMMSupported() {
+		return "", errMMUnsupported
+	}
+	ret := cppGenerateMM(h, prompt, imagesJSON, optsJSON)
+	if ret == 0 {
+		return "", fmt.Errorf("dllm: generate_mm failed: %s", lastErrorOr(h, "unknown error"))
+	}
+	out := goStringFromCPtr(ret)
+	cppFreeString(ret)
+	return out, nil
+}
+
+// streamCallState carries the Go callbacks for one in-flight
+// cGenerateStream call; the registry key travels through C as user_data.
+// The map shape mirrors the whisper backend's streamCallStates: only one
+// entry per ctx is ever live (the C-ABI is serialized per ctx), but keying
+// by call survives multiple models/processes sharing the package.
+type streamCallState struct {
+	onBlock func(text string)
+	onStep  func(step, total int, preview string)
+}
+
+var (
+	streamCallStates sync.Map // uint64 -> *streamCallState
+	streamCallSeq    atomic.Uint64
+
+	// purego.NewCallback allocates a finite, never-released callback slot, so
+	// the two trampolines are created exactly once and reused across calls.
+	streamCbOnce sync.Once
+	blockCbPtr   uintptr
+	stepCbPtr    uintptr
+)
+
+// onBlockTrampoline is the Go side of dllm_block_cb. It runs on the C
+// calling thread, mid-generate: keep it tiny and non-blocking (callers that
+// bridge to goroutines must hand off via buffered channels). The text
+// pointer is only valid for the duration of the invocation, so it is copied
+// to a Go string immediately.
+func onBlockTrampoline(text uintptr, userData uintptr) {
+	v, ok := streamCallStates.Load(uint64(userData))
+	if !ok {
+		return // call already torn down
+	}
+	state := v.(*streamCallState)
+	if state.onBlock != nil {
+		state.onBlock(goStringFromCPtr(text))
+	}
+}
+
+// onStepTrampoline is the Go side of dllm_step_cb; same threading and
+// lifetime caveats as onBlockTrampoline.
+func onStepTrampoline(step int32, totalSteps int32, canvasPreview uintptr, userData uintptr) {
+	v, ok := streamCallStates.Load(uint64(userData))
+	if !ok {
+		return
+	}
+	state := v.(*streamCallState)
+	if state.onStep != nil {
+		state.onStep(int(step), int(totalSteps), goStringFromCPtr(canvasPreview))
+	}
+}
+
+// withStreamCallbacks registers onBlock/onStep in the trampoline registry
+// for the duration of one streaming C call and invokes call with the C
+// function pointers (NULL for absent callbacks, so the C side skips the
+// per-block / per-step detokenize work entirely) plus the registry key to
+// pass as user_data. Shared by the text and multimodal stream wrappers.
+func withStreamCallbacks(onBlock func(text string), onStep func(step, total int, preview string), call func(blockPtr, stepPtr, userData uintptr) int32) int32 {
+	streamCbOnce.Do(func() {
+		blockCbPtr = purego.NewCallback(onBlockTrampoline)
+		stepCbPtr = purego.NewCallback(onStepTrampoline)
+	})
+
+	id := streamCallSeq.Add(1)
+	streamCallStates.Store(id, &streamCallState{onBlock: onBlock, onStep: onStep})
+	defer streamCallStates.Delete(id)
+
+	var blockPtr, stepPtr uintptr
+	if onBlock != nil {
+		blockPtr = blockCbPtr
+	}
+	if onStep != nil {
+		stepPtr = stepCbPtr
+	}
+	return call(blockPtr, stepPtr, uintptr(id))
+}
+
+// cGenerateStream runs a generation with per-committed-block (onBlock) and
+// per-denoising-step (onStep) callbacks; either may be nil. The callbacks
+// run on the C thread (see the trampoline docs). Returns an error carrying
+// last_error on failure; cancellation surfaces as the "cancelled" message.
+func cGenerateStream(h uintptr, prompt, optsJSON string, onBlock func(text string), onStep func(step, total int, preview string)) error {
+	rc := withStreamCallbacks(onBlock, onStep, func(blockPtr, stepPtr, userData uintptr) int32 {
+		return cppGenerateStream(h, prompt, optsJSON, blockPtr, stepPtr, userData)
+	})
+	if rc != 0 {
+		return fmt.Errorf("dllm: generate_stream failed: %s", lastErrorOr(h, "unknown error"))
+	}
+	return nil
+}
+
+// cGenerateStreamMM is cGenerateStream's multimodal counterpart; see
+// cGenerateMM for the imagesJSON/marker contract.
+func cGenerateStreamMM(h uintptr, prompt, imagesJSON, optsJSON string, onBlock func(text string), onStep func(step, total int, preview string)) error {
+	if !cMMSupported() {
+		return errMMUnsupported
+	}
+	rc := withStreamCallbacks(onBlock, onStep, func(blockPtr, stepPtr, userData uintptr) int32 {
+		return cppGenerateStreamMM(h, prompt, imagesJSON, optsJSON, blockPtr, stepPtr, userData)
+	})
+	if rc != 0 {
+		return fmt.Errorf("dllm: generate_stream_mm failed: %s", lastErrorOr(h, "unknown error"))
+	}
+	return nil
+}
+
+// cCancel requests cancellation of the in-flight generate on h. This is the
+// ONE entry point safe to call from any goroutine while a generate runs (it
+// only flips an atomic). Note the cancel-reset race from the header: each
+// generate resets the flag on entry, so a watchdog should re-issue cancel if
+// the call has not returned.
+func cCancel(h uintptr) {
+	cppCancel(h)
+}
+
+// buildOptsJSON renders generation options as the flat JSON object the
+// C-ABI expects (known keys: n_predict, blocks, seed, eb_*, kv_cache). The
+// C-side scanner only understands scalar number/string values and rejects
+// nested objects/arrays loudly; bools are rejected here too because the
+// scanner has no concept of them. Fail loud rather than let an option be
+// silently misread.
+//
+// CAVEAT: json.Marshal HTML-escapes <, > and & inside string values (e.g.
+// "<" becomes the six-byte \u003c sequence). None of the known string-valued keys
+// (kv_cache: auto|on|off) can contain those bytes today; if one ever does,
+// switch to an Encoder with SetEscapeHTML(false) like gemma4JSONString.
+func buildOptsJSON(opts map[string]any) (string, error) {
+	if len(opts) == 0 {
+		return "{}", nil
+	}
+	for k, v := range opts {
+		switch v.(type) {
+		case string,
+			int, int8, int16, int32, int64,
+			uint, uint8, uint16, uint32, uint64,
+			float32, float64,
+			json.Number:
+			// scalar: fine
+		default:
+			return "", fmt.Errorf("dllm: opts key %q has non-scalar value %T (the C-ABI only accepts flat number/string scalars)", k, v)
+		}
+	}
+	b, err := json.Marshal(opts)
+	if err != nil {
+		return "", fmt.Errorf("dllm: marshal opts: %w", err)
+	}
+	return string(b), nil
+}
+
+// goStringFromCPtr copies a NUL-terminated C string into Go memory. cptr is
+// the raw pointer returned by purego from the C-ABI (a malloc'd buffer the
+// caller owns, or a callback argument only valid during the invocation);
+// owning callers must free it via cppFreeString after the copy lands.
+//
+// A direct unsafe.Pointer(cptr) conversion trips go vet's unsafeptr check,
+// which can't distinguish a C-owned heap pointer from Go-managed memory (the
+// parakeet-cpp and whisper backends tolerate that warning). Reinterpreting
+// through &cptr below is equivalent at runtime and keeps plain `go vet`
+// clean. It is safe either way: the pointer addresses C memory the Go GC
+// neither tracks nor moves, and we dereference it immediately to copy the
+// bytes out.
+func goStringFromCPtr(cptr uintptr) string {
+	if cptr == 0 {
+		return ""
+	}
+	p := *(*unsafe.Pointer)(unsafe.Pointer(&cptr)) // C-owned buffer, not Go-GC memory (see doc above)
+	n := 0
+	for *(*byte)(unsafe.Add(p, n)) != 0 {
+		n++
+	}
+	return string(unsafe.Slice((*byte)(p), n))
+}

backend/go/dllm/dllm.go

@@ -0,0 +1,622 @@
+package main
+
+// LocalAI gRPC backend for dllm.cpp (DiffusionGemma block-diffusion models).
+//
+// Wiring overview:
+//   - Load opens the GGUF via dllm_capi_load and starts the per-model worker
+//     goroutine that serializes every C call (see submit).
+//   - PredictRich / PredictStreamRich implement grpc.AIModelRich: when the
+//     request carries raw messages (use_tokenizer_template), the backend owns
+//     templating (RenderGemma4) and output parsing (Gemma4Parser) and replies
+//     with ChatDeltas, like the llama.cpp autoparser and the ds4 backend.
+//   - The legacy Predict / PredictStream methods delegate to the rich pair
+//     (cloud-proxy precedent); the gRPC server prefers the rich path anyway.
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strconv"
+	"strings"
+	"sync"
+	"unicode/utf8"
+
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	"github.com/mudler/LocalAI/pkg/grpc/grpcerrors"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// The gRPC server cancels in-flight generations on client disconnect only
+// for backends advertising the Cancellable capability; keep Dllm pinned to
+// it so a signature drift fails the build, not the disconnect path.
+var _ grpc.Cancellable = (*Dllm)(nil)
+
+// generator is the seam between the backend wiring and the dllm.cpp C-ABI:
+// the real implementation (capiGenerator) wraps the cGenerate/cTokenizeJSON
+// family, while tests substitute a fake to exercise prompt construction,
+// parsing and serialization without libdllm.so.
+type generator interface {
+	generate(prompt, optsJSON string) (string, error)
+	// generateStream invokes onBlock once per committed diffusion block, on
+	// the thread running the C call, before returning.
+	generateStream(prompt, optsJSON string, onBlock func(text string)) error
+	// generateMM / generateStreamMM are the multimodal counterparts:
+	// imagesJSON is a flat JSON array of data: base64 URIs and the prompt
+	// carries one mmImageMarker per entry (dllm_capi.h placeholder
+	// contract). Against an old text-only libdllm.so they fail with
+	// errMMUnsupported.
+	generateMM(prompt, imagesJSON, optsJSON string) (string, error)
+	generateStreamMM(prompt, imagesJSON, optsJSON string, onBlock func(text string)) error
+	tokenizeJSON(text string) (string, error)
+	// cancel is the ONE entry point safe to call concurrently with an
+	// in-flight generate on the same ctx (dllm_capi.h: it only flips an
+	// atomic; everything else must be externally serialized per ctx).
+	cancel()
+	free()
+}
+
+// capiGenerator is the production generator over one dllm_ctx handle.
+type capiGenerator struct {
+	h uintptr
+}
+
+func (g *capiGenerator) generate(prompt, optsJSON string) (string, error) {
+	return cGenerate(g.h, prompt, optsJSON)
+}
+
+func (g *capiGenerator) generateStream(prompt, optsJSON string, onBlock func(text string)) error {
+	// on_step (per-denoise-step canvas preview, dllm.cpp's --visual) is
+	// passed as nil for now: a future progress hook for the React UI can
+	// plumb it through without touching the C binding.
+	return cGenerateStream(g.h, prompt, optsJSON, onBlock, nil)
+}
+
+func (g *capiGenerator) generateMM(prompt, imagesJSON, optsJSON string) (string, error) {
+	return cGenerateMM(g.h, prompt, imagesJSON, optsJSON)
+}
+
+func (g *capiGenerator) generateStreamMM(prompt, imagesJSON, optsJSON string, onBlock func(text string)) error {
+	// on_step is nil for the same reason as generateStream.
+	return cGenerateStreamMM(g.h, prompt, imagesJSON, optsJSON, onBlock, nil)
+}
+
+func (g *capiGenerator) tokenizeJSON(text string) (string, error) {
+	return cTokenizeJSON(g.h, text)
+}
+
+func (g *capiGenerator) cancel() {
+	cCancel(g.h)
+}
+
+func (g *capiGenerator) free() {
+	cFree(g.h)
+}
+
+// Dllm is the gRPC backend instance: one per loaded model (LocalAI starts
+// one backend process per model).
+type Dllm struct {
+	base.Base
+
+	gen generator
+	// genOpts holds the model-level generation overrides parsed from
+	// ModelOptions.Options at Load (eb_*, blocks, kv_cache). The C-ABI takes
+	// them per-generate, not per-load, so they are merged into every
+	// request's opts JSON (requestOptsJSON).
+	genOpts map[string]any
+
+	// jobs is the per-model worker queue. dllm_capi.h requires every entry
+	// point EXCEPT dllm_capi_cancel to be externally serialized per ctx (one
+	// ctx = one concurrent generate/tokenize; last_error is unsafe to read
+	// while a call is in flight). A single goroutine owning all C calls makes
+	// that contract structural instead of relying on lock discipline.
+	jobs     chan func()
+	workerWG sync.WaitGroup
+
+	// genMu guards gen against Free racing in-flight requests: requests hold
+	// the read lock for their full duration (they stay concurrent with each
+	// other - the worker still serializes the C calls), Free takes the write
+	// lock so it can only run when no request is in flight.
+	genMu sync.RWMutex
+}
+
+func (d *Dllm) startWorker() {
+	d.jobs = make(chan func())
+	d.workerWG.Add(1)
+	go func() {
+		defer d.workerWG.Done()
+		for job := range d.jobs {
+			job()
+		}
+	}()
+}
+
+// submit runs job on the worker goroutine and waits for it to finish.
+// Concurrent gRPC requests therefore queue up and execute one at a time
+// against the single dllm_ctx.
+func (d *Dllm) submit(job func()) {
+	done := make(chan struct{})
+	d.jobs <- func() {
+		defer close(done)
+		job()
+	}
+	<-done
+}
+
+// Load opens the GGUF and prepares the worker. Load-time engine parameters
+// travel as the flat params JSON of dllm_capi_load; generation overrides
+// from Options are stored for per-request opts JSON instead (the C-ABI has
+// no per-load sampler state).
+func (d *Dllm) Load(opts *pb.ModelOptions) error {
+	if d.gen != nil {
+		return errors.New("dllm: model already loaded")
+	}
+
+	params := map[string]any{
+		"n_gpu_layers": opts.GetNGPULayers(),
+	}
+	if opts.GetThreads() > 0 {
+		params["n_threads"] = opts.GetThreads()
+	}
+	if opts.GetContextSize() > 0 {
+		params["ctx_len"] = opts.GetContextSize()
+	}
+	paramsJSON, err := buildOptsJSON(params)
+	if err != nil {
+		return err
+	}
+
+	d.genOpts = parseModelGenOpts(opts.GetOptions())
+
+	h := cLoad(opts.GetModelFile(), paramsJSON)
+	if h == 0 {
+		// No ctx exists on load failure, so last_error(NULL) only carries the
+		// static NULL-ctx message; the real reason is on the backend's stderr.
+		return fmt.Errorf("dllm: load %q failed: %s (see backend log for details)",
+			opts.GetModelFile(), lastErrorOr(0, "unknown error"))
+	}
+	d.gen = &capiGenerator{h: h}
+	d.startWorker()
+	xlog.Info("dllm: model loaded", "model", opts.GetModelFile(), "params", paramsJSON, "gen_opts", d.genOpts)
+	return nil
+}
+
+// Free releases the dllm ctx and stops the worker. Safe when never loaded.
+//
+// The write lock is essential: the gRPC server (pkg/grpc/server.go, see the
+// model-unload path around line 764) calls Free with no locking of its own,
+// and base.Base provides none either. Without it a request racing Free would
+// panic sending on the closed jobs channel - or worse, generate on a freed C
+// ctx. Holding genMu until gen is nil also turns post-Free requests into a
+// clean "model not loaded" error instead of a crash.
+func (d *Dllm) Free() error {
+	d.genMu.Lock()
+	defer d.genMu.Unlock()
+	if d.gen == nil {
+		return nil
+	}
+	d.submit(d.gen.free)
+	close(d.jobs)
+	d.workerWG.Wait()
+	d.gen = nil
+	return nil
+}
+
+// Cancel requests cancellation of the in-flight generate (the
+// grpc.Cancellable capability). The gRPC server arms it via
+// context.AfterFunc on the request/stream context, so a client
+// disconnect or timeout aborts the generation server-side - the same
+// semantics the llama.cpp C++ backend gets from polling IsCancelled().
+// It deliberately bypasses the worker queue: dllm_capi_cancel is the one
+// call the C-ABI allows from any goroutine mid-generate (it only flips
+// an atomic).
+//
+// Note dllm_capi.h's cancel-reset race: each generate resets the flag on
+// entry, so a Cancel racing a NEW generate on the same ctx can be lost
+// (and, with requests queued on the worker, it aborts whichever generate
+// is currently running). The single-flag granularity is acceptable here
+// because the server de-registers the hook on normal completion and one
+// backend process serves one model.
+func (d *Dllm) Cancel() {
+	// RLock so a server-side AfterFunc firing in the window between a
+	// request finishing and a model unload cannot touch a freed C ctx
+	// (Free holds the write lock while tearing gen down). cancel() is the
+	// one C call that is safe concurrently with an in-flight generate, so
+	// taking a read lock here cannot deadlock against request holders.
+	d.genMu.RLock()
+	defer d.genMu.RUnlock()
+	if d.gen != nil {
+		d.gen.cancel()
+	}
+}
+
+// dllmGenOptKeys are the ModelOptions.Options keys this backend forwards to
+// the engine. Options is a shared free-form bag (other layers put their own
+// entries there), so unknown keys are skipped with a warning, not an error.
+var dllmGenOptKeys = map[string]bool{
+	"blocks":   true,
+	"kv_cache": true, // "auto"|"on"|"off"; honored by the engine from P3
+}
+
+// parseModelGenOpts parses "key:value" Options entries into the flat scalar
+// map merged into every generate's opts JSON. eb_* (Entropy-Bound sampler
+// knobs) and the keys in dllmGenOptKeys are recognized; values are typed by
+// first successful parse (int, then float, else string) to match the C
+// scanner's number/string scalars.
+func parseModelGenOpts(options []string) map[string]any {
+	out := map[string]any{}
+	for _, o := range options {
+		key, val, found := strings.Cut(o, ":")
+		if !found {
+			xlog.Warn("dllm: ignoring malformed option (want key:value)", "option", o)
+			continue
+		}
+		if !strings.HasPrefix(key, "eb_") && !dllmGenOptKeys[key] {
+			xlog.Debug("dllm: ignoring unrecognized option", "key", key)
+			continue
+		}
+		out[key] = parseScalarOpt(val)
+	}
+	return out
+}
+
+func parseScalarOpt(v string) any {
+	if iv, err := strconv.ParseInt(v, 10, 64); err == nil {
+		return iv
+	}
+	if fv, err := strconv.ParseFloat(v, 64); err == nil {
+		return fv
+	}
+	return v
+}
+
+// metadataEnableThinking reads the enable_thinking gate. Unlike ds4 (default
+// ON, matching ds4-server), dllm defaults OFF: DiffusionGemma's chat
+// template guards every thinking branch with `enable_thinking is defined and
+// enable_thinking`, i.e. thinking is opt-in for this model family, and the
+// no-thinking render pre-closes an empty thought channel that the OFF
+// default must produce.
+func metadataEnableThinking(opts *pb.PredictOptions) bool {
+	v := opts.GetMetadata()["enable_thinking"]
+	return v == "true" || v == "1"
+}
+
+// buildPrompt resolves the prompt for a request. With use_tokenizer_template
+// and raw messages the backend owns templating (RenderGemma4, including the
+// mmImageMarker injection for opts.Images) and the output is in the known
+// gemma4 format, so parse=true. Without it the caller templated the prompt
+// themselves (LocalAI's Go templates + PEG fallback, or a bare completion):
+// the prompt passes through verbatim - for image requests it must already
+// carry one literal mmImageMarker per image (the engine enforces the 1:1
+// match) - and the output is NOT gemma4-parsed - it is emitted as plain
+// content and the Go side's extraction applies, as for any non-autoparsing
+// backend.
+func buildPrompt(opts *pb.PredictOptions) (prompt string, parse bool, err error) {
+	if opts.GetUseTokenizerTemplate() && len(opts.GetMessages()) > 0 {
+		prompt, err = RenderGemma4(opts.GetMessages(), opts.GetTools(), len(opts.GetImages()), metadataEnableThinking(opts), true)
+		return prompt, true, err
+	}
+	return opts.GetPrompt(), false, nil
+}
+
+// imagesJSON renders opts.Images as the flat JSON array of data: URIs the mm
+// C-ABI expects, or "" when the request carries no images. The entries arrive
+// as RAW base64 payloads: LocalAI's OpenAI layer decodes every image_url /
+// image content part (URL download or data: URI) to plain base64 via
+// utils.GetContentURIAsBase64 (core/http/middleware/request.go) and core
+// flattens them into PredictOptions.Images (core/backend/llm.go). The
+// hardcoded image/jpeg mime mirrors the llama.cpp backend's re-wrapping
+// convention (grpc-server.cpp, "data:image/jpeg;base64," + images(i)); the
+// engine ignores the declared mime and sniffs the real format from the
+// decoded bytes (stb_image), so PNG/BMP payloads work through it too.
+func imagesJSON(images []string) (string, error) {
+	if len(images) == 0 {
+		return "", nil
+	}
+	uris := make([]string, len(images))
+	for i, img := range images {
+		// dllm_capi.h: array entries are read VERBATIM up to the closing
+		// quote, with NO escape handling. json.Marshal would escape these
+		// bytes and the C side would misparse the entry, so fail loud (they
+		// can never appear in genuine base64 anyway).
+		if strings.ContainsAny(img, "\"\\") {
+			return "", fmt.Errorf("dllm: image %d is not base64 (contains a quote or backslash; PredictOptions.Images entries must be raw base64 payloads)", i)
+		}
+		uris[i] = "data:image/jpeg;base64," + img
+	}
+	b, err := json.Marshal(uris)
+	if err != nil {
+		return "", fmt.Errorf("dllm: marshal images: %w", err)
+	}
+	return string(b), nil
+}
+
+// requestOptsJSON merges the model-level overrides with the request's
+// sampling fields into the flat opts JSON for one generate call.
+func (d *Dllm) requestOptsJSON(opts *pb.PredictOptions) (string, error) {
+	m := make(map[string]any, len(d.genOpts)+2)
+	for k, v := range d.genOpts {
+		m[k] = v
+	}
+	if n := opts.GetTokens(); n > 0 {
+		// The engine rounds n_predict UP to a whole number of diffusion
+		// blocks (the canvas is denoised block-wise), so the completion may
+		// run slightly past the requested budget. Tokens==0 omits the key so
+		// the C-ABI default of 256 applies (hardcoded in capi.cpp's
+		// parse_gen_opts, independent of canvas_length).
+		m["n_predict"] = n
+	}
+	if s := opts.GetSeed(); s > 0 {
+		// The engine seeds mt19937 with explicit non-negative seeds. Seed<=0
+		// is omitted: proto3 cannot distinguish 0 from unset, and negative
+		// values conventionally mean "random" across LocalAI backends.
+		m["seed"] = s
+	}
+	return buildOptsJSON(m)
+}
+
+// prepareRequest is the shared prologue of the rich methods: resolve the
+// prompt (and whether the output gets gemma4-parsed) and build the per-call
+// opts JSON plus the images JSON ("" for text-only requests, which routes
+// the call through the text generate entry points).
+func (d *Dllm) prepareRequest(opts *pb.PredictOptions) (prompt string, parse bool, optsJSON, imgJSON string, err error) {
+	// Fail loud on media the engine has no path for, instead of silently
+	// generating from a prompt that ignores them.
+	if len(opts.GetVideos()) > 0 || len(opts.GetAudios()) > 0 {
+		return "", false, "", "", errors.New("dllm: video/audio input is not supported (images only)")
+	}
+	prompt, parse, err = buildPrompt(opts)
+	if err != nil {
+		return "", false, "", "", err
+	}
+	optsJSON, err = d.requestOptsJSON(opts)
+	if err != nil {
+		return "", false, "", "", err
+	}
+	imgJSON, err = imagesJSON(opts.GetImages())
+	if err != nil {
+		return "", false, "", "", err
+	}
+	return prompt, parse, optsJSON, imgJSON, nil
+}
+
+// sanitizeUTF8 makes s safe for a proto3 string field. Block-boundary
+// detokenization and byte-fallback tokens can produce invalid UTF-8, and
+// grpc-go refuses to marshal it ("string field contains invalid UTF-8"), so
+// every string destined for a Reply/ChatDelta must pass through here (or
+// through splitValidUTF8, which calls it). Lone malformed bytes are genuinely
+// undecodable: replace with U+FFFD rather than crash the stream.
+func sanitizeUTF8(s string) string {
+	if utf8.ValidString(s) {
+		return s
+	}
+	return strings.ToValidUTF8(s, "<22>")
+}
+
+// utf8SeqLen returns the declared sequence length of a UTF-8 leading byte
+// (1 for bytes that can never lead a multi-byte sequence, so they are never
+// held back and fall through to sanitizeUTF8's replacement).
+func utf8SeqLen(b byte) int {
+	switch {
+	case b&0xE0 == 0xC0:
+		return 2
+	case b&0xF0 == 0xE0:
+		return 3
+	case b&0xF8 == 0xF0:
+		return 4
+	default:
+		return 1
+	}
+}
+
+// splitValidUTF8 prepends the previous block's carry to the new block and
+// splits the result into text safe to emit now and a trailing INCOMPLETE
+// UTF-8 sequence (at most utf8.UTFMax-1 bytes) to carry into the next block:
+// the per-block detokenize can split a multi-byte character across block
+// boundaries (llama.cpp's grpc-server holds back the same way). Only a
+// suffix that can still become a valid rune is withheld; bytes that are
+// already undecodable are replaced immediately so the carry stays bounded.
+func splitValidUTF8(carry, block string) (emit, newCarry string) {
+	s := carry + block
+	cut := len(s)
+	for i := len(s) - 1; i >= 0 && len(s)-i < utf8.UTFMax; i-- {
+		b := s[i]
+		if b < utf8.RuneSelf {
+			break // ASCII: everything before the tail scan is complete
+		}
+		if !utf8.RuneStart(b) {
+			continue // continuation byte: keep looking for its leading byte
+		}
+		// Leading byte: hold the sequence back iff it declares more bytes
+		// than the stream has produced so far (it may complete next block).
+		if utf8SeqLen(b) > len(s)-i {
+			cut = i
+		}
+		break
+	}
+	return sanitizeUTF8(s[:cut]), s[cut:]
+}
+
+// PredictRich is the non-streaming inference path (grpc.AIModelRich).
+// Returns one Reply whose Message is the aggregated assistant content and
+// whose ChatDeltas carry the parsed content/reasoning/tool-call events.
+func (d *Dllm) PredictRich(opts *pb.PredictOptions) (*pb.Reply, error) {
+	d.genMu.RLock()
+	defer d.genMu.RUnlock()
+	if d.gen == nil {
+		return nil, grpcerrors.ModelNotLoaded("dllm")
+	}
+	prompt, parse, optsJSON, imgJSON, err := d.prepareRequest(opts)
+	if err != nil {
+		return nil, err
+	}
+
+	var out string
+	var genErr error
+	d.submit(func() {
+		if imgJSON != "" {
+			out, genErr = d.gen.generateMM(prompt, imgJSON, optsJSON)
+		} else {
+			out, genErr = d.gen.generate(prompt, optsJSON)
+		}
+	})
+	if genErr != nil {
+		return nil, genErr
+	}
+	// Byte-fallback tokens can detokenize to invalid UTF-8; proto3 strings
+	// must be valid or grpc-go fails the whole reply at marshal time.
+	out = sanitizeUTF8(out)
+
+	if !parse {
+		// Raw-prompt mode: plain content, no gemma4 parsing (see buildPrompt).
+		return &pb.Reply{Message: []byte(out), ChatDeltas: []*pb.ChatDelta{{Content: out}}}, nil
+	}
+
+	// The prompt renders with add_generation_prompt; both thinking modes
+	// leave the model starting in content state (see the Gemma4Parser header
+	// comment), hence NewGemma4Parser(false).
+	parser := NewGemma4Parser(false)
+	if reply := replyFromDeltas(append(parser.Feed(out), parser.Close()...)); reply != nil {
+		return reply, nil
+	}
+	// Everything was markers (or out was empty): an empty but non-nil Reply.
+	return &pb.Reply{}, nil
+}
+
+// PredictStreamRich is the streaming counterpart (grpc.AIModelRich): one
+// Reply per committed diffusion block that produced deltas. Per the
+// interface contract the channel is only sent into here - the gRPC server
+// closes it after this returns (opposite to legacy PredictStream).
+func (d *Dllm) PredictStreamRich(opts *pb.PredictOptions, results chan<- *pb.Reply) error {
+	d.genMu.RLock()
+	defer d.genMu.RUnlock()
+	if d.gen == nil {
+		return grpcerrors.ModelNotLoaded("dllm")
+	}
+	prompt, parse, optsJSON, imgJSON, err := d.prepareRequest(opts)
+	if err != nil {
+		return err
+	}
+
+	var parser *Gemma4Parser
+	if parse {
+		parser = NewGemma4Parser(false)
+	}
+	// emit runs inside onBlock, i.e. on the thread driving the C generate.
+	// Sending on results can block on a slow consumer, but the server-side
+	// pump (pkg/grpc/server.go PredictStream) drains continuously and drops
+	// undeliverable sends, so this backpressure is brief and bounded - and
+	// pausing the diffusion loop under it is the desired behavior anyway.
+	emit := func(text string) {
+		if !parse {
+			if text != "" {
+				results <- &pb.Reply{Message: []byte(text), ChatDeltas: []*pb.ChatDelta{{Content: text}}}
+			}
+			return
+		}
+		deltas := parser.Feed(text)
+		if reply := replyFromDeltas(deltas); reply != nil {
+			results <- reply
+		}
+	}
+	// onBlock guards emit (and through it the parser) against invalid UTF-8:
+	// a multi-byte character split across block boundaries is held back until
+	// it completes (see splitValidUTF8), so proto3 marshaling never fails.
+	var carry string
+	onBlock := func(block string) {
+		var text string
+		text, carry = splitValidUTF8(carry, block)
+		emit(text)
+	}
+
+	var genErr error
+	d.submit(func() {
+		if imgJSON != "" {
+			genErr = d.gen.generateStreamMM(prompt, imgJSON, optsJSON, onBlock)
+		} else {
+			genErr = d.gen.generateStream(prompt, optsJSON, onBlock)
+		}
+	})
+	if genErr != nil {
+		return genErr
+	}
+	if carry != "" {
+		// The stream ended mid-sequence: the held-back bytes can no longer
+		// complete, so flush them through the U+FFFD last resort.
+		emit(sanitizeUTF8(carry))
+	}
+	if parse {
+		if reply := replyFromDeltas(parser.Close()); reply != nil {
+			results <- reply
+		}
+	}
+	return nil
+}
+
+// replyFromDeltas wraps one batch of parsed deltas into a streaming Reply,
+// or nil when the batch is empty (markers consumed, nothing emitted yet).
+// Message mirrors the batch's content text so legacy chan-string consumers
+// see exactly the displayed tokens.
+func replyFromDeltas(deltas []*pb.ChatDelta) *pb.Reply {
+	if len(deltas) == 0 {
+		return nil
+	}
+	var content strings.Builder
+	for _, delta := range deltas {
+		content.WriteString(delta.GetContent())
+	}
+	return &pb.Reply{Message: []byte(content.String()), ChatDeltas: deltas}
+}
+
+// Predict is the legacy (string, error) signature; the gRPC server prefers
+// PredictRich, this exists for non-rich callers (cloud-proxy precedent).
+func (d *Dllm) Predict(opts *pb.PredictOptions) (string, error) {
+	reply, err := d.PredictRich(opts)
+	if err != nil {
+		return "", err
+	}
+	return string(reply.GetMessage()), nil
+}
+
+// PredictStream is the legacy chan-string path: rich replies reduced to
+// their content text. Note the inverted channel ownership - the LEGACY
+// contract requires the impl to close the channel.
+func (d *Dllm) PredictStream(opts *pb.PredictOptions, results chan string) error {
+	defer close(results)
+	richCh := make(chan *pb.Reply)
+	errCh := make(chan error, 1)
+	go func() {
+		errCh <- d.PredictStreamRich(opts, richCh)
+		close(richCh)
+	}()
+	for reply := range richCh {
+		if msg := reply.GetMessage(); len(msg) > 0 {
+			results <- string(msg)
+		}
+	}
+	return <-errCh
+}
+
+// TokenizeString tokenizes opts.Prompt via dllm_capi_tokenize_json (the C
+// side prepends bos per the vocab) and decodes the returned id array.
+func (d *Dllm) TokenizeString(opts *pb.PredictOptions) (pb.TokenizationResponse, error) {
+	d.genMu.RLock()
+	defer d.genMu.RUnlock()
+	if d.gen == nil {
+		return pb.TokenizationResponse{}, grpcerrors.ModelNotLoaded("dllm")
+	}
+	var out string
+	var tokErr error
+	d.submit(func() {
+		out, tokErr = d.gen.tokenizeJSON(opts.GetPrompt())
+	})
+	if tokErr != nil {
+		return pb.TokenizationResponse{}, tokErr
+	}
+	var tokens []int32
+	if err := json.Unmarshal([]byte(out), &tokens); err != nil {
+		return pb.TokenizationResponse{}, fmt.Errorf("dllm: decode tokenize result %q: %w", out, err)
+	}
+	return pb.TokenizationResponse{Length: int32(len(tokens)), Tokens: tokens}, nil
+}

backend/go/dllm/gemma4_parser.go

@@ -0,0 +1,562 @@
+// Gemma4 (DiffusionGemma) streaming output parser: raw model text, fed in
+// arbitrary fragments (per committed diffusion block; a fragment can split
+// anywhere, including mid-marker and mid-payload), is turned into
+// pb.ChatDelta events (content / reasoning_content / tool_calls).
+//
+// Normative sources:
+//   - The chat template embedded at the top of gemma4_renderer.go ("tpl L<n>"
+//     citations below refer to its numbered lines). The OUTPUT format mirrors
+//     what the template renders for assistant history: thought channels
+//     (<|channel>thought\n ... <channel|>, tpl L240), tool calls
+//     (<|tool_call>call:name{...}<tool_call|>, tpl L246-L257) and turn ends
+//     (<turn|>, tpl L351).
+//   - vLLM PR #45163: vllm/tool_parsers/gemma4_tool_parser.py (marker
+//     handling, the call:name{...} argument grammar and its decoder, ported
+//     below) and vllm/reasoning/gemma4_reasoning_parser.py (channel markers,
+//     the "thought\n" role label, is_reasoning_end semantics).
+//
+// Initial state (derived from the generation prompt, tpl L356-L362, see
+// RenderGemma4):
+//   - enable_thinking=false: the prompt ends with "<|turn>model\n" +
+//     "<|channel>thought\n<channel|>" - an EMPTY thought channel, pre-opened
+//     AND pre-closed by the template. The model's output therefore starts in
+//     plain content. Use NewGemma4Parser(false).
+//   - enable_thinking=true: the prompt ends at "<|turn>model\n" and the model
+//     opens and closes its own thought channel in the OUTPUT
+//     ("<|channel>thought\n...reasoning...<channel|>final answer", per the
+//     vLLM Gemma4ReasoningParser docstring). The parser still starts in
+//     content state - the channel markers in the output drive the switch.
+//     Use NewGemma4Parser(false) here too.
+//   - NewGemma4Parser(true) is for callers that pre-open the thought channel
+//     in the prompt themselves (appending "<|channel>thought\n" after the
+//     generation prompt to force thinking): the output then begins mid-thought
+//     and everything is reasoning until the first <channel|>.
+//
+// State diagram (markers are consumed, never emitted):
+//
+//	             <|channel>                  \n (channel name dropped: the
+//	[content] --------------> [chan-header] ----> [thought]   "thought\n" role
+//	   ^ |  <channel|> (stray close: swallowed,                label, stripped
+//	   +-+  strip_thinking semantics, tpl L148-L158)           like vLLM does)
+//	   ^                  <channel|>
+//	   +----------------------------------------- [thought]
+//	   ^                  <tool_call|>                 | <|tool_call> (implicit
+//	   +-------------- [tool-call] <-------------------+  reasoning end, vLLM
+//	   |  <|tool_call>     ^                               is_reasoning_end)
+//	   +-------------------+
+//	[content]/[thought] --- <turn|> ---> [done]  (everything after is dropped)
+//
+// Buffering rules:
+//   - content/thought states hold back at most len(longest marker)-1 bytes:
+//     the longest tail that is still a proper prefix of a watched marker.
+//     Content is otherwise emitted immediately (no unbounded buffering).
+//   - the tool-call state buffers the whole payload until <tool_call|>. This
+//     is unbounded in principle but bounded in practice by the model's
+//     diffusion canvas, and is required because the call:name{...} payload
+//     only becomes decodable (and trustworthy) once complete - the same
+//     reason vLLM's parser accumulates before parsing.
+//   - Close() flushes whatever is still held: partial markers come out as
+//     content/reasoning (per the state that held them); an unterminated
+//     channel header or tool-call payload is re-emitted RAW (including its
+//     opening marker) as content - malformed output is never silently
+//     dropped (mirrors vLLM extract_tool_calls returning the raw text as
+//     content when its regex does not match).
+//
+// Streaming granularity DIVERGENCE from vLLM: vLLM re-parses the partial
+// payload on every token and streams argument-JSON diffs (its `partial=True`
+// decoder mode plus withholding logic exist only for that). Our fragments are
+// whole committed diffusion blocks, so each completed tool call is emitted
+// once, as a single ToolCallDelta carrying index + id + name + the full
+// arguments JSON - exactly the shape backend/python/vllm/backend.py emits
+// per call and pkg/functions.ToolCallsFromChatDeltas re-accumulates.
+package main
+
+import (
+	"encoding/json"
+	"regexp"
+	"strconv"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// gemma4CallRE is vLLM's tool_call_regex
+// (`<\|tool_call>call:([\w\-\.]+)\{(.*?)\}<tool_call\|>`, DOTALL) anchored to
+// a single already-extracted payload: name charset [\w\-.], braces mandatory.
+var gemma4CallRE = regexp.MustCompile(`(?s)^call:([\w\-.]+)\{(.*)\}$`)
+
+type g4State int
+
+const (
+	g4Content g4State = iota
+	g4ChanHeader
+	g4Thought
+	g4ToolCall
+	g4Done
+)
+
+// Markers watched per emitting state. A stray <tool_call|> outside a tool
+// call is deliberately NOT watched: it passes through verbatim, consistent
+// with the malformed-payload fallback re-emitting it as content.
+var (
+	gemma4ContentMarkers = []string{gemma4ChannelOpen, gemma4ChannelClose, gemma4ToolCallOpen, gemma4TurnEnd}
+	gemma4ThoughtMarkers = []string{gemma4ChannelClose, gemma4ToolCallOpen, gemma4TurnEnd}
+)
+
+type Gemma4Parser struct {
+	state g4State
+	// held is the per-state carry-over between Feed calls: a partial marker
+	// (content/thought), a partial channel header (chan-header) or the
+	// payload accumulated so far (tool-call).
+	held    string
+	toolIdx int
+}
+
+// NewGemma4Parser returns a parser positioned per the initial-state rules in
+// the header comment: startInThought=true only when the caller pre-opened a
+// thought channel in the prompt.
+func NewGemma4Parser(startInThought bool) *Gemma4Parser {
+	state := g4Content
+	if startInThought {
+		state = g4Thought
+	}
+	return &Gemma4Parser{state: state}
+}
+
+// Feed consumes the next output fragment and returns the deltas it completes.
+func (p *Gemma4Parser) Feed(text string) []*pb.ChatDelta {
+	if text == "" || p.state == g4Done {
+		return nil
+	}
+	pending := p.held + text
+	p.held = ""
+	var em g4Emitter
+	for pending != "" {
+		switch p.state {
+		case g4Content, g4Thought:
+			markers := gemma4ContentMarkers
+			if p.state == g4Thought {
+				markers = gemma4ThoughtMarkers
+			}
+			idx, marker := findEarliestGemma4Marker(pending, markers)
+			if idx == -1 {
+				hold := gemma4MarkerHoldback(pending, markers)
+				p.emitText(&em, pending[:len(pending)-hold])
+				p.held = pending[len(pending)-hold:]
+				pending = ""
+				continue
+			}
+			p.emitText(&em, pending[:idx])
+			pending = pending[idx+len(marker):]
+			switch marker {
+			case gemma4ChannelOpen:
+				p.state = g4ChanHeader
+			case gemma4ChannelClose:
+				// In thought: channel ends. In content: stray close,
+				// swallowed (strip_thinking keeps both sides, tpl L148-L158).
+				p.state = g4Content
+			case gemma4ToolCallOpen:
+				p.state = g4ToolCall
+			case gemma4TurnEnd:
+				p.state = g4Done
+			}
+		case g4ChanHeader:
+			// The channel header is "<name>\n"; the template only ever writes
+			// "thought" (tpl L240/L360) and the label is structural, so it is
+			// dropped, not emitted (vLLM strips the same "thought\n" prefix).
+			nl := strings.IndexByte(pending, '\n')
+			if nl == -1 {
+				p.held = pending
+				pending = ""
+				continue
+			}
+			pending = pending[nl+1:]
+			p.state = g4Thought
+		case g4ToolCall:
+			end := strings.Index(pending, gemma4ToolCallClose)
+			if end == -1 {
+				p.held = pending
+				pending = ""
+				continue
+			}
+			p.emitToolCall(&em, pending[:end])
+			pending = pending[end+len(gemma4ToolCallClose):]
+			p.state = g4Content
+		case g4Done:
+			pending = ""
+		}
+	}
+	return em.deltas
+}
+
+// Close flushes held-back partials. Incomplete structures (open channel
+// header, unterminated tool payload) are re-emitted raw as content rather
+// than dropped. The parser is finished afterwards.
+func (p *Gemma4Parser) Close() []*pb.ChatDelta {
+	var em g4Emitter
+	switch p.state {
+	case g4Content:
+		em.content(p.held)
+	case g4Thought:
+		em.reasoning(p.held)
+	case g4ChanHeader:
+		em.content(gemma4ChannelOpen + p.held)
+	case g4ToolCall:
+		em.content(gemma4ToolCallOpen + p.held)
+	case g4Done:
+	}
+	p.held = ""
+	p.state = g4Done
+	return em.deltas
+}
+
+func (p *Gemma4Parser) emitText(em *g4Emitter, s string) {
+	if p.state == g4Thought {
+		em.reasoning(s)
+		return
+	}
+	em.content(s)
+}
+
+// emitToolCall decodes one complete <|tool_call>...<tool_call|> payload. On a
+// payload that does not match call:name{...} the raw text (markers included)
+// is emitted as content, mirroring vLLM's extract_tool_calls fallback.
+func (p *Gemma4Parser) emitToolCall(em *g4Emitter, payload string) {
+	m := gemma4CallRE.FindStringSubmatch(payload)
+	if m == nil {
+		em.content(gemma4ToolCallOpen + payload + gemma4ToolCallClose)
+		return
+	}
+	// Index-based ids: deterministic (the split-invariance property relies
+	// on it) and matching the call_<n> convention of pkg/grpc/rich_test.go;
+	// core only needs ids to be non-empty and unique within the response.
+	em.tool(p.toolIdx, "call_"+strconv.Itoa(p.toolIdx), m[1], decodeGemma4Args(m[2], 0))
+	p.toolIdx++
+}
+
+// g4Emitter collects ChatDeltas; empty text events are dropped.
+type g4Emitter struct {
+	deltas []*pb.ChatDelta
+}
+
+func (e *g4Emitter) content(s string) {
+	if s != "" {
+		e.deltas = append(e.deltas, &pb.ChatDelta{Content: s})
+	}
+}
+
+func (e *g4Emitter) reasoning(s string) {
+	if s != "" {
+		e.deltas = append(e.deltas, &pb.ChatDelta{ReasoningContent: s})
+	}
+}
+
+func (e *g4Emitter) tool(index int, id, name, argsJSON string) {
+	e.deltas = append(e.deltas, &pb.ChatDelta{ToolCalls: []*pb.ToolCallDelta{{
+		Index:     int32(index),
+		Id:        id,
+		Name:      name,
+		Arguments: argsJSON,
+	}}})
+}
+
+// findEarliestGemma4Marker returns the position and value of the first
+// complete marker occurrence, or (-1, "").
+func findEarliestGemma4Marker(s string, markers []string) (int, string) {
+	best, bestMarker := -1, ""
+	for _, m := range markers {
+		if idx := strings.Index(s, m); idx >= 0 && (best == -1 || idx < best) {
+			best, bestMarker = idx, m
+		}
+	}
+	return best, bestMarker
+}
+
+// gemma4MarkerHoldback returns the length of the longest suffix of s that is
+// a proper prefix of a watched marker - the only bytes that may still grow
+// into a marker and therefore must not be emitted yet (bounded by the
+// longest marker, so content is never buffered unboundedly).
+func gemma4MarkerHoldback(s string, markers []string) int {
+	maxHold := 0
+	for _, m := range markers {
+		if len(m)-1 > maxHold {
+			maxHold = len(m) - 1
+		}
+	}
+	if len(s) < maxHold {
+		maxHold = len(s)
+	}
+	for k := maxHold; k >= 1; k-- {
+		tail := s[len(s)-k:]
+		for _, m := range markers {
+			if strings.HasPrefix(m, tail) {
+				return k
+			}
+		}
+	}
+	return 0
+}
+
+// ---------------------------------------------------------------------------
+// call:name{...} argument decoder
+//
+// Port of vLLM's _parse_gemma4_args / _parse_gemma4_array /
+// _parse_gemma4_value (gemma4_tool_parser.py) in non-partial mode only: this
+// parser decodes exclusively COMPLETE payloads (incomplete ones fall back to
+// raw content at Close), so vLLM's partial-withholding machinery
+// (trailing-dot floats, withheld bare tails) is intentionally not ported.
+//
+// Grammar (inverse of the renderer's formatGemma4Argument, tpl L118-L147):
+//
+//	args    := pair (',' pair)*
+//	pair    := key ':' value          (keys unquoted, up to the first ':')
+//	value   := string | object | array | bare
+//	string  := '<|"|>' ... '<|"|>'    (no escapes; unterminated -> rest)
+//	object  := '{' args '}'           (delimited strings skipped when
+//	array   := '[' value,* ']'         counting braces/brackets)
+//	bare    := true | false | null/none/nil | number | bare-string
+//
+// Output is a JSON object/array string with keys in payload order (Python
+// dict insertion order), built with HTML escaping off so payload text
+// survives byte-for-byte.
+// ---------------------------------------------------------------------------
+
+func isGemma4Space(c byte) bool { return c == ' ' || c == '\n' || c == '\t' }
+
+// gemma4MaxArgsDepth caps the mutual recursion between decodeGemma4Args and
+// decodeGemma4Array. Defense against model-generated deep nesting: a Go stack
+// overflow is a fatal process kill, not a recoverable error, so past the cap
+// a nested body gracefully degrades to a JSON string of its raw text.
+const gemma4MaxArgsDepth = 100
+
+// decodeGemma4Args decodes one args body (the text between the outer braces
+// of call:name{...}) into a JSON object string. depth is the current nesting
+// level (0 at the payload root); see gemma4MaxArgsDepth.
+func decodeGemma4Args(s string, depth int) string {
+	if depth > gemma4MaxArgsDepth {
+		return gemma4JSONString(s)
+	}
+	var b strings.Builder
+	b.WriteString("{")
+	first := true
+	pair := func(key, val string) {
+		if !first {
+			b.WriteString(",")
+		}
+		first = false
+		b.WriteString(gemma4JSONString(key))
+		b.WriteString(":")
+		b.WriteString(val)
+	}
+	i, n := 0, len(s)
+	for i < n {
+		for i < n && (isGemma4Space(s[i]) || s[i] == ',') {
+			i++
+		}
+		if i >= n {
+			break
+		}
+		keyStart := i
+		for i < n && s[i] != ':' {
+			i++
+		}
+		if i >= n {
+			break // no ':' -> trailing junk, dropped (vLLM does the same)
+		}
+		key := strings.TrimSpace(s[keyStart:i])
+		i++ // skip ':'
+		for i < n && isGemma4Space(s[i]) {
+			i++
+		}
+		if i >= n {
+			pair(key, `""`) // "key:" with nothing after -> empty string
+			break
+		}
+		switch {
+		case strings.HasPrefix(s[i:], gemma4StringDelim):
+			i += len(gemma4StringDelim)
+			if end := strings.Index(s[i:], gemma4StringDelim); end == -1 {
+				pair(key, gemma4JSONString(s[i:])) // unterminated -> take rest
+				i = n
+			} else {
+				pair(key, gemma4JSONString(s[i:i+end]))
+				i += end + len(gemma4StringDelim)
+			}
+		case s[i] == '{':
+			inner, next := scanGemma4Balanced(s, i, '{', '}')
+			pair(key, decodeGemma4Args(inner, depth+1))
+			i = next
+		case s[i] == '[':
+			inner, next := scanGemma4Balanced(s, i, '[', ']')
+			pair(key, decodeGemma4Array(inner, depth+1))
+			i = next
+		default:
+			valStart := i
+			for i < n && s[i] != ',' && s[i] != '}' && s[i] != ']' {
+				i++
+			}
+			if i == valStart {
+				// No progress (value starts on a stray '}'/']'): abort on
+				// malformed input rather than loop, like vLLM.
+				i = n
+				continue
+			}
+			pair(key, decodeGemma4Bare(s[valStart:i]))
+		}
+	}
+	b.WriteString("}")
+	return b.String()
+}
+
+// decodeGemma4Array decodes one array body (the text between '[' and ']')
+// into a JSON array string. depth is the current nesting level; see
+// gemma4MaxArgsDepth.
+func decodeGemma4Array(s string, depth int) string {
+	if depth > gemma4MaxArgsDepth {
+		return gemma4JSONString(s)
+	}
+	var b strings.Builder
+	b.WriteString("[")
+	first := true
+	item := func(val string) {
+		if !first {
+			b.WriteString(",")
+		}
+		first = false
+		b.WriteString(val)
+	}
+	i, n := 0, len(s)
+	for i < n {
+		for i < n && (isGemma4Space(s[i]) || s[i] == ',') {
+			i++
+		}
+		if i >= n {
+			break
+		}
+		switch {
+		case strings.HasPrefix(s[i:], gemma4StringDelim):
+			i += len(gemma4StringDelim)
+			if end := strings.Index(s[i:], gemma4StringDelim); end == -1 {
+				item(gemma4JSONString(s[i:]))
+				i = n
+			} else {
+				item(gemma4JSONString(s[i : i+end]))
+				i += end + len(gemma4StringDelim)
+			}
+		case s[i] == '{':
+			inner, next := scanGemma4Balanced(s, i, '{', '}')
+			item(decodeGemma4Args(inner, depth+1))
+			i = next
+		case s[i] == '[':
+			inner, next := scanGemma4Balanced(s, i, '[', ']')
+			item(decodeGemma4Array(inner, depth+1))
+			i = next
+		default:
+			valStart := i
+			for i < n && s[i] != ',' && s[i] != ']' {
+				i++
+			}
+			if i == valStart {
+				i = n // no progress: abort on malformed input, like vLLM
+				continue
+			}
+			item(decodeGemma4Bare(s[valStart:i]))
+		}
+	}
+	b.WriteString("]")
+	return b.String()
+}
+
+// scanGemma4Balanced scans a brace/bracket-balanced span starting at the
+// opener s[start], skipping over <|"|>-delimited strings so structural
+// characters inside them do not count (vLLM's depth scan). Returns the inner
+// text and the index just past the closer; an unterminated span yields the
+// rest of the string (the inner decoder still extracts what is there - this
+// path is only reachable from genuinely malformed complete payloads).
+func scanGemma4Balanced(s string, start int, open, close byte) (string, int) {
+	depth := 1
+	i := start + 1
+	innerStart := i
+	n := len(s)
+	for i < n && depth > 0 {
+		if strings.HasPrefix(s[i:], gemma4StringDelim) {
+			i += len(gemma4StringDelim)
+			if nd := strings.Index(s[i:], gemma4StringDelim); nd == -1 {
+				i = n
+			} else {
+				i += nd + len(gemma4StringDelim)
+			}
+			continue
+		}
+		switch s[i] {
+		case open:
+			depth++
+		case close:
+			depth--
+		}
+		i++
+	}
+	if depth > 0 {
+		return s[innerStart:], n
+	}
+	return s[innerStart : i-1], i
+}
+
+// decodeGemma4Bare maps an undelimited value to its JSON form: booleans,
+// null aliases (null/none/nil, case-insensitive - the renderer writes
+// Python None as "None", tpl L144-L145 via format_argument's else branch),
+// numbers (vLLM's rule: a '.' tries float, otherwise int; anything that
+// fails parses as a bare string).
+func decodeGemma4Bare(raw string) string {
+	v := strings.TrimSpace(raw)
+	if v == "" {
+		return `""`
+	}
+	if v == "true" || v == "false" {
+		return v
+	}
+	switch strings.ToLower(v) {
+	case "null", "none", "nil":
+		return "null"
+	}
+	if strings.Contains(v, ".") {
+		if f, err := strconv.ParseFloat(v, 64); err == nil {
+			return formatGemma4Float(f)
+		}
+	} else if iv, err := strconv.ParseInt(v, 10, 64); err == nil {
+		return strconv.FormatInt(iv, 10)
+	}
+	return gemma4JSONString(v)
+}
+
+// formatGemma4Float renders like Python's json.dumps(float): integral floats
+// keep a ".0" suffix ("108." decodes to 108.0, not 108), so the arguments
+// JSON matches what vLLM would have produced for the same payload.
+func formatGemma4Float(f float64) string {
+	s := strconv.FormatFloat(f, 'g', -1, 64)
+	if !strings.ContainsAny(s, ".eE") {
+		s += ".0"
+	}
+	return s
+}
+
+// gemma4JSONString encodes a JSON string WITHOUT HTML escaping (json.Marshal
+// would escape the angle brackets in "<div>" to \u003c / \u003e sequences;
+// payload text should survive
+// byte-for-byte, like Python's json.dumps(ensure_ascii=False)).
+func gemma4JSONString(s string) string {
+	var sb strings.Builder
+	enc := json.NewEncoder(&sb)
+	enc.SetEscapeHTML(false)
+	if err := enc.Encode(s); err != nil {
+		// Unreachable for plain strings; fall back to default escaping
+		// rather than emitting invalid JSON.
+		b, mErr := json.Marshal(s)
+		if mErr != nil {
+			return `""`
+		}
+		return string(b)
+	}
+	// Encode appends a trailing newline.
+	return strings.TrimSuffix(sb.String(), "\n")
+}

backend/go/dllm/gemma4_parser_test.go

@@ -0,0 +1,592 @@
+package main
+
+// Parser specs for Gemma4Parser (model output text -> pb.ChatDelta events).
+//
+// Fixture provenance:
+//   - Entries marked "vLLM: <name>" are direct ports of the named test from
+//     vLLM PR #45163, tests/tool_parsers/test_gemma4_tool_parser.py (the
+//     authoritative test-suite for the gemma4 tool-call wire format). The
+//     streaming tests' chunk lists are reused verbatim as Feed fragments.
+//   - Decoder entries port the TestParseGemma4Args / TestParseGemma4Array
+//     classes from the same file (non-partial mode only; this parser never
+//     decodes partial payloads, see the divergence note in gemma4_parser.go).
+//   - Channel/turn-marker expectations come from the chat template embedded
+//     in gemma4_renderer.go (tpl L356-L362 generation prompt, L148-L158
+//     strip_thinking) and vLLM's Gemma4ReasoningParser
+//     (vllm/reasoning/gemma4_reasoning_parser.py).
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// flatGemma4Tool is one accumulated tool call, mirroring how LocalAI core
+// folds ToolCallDelta streams (pkg/functions/chat_deltas.go
+// ToolCallsFromChatDeltas: name/id latch on first non-empty, arguments
+// concatenate per index). Tests flatten through the same rules so they
+// assert exactly what core will reconstruct.
+type flatGemma4Tool struct {
+	id   string
+	name string
+	args string
+}
+
+func flattenGemma4Deltas(deltas []*pb.ChatDelta) (string, string, []flatGemma4Tool) {
+	var content, reasoning strings.Builder
+	byIndex := map[int32]*flatGemma4Tool{}
+	maxIdx := int32(-1)
+	for _, d := range deltas {
+		content.WriteString(d.GetContent())
+		reasoning.WriteString(d.GetReasoningContent())
+		for _, tc := range d.GetToolCalls() {
+			acc, ok := byIndex[tc.GetIndex()]
+			if !ok {
+				acc = &flatGemma4Tool{}
+				byIndex[tc.GetIndex()] = acc
+			}
+			if tc.GetName() != "" {
+				acc.name = tc.GetName()
+			}
+			if tc.GetId() != "" {
+				acc.id = tc.GetId()
+			}
+			acc.args += tc.GetArguments()
+			if tc.GetIndex() > maxIdx {
+				maxIdx = tc.GetIndex()
+			}
+		}
+	}
+	var tools []flatGemma4Tool
+	for i := int32(0); i <= maxIdx; i++ {
+		if acc, ok := byIndex[i]; ok {
+			tools = append(tools, *acc)
+		}
+	}
+	return content.String(), reasoning.String(), tools
+}
+
+type wantGemma4Tool struct {
+	name     string
+	argsJSON string // compared with MatchJSON (key order irrelevant)
+}
+
+type parseGemma4Case struct {
+	startInThought bool
+	fragments      []string
+	wantContent    string
+	wantReasoning  string
+	wantTools      []wantGemma4Tool
+}
+
+func parseGemma4Fragments(startInThought bool, fragments []string) []*pb.ChatDelta {
+	p := NewGemma4Parser(startInThought)
+	var all []*pb.ChatDelta
+	for _, f := range fragments {
+		all = append(all, p.Feed(f)...)
+	}
+	return append(all, p.Close()...)
+}
+
+var _ = Describe("Gemma4Parser", func() {
+	DescribeTable("parses streamed gemma4 output into ChatDeltas",
+		func(c parseGemma4Case) {
+			content, reasoning, tools := flattenGemma4Deltas(parseGemma4Fragments(c.startInThought, c.fragments))
+			Expect(content).To(Equal(c.wantContent))
+			Expect(reasoning).To(Equal(c.wantReasoning))
+			Expect(tools).To(HaveLen(len(c.wantTools)))
+			seenIDs := map[string]bool{}
+			for i, want := range c.wantTools {
+				Expect(tools[i].name).To(Equal(want.name), "tool %d name", i)
+				Expect(tools[i].args).To(MatchJSON(want.argsJSON), "tool %d arguments", i)
+				Expect(tools[i].id).ToNot(BeEmpty(), "tool %d id", i)
+				Expect(seenIDs).ToNot(HaveKey(tools[i].id), "tool %d id must be unique", i)
+				seenIDs[tools[i].id] = true
+			}
+		},
+
+		// --- (1) pure content -------------------------------------------------
+		// vLLM: test_no_tool_calls
+		Entry("pure content, single fragment", parseGemma4Case{
+			fragments:   []string{"Hello, how can I help you today?"},
+			wantContent: "Hello, how can I help you today?",
+		}),
+
+		// --- (2) thought -> final transition ----------------------------------
+		// enable_thinking render: prompt ends at <|turn>model\n and the model
+		// opens/closes its own thought channel in the OUTPUT (vLLM
+		// Gemma4ReasoningParser docstring; tpl L356-L362). The "thought\n"
+		// role label after <|channel> is structural and must be stripped
+		// (vLLM _THOUGHT_PREFIX handling).
+		Entry("thought channel then final content", parseGemma4Case{
+			fragments:     []string{"<|channel>thought\nLet me think about this.\n<channel|>The answer is 42."},
+			wantReasoning: "Let me think about this.\n",
+			wantContent:   "The answer is 42.",
+		}),
+
+		// --- (3) startInThought both ways -------------------------------------
+		Entry("startInThought=true routes initial text to reasoning until <channel|>", parseGemma4Case{
+			startInThought: true,
+			fragments:      []string{"I am thinking hard.<channel|>Done."},
+			wantReasoning:  "I am thinking hard.",
+			wantContent:    "Done.",
+		}),
+		// A stray <channel|> with no open channel is swallowed, matching the
+		// template's strip_thinking (tpl L148-L158: the marker is dropped,
+		// text on both sides is kept).
+		Entry("startInThought=false keeps the same text as content, stray <channel|> swallowed", parseGemma4Case{
+			startInThought: false,
+			fragments:      []string{"I am thinking hard.<channel|>Done."},
+			wantContent:    "I am thinking hard.Done.",
+		}),
+
+		// --- (4) one tool call, full payload type zoo --------------------------
+		Entry("single tool call: strings, numbers, bools, null, nested object and array", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:complex_function{text:<|"|>with, comma and {braces}<|"|>,count:42,score:3.14,yes:true,no:false,nothing:null,obj:{inner:<|"|>v<|"|>,k:1},arr:[<|"|>a<|"|>,2,true]}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{
+				name:     "complex_function",
+				argsJSON: `{"text":"with, comma and {braces}","count":42,"score":3.14,"yes":true,"no":false,"nothing":null,"obj":{"inner":"v","k":1},"arr":["a",2,true]}`,
+			}},
+		}),
+
+		// --- (5) payload split across 3 fragments ------------------------------
+		Entry("tool-call payload split across three fragments", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>call:get_weather{loc",
+				`ation:<|"|>Paris, Fra`,
+				`nce<|"|>}<tool_call|>`,
+			},
+			wantTools: []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"Paris, France"}`}},
+		}),
+
+		// --- (6) marker split across fragments ----------------------------------
+		Entry("tool-call open marker split across fragments", parseGemma4Case{
+			fragments: []string{
+				"<|tool_ca",
+				`ll>call:get_weather{location:<|"|>London<|"|>}<tool_call|>`,
+			},
+			wantTools: []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"London"}`}},
+		}),
+		Entry("channel open marker split across fragments", parseGemma4Case{
+			fragments: []string{
+				"<|chan",
+				"nel>thought\ndeep thought<channel|>final",
+			},
+			wantReasoning: "deep thought",
+			wantContent:   "final",
+		}),
+
+		// --- (7) trailing partial marker held, flushed by Close -----------------
+		Entry("trailing partial marker is held back and flushed by Close", parseGemma4Case{
+			fragments:   []string{"Hello <|tool"},
+			wantContent: "Hello <|tool",
+		}),
+
+		// --- (8) malformed/incomplete payload -> content fallback ---------------
+		// vLLM: test_incomplete_tool_call (no end marker: the whole text stays
+		// content, never silently dropped).
+		Entry("incomplete tool payload at Close is emitted as raw content", parseGemma4Case{
+			fragments:   []string{`<|tool_call>call:get_weather{location:<|"|>London`},
+			wantContent: `<|tool_call>call:get_weather{location:<|"|>London`,
+		}),
+		Entry("malformed complete payload is emitted as raw content, parsing continues", parseGemma4Case{
+			fragments:   []string{"<|tool_call>oops no call syntax<tool_call|> done"},
+			wantContent: "<|tool_call>oops no call syntax<tool_call|> done",
+		}),
+
+		// --- (9) <turn|> ends the turn -------------------------------------------
+		Entry("text after <turn|> is ignored, including later fragments", parseGemma4Case{
+			fragments: []string{
+				"before<turn|>after",
+				`more <|tool_call>call:f{}<tool_call|>`,
+			},
+			wantContent: "before",
+		}),
+		Entry("<turn|> inside a thought channel ends the turn", parseGemma4Case{
+			startInThought: true,
+			fragments:      []string{"thinking<turn|>ignored"},
+			wantReasoning:  "thinking",
+		}),
+
+		// --- (10) ported vLLM non-streaming cases ---------------------------------
+		// vLLM: test_single_tool_call
+		Entry("vLLM: test_single_tool_call", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:get_weather{location:<|"|>London<|"|>}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"London"}`}},
+		}),
+		// vLLM: test_multiple_arguments
+		Entry("vLLM: test_multiple_arguments", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:get_weather{location:<|"|>San Francisco<|"|>,unit:<|"|>celsius<|"|>}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"San Francisco","unit":"celsius"}`}},
+		}),
+		// vLLM: test_text_before_tool_call. DIVERGENCE: vLLM's non-streaming
+		// extractor trims the content ("...you."); a streaming parser cannot
+		// retroactively trim already-emitted text, so the trailing space is
+		// kept (vLLM's own streaming path keeps it too, see
+		// test_streaming_text_before_tool_call which only checks a prefix).
+		Entry("vLLM: test_text_before_tool_call (streaming semantics: no trim)", parseGemma4Case{
+			fragments:   []string{`Let me check the weather for you. <|tool_call>call:get_weather{location:<|"|>Paris<|"|>}<tool_call|>`},
+			wantContent: "Let me check the weather for you. ",
+			wantTools:   []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"Paris"}`}},
+		}),
+		// vLLM: test_multiple_tool_calls (also covers case 11: multi-tool sequence)
+		Entry("vLLM: test_multiple_tool_calls", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:get_weather{location:<|"|>London<|"|>}<tool_call|><|tool_call>call:get_time{location:<|"|>London<|"|>}<tool_call|>`},
+			wantTools: []wantGemma4Tool{
+				{name: "get_weather", argsJSON: `{"location":"London"}`},
+				{name: "get_time", argsJSON: `{"location":"London"}`},
+			},
+		}),
+		// vLLM: test_nested_arguments
+		Entry("vLLM: test_nested_arguments", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:complex_function{nested:{inner:<|"|>value<|"|>},list:[<|"|>a<|"|>,<|"|>b<|"|>]}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{name: "complex_function", argsJSON: `{"nested":{"inner":"value"},"list":["a","b"]}`}},
+		}),
+		// vLLM: test_tool_call_with_number_and_boolean
+		Entry("vLLM: test_tool_call_with_number_and_boolean", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:set_status{is_active:true,count:42,score:3.14}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{name: "set_status", argsJSON: `{"is_active":true,"count":42,"score":3.14}`}},
+		}),
+		// vLLM: test_hyphenated_function_name
+		Entry("vLLM: test_hyphenated_function_name", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:get-weather{location:<|"|>London<|"|>}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{name: "get-weather", argsJSON: `{"location":"London"}`}},
+		}),
+		// vLLM: test_dotted_function_name
+		Entry("vLLM: test_dotted_function_name", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:weather.get{location:<|"|>London<|"|>}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{name: "weather.get", argsJSON: `{"location":"London"}`}},
+		}),
+		// vLLM: test_no_arguments
+		Entry("vLLM: test_no_arguments", parseGemma4Case{
+			fragments: []string{"<|tool_call>call:get_status{}<tool_call|>"},
+			wantTools: []wantGemma4Tool{{name: "get_status", argsJSON: `{}`}},
+		}),
+
+		// --- ported vLLM streaming cases (chunk lists reused as fragments) --------
+		// vLLM: test_basic_streaming_single_tool
+		Entry("vLLM: test_basic_streaming_single_tool", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:get_weather{",
+				`location:<|"|>Paris`,
+				", France",
+				`<|"|>}`,
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"Paris, France"}`}},
+		}),
+		// vLLM: test_streaming_multi_arg
+		Entry("vLLM: test_streaming_multi_arg", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:get_weather{",
+				`location:<|"|>Tokyo<|"|>,`,
+				`unit:<|"|>celsius<|"|>}`,
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"Tokyo","unit":"celsius"}`}},
+		}),
+		// vLLM: test_streaming_text_before_tool_call
+		Entry("vLLM: test_streaming_text_before_tool_call", parseGemma4Case{
+			fragments: []string{
+				"Let me check ",
+				"the weather. ",
+				"<|tool_call>",
+				"call:get_weather{",
+				`location:<|"|>London<|"|>}`,
+				"<tool_call|>",
+			},
+			wantContent: "Let me check the weather. ",
+			wantTools:   []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"London"}`}},
+		}),
+		// vLLM: test_streaming_numeric_args
+		Entry("vLLM: test_streaming_numeric_args", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:set_config{",
+				"count:42,",
+				"active:true}",
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "set_config", argsJSON: `{"count":42,"active":true}`}},
+		}),
+		// vLLM: test_streaming_boolean_split_across_chunks
+		Entry("vLLM: test_streaming_boolean_split_across_chunks", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:search{input:{all:tru",
+				"e}}",
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "search", argsJSON: `{"input":{"all":true}}`}},
+		}),
+		// vLLM: test_streaming_false_split_across_chunks
+		Entry("vLLM: test_streaming_false_split_across_chunks", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:set{flag:fals",
+				"e}",
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "set", argsJSON: `{"flag":false}`}},
+		}),
+		// vLLM: test_streaming_number_split_across_chunks
+		Entry("vLLM: test_streaming_number_split_across_chunks", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:set{count:4",
+				"2}",
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "set", argsJSON: `{"count":42}`}},
+		}),
+		// vLLM: test_streaming_empty_args
+		Entry("vLLM: test_streaming_empty_args", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:get_status{}",
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "get_status", argsJSON: `{}`}},
+		}),
+		// vLLM: test_streaming_split_delimiter_no_invalid_json (string
+		// delimiter <|"|> split across fragments must not leak fragments).
+		Entry("vLLM: test_streaming_split_delimiter_no_invalid_json", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:todowrite{",
+				`content:<|"|>Buy milk<|`,
+				`"|>}`,
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{name: "todowrite", argsJSON: `{"content":"Buy milk"}`}},
+		}),
+		// vLLM: test_streaming_does_not_duplicate_plain_text_after_tool_call
+		Entry("vLLM: test_streaming_does_not_duplicate_plain_text_after_tool_call", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:get_weather{",
+				`location:<|"|>Paris<|"|>}`,
+				"<tool_call|><",
+				"div>",
+			},
+			wantContent: "<div>",
+			wantTools:   []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"Paris"}`}},
+		}),
+		// vLLM: test_streaming_html_argument_does_not_duplicate_tag_prefixes
+		Entry("vLLM: test_streaming_html_argument_does_not_duplicate_tag_prefixes", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:write_file{",
+				`path:<|"|>index.html<|"|>,`,
+				`content:<|"|><!DOCTYPE html>` + "\n<",
+				`html lang="zh-CN">` + "\n<",
+				"head>\n    <",
+				`meta charset="UTF-8">` + "\n    <",
+				`meta name="viewport" content="width=device-width">` + "\n",
+				`<|"|>}`,
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{
+				name:     "write_file",
+				argsJSON: `{"path":"index.html","content":"<!DOCTYPE html>\n<html lang=\"zh-CN\">\n<head>\n    <meta charset=\"UTF-8\">\n    <meta name=\"viewport\" content=\"width=device-width\">\n"}`,
+			}},
+		}),
+		// vLLM: test_streaming_single_chunk_complete_tool_call
+		Entry("vLLM: test_streaming_single_chunk_complete_tool_call", parseGemma4Case{
+			fragments: []string{`<|tool_call>call:name_a_color{color_hex:<|"|>00ff11<|"|>}<tool_call|>`},
+			wantTools: []wantGemma4Tool{{name: "name_a_color", argsJSON: `{"color_hex":"00ff11"}`}},
+		}),
+		// vLLM: test_streaming_multi_chunk_batched_tool_calls (two complete
+		// calls in ONE fragment; both must come out with distinct indices)
+		Entry("vLLM: test_streaming_multi_chunk_batched_tool_calls", parseGemma4Case{
+			fragments: []string{
+				`<|tool_call>call:get_weather{location:<|"|>London<|"|>}<tool_call|>` +
+					`<|tool_call>call:get_time{timezone:<|"|>GMT<|"|>}<tool_call|>`,
+			},
+			wantTools: []wantGemma4Tool{
+				{name: "get_weather", argsJSON: `{"location":"London"}`},
+				{name: "get_time", argsJSON: `{"timezone":"GMT"}`},
+			},
+		}),
+		// vLLM: test_streaming_trailing_bare_bool_not_duplicated
+		Entry("vLLM: test_streaming_trailing_bare_bool_not_duplicated", parseGemma4Case{
+			fragments: []string{
+				"<|tool_call>",
+				"call:Edit{",
+				`file_path:<|"|>src/env.py<|"|>,`,
+				`old_string:<|"|>old_val<|"|>,`,
+				`new_string:<|"|>new_val<|"|>,`,
+				"replace_all:",
+				"false}",
+				"<tool_call|>",
+			},
+			wantTools: []wantGemma4Tool{{
+				name:     "Edit",
+				argsJSON: `{"file_path":"src/env.py","old_string":"old_val","new_string":"new_val","replace_all":false}`,
+			}},
+		}),
+
+		// --- implicit reasoning end on <|tool_call> (vLLM is_reasoning_end:
+		// a tool_call token means reasoning is over) -----------------------------
+		Entry("tool call inside an open thought channel ends the reasoning", parseGemma4Case{
+			startInThought: true,
+			fragments:      []string{`need the weather<|tool_call>call:get_weather{location:<|"|>Rome<|"|>}<tool_call|>`},
+			wantReasoning:  "need the weather",
+			wantTools:      []wantGemma4Tool{{name: "get_weather", argsJSON: `{"location":"Rome"}`}},
+		}),
+
+		// --- (12) empty fragments are no-ops --------------------------------------
+		Entry("empty fragments are no-ops", parseGemma4Case{
+			fragments:   []string{"", "Hello", "", "", " world", ""},
+			wantContent: "Hello world",
+		}),
+	)
+
+	It("returns no deltas for an empty fragment and after Close", func() {
+		p := NewGemma4Parser(false)
+		Expect(p.Feed("")).To(BeEmpty())
+		Expect(p.Feed("hi")).ToNot(BeEmpty())
+		Expect(p.Close()).To(BeEmpty()) // nothing held back
+		// The parser is finished after Close: further input is dropped.
+		Expect(p.Feed("more")).To(BeEmpty())
+		Expect(p.Close()).To(BeEmpty())
+	})
+
+	It("generates index-based tool call ids (call_<index>)", func() {
+		// Mirrors the index-based id convention of pkg/grpc/rich_test.go and
+		// keeps ids deterministic for the split-invariance property below.
+		deltas := parseGemma4Fragments(false, []string{
+			`<|tool_call>call:a{}<tool_call|><|tool_call>call:b{}<tool_call|>`,
+		})
+		_, _, tools := flattenGemma4Deltas(deltas)
+		Expect(tools).To(HaveLen(2))
+		Expect(tools[0].id).To(Equal("call_0"))
+		Expect(tools[1].id).To(Equal("call_1"))
+	})
+
+	// Property: for a fixed full output, EVERY 2-split position must yield
+	// exactly the same flattened result as the unsplit parse. This kills
+	// fragment-boundary bugs (mid-marker, mid-delimiter, mid-payload splits).
+	DescribeTable("2-split fragment invariance",
+		func(startInThought bool, full string) {
+			refContent, refReasoning, refTools := flattenGemma4Deltas(
+				parseGemma4Fragments(startInThought, []string{full}))
+			for i := 0; i <= len(full); i++ {
+				content, reasoning, tools := flattenGemma4Deltas(
+					parseGemma4Fragments(startInThought, []string{full[:i], full[i:]}))
+				Expect(content).To(Equal(refContent), fmt.Sprintf("content diverged at split %d", i))
+				Expect(reasoning).To(Equal(refReasoning), fmt.Sprintf("reasoning diverged at split %d", i))
+				Expect(tools).To(Equal(refTools), fmt.Sprintf("tool calls diverged at split %d", i))
+			}
+		},
+		Entry("thought + content + two tool calls + turn end", false,
+			"<|channel>thought\nPondering the request...\n<channel|>Sure - calling tools now. "+
+				`<|tool_call>call:get_weather{location:<|"|>Paris, France<|"|>,unit:<|"|>celsius<|"|>,days:3,detailed:true}<tool_call|>`+
+				`<|tool_call>call:get_time{timezone:<|"|>Europe/Lisbon<|"|>,nested:{flag:false,vals:[1,2.5,<|"|>x<|"|>]}}<tool_call|>`+
+				"Done.<turn|>ignored tail"),
+		Entry("startInThought + tool call + trailing partial marker", true,
+			`Deep thought<channel|>final answer <|tool_call>call:noop{}<tool_call|> trailing <|tool`),
+		Entry("malformed payload fallback", false,
+			`pre <|tool_call>not a call<tool_call|> post`),
+	)
+})
+
+// Decoder-level ports of vLLM's TestParseGemma4Args / TestParseGemma4Array
+// (non-partial mode; the partial-withholding tests do not apply because this
+// parser only ever decodes COMPLETE payloads, see gemma4_parser.go).
+var _ = Describe("decodeGemma4Args", func() {
+	DescribeTable("decodes the gemma4 call syntax into JSON arguments",
+		func(in, wantJSON string) {
+			Expect(decodeGemma4Args(in, 0)).To(MatchJSON(wantJSON))
+		},
+		// vLLM: test_empty_string / test_whitespace_only
+		Entry("empty string", "", `{}`),
+		Entry("whitespace only", "   ", `{}`),
+		// vLLM: test_single_string_value
+		Entry("single string value", `location:<|"|>Paris<|"|>`, `{"location":"Paris"}`),
+		// vLLM: test_string_value_with_comma
+		Entry("string value with comma", `location:<|"|>Paris, France<|"|>`, `{"location":"Paris, France"}`),
+		// vLLM: test_multiple_string_values
+		Entry("multiple string values", `location:<|"|>San Francisco<|"|>,unit:<|"|>celsius<|"|>`, `{"location":"San Francisco","unit":"celsius"}`),
+		// vLLM: test_integer_value / test_float_value
+		Entry("integer value", "count:42", `{"count":42}`),
+		Entry("float value", "score:3.14", `{"score":3.14}`),
+		// vLLM: test_boolean_true / test_boolean_false
+		Entry("boolean true", "flag:true", `{"flag":true}`),
+		Entry("boolean false", "flag:false", `{"flag":false}`),
+		// vLLM: test_null_value (bare null must become JSON null, not "null")
+		Entry("null value", "param:null", `{"param":null}`),
+		// vLLM: test_mixed_types
+		Entry("mixed types", `name:<|"|>test<|"|>,count:42,active:true,score:3.14`,
+			`{"name":"test","count":42,"active":true,"score":3.14}`),
+		// vLLM: test_nested_object
+		Entry("nested object", `nested:{inner:<|"|>value<|"|>}`, `{"nested":{"inner":"value"}}`),
+		// vLLM: test_array_of_strings
+		Entry("array of strings", `items:[<|"|>a<|"|>,<|"|>b<|"|>]`, `{"items":["a","b"]}`),
+		// vLLM: test_unterminated_string (take everything after the delimiter)
+		Entry("unterminated string", `key:<|"|>unterminated`, `{"key":"unterminated"}`),
+		// vLLM: test_empty_value (key with no value after colon)
+		Entry("empty value", "key:", `{"key":""}`),
+		// vLLM: test_trailing_dot_float_partial_withheld, non-partial branch
+		// (trailing-dot floats parse normally outside streaming).
+		Entry("trailing dot float, complete payload", "left:108.,right:22.8", `{"left":108.0,"right":22.8}`),
+	)
+
+	It("terminates and yields valid JSON on malformed input", func() {
+		// vLLM: test_malformed_partial_array (the assertion there is only
+		// "returns a dict without hanging"; ours is "valid JSON object").
+		out := decodeGemma4Args(":[t:[]", 0)
+		var v map[string]any
+		Expect(json.Unmarshal([]byte(out), &v)).To(Succeed())
+	})
+
+	It("degrades nesting beyond the recursion cap to a string value", func() {
+		// 200 levels of a:{a:{...a:1...}}. Without the depth cap the mutual
+		// recursion would grow the stack with the model's output; a Go stack
+		// overflow is a fatal process kill, so levels past gemma4MaxArgsDepth
+		// must gracefully fall back to the raw inner text as a JSON string.
+		const depth = 200
+		body := strings.Repeat("a:{", depth-1) + "a:1" + strings.Repeat("}", depth-1)
+		out := decodeGemma4Args(body, 0)
+		var v map[string]any
+		Expect(json.Unmarshal([]byte(out), &v)).To(Succeed())
+		levels := 0
+		var cur any = v
+		for {
+			m, ok := cur.(map[string]any)
+			if !ok {
+				break
+			}
+			Expect(m).To(HaveKey("a"))
+			cur = m["a"]
+			levels++
+		}
+		Expect(levels).To(Equal(gemma4MaxArgsDepth + 1))
+		Expect(cur).To(BeAssignableToTypeOf(""))
+		Expect(cur).To(ContainSubstring("a:{"))
+	})
+})
+
+var _ = Describe("decodeGemma4Array", func() {
+	DescribeTable("decodes gemma4 array bodies into JSON arrays",
+		func(in, wantJSON string) {
+			Expect(decodeGemma4Array(in, 0)).To(MatchJSON(wantJSON))
+		},
+		// vLLM: test_string_array / test_empty_array / test_bare_values
+		Entry("string array", `<|"|>a<|"|>,<|"|>b<|"|>`, `["a","b"]`),
+		Entry("empty array", "", `[]`),
+		Entry("bare values", "42,true,3.14", `[42,true,3.14]`),
+		// vLLM: test_string_element_with_closing_bracket (a ']' inside a
+		// delimited string must not close the array)
+		Entry("string element with closing bracket", `[<|"|>a]b<|"|>,<|"|>c<|"|>],<|"|>tail<|"|>`, `[["a]b","c"],"tail"]`),
+		// vLLM: test_stray_closing_bracket (no-progress abort, keep prefix)
+		Entry("stray closing bracket", "42,]trailing", `[42]`),
+	)
+})

backend/go/dllm/gemma4_renderer_test.go

@@ -0,0 +1,406 @@
+package main
+
+// Renderer specs for RenderGemma4 against the canonical gemma4 chat template
+// (see the normative template comment in gemma4_renderer.go).
+//
+// Fixture provenance:
+//   - "single user message" and "enable_thinking" are the EXACT expected
+//     decodes from transformers tests/models/diffusion_gemma/
+//     test_modeling_diffusion_gemma.py (test_diffusion_gemma_chat_template
+//     and ..._with_thinking) with ONE difference: the transformers fixtures
+//     start with "<bos>" because apply_chat_template tokenizes the rendered
+//     text with add_bos. Our prompt goes through dllm_capi_generate, whose
+//     run_generate already tokenizes with prepend_bos = vocab.add_bos
+//     (dllm.cpp src/capi.cpp:230-231, true for gemma4), so the renderer must
+//     NOT emit a literal <bos> (it would double) and every expected string
+//     here drops that leading token.
+//   - All other expected strings were produced by rendering the verbatim
+//     GGUF template with jinja2 3.1.2 (bos_token="<bos>") and dropping the
+//     leading "<bos>" for the same reason.
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// Two-function tools array used by the tool fixtures (OpenAI wire shape, as
+// LocalAI passes it through PredictOptions.Tools).
+const testToolsJSON = `[{"type":"function","function":{"name":"get_weather","description":"Get the current weather in a location.","parameters":{"type":"object","properties":{"location":{"type":"string","description":"The city name."},"unit":{"type":"string","enum":["celsius","fahrenheit"]}},"required":["location"]}}},{"type":"function","function":{"name":"get_time","description":"Get the current time in a timezone.","parameters":{"type":"object","properties":{"timezone":{"type":"string","description":"IANA timezone name."}},"required":["timezone"]}}}]`
+
+// The <|tool>...<tool|> block the template renders for testToolsJSON inside
+// the system turn (jinja2-verified).
+const testToolsBlock = `<|tool>declaration:get_weather{description:<|"|>Get the current weather in a location.<|"|>,parameters:{properties:{location:{description:<|"|>The city name.<|"|>,type:<|"|>STRING<|"|>},unit:{enum:[<|"|>celsius<|"|>,<|"|>fahrenheit<|"|>],type:<|"|>STRING<|"|>}},required:[<|"|>location<|"|>],type:<|"|>OBJECT<|"|>}}<tool|><|tool>declaration:get_time{description:<|"|>Get the current time in a timezone.<|"|>,parameters:{properties:{timezone:{description:<|"|>IANA timezone name.<|"|>,type:<|"|>STRING<|"|>}},required:[<|"|>timezone<|"|>],type:<|"|>OBJECT<|"|>}}<tool|>`
+
+// A single tool exercising the deep format_parameters branches: array items
+// (string-typed and nested-array), nullable, enum+nullable, nested object
+// properties/required, and a response declaration.
+const complexToolsJSON = `[{"type":"function","function":{"name":"complex_tool","description":"A complex tool.","parameters":{"type":"object","properties":{"tags":{"type":"array","description":"Tags.","items":{"type":"string"}},"matrix":{"type":"array","items":{"type":"array","items":{"type":"number"}}},"opts":{"type":"object","description":"Options.","properties":{"depth":{"type":"integer","nullable":true}},"required":["depth"]},"mode":{"type":"string","enum":["a","b"],"nullable":true}},"required":["tags","opts"]},"response":{"description":"The result.","type":"object"}}}]`
+
+// jinja2-verified render of complexToolsJSON. Notable template quirks pinned
+// here: nested array items go through format_argument with ESCAPED keys and
+// an un-uppercased type (<|"|>type<|"|>:<|"|>number<|"|>), while direct item
+// types are uppercased; properties dictsort case-insensitively.
+const complexToolsBlock = `<|tool>declaration:complex_tool{description:<|"|>A complex tool.<|"|>,parameters:{properties:{matrix:{items:{items:{<|"|>type<|"|>:<|"|>number<|"|>},type:<|"|>ARRAY<|"|>},type:<|"|>ARRAY<|"|>},mode:{enum:[<|"|>a<|"|>,<|"|>b<|"|>],nullable:true,type:<|"|>STRING<|"|>},opts:{description:<|"|>Options.<|"|>,properties:{depth:{nullable:true,type:<|"|>INTEGER<|"|>}},required:[<|"|>depth<|"|>],type:<|"|>OBJECT<|"|>},tags:{description:<|"|>Tags.<|"|>,items:{type:<|"|>STRING<|"|>},type:<|"|>ARRAY<|"|>}},required:[<|"|>tags<|"|>,<|"|>opts<|"|>],type:<|"|>OBJECT<|"|>},response:{description:<|"|>The result.<|"|>,type:<|"|>OBJECT<|"|>}}<tool|>`
+
+type renderGemma4Case struct {
+	msgs      []*pb.Message
+	toolsJSON string
+	// nImages mirrors len(PredictOptions.Images): the OpenAI layer strips
+	// image content parts out of the messages, so the renderer re-injects
+	// one engine marker per image on the last user message (see the IMAGE
+	// NOTE on RenderGemma4).
+	nImages            int
+	enableThinking     bool
+	noGenerationPrompt bool // inverted so the zero value is the common case
+	expected           string
+}
+
+var _ = Describe("RenderGemma4", func() {
+	DescribeTable("renders the canonical gemma4 prompt",
+		func(c renderGemma4Case) {
+			out, err := RenderGemma4(c.msgs, c.toolsJSON, c.nImages, c.enableThinking, !c.noGenerationPrompt)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(out).To(Equal(c.expected))
+			// The C-ABI generate prepends BOS itself: a literal <bos>
+			// anywhere in the rendered prompt would double-encode it.
+			Expect(out).ToNot(ContainSubstring("<bos>"))
+		},
+
+		// transformers fixture (test_diffusion_gemma_chat_template), sans <bos>:
+		// default thinking pre-opens an EMPTY thought channel in the
+		// generation prompt.
+		Entry("single user message, default (no thinking)", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "Write a long essay about Portugal."},
+			},
+			expected: "<|turn>user\nWrite a long essay about Portugal.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// transformers fixture (test_diffusion_gemma_chat_template_with_thinking),
+		// sans <bos>: a system turn carrying <|think|> and NO auto-opened
+		// thought channel.
+		Entry("enable_thinking=true", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "Write a long essay about Portugal."},
+			},
+			enableThinking: true,
+			expected:       "<|turn>system\n<|think|>\n<turn|>\n<|turn>user\nWrite a long essay about Portugal.<turn|>\n<|turn>model\n",
+		}),
+
+		Entry("multi-turn user/assistant/user", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "Hello, who are you?"},
+				{Role: "assistant", Content: "I am Gemma, a helpful assistant."},
+				{Role: "user", Content: "Tell me a joke."},
+			},
+			expected: "<|turn>user\nHello, who are you?<turn|>\n<|turn>model\nI am Gemma, a helpful assistant.<turn|>\n<|turn>user\nTell me a joke.<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// tpl L178-L195: a leading system message is folded into the system
+		// turn (trimmed) and consumed from the loop.
+		Entry("system message folds into the system turn", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "system", Content: "You are a pirate."},
+				{Role: "user", Content: "Hello!"},
+			},
+			expected: "<|turn>system\nYou are a pirate.<turn|>\n<|turn>user\nHello!<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// tpl L182-L185: <|think|> goes at the very top of the SAME system
+		// turn, before the system prompt text.
+		Entry("system message with enable_thinking shares the turn", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "system", Content: "You are a pirate."},
+				{Role: "user", Content: "Hello!"},
+			},
+			enableThinking: true,
+			expected:       "<|turn>system\n<|think|>\nYou are a pirate.<turn|>\n<|turn>user\nHello!<turn|>\n<|turn>model\n",
+		}),
+
+		// tpl L196-L203: tool declarations render in the system turn, one
+		// <|tool>declaration:...<tool|> block per tool, no separators.
+		Entry("tools array (two functions)", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "What is the weather in Tokyo?"},
+			},
+			toolsJSON: testToolsJSON,
+			expected:  "<|turn>system\n" + testToolsBlock + "<turn|>\n<|turn>user\nWhat is the weather in Tokyo?<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// format_parameters deep branches (tpl L1-L85) + response declaration
+		// (tpl L106-L116).
+		Entry("complex tool schema (array items, nullable, nested object, response)", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "go"},
+			},
+			toolsJSON: complexToolsJSON,
+			expected:  "<|turn>system\n" + complexToolsBlock + "<turn|>\n<|turn>user\ngo<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// tpl L243-L313: assistant tool_calls render as
+		// <|tool_call>call:name{args}<tool_call|>; the following role=tool
+		// message renders inline as <|tool_response>response:name{value:..}
+		// <tool_response|>; the model turn stays OPEN (no <turn|>, no new
+		// generation prompt) so the model continues after the response.
+		Entry("assistant tool_calls + role=tool result", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "What is the weather in Tokyo?"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"Tokyo\",\"unit\":\"celsius\"}"}}]`},
+				{Role: "tool", ToolCallId: "call_1", Content: "Sunny, 22 degrees celsius."},
+			},
+			toolsJSON: testToolsJSON,
+			expected:  "<|turn>system\n" + testToolsBlock + "<turn|>\n<|turn>user\nWhat is the weather in Tokyo?<turn|>\n<|turn>model\n" + `<|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>,unit:<|"|>celsius<|"|>}<tool_call|><|tool_response>response:get_weather{value:<|"|>Sunny, 22 degrees celsius.<|"|>}<tool_response|>`,
+		}),
+
+		// tpl L348-L349: a tool_calls turn with no rendered responses ends
+		// on an OPEN <|tool_response> marker for the runtime to fill, and
+		// add_generation_prompt adds nothing (tpl L357).
+		Entry("assistant tool_calls without a result leaves <|tool_response> open", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "What is the weather in Tokyo?"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"Tokyo\",\"unit\":\"celsius\"}"}}]`},
+			},
+			toolsJSON: testToolsJSON,
+			expected:  "<|turn>system\n" + testToolsBlock + "<turn|>\n<|turn>user\nWhat is the weather in Tokyo?<turn|>\n<|turn>model\n" + `<|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>,unit:<|"|>celsius<|"|>}<tool_call|><|tool_response>`,
+		}),
+
+		// tpl L237-L241: reasoning_content renders as a thought channel only
+		// on a tool-calling turn after the last user message.
+		Entry("reasoning_content with tool_calls renders the thought channel", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "weather?"},
+				{Role: "assistant", Content: "", ReasoningContent: "I should call the tool", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"Tokyo\"}"}}]`},
+				{Role: "tool", ToolCallId: "c1", Content: "Sunny"},
+			},
+			expected: "<|turn>user\nweather?<turn|>\n<|turn>model\n<|channel>thought\nI should call the tool\n<channel|>" + `<|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|><|tool_response>response:get_weather{value:<|"|>Sunny<|"|>}<tool_response|>`,
+		}),
+
+		// tpl L220-L235: the assistant answer following its own tool round
+		// continues the SAME model turn (no second <|turn>model).
+		Entry("tool round then final assistant answer then user", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "weather?"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"Tokyo\"}"}}]`},
+				{Role: "tool", ToolCallId: "c1", Content: "Sunny"},
+				{Role: "assistant", Content: "It is sunny."},
+				{Role: "user", Content: "thanks"},
+			},
+			expected: "<|turn>user\nweather?<turn|>\n<|turn>model\n" + `<|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|><|tool_response>response:get_weather{value:<|"|>Sunny<|"|>}<tool_response|>` + "It is sunny.<turn|>\n<|turn>user\nthanks<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// format_argument (tpl L118-L147): numbers keep their JSON literal,
+		// booleans lower-case, nested maps have unquoted dictsorted keys,
+		// arrays bracketed; top-level args are dictsorted case-insensitively.
+		Entry("tool_call argument types (number/bool/nested/array)", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "go"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"f","arguments":"{\"count\":42,\"ratio\":3.5,\"flag\":true,\"off\":false,\"nested\":{\"x\":\"y\",\"n\":7},\"list\":[\"a\",1,true]}"}}]`},
+			},
+			expected: "<|turn>user\ngo<turn|>\n<|turn>model\n" + `<|tool_call>call:f{count:42,flag:true,list:[<|"|>a<|"|>,1,true],nested:{n:7,x:<|"|>y<|"|>},off:false,ratio:3.5}<tool_call|><|tool_response>`,
+		}),
+
+		// jinja dictsort is case-insensitive: alpha sorts before Beta.
+		Entry("tool_call argument dictsort is case-insensitive", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "go"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"f","arguments":"{\"Beta\":1,\"alpha\":2}"}}]`},
+			},
+			expected: "<|turn>user\ngo<turn|>\n<|turn>model\n<|tool_call>call:f{alpha:2,Beta:1}<tool_call|><|tool_response>",
+		}),
+
+		// jinja renders Python None as "None" (round-trips through vLLM's
+		// parser, which lowers "none" back to null).
+		Entry("tool_call null argument renders as None", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "go"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"f","arguments":"{\"maybe\":null}"}}]`},
+			},
+			expected: "<|turn>user\ngo<turn|>\n<|turn>model\n<|tool_call>call:f{maybe:None}<tool_call|><|tool_response>",
+		}),
+
+		Entry("tool_call empty arguments render empty braces", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "go"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"f","arguments":"{}"}}]`},
+			},
+			expected: "<|turn>user\ngo<turn|>\n<|turn>model\n<|tool_call>call:f{}<tool_call|><|tool_response>",
+		}),
+
+		// tpl L253-L254: a non-object arguments string renders verbatim.
+		Entry("tool_call non-object string arguments render verbatim", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "go"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"f","arguments":"just text"}}]`},
+			},
+			expected: "<|turn>user\ngo<turn|>\n<|turn>model\n<|tool_call>call:f{just text}<tool_call|><|tool_response>",
+		}),
+
+		// tpl L278-L285: unmatched tool_call_id falls back to the tool
+		// message's own name.
+		Entry("tool result name falls back when tool_call_id does not match", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "go"},
+				{Role: "assistant", Content: "", ToolCalls: `[{"index":0,"id":"c1","type":"function","function":{"name":"f","arguments":"{}"}}]`},
+				{Role: "tool", ToolCallId: "OTHER", Name: "named_tool", Content: "out"},
+			},
+			expected: "<|turn>user\ngo<turn|>\n<|turn>model\n" + `<|tool_call>call:f{}<tool_call|><|tool_response>response:named_tool{value:<|"|>out<|"|>}<tool_response|>`,
+		}),
+
+		// strip_thinking (tpl L148-L158): historical assistant content loses
+		// its <|channel>...<channel|> spans.
+		Entry("assistant content thinking channels are stripped", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "hi"},
+				{Role: "assistant", Content: "<|channel>thought\nsecret\n<channel|>visible answer"},
+				{Role: "user", Content: "more"},
+			},
+			expected: "<|turn>user\nhi<turn|>\n<|turn>model\nvisible answer<turn|>\n<|turn>user\nmore<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// tpl L220-L235: consecutive assistant messages suppress the second
+		// <|turn>model (continuation), but each still closes with <turn|>.
+		Entry("consecutive assistant messages continue the model turn", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "hi"},
+				{Role: "assistant", Content: "part one"},
+				{Role: "assistant", Content: "part two"},
+				{Role: "user", Content: "ok"},
+			},
+			expected: "<|turn>user\nhi<turn|>\n<|turn>model\npart one<turn|>\npart two<turn|>\n<|turn>user\nok<turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		Entry("add_generation_prompt=false renders no model turn", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "hi"},
+			},
+			noGenerationPrompt: true,
+			expected:           "<|turn>user\nhi<turn|>\n",
+		}),
+
+		// One engine marker per image, appended directly after the user
+		// text with no separator (tpl L323-L341 emits parts back-to-back;
+		// "<image>" is dllm_capi.h's splice marker, not the template's
+		// <|image|> text token - see the IMAGE NOTE on RenderGemma4).
+		Entry("one image appends one engine marker to the user message", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "What is in this picture?"},
+			},
+			nImages:  1,
+			expected: "<|turn>user\nWhat is in this picture?<image><turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		Entry("multiple images append markers in image order", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "Compare these."},
+			},
+			nImages:  3,
+			expected: "<|turn>user\nCompare these.<image><image><image><turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// Flattened delivery loses per-message attribution, so all images
+		// attach to the LAST user message (llama.cpp grpc-server convention).
+		Entry("images attach to the last user message in multi-turn", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: "hi"},
+				{Role: "assistant", Content: "hello"},
+				{Role: "user", Content: "and this?"},
+			},
+			nImages:  1,
+			expected: "<|turn>user\nhi<turn|>\n<|turn>model\nhello<turn|>\n<|turn>user\nand this?<image><turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+
+		// tpl L346: the markers count as captured_content, so an image-only
+		// user message still has content and closes its turn normally.
+		Entry("image with empty user text still closes the turn", renderGemma4Case{
+			msgs: []*pb.Message{
+				{Role: "user", Content: ""},
+			},
+			nImages:  1,
+			expected: "<|turn>user\n<image><turn|>\n<|turn>model\n<|channel>thought\n<channel|>",
+		}),
+	)
+
+	Describe("error handling", func() {
+		It("fails loud on an unknown role", func() {
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "narrator", Content: "Meanwhile..."},
+			}, "", 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring(`unknown role "narrator"`))
+		})
+
+		It("fails on invalid tools JSON", func() {
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "user", Content: "hi"},
+			}, "{not json", 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("tools JSON"))
+		})
+
+		It("fails on invalid tool_calls JSON", func() {
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "user", Content: "hi"},
+				{Role: "assistant", Content: "", ToolCalls: "{not json"},
+			}, "", 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("tool_calls JSON"))
+		})
+
+		It("fails on an orphan tool message, naming its index", func() {
+			// A role:tool message with no preceding assistant tool_calls turn
+			// would be silently dropped by the jinja; we fail loud instead.
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "user", Content: "hi"},
+				{Role: "tool", Content: `{"temp": 20}`, ToolCallId: "call_1"},
+			}, "", 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("orphan tool message 1"))
+		})
+
+		It("fails on trailing garbage after the tools JSON array", func() {
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "user", Content: "hi"},
+			}, "[] junk", 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("tools JSON"))
+		})
+
+		It("fails when the tools JSON is not an array", func() {
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "user", Content: "hi"},
+			}, `{"type":"function"}`, 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("tools JSON is not an array"))
+		})
+
+		It("fails when a tools array element is not an object", func() {
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "user", Content: "hi"},
+			}, `[42]`, 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("tools[0] is not an object"))
+		})
+
+		It("rejects a nil message via the unknown-role check", func() {
+			// Pins current behavior: pb getters are nil-safe, so a nil message
+			// reads as role "" and trips the fail-loud unknown-role guard.
+			_, err := RenderGemma4([]*pb.Message{nil}, "", 0, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring(`unknown role "" in message 0`))
+		})
+
+		It("fails loud on images with no user message to attach them to", func() {
+			// The engine would reject the markerless prompt anyway
+			// (marker/image count mismatch); the renderer surfaces the bad
+			// request with a usable message instead.
+			_, err := RenderGemma4([]*pb.Message{
+				{Role: "system", Content: "sys"},
+				{Role: "assistant", Content: "hi"},
+			}, "", 1, false, true)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("no user message"))
+		})
+	})
+})

backend/go/dllm/main.go

@@ -0,0 +1,98 @@
+package main
+
+// Started internally by LocalAI - one gRPC server per loaded model.
+//
+// Loads libdllm.so via purego and registers the flat C-ABI declared in
+// dllm.cpp's include/dllm_capi.h (ABI v1): 9 mandatory symbols plus the
+// Dlsym-probed optional multimodal pair. The library name can
+// be overridden with DLLM_LIBRARY (mirrors the PARAKEET_LIBRARY /
+// WHISPER_LIBRARY convention in the sibling backends); the default looks
+// for the .so next to this binary (run.sh puts the package dir on
+// LD_LIBRARY_PATH).
+import (
+	"flag"
+	"fmt"
+	"os"
+
+	"github.com/ebitengine/purego"
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+type LibFuncs struct {
+	FuncPtr any
+	Name    string
+}
+
+// loadCAPI dlopens libName and binds the 9 dllm_capi_* entry points 1:1 to
+// dllm_capi.h, so an `nm libdllm.so | grep dllm_capi` is enough to spot
+// drift. Shared with the test suite (ensureLibLoaded), which drives the
+// bridge without the gRPC server.
+//
+// The C-ABI returns malloc'd char* buffers from tokenize_json/generate; we
+// register those as uintptr so we get the raw pointer back and can call
+// dllm_capi_free_string on it (purego's string return would copy and forget
+// the original pointer, leaking it on every call). last_error returns a
+// BORROWED pointer instead, so it is registered as a plain string: purego
+// copies it and nothing must be freed.
+func loadCAPI(libName string) error {
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		return fmt.Errorf("dllm: dlopen %q: %w", libName, err)
+	}
+
+	libFuncs := []LibFuncs{
+		{&cppAbiVersion, "dllm_capi_abi_version"},
+		{&cppLoad, "dllm_capi_load"},
+		{&cppFree, "dllm_capi_free"},
+		{&cppLastError, "dllm_capi_last_error"},
+		{&cppFreeString, "dllm_capi_free_string"},
+		{&cppTokenizeJSON, "dllm_capi_tokenize_json"},
+		{&cppGenerate, "dllm_capi_generate"},
+		{&cppGenerateStream, "dllm_capi_generate_stream"},
+		{&cppCancel, "dllm_capi_cancel"},
+	}
+	for _, lf := range libFuncs {
+		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
+	}
+
+	// Multimodal entry points (dllm_capi.h's P4 surface). Additive: the ABI
+	// version stays 1 and consumers detect the surface by probing the symbols
+	// (the parakeet-cpp optional-symbol pattern), so the backend still loads
+	// against an older text-only libdllm.so - image requests then fail with
+	// errMMUnsupported instead of a boot failure.
+	if sym, err := purego.Dlsym(lib, "dllm_capi_generate_mm"); err == nil && sym != 0 {
+		purego.RegisterLibFunc(&cppGenerateMM, lib, "dllm_capi_generate_mm")
+	}
+	if sym, err := purego.Dlsym(lib, "dllm_capi_generate_stream_mm"); err == nil && sym != 0 {
+		purego.RegisterLibFunc(&cppGenerateStreamMM, lib, "dllm_capi_generate_stream_mm")
+	}
+	return nil
+}
+
+func main() {
+	libName := os.Getenv("DLLM_LIBRARY")
+	if libName == "" {
+		libName = "libdllm.so"
+	}
+
+	if err := loadCAPI(libName); err != nil {
+		panic(err)
+	}
+
+	// Hard-fail on an ABI mismatch: the flat-pointer bindings above would
+	// otherwise misbehave silently against a future libdllm.so.
+	if v := cAbiVersion(); v != dllmABIVersion {
+		panic(fmt.Errorf("dllm: libdllm.so ABI=%d, this backend speaks ABI=%d", v, dllmABIVersion))
+	}
+	fmt.Fprintf(os.Stderr, "[dllm] ABI=%d multimodal=%t\n", cAbiVersion(), cMMSupported())
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &Dllm{}); err != nil {
+		panic(err)
+	}
+}

backend/go/dllm/package.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+# T1 packaging stub: copy the binary, run.sh and libdllm.so into package/.
+# The full ldd walk (libc, libstdc++, libgomp, GPU runtimes, arch
+# detection) lands with the registration task, mirroring
+# backend/go/whisper/package.sh.
+
+set -e
+
+CURDIR=$(dirname "$(realpath "$0")")
+
+mkdir -p "$CURDIR/package/lib"
+
+cp -avf "$CURDIR/dllm-grpc" "$CURDIR/package/"
+cp -avf "$CURDIR/run.sh" "$CURDIR/package/"
+
+# libdllm.so + any soname symlinks, should upstream ever add them.
+cp -avf "$CURDIR"/libdllm.so* "$CURDIR/package/lib/" 2>/dev/null || {
+	echo "ERROR: libdllm.so not found in $CURDIR, run 'make' first" >&2
+	exit 1
+}
+
+echo "T1 package layout (full ldd walk lands with registration):"
+ls -liah "$CURDIR/package/" "$CURDIR/package/lib/"

backend/go/dllm/run.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath "$0")")
+
+export LD_LIBRARY_PATH="$CURDIR/lib:$CURDIR:${LD_LIBRARY_PATH:-}"
+
+# If a self-contained ld.so was packaged, route through it so the
+# packaged libc / libstdc++ are used instead of the host's (matches the
+# whisper / parakeet-cpp backends' runtime layout).
+if [ -f "$CURDIR/lib/ld.so" ]; then
+	echo "Using lib/ld.so"
+	exec "$CURDIR/lib/ld.so" "$CURDIR/dllm-grpc" "$@"
+fi
+
+exec "$CURDIR/dllm-grpc" "$@"

backend/go/locate-anything-cpp/.gitignore

@@ -1,7 +0,0 @@
-sources/
-build*/
-package/
-liblocateanythingcpp*.so
-locate-anything-cpp
-test-models/
-test-data/

backend/go/locate-anything-cpp/CMakeLists.txt

@@ -1,57 +0,0 @@
-cmake_minimum_required(VERSION 3.18)
-project(liblocateanythingcpp LANGUAGES C CXX)
-
-set(CMAKE_POSITION_INDEPENDENT_CODE ON)
-set(CMAKE_CXX_STANDARD 17)
-set(CMAKE_CXX_STANDARD_REQUIRED ON)
-
-# Static-link ggml + locate_anything so the resulting .so has no runtime
-# dependency on extra ggml/locate_anything shared libraries — only on
-# libc/libstdc++/libgomp, which the LocalAI package step bundles into the
-# docker image.
-set(BUILD_SHARED_LIBS OFF CACHE BOOL "Build static libraries" FORCE)
-
-# locate-anything.cpp build switches: skip CLI/tests, keep static lib.
-set(LA_BUILD_CLI OFF CACHE BOOL "Disable locate-anything CLI" FORCE)
-set(LA_BUILD_TESTS OFF CACHE BOOL "Disable locate-anything tests" FORCE)
-set(LA_SHARED OFF CACHE BOOL "Build locate_anything as static lib" FORCE)
-
-# Unlike rt-detr.cpp, locate-anything.cpp ships no in-tree ggml patches, so
-# there is no apply_ggml_patches.sh hook to shim here.
-add_subdirectory(./sources/locate-anything.cpp)
-
-# locate-anything.cpp's top-level CMakeLists points its own target's include
-# dirs at ${CMAKE_SOURCE_DIR}/{include,src,third_party,...}. CMAKE_SOURCE_DIR
-# is the *top-level* source dir of the whole CMake tree, so when we pull it in
-# via add_subdirectory it resolves to OUR directory, not theirs, and the
-# locate_anything target fails to find its own headers (la_capi.h, stb_image.h,
-# la_gguf_keys.h). Re-add the correct, subdir-relative include paths to the
-# already-defined target so it compiles regardless of where it's nested.
-set(LA_SRC ${CMAKE_CURRENT_SOURCE_DIR}/sources/locate-anything.cpp)
-target_include_directories(locate_anything PRIVATE
-    ${LA_SRC}/include
-    ${LA_SRC}/src
-    ${LA_SRC}/third_party
-    ${LA_SRC}/third_party/stb)
-
-# locate-anything.cpp's C-API symbols already live inside liblocate_anything
-# (src/la_capi.cpp is compiled into the lib). We re-export them via a MODULE
-# library that links locate_anything so the symbols are visible at dlopen time.
-add_library(locateanythingcpp MODULE
-    sources/locate-anything.cpp/src/la_capi.cpp)
-
-target_include_directories(locateanythingcpp PRIVATE
-    sources/locate-anything.cpp/include
-    sources/locate-anything.cpp/src
-    sources/locate-anything.cpp/third_party
-    sources/locate-anything.cpp/third_party/stb
-)
-
-target_link_libraries(locateanythingcpp PRIVATE locate_anything ggml)
-
-if(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9.0)
-    target_link_libraries(locateanythingcpp PRIVATE stdc++fs)
-endif()
-
-set_property(TARGET locateanythingcpp PROPERTY CXX_STANDARD 17)
-set_target_properties(locateanythingcpp PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/locate-anything-cpp/Makefile

@@ -1,134 +0,0 @@
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-
-GOCMD?=go
-GO_TAGS?=
-JOBS?=$(shell nproc --ignore=1)
-
-# locate-anything.cpp. Pin to a specific commit for a stable build; leaving
-# this on `master` always picks up the latest C-API surface (incl. the
-# per-detection accessor functions used by golocateanythingcpp.go).
-LOCATEANYTHING_REPO?=https://github.com/mudler/locate-anything.cpp.git
-LOCATEANYTHING_VERSION?=92c1682da792c1e8a5dec91acc2be4b02c742ded
-
-ifeq ($(NATIVE),false)
-	CMAKE_ARGS+=-DGGML_NATIVE=OFF
-endif
-
-# Forward LocalAI's BUILD_TYPE to the matching ggml backend switch.
-ifeq ($(BUILD_TYPE),cublas)
-	CMAKE_ARGS+=-DGGML_CUDA=ON -DLA_GGML_CUDA=ON
-else ifeq ($(BUILD_TYPE),openblas)
-	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
-else ifeq ($(BUILD_TYPE),clblas)
-	CMAKE_ARGS+=-DGGML_CLBLAST=ON
-else ifeq ($(BUILD_TYPE),hipblas)
-	ROCM_HOME ?= /opt/rocm
-	ROCM_PATH ?= /opt/rocm
-	export CXX=$(ROCM_HOME)/llvm/bin/clang++
-	export CC=$(ROCM_HOME)/llvm/bin/clang
-	AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1200,gfx1201
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
-else ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DGGML_VULKAN=ON -DLA_GGML_VULKAN=ON
-else ifeq ($(OS),Darwin)
-	ifneq ($(BUILD_TYPE),metal)
-		CMAKE_ARGS+=-DGGML_METAL=OFF
-	else
-		CMAKE_ARGS+=-DGGML_METAL=ON
-		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
-		CMAKE_ARGS+=-DLA_GGML_METAL=ON
-	endif
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f16)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx \
-		-DGGML_SYCL_F16=ON
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f32)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx
-endif
-
-sources/locate-anything.cpp:
-	mkdir -p sources && \
-	git clone --recursive $(LOCATEANYTHING_REPO) sources/locate-anything.cpp && \
-	cd sources/locate-anything.cpp && \
-	git checkout $(LOCATEANYTHING_VERSION) && \
-	git submodule update --init --recursive --depth 1 --single-branch
-
-# Detect OS
-UNAME_S := $(shell uname -s)
-
-# Only build CPU variants on Linux
-ifeq ($(UNAME_S),Linux)
-	VARIANT_TARGETS = liblocateanythingcpp-avx.so liblocateanythingcpp-avx2.so liblocateanythingcpp-avx512.so liblocateanythingcpp-fallback.so
-else
-	# On non-Linux (e.g., Darwin), build only fallback variant
-	VARIANT_TARGETS = liblocateanythingcpp-fallback.so
-endif
-
-locate-anything-cpp: main.go golocateanythingcpp.go $(VARIANT_TARGETS)
-	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o locate-anything-cpp ./
-
-package: locate-anything-cpp
-	bash package.sh
-
-build: package
-
-clean: purge
-	rm -rf liblocateanythingcpp*.so locate-anything-cpp package sources
-
-purge:
-	rm -rf build*
-
-# Build all variants (Linux only)
-ifeq ($(UNAME_S),Linux)
-liblocateanythingcpp-avx.so: sources/locate-anything.cpp
-	rm -rfv build-$@
-	$(info ${GREEN}I locate-anything-cpp build info:avx${RESET})
-	SO_TARGET=$@ CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) liblocateanythingcpp-custom
-	rm -rfv build-$@
-
-liblocateanythingcpp-avx2.so: sources/locate-anything.cpp
-	rm -rfv build-$@
-	$(info ${GREEN}I locate-anything-cpp build info:avx2${RESET})
-	SO_TARGET=$@ CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) liblocateanythingcpp-custom
-	rm -rfv build-$@
-
-liblocateanythingcpp-avx512.so: sources/locate-anything.cpp
-	rm -rfv build-$@
-	$(info ${GREEN}I locate-anything-cpp build info:avx512${RESET})
-	SO_TARGET=$@ CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) liblocateanythingcpp-custom
-	rm -rfv build-$@
-endif
-
-# Build fallback variant (all platforms)
-liblocateanythingcpp-fallback.so: sources/locate-anything.cpp
-	rm -rfv build-$@
-	$(info ${GREEN}I locate-anything-cpp build info:fallback${RESET})
-	SO_TARGET=$@ CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) liblocateanythingcpp-custom
-	rm -rfv build-$@
-
-liblocateanythingcpp-custom: CMakeLists.txt
-	mkdir -p build-$(SO_TARGET) && \
-	cd build-$(SO_TARGET) && \
-	cmake .. $(CMAKE_ARGS) && \
-	cmake --build . --config Release -j$(JOBS) && \
-	cd .. && \
-	mv build-$(SO_TARGET)/liblocateanythingcpp.so ./$(SO_TARGET)
-
-all: locate-anything-cpp package
-
-# `test` is invoked by the top-level Makefile's `test-extra` target. It builds
-# the backend binary + the fallback shared library (needed for dlopen at
-# runtime), then runs test.sh which downloads the q8_0 GGUF + COCO image and
-# exercises the gRPC Load/Detect wire path via the Go smoke test in
-# main_test.go.
-test: locate-anything-cpp liblocateanythingcpp-fallback.so
-	bash test.sh

backend/go/locate-anything-cpp/golocateanythingcpp.go

@@ -1,174 +0,0 @@
-package main
-
-// golocateanythingcpp.go - gRPC handlers (Load, Detect) for the
-// locate-anything-cpp backend.
-//
-// Embeds base.SingleThread to default unimplemented RPCs to "not supported"
-// while we only implement open-vocabulary object detection (Detect).
-
-import (
-	"encoding/base64"
-	"fmt"
-	"os"
-	"path/filepath"
-	"unsafe"
-
-	"github.com/mudler/LocalAI/pkg/grpc/base"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-)
-
-// la_ctx* is an opaque handle. la_capi_load returns it directly (0 == failure),
-// unlike rfdetr's out-parameter convention.
-var (
-	// la_capi_load(const char* gguf_path, int n_threads) -> la_ctx* (0 = fail)
-	CapiLoad func(gguf string, nThreads int32) uintptr
-	// la_capi_free(la_ctx* ctx)
-	CapiFree func(handle uintptr)
-	// la_capi_locate_path(ctx, image_path, prompt, mode) -> char* json (0 = err)
-	CapiLocatePath func(handle uintptr, imagePath string, prompt string, mode int32) uintptr
-	// la_capi_locate_buffer(ctx, bytes, len, prompt, mode) -> char* json (0 = err)
-	CapiLocateBuffer func(handle uintptr, bytes uintptr, length uintptr, prompt string, mode int32) uintptr
-	// la_capi_get_n_detections(ctx) -> int
-	CapiGetNDetections func(handle uintptr) int32
-	// la_capi_get_detection_box(ctx, i, out_xyxy[4]) -> int (0 on success)
-	CapiGetDetectionBox func(handle uintptr, i int32, outXYXY uintptr) int32
-	// la_capi_get_detection_label(ctx, i, buf, buf_size) -> int (required size incl NUL; two-call sizing)
-	CapiGetDetectionLabel func(handle uintptr, i int32, buf uintptr, bufSize int32) int32
-	// la_capi_free_string(char* s)
-	CapiFreeString func(s uintptr)
-	// la_capi_last_error(ctx) -> const char* (owned by ctx, "" if none / null ctx).
-	// purego marshals the returned C string into a Go string (a copy), so we
-	// never free it and avoid raw pointer arithmetic.
-	CapiLastError func(handle uintptr) string
-)
-
-type LocateAnythingCpp struct {
-	base.SingleThread
-	handle uintptr
-}
-
-// Load loads the GGUF model at opts.ModelFile (joined with opts.ModelPath if
-// relative) and stores the la_ctx handle for later Detect calls.
-func (r *LocateAnythingCpp) Load(opts *pb.ModelOptions) error {
-	modelFile := opts.ModelFile
-	if modelFile == "" {
-		modelFile = opts.Model
-	}
-	if modelFile == "" {
-		return fmt.Errorf("locate-anything-cpp: ModelFile is empty")
-	}
-
-	var modelPath string
-	if filepath.IsAbs(modelFile) {
-		modelPath = modelFile
-	} else {
-		modelPath = filepath.Join(opts.ModelPath, modelFile)
-	}
-
-	if _, err := os.Stat(modelPath); err != nil {
-		return fmt.Errorf("locate-anything-cpp: model file not found: %s: %w", modelPath, err)
-	}
-
-	threads := opts.Threads
-	if threads <= 0 {
-		threads = 4
-	}
-
-	// Release previous model if any (re-Load).
-	if r.handle != 0 {
-		CapiFree(r.handle)
-		r.handle = 0
-	}
-
-	h := CapiLoad(modelPath, threads)
-	if h == 0 {
-		// la_capi_last_error needs a ctx; on a failed load we have none (it
-		// returns "" for a null ctx), so the text is best-effort. Surface it
-		// when present.
-		if msg := CapiLastError(0); msg != "" {
-			return fmt.Errorf("locate-anything-cpp: la_capi_load failed for %s: %s", modelPath, msg)
-		}
-		return fmt.Errorf("locate-anything-cpp: la_capi_load failed for %s", modelPath)
-	}
-	r.handle = h
-	return nil
-}
-
-// Detect runs open-vocabulary detection on the base64-encoded image in opts.Src
-// using the required text prompt in opts.Prompt, returning one pb.Detection per
-// located object with its predicted label as ClassName.
-func (r *LocateAnythingCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, error) {
-	if r.handle == 0 {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: model not loaded")
-	}
-
-	// Open-vocabulary detection is prompt-driven; without a prompt there is
-	// nothing to locate.
-	prompt := opts.Prompt
-	if prompt == "" {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: a text prompt is required (open-vocabulary detection)")
-	}
-
-	// Decode base64 image and write to temp file.
-	imgData, err := base64.StdEncoding.DecodeString(opts.Src)
-	if err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to decode base64 image: %w", err)
-	}
-
-	tmpFile, err := os.CreateTemp("", "locate-anything-*.img")
-	if err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to create temp file: %w", err)
-	}
-	defer func() { _ = os.Remove(tmpFile.Name()) }()
-
-	if _, err := tmpFile.Write(imgData); err != nil {
-		_ = tmpFile.Close()
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to write temp file: %w", err)
-	}
-	if err := tmpFile.Close(); err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to close temp file: %w", err)
-	}
-
-	// mode 0 = hybrid (Parallel Box Decoding). The JSON return value is unused:
-	// structured detections are read via the accessor functions. Still must
-	// free the returned string.
-	jsonPtr := CapiLocatePath(r.handle, tmpFile.Name(), prompt, 0)
-	if jsonPtr != 0 {
-		CapiFreeString(jsonPtr)
-	}
-
-	n := CapiGetNDetections(r.handle)
-	if n < 0 {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: invalid n_detections=%d", n)
-	}
-
-	detections := make([]*pb.Detection, 0, n)
-	for i := int32(0); i < n; i++ {
-		var xyxy [4]float32 // x1, y1, x2, y2
-		if CapiGetDetectionBox(r.handle, i, uintptr(unsafe.Pointer(&xyxy[0]))) != 0 {
-			continue
-		}
-
-		// Two-call sizing for the label string.
-		label := ""
-		need := CapiGetDetectionLabel(r.handle, i, 0, 0)
-		if need > 0 {
-			buf := make([]byte, need)
-			CapiGetDetectionLabel(r.handle, i, uintptr(unsafe.Pointer(&buf[0])), need)
-			label = string(buf[:need-1])
-		}
-
-		detections = append(detections, &pb.Detection{
-			X:          xyxy[0],
-			Y:          xyxy[1],
-			Width:      xyxy[2] - xyxy[0],
-			Height:     xyxy[3] - xyxy[1],
-			Confidence: 1.0,
-			ClassName:  label,
-		})
-	}
-
-	return pb.DetectResponse{
-		Detections: detections,
-	}, nil
-}

backend/go/locate-anything-cpp/main.go

@@ -1,59 +0,0 @@
-package main
-
-// main.go - entry point for the locate-anything-cpp gRPC backend.
-//
-// Dlopens liblocateanythingcpp-<variant>.so via purego at the path in
-// LOCATEANYTHING_LIBRARY (set by run.sh based on /proc/cpuinfo), registers
-// the la_capi_* C ABI symbols, then starts the gRPC server.
-
-import (
-	"flag"
-	"os"
-
-	"github.com/ebitengine/purego"
-	grpc "github.com/mudler/LocalAI/pkg/grpc"
-)
-
-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
-)
-
-type LibFuncs struct {
-	FuncPtr any
-	Name    string
-}
-
-func main() {
-	// Get library name from environment variable, default to fallback
-	libName := os.Getenv("LOCATEANYTHING_LIBRARY")
-	if libName == "" {
-		libName = "./liblocateanythingcpp-fallback.so"
-	}
-
-	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-	if err != nil {
-		panic(err)
-	}
-
-	libFuncs := []LibFuncs{
-		{&CapiLoad, "la_capi_load"},
-		{&CapiFree, "la_capi_free"},
-		{&CapiLocatePath, "la_capi_locate_path"},
-		{&CapiLocateBuffer, "la_capi_locate_buffer"},
-		{&CapiGetNDetections, "la_capi_get_n_detections"},
-		{&CapiGetDetectionBox, "la_capi_get_detection_box"},
-		{&CapiGetDetectionLabel, "la_capi_get_detection_label"},
-		{&CapiFreeString, "la_capi_free_string"},
-		{&CapiLastError, "la_capi_last_error"},
-	}
-
-	for _, lf := range libFuncs {
-		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
-	}
-
-	flag.Parse()
-
-	if err := grpc.StartServer(*addr, &LocateAnythingCpp{}); err != nil {
-		panic(err)
-	}
-}

backend/go/locate-anything-cpp/main_test.go

@@ -1,176 +0,0 @@
-package main
-
-// main_test.go - end-to-end smoke test for the locate-anything-cpp gRPC backend.
-//
-// Spawns the compiled locate-anything-cpp binary on a free local port, dials it
-// via gRPC, and exercises LoadModel + Detect against the test fixtures
-// downloaded by test.sh: the q8_0 GGUF of nvidia/LocateAnything-3B and a real
-// COCO image with people + cars. Asserts that open-vocabulary detection driven
-// by a text prompt returns at least one detection, each carrying a non-empty
-// class name and a bounding box of non-zero size.
-//
-// The spec Skip()s cleanly if its fixtures (the ~6.3 GB model, the test image,
-// the built binary, or the fallback .so) are missing, so the test target stays
-// usable on a fresh checkout / on CI runners where the large model hasn't been
-// downloaded.
-
-import (
-	"context"
-	"encoding/base64"
-	"fmt"
-	"net"
-	"os"
-	"os/exec"
-	"path/filepath"
-	"testing"
-	"time"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-	"google.golang.org/grpc"
-	"google.golang.org/grpc/credentials/insecure"
-)
-
-func TestDetect(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "locate-anything-cpp backend smoke suite")
-}
-
-// freePort grabs an ephemeral TCP port and immediately releases it so the
-// spawned backend can bind to it. There is a tiny TOCTOU window here but in
-// practice it's adequate for a smoke test on a quiet runner.
-func freePort() int {
-	l, err := net.Listen("tcp", "127.0.0.1:0")
-	Expect(err).ToNot(HaveOccurred(), "freePort listen")
-	port := l.Addr().(*net.TCPAddr).Port
-	Expect(l.Close()).To(Succeed())
-	return port
-}
-
-// startBackend spawns the locate-anything-cpp binary on the given port and
-// waits until it accepts TCP connections (up to 10s). It mirrors how main.go
-// resolves the purego library: the LOCATEANYTHING_LIBRARY env var points the
-// dlopen at the freshly built fallback .so, and the la_capi_* symbols are
-// registered there. The returned cleanup func kills the process and reaps it.
-func startBackend(port int) func() {
-	binary, err := filepath.Abs("./locate-anything-cpp")
-	Expect(err).ToNot(HaveOccurred())
-	if _, err := os.Stat(binary); err != nil {
-		Skip(fmt.Sprintf("backend binary not built: %s (run `make locate-anything-cpp` first)", binary))
-	}
-
-	libPath, err := filepath.Abs("./liblocateanythingcpp-fallback.so")
-	Expect(err).ToNot(HaveOccurred())
-	if _, err := os.Stat(libPath); err != nil {
-		Skip(fmt.Sprintf("fallback library not built: %s (run `make liblocateanythingcpp-fallback.so` first)", libPath))
-	}
-
-	addr := fmt.Sprintf("127.0.0.1:%d", port)
-	cmd := exec.Command(binary, "--addr", addr)
-	cmd.Env = append(os.Environ(), "LOCATEANYTHING_LIBRARY="+libPath)
-	cmd.Stdout = os.Stderr
-	cmd.Stderr = os.Stderr
-	Expect(cmd.Start()).To(Succeed())
-
-	cleanup := func() {
-		if cmd.Process != nil {
-			_ = cmd.Process.Kill()
-			_, _ = cmd.Process.Wait()
-		}
-	}
-
-	deadline := time.Now().Add(10 * time.Second)
-	for time.Now().Before(deadline) {
-		c, err := net.DialTimeout("tcp", addr, 200*time.Millisecond)
-		if err == nil {
-			_ = c.Close()
-			return cleanup
-		}
-		time.Sleep(200 * time.Millisecond)
-	}
-
-	cleanup()
-	Fail(fmt.Sprintf("backend did not become ready on %s within 10s", addr))
-	return func() {}
-}
-
-// loadTestImage reads the COCO test image downloaded by test.sh and returns its
-// base64-encoded content (the wire format accepted by the Detect RPC).
-func loadTestImage() string {
-	imgPath, err := filepath.Abs("test-data/test.jpg")
-	Expect(err).ToNot(HaveOccurred())
-	imgBytes, err := os.ReadFile(imgPath)
-	if err != nil {
-		Skip(fmt.Sprintf("test image not present: %s (run test.sh first)", imgPath))
-	}
-	return base64.StdEncoding.EncodeToString(imgBytes)
-}
-
-// dialBackend opens a gRPC client connection to the spawned backend.
-func dialBackend(port int) (pb.BackendClient, func()) {
-	addr := fmt.Sprintf("127.0.0.1:%d", port)
-	conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
-	Expect(err).ToNot(HaveOccurred())
-	return pb.NewBackendClient(conn), func() { _ = conn.Close() }
-}
-
-// modelPathOrSkip resolves the model file under ./test-models/ and Skip()s the
-// current spec if it's missing (the ~6.3 GB GGUF is not present on a fresh
-// checkout / on CI runners without the download).
-func modelPathOrSkip(name string) string {
-	modelDir, err := filepath.Abs("test-models")
-	Expect(err).ToNot(HaveOccurred())
-	modelPath := filepath.Join(modelDir, name)
-	if _, err := os.Stat(modelPath); err != nil {
-		Skip(fmt.Sprintf("model not present: %s (run test.sh first)", modelPath))
-	}
-	return modelPath
-}
-
-var _ = Describe("locate-anything-cpp backend", func() {
-	It("runs open-vocabulary detection against a known-good COCO image", func() {
-		modelPath := modelPathOrSkip("locate-anything-q8_0.gguf")
-		imgB64 := loadTestImage()
-
-		port := freePort()
-		cleanup := startBackend(port)
-		defer cleanup()
-
-		client, closeConn := dialBackend(port)
-		defer closeConn()
-
-		// The q8_0 model is ~6.3 GB and hybrid Parallel Box Decoding on CPU is
-		// not cheap, so give LoadModel + Detect a generous deadline.
-		ctx, cancel := context.WithTimeout(context.Background(), 20*time.Minute)
-		defer cancel()
-
-		loadResp, err := client.LoadModel(ctx, &pb.ModelOptions{
-			Model:     "locate-anything-q8_0.gguf",
-			ModelFile: modelPath,
-			Threads:   4,
-		})
-		Expect(err).ToNot(HaveOccurred(), "LoadModel")
-		Expect(loadResp.GetSuccess()).To(BeTrue(), "LoadModel reported failure: %s", loadResp.GetMessage())
-
-		// Open-vocabulary detection is prompt-driven; the prompt names the
-		// classes to locate (people + cars), separated by the </c> control token.
-		detResp, err := client.Detect(ctx, &pb.DetectOptions{
-			Src:    imgB64,
-			Prompt: "Locate all the instances that matches the following description: person</c>car.",
-		})
-		Expect(err).ToNot(HaveOccurred(), "Detect")
-		Expect(detResp.GetDetections()).ToNot(BeEmpty(), "no detections returned on a known-good COCO image")
-
-		_, _ = fmt.Fprintf(GinkgoWriter, "detection OK: %d detections\n", len(detResp.GetDetections()))
-		for i, d := range detResp.GetDetections() {
-			Expect(d.GetClassName()).ToNot(BeEmpty(), "detection %d has empty class_name", i)
-			Expect(d.GetWidth()).To(BeNumerically(">", float32(0)),
-				"detection %d has non-positive width", i)
-			Expect(d.GetHeight()).To(BeNumerically(">", float32(0)),
-				"detection %d has non-positive height", i)
-			_, _ = fmt.Fprintf(GinkgoWriter, "  [%d] %s box=(%.1f,%.1f,%.1fx%.1f)\n",
-				i, d.GetClassName(), d.GetX(), d.GetY(), d.GetWidth(), d.GetHeight())
-		}
-	})
-})

backend/go/locate-anything-cpp/package.sh

@@ -1,59 +0,0 @@
-#!/bin/bash
-
-# Script to copy the appropriate libraries based on architecture
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-# Create lib directory
-mkdir -p $CURDIR/package/lib
-
-cp -avf $CURDIR/liblocateanythingcpp-*.so $CURDIR/package/
-cp -avf $CURDIR/locate-anything-cpp $CURDIR/package/
-cp -fv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    # x86_64 architecture
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    # ARM64 architecture
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ $(uname -s) = "Darwin" ]; then
-    echo "Detected Darwin"
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/go/locate-anything-cpp/run.sh

@@ -1,52 +0,0 @@
-#!/bin/bash
-set -ex
-
-# Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath $0)")
-
-cd /
-
-echo "CPU info:"
-if [ "$(uname)" != "Darwin" ]; then
-	grep -e "model\sname" /proc/cpuinfo | head -1
-	grep -e "flags" /proc/cpuinfo | head -1
-fi
-
-LIBRARY="$CURDIR/liblocateanythingcpp-fallback.so"
-
-if [ "$(uname)" != "Darwin" ]; then
-	if grep -q -e "\savx\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX    found OK"
-		if [ -e $CURDIR/liblocateanythingcpp-avx.so ]; then
-			LIBRARY="$CURDIR/liblocateanythingcpp-avx.so"
-		fi
-	fi
-
-	if grep -q -e "\savx2\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX2   found OK"
-		if [ -e $CURDIR/liblocateanythingcpp-avx2.so ]; then
-			LIBRARY="$CURDIR/liblocateanythingcpp-avx2.so"
-		fi
-	fi
-
-	# Check avx 512
-	if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX512F found OK"
-		if [ -e $CURDIR/liblocateanythingcpp-avx512.so ]; then
-			LIBRARY="$CURDIR/liblocateanythingcpp-avx512.so"
-		fi
-	fi
-fi
-
-export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
-export LOCATEANYTHING_LIBRARY=$LIBRARY
-
-# If there is a lib/ld.so, use it
-if [ -f $CURDIR/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using library: $LIBRARY"
-	exec $CURDIR/lib/ld.so $CURDIR/locate-anything-cpp "$@"
-fi
-
-echo "Using library: $LIBRARY"
-exec $CURDIR/locate-anything-cpp "$@"

backend/go/locate-anything-cpp/test.sh

@@ -1,47 +0,0 @@
-#!/bin/bash
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-
-echo "Running locate-anything-cpp backend tests..."
-
-# Test model from the mudler/locate-anything.cpp-gguf HuggingFace repo. This is
-# the q8_0 quantization of nvidia/LocateAnything-3B (~6.3 GB), so the download
-# is the slow step. It is resumed with `curl -C -` and skipped entirely if the
-# file is already present.
-LOCATEANYTHING_MODEL_DIR="${LOCATEANYTHING_MODEL_DIR:-$CURDIR/test-models}"
-
-LOCATEANYTHING_MODEL_FILE="${LOCATEANYTHING_MODEL_FILE:-locate-anything-q8_0.gguf}"
-LOCATEANYTHING_MODEL_URL="${LOCATEANYTHING_MODEL_URL:-https://huggingface.co/mudler/locate-anything.cpp-gguf/resolve/main/locate-anything-q8_0.gguf}"
-
-mkdir -p "$LOCATEANYTHING_MODEL_DIR"
-
-if [ ! -f "$LOCATEANYTHING_MODEL_DIR/$LOCATEANYTHING_MODEL_FILE" ]; then
-    echo "Downloading locate-anything q8_0 model (~6.3 GB, this is slow)..."
-    # -C - resumes a partial download so an interrupted run doesn't restart from 0.
-    curl -L -C - -o "$LOCATEANYTHING_MODEL_DIR/$LOCATEANYTHING_MODEL_FILE" "$LOCATEANYTHING_MODEL_URL" --progress-bar
-fi
-
-# Use a real COCO test image (people + cars) from the upstream rf-detr.cpp repo
-# (~46 KB). Open-vocabulary detection needs real content to locate, so a
-# synthetic image would trivially yield zero detections.
-TEST_IMAGE_DIR="$CURDIR/test-data"
-TEST_IMAGE_FILE="$TEST_IMAGE_DIR/test.jpg"
-TEST_IMAGE_URL="${TEST_IMAGE_URL:-https://raw.githubusercontent.com/mudler/rf-detr.cpp/main/tests/fixtures/ci/test_image.jpg}"
-
-mkdir -p "$TEST_IMAGE_DIR"
-if [ ! -f "$TEST_IMAGE_FILE" ]; then
-    echo "Downloading COCO test image..."
-    curl -L -o "$TEST_IMAGE_FILE" "$TEST_IMAGE_URL" --progress-bar
-fi
-
-echo "locate-anything-cpp test setup complete."
-echo "  model:      $LOCATEANYTHING_MODEL_DIR/$LOCATEANYTHING_MODEL_FILE"
-echo "  test image: $TEST_IMAGE_FILE"
-
-# Run the Go smoke test: spawns the backend binary on a free port, calls
-# LoadModel + Detect via gRPC against the downloaded GGUF + COCO image.
-echo ""
-echo "Running Go smoke test..."
-cd "$CURDIR"
-go test -v -timeout 30m ./...

backend/go/omnivoice-cpp/.gitignore

@@ -1,17 +0,0 @@
-# Fetched upstream sources
-sources/
-
-# CMake build directories
-build*/
-
-# Compiled shared libraries
-*.so
-
-# Compiled backend binary
-omnivoice-cpp
-
-# Packaging output
-package/
-
-# Downloaded e2e models
-omnivoice-models/

backend/go/omnivoice-cpp/CMakeLists.txt

@@ -1,53 +0,0 @@
-cmake_minimum_required(VERSION 3.14)
-project(gomnivoicecpp LANGUAGES C CXX)
-set(CMAKE_POSITION_INDEPENDENT_CODE ON)
-set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
-
-set(OMNIVOICE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/omnivoice.cpp)
-
-# Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
-if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
-    set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
-endif()
-
-# Add the upstream project. Its own CMakeLists adds ggml + builds
-# omnivoice-core (STATIC, contains src/omnivoice.cpp i.e. the ov_* impl).
-# EXCLUDE_FROM_ALL keeps its CLI tools/tests from building unless referenced.
-add_subdirectory(${OMNIVOICE_DIR} omnivoice EXCLUDE_FROM_ALL)
-
-# Upstream generates version.h into its own CMAKE_CURRENT_BINARY_DIR and adds
-# the top-level ${CMAKE_BINARY_DIR} to omnivoice-core's include path. When the
-# project is nested under add_subdirectory those two directories differ
-# (<build>/omnivoice vs <build>), so omnivoice.cpp cannot find version.h. Point
-# omnivoice-core at the subproject binary dir where version.h is actually
-# generated. (Fix lives here, never in the fetched upstream checkout.)
-target_include_directories(omnivoice-core PRIVATE ${CMAKE_BINARY_DIR}/omnivoice)
-
-add_library(gomnivoicecpp MODULE cpp/gomnivoicecpp.cpp)
-target_link_libraries(gomnivoicecpp PRIVATE omnivoice-core)
-
-target_include_directories(gomnivoicecpp PRIVATE ${OMNIVOICE_DIR}/src)
-target_include_directories(gomnivoicecpp SYSTEM PRIVATE ${OMNIVOICE_DIR}/ggml/include)
-
-# Link GPU backends if the upstream ggml created them.
-foreach(backend blas cuda metal vulkan sycl)
-    if(TARGET ggml-${backend})
-        target_link_libraries(gomnivoicecpp PRIVATE ggml-${backend})
-        if(backend STREQUAL "cuda")
-            find_package(CUDAToolkit QUIET)
-            if(CUDAToolkit_FOUND)
-                target_link_libraries(gomnivoicecpp PRIVATE CUDA::cudart)
-            endif()
-        endif()
-    endif()
-endforeach()
-
-if(MSVC)
-    target_compile_options(gomnivoicecpp PRIVATE /W4 /wd4100 /wd4505)
-else()
-    target_compile_options(gomnivoicecpp PRIVATE -Wall -Wextra
-                          -Wno-unused-parameter -Wno-unused-function)
-endif()
-
-set_property(TARGET gomnivoicecpp PROPERTY CXX_STANDARD 17)
-set_target_properties(gomnivoicecpp PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/omnivoice-cpp/Makefile

@@ -1,122 +0,0 @@
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-
-GOCMD?=go
-GO_TAGS?=
-JOBS?=$(shell nproc --ignore=1)
-
-# omnivoice.cpp version
-OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=2603355a5dfacae5cfc33531d5d0933221843509
-SO_TARGET?=libgomnivoicecpp.so
-
-CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
-
-ifeq ($(NATIVE),false)
-	CMAKE_ARGS+=-DGGML_NATIVE=OFF
-endif
-
-ifeq ($(BUILD_TYPE),cublas)
-	CMAKE_ARGS+=-DGGML_CUDA=ON
-else ifeq ($(BUILD_TYPE),openblas)
-	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
-else ifeq ($(BUILD_TYPE),clblas)
-	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
-else ifeq ($(BUILD_TYPE),hipblas)
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON
-else ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DGGML_VULKAN=ON
-else ifeq ($(OS),Darwin)
-	ifneq ($(BUILD_TYPE),metal)
-		CMAKE_ARGS+=-DGGML_METAL=OFF
-	else
-		CMAKE_ARGS+=-DGGML_METAL=ON
-		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
-	endif
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f16)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx \
-		-DGGML_SYCL_F16=ON
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f32)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx
-endif
-
-sources/omnivoice.cpp:
-	mkdir -p sources/omnivoice.cpp
-	cd sources/omnivoice.cpp && \
-	git init && \
-	git remote add origin $(OMNIVOICE_REPO) && \
-	git fetch origin && \
-	git checkout $(OMNIVOICE_VERSION) && \
-	git submodule update --init --recursive --depth 1 --single-branch
-
-# Detect OS
-UNAME_S := $(shell uname -s)
-
-# Only build CPU variants on Linux
-ifeq ($(UNAME_S),Linux)
-	VARIANT_TARGETS = libgomnivoicecpp-avx.so libgomnivoicecpp-avx2.so libgomnivoicecpp-avx512.so libgomnivoicecpp-fallback.so
-else
-	VARIANT_TARGETS = libgomnivoicecpp-fallback.so
-endif
-
-omnivoice-cpp: main.go gomnivoicecpp.go $(VARIANT_TARGETS)
-	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o omnivoice-cpp ./
-
-package: omnivoice-cpp
-	bash package.sh
-
-build: package
-
-clean: purge
-	rm -rf libgomnivoicecpp*.so package sources/omnivoice.cpp omnivoice-cpp
-
-purge:
-	rm -rf build*
-
-.NOTPARALLEL:
-
-ifeq ($(UNAME_S),Linux)
-libgomnivoicecpp-avx.so: sources/omnivoice.cpp
-	$(info ${GREEN}I omnivoice-cpp build info:avx${RESET})
-	SO_TARGET=libgomnivoicecpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomnivoicecpp-custom
-	rm -rf build-libgomnivoicecpp-avx.so
-
-libgomnivoicecpp-avx2.so: sources/omnivoice.cpp
-	$(info ${GREEN}I omnivoice-cpp build info:avx2${RESET})
-	SO_TARGET=libgomnivoicecpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomnivoicecpp-custom
-	rm -rf build-libgomnivoicecpp-avx2.so
-
-libgomnivoicecpp-avx512.so: sources/omnivoice.cpp
-	$(info ${GREEN}I omnivoice-cpp build info:avx512${RESET})
-	SO_TARGET=libgomnivoicecpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgomnivoicecpp-custom
-	rm -rf build-libgomnivoicecpp-avx512.so
-endif
-
-libgomnivoicecpp-fallback.so: sources/omnivoice.cpp
-	$(info ${GREEN}I omnivoice-cpp build info:fallback${RESET})
-	SO_TARGET=libgomnivoicecpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgomnivoicecpp-custom
-	rm -rf build-libgomnivoicecpp-fallback.so
-
-libgomnivoicecpp-custom: CMakeLists.txt cpp/gomnivoicecpp.cpp cpp/gomnivoicecpp.h
-	mkdir -p build-$(SO_TARGET) && \
-	cd build-$(SO_TARGET) && \
-	cmake .. $(CMAKE_ARGS) && \
-	cmake --build . --config Release -j$(JOBS) --target gomnivoicecpp && \
-	cd .. && \
-	mv build-$(SO_TARGET)/libgomnivoicecpp.so ./$(SO_TARGET)
-
-test: omnivoice-cpp
-	@echo "Running omnivoice-cpp tests..."
-	bash test.sh
-	@echo "omnivoice-cpp tests completed."
-
-all: omnivoice-cpp package

backend/go/omnivoice-cpp/audio.go

@@ -1,129 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"encoding/binary"
-	"fmt"
-	"os"
-	"runtime"
-
-	"github.com/go-audio/audio"
-	"github.com/go-audio/wav"
-)
-
-const omnivoiceSampleRate = 24000
-
-// wavHeader24k returns a 44-byte WAV header for a streaming 24 kHz mono 16-bit
-// PCM stream, with placeholder (0xFFFFFFFF) sizes since the total length is
-// unknown up front. Emitted as the first chunk of TTSStream so the HTTP layer
-// receives a self-describing WAV (the gRPC TTSStream path never sets Message,
-// so the backend owns the header - see core/backend/tts.go:ModelTTSStream).
-func wavHeader24k() []byte {
-	var buf bytes.Buffer
-	w := func(v any) { _ = binary.Write(&buf, binary.LittleEndian, v) }
-	buf.WriteString("RIFF")
-	w(uint32(0xFFFFFFFF))
-	buf.WriteString("WAVE")
-	buf.WriteString("fmt ")
-	w(uint32(16))                      // Subchunk1Size
-	w(uint16(1))                       // PCM
-	w(uint16(1))                       // mono
-	w(uint32(omnivoiceSampleRate))     // sample rate
-	w(uint32(omnivoiceSampleRate * 2)) // byte rate = SR * blockAlign
-	w(uint16(2))                       // block align (16-bit mono)
-	w(uint16(16))                      // bits per sample
-	buf.WriteString("data")
-	w(uint32(0xFFFFFFFF))
-	return buf.Bytes()
-}
-
-// floatToPCM16LE clamps each sample to [-1,1] and encodes it as little-endian
-// signed 16-bit PCM.
-func floatToPCM16LE(samples []float32) []byte {
-	out := make([]byte, len(samples)*2)
-	for i, s := range samples {
-		if s > 1 {
-			s = 1
-		} else if s < -1 {
-			s = -1
-		}
-		v := int16(s * 32767)
-		out[i*2] = byte(v)
-		out[i*2+1] = byte(v >> 8)
-	}
-	return out
-}
-
-// writeWAV24k writes samples as a finalized 24 kHz mono 16-bit WAV at dst.
-func writeWAV24k(dst string, samples []float32) error {
-	f, err := os.Create(dst)
-	if err != nil {
-		return fmt.Errorf("omnivoice: create %q: %w", dst, err)
-	}
-	enc := wav.NewEncoder(f, omnivoiceSampleRate, 16, 1, 1)
-	ints := make([]int, len(samples))
-	for i, s := range samples {
-		if s > 1 {
-			s = 1
-		} else if s < -1 {
-			s = -1
-		}
-		ints[i] = int(s * 32767)
-	}
-	b := &audio.IntBuffer{
-		Format:         &audio.Format{NumChannels: 1, SampleRate: omnivoiceSampleRate},
-		Data:           ints,
-		SourceBitDepth: 16,
-	}
-	if err := enc.Write(b); err != nil {
-		_ = enc.Close()
-		_ = f.Close()
-		return fmt.Errorf("omnivoice: encode WAV: %w", err)
-	}
-	if err := enc.Close(); err != nil {
-		_ = f.Close()
-		return fmt.Errorf("omnivoice: finalize WAV: %w", err)
-	}
-	return f.Close()
-}
-
-// readWAVAsFloat decodes a WAV file (any sample rate/channels) to a mono
-// float32 slice in [-1,1] for use as reference audio. OmniVoice expects 24 kHz;
-// callers should supply 24 kHz reference clips.
-func readWAVAsFloat(path string) ([]float32, error) {
-	f, err := os.Open(path)
-	if err != nil {
-		return nil, fmt.Errorf("omnivoice: open ref %q: %w", path, err)
-	}
-	defer func() { _ = f.Close() }()
-
-	dec := wav.NewDecoder(f)
-	buf, err := dec.FullPCMBuffer()
-	if err != nil {
-		return nil, fmt.Errorf("omnivoice: decode ref %q: %w", path, err)
-	}
-	ch := int(buf.Format.NumChannels)
-	if ch < 1 {
-		ch = 1
-	}
-	bitDepth := int(buf.SourceBitDepth)
-	if bitDepth == 0 {
-		bitDepth = 16
-	}
-	scale := float32(int64(1) << uint(bitDepth-1))
-	n := len(buf.Data) / ch
-	out := make([]float32, n)
-	for i := 0; i < n; i++ {
-		// Downmix to mono by averaging channels.
-		var acc int
-		for c := 0; c < ch; c++ {
-			acc += buf.Data[i*ch+c]
-		}
-		out[i] = float32(acc) / float32(ch) / scale
-	}
-	return out, nil
-}
-
-// runtimeKeepAlive prevents the GC from reclaiming the reference-audio slice
-// while its backing pointer is in use across the C call.
-func runtimeKeepAlive(v any) { runtime.KeepAlive(v) }

backend/go/omnivoice-cpp/cpp/gomnivoicecpp.cpp

@@ -1,166 +0,0 @@
-#include "gomnivoicecpp.h"
-#include "ggml-backend.h"
-#include "omnivoice.h"
-
-#include <cstdio>
-#include <cstdlib>
-#include <cstring>
-
-static ov_context *g_ctx = nullptr;
-
-static void ggml_log_cb(enum ggml_log_level level, const char *log,
-                        void * /*data*/) {
-    if (!log)
-        return;
-    const char *lvl = "?????";
-    switch (level) {
-    case GGML_LOG_LEVEL_DEBUG: lvl = "DEBUG"; break;
-    case GGML_LOG_LEVEL_INFO:  lvl = "INFO";  break;
-    case GGML_LOG_LEVEL_WARN:  lvl = "WARN";  break;
-    case GGML_LOG_LEVEL_ERROR: lvl = "ERROR"; break;
-    default: break;
-    }
-    fprintf(stderr, "[%-5s] %s", lvl, log);
-    fflush(stderr);
-}
-
-int omni_load(const char *model_path, const char *codec_path, int use_fa,
-              int clamp_fp16) {
-    ggml_log_set(ggml_log_cb, nullptr);
-    ggml_backend_load_all();
-
-    if (!model_path || model_path[0] == '\0') {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: model_path is required\n");
-        return 1;
-    }
-    if (!codec_path || codec_path[0] == '\0') {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: codec_path is required\n");
-        return 2;
-    }
-
-    ov_init_params p;
-    ov_init_default_params(&p);
-    p.model_path = model_path;
-    p.codec_path = codec_path;
-    p.use_fa = use_fa != 0;
-    p.clamp_fp16 = clamp_fp16 != 0;
-
-    fprintf(stderr, "[omnivoice-cpp] Loading model=%s codec=%s\n", model_path,
-            codec_path);
-
-    g_ctx = ov_init(&p);
-    if (!g_ctx) {
-        fprintf(stderr, "[omnivoice-cpp] FATAL: ov_init failed: %s\n",
-                ov_last_error());
-        return 3;
-    }
-    fprintf(stderr, "[omnivoice-cpp] Model loaded (%s)\n", ov_version());
-    return 0;
-}
-
-// Fill an ov_tts_params from the flat wrapper arguments.
-static void fill_params(ov_tts_params *tp, const char *text, const char *lang,
-                        const char *instruct, const float *ref_samples,
-                        int ref_n, const char *ref_text, long long seed,
-                        int denoise) {
-    ov_tts_default_params(tp);
-    tp->text = text ? text : "";
-    tp->lang = lang ? lang : "";
-    if (instruct && instruct[0] != '\0')
-        tp->instruct = instruct;
-    if (ref_samples && ref_n > 0) {
-        tp->ref_audio_24k = ref_samples;
-        tp->ref_n_samples = ref_n;
-        if (ref_text && ref_text[0] != '\0')
-            tp->ref_text = ref_text;
-        tp->denoise = denoise != 0;
-    }
-    if (seed >= 0)
-        tp->mg_seed = (uint64_t)seed;
-}
-
-float *omni_tts(const char *text, const char *lang, const char *instruct,
-                const float *ref_samples, int ref_n, const char *ref_text,
-                long long seed, int denoise, int *out_n) {
-    if (out_n)
-        *out_n = 0;
-    if (!g_ctx) {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: model not loaded\n");
-        return nullptr;
-    }
-    if (!text || text[0] == '\0') {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: text is required\n");
-        return nullptr; // omni_tts: out_n already 0
-    }
-    ov_tts_params tp;
-    fill_params(&tp, text, lang, instruct, ref_samples, ref_n, ref_text, seed,
-                denoise);
-
-    ov_audio out = {0};
-    enum ov_status rc = ov_synthesize(g_ctx, &tp, &out);
-    if (rc != OV_STATUS_OK || out.n_samples <= 0 || !out.samples) {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: synthesize failed (rc=%d): %s\n",
-                (int)rc, ov_last_error());
-        ov_audio_free(&out);
-        return nullptr;
-    }
-
-    // Copy into a plain malloc buffer the Go side can free symmetrically via
-    // omni_pcm_free; then release the ov_audio-owned buffer.
-    size_t bytes = (size_t)out.n_samples * sizeof(float);
-    float *buf = (float *)malloc(bytes);
-    if (!buf) {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: malloc(%zu) failed\n", bytes);
-        ov_audio_free(&out);
-        return nullptr;
-    }
-    memcpy(buf, out.samples, bytes);
-    if (out_n)
-        *out_n = out.n_samples;
-    ov_audio_free(&out);
-    return buf;
-}
-
-int omni_tts_stream(const char *text, const char *lang, const char *instruct,
-                    const float *ref_samples, int ref_n, const char *ref_text,
-                    long long seed, int denoise, omni_pcm_chunk_cb cb,
-                    void *user_data) {
-    if (!g_ctx) {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: model not loaded\n");
-        return 1;
-    }
-    if (!cb) {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: stream callback is null\n");
-        return 2;
-    }
-    if (!text || text[0] == '\0') {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: text is required\n");
-        return 4;
-    }
-    ov_tts_params tp;
-    fill_params(&tp, text, lang, instruct, ref_samples, ref_n, ref_text, seed,
-                denoise);
-    // ov_audio_chunk_cb has the identical signature to omni_pcm_chunk_cb
-    // (bool vs int return are ABI-compatible; non-zero == true).
-    tp.on_chunk = (ov_audio_chunk_cb)cb;
-    tp.on_chunk_user_data = user_data;
-
-    ov_audio out = {0}; // stays empty in streaming mode
-    enum ov_status rc = ov_synthesize(g_ctx, &tp, &out);
-    ov_audio_free(&out);
-    if (rc != OV_STATUS_OK && rc != OV_STATUS_CANCELLED) {
-        fprintf(stderr, "[omnivoice-cpp] ERROR: stream synth failed (rc=%d): %s\n",
-                (int)rc, ov_last_error());
-        return 3;
-    }
-    return 0;
-}
-
-void omni_pcm_free(float *p) { free(p); }
-
-void omni_unload(void) {
-    if (g_ctx) {
-        ov_free(g_ctx);
-        g_ctx = nullptr;
-    }
-}

backend/go/omnivoice-cpp/cpp/gomnivoicecpp.h

@@ -1,38 +0,0 @@
-#pragma once
-
-#include <cstdint>
-
-extern "C" {
-
-// Streaming PCM chunk callback. samples is mono float PCM at 24 kHz, valid
-// only for the duration of the call. Return non-zero to continue, 0 to abort.
-typedef int (*omni_pcm_chunk_cb)(const float *samples, int n_samples,
-                                 void *user_data);
-
-// Load the LM (model_path) + codec (codec_path) GGUFs. use_fa / clamp_fp16
-// map to ov_init_params. Returns 0 on success, non-zero on failure.
-int omni_load(const char *model_path, const char *codec_path, int use_fa,
-              int clamp_fp16);
-
-// Synthesize to a malloc'd float PCM buffer (caller frees via omni_pcm_free).
-// ref_samples != null && ref_n > 0 => voice cloning (ref_text optional).
-// instruct != null && non-empty => voice design. seed < 0 keeps the default
-// MaskGIT seed. denoise toggles the <|denoise|> marker (only with a reference).
-// Writes the sample count to *out_n. Returns NULL on failure (out_n set to 0).
-float *omni_tts(const char *text, const char *lang, const char *instruct,
-                const float *ref_samples, int ref_n, const char *ref_text,
-                long long seed, int denoise, int *out_n);
-
-// Streaming synthesis: cb is invoked per PCM chunk as audio is produced.
-// Same reference/design/seed semantics as omni_tts. Returns 0 on success.
-int omni_tts_stream(const char *text, const char *lang, const char *instruct,
-                    const float *ref_samples, int ref_n, const char *ref_text,
-                    long long seed, int denoise, omni_pcm_chunk_cb cb,
-                    void *user_data);
-
-// Free a buffer returned by omni_tts.
-void omni_pcm_free(float *p);
-
-// Release the OmniVoice context.
-void omni_unload(void);
-}

backend/go/omnivoice-cpp/e2e_test.go

@@ -1,74 +0,0 @@
-package main
-
-import (
-	"os"
-	"strings"
-
-	"github.com/ebitengine/purego"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func ttsReq(text, voice string, lang *string, dst string) *pb.TTSRequest {
-	return &pb.TTSRequest{Text: text, Voice: voice, Language: lang, Dst: dst}
-}
-
-var _ = Describe("OmniVoice e2e", Label("e2e"), func() {
-	var loaded bool
-
-	BeforeEach(func() {
-		modelPath := os.Getenv("OMNIVOICE_MODEL")
-		codecPath := os.Getenv("OMNIVOICE_CODEC")
-		if modelPath == "" || codecPath == "" {
-			Skip("OMNIVOICE_MODEL / OMNIVOICE_CODEC not set; skipping e2e")
-		}
-		if !loaded {
-			lib := os.Getenv("OMNIVOICE_LIBRARY")
-			if lib == "" {
-				lib = "./libgomnivoicecpp-fallback.so"
-			}
-			h, err := purego.Dlopen(lib, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-			Expect(err).ToNot(HaveOccurred())
-			purego.RegisterLibFunc(&CppLoad, h, "omni_load")
-			purego.RegisterLibFunc(&CppTTS, h, "omni_tts")
-			purego.RegisterLibFunc(&CppTTSStream, h, "omni_tts_stream")
-			purego.RegisterLibFunc(&CppPCMFree, h, "omni_pcm_free")
-			purego.RegisterLibFunc(&CppUnload, h, "omni_unload")
-			Expect(CppLoad(modelPath, codecPath, 0, 0)).To(Equal(0))
-			loaded = true
-		}
-	})
-
-	It("synthesizes a WAV file via TTS", func() {
-		b := &OmnivoiceCpp{opts: loadOptions{seed: 42, denoise: true}}
-		dst := GinkgoT().TempDir() + "/out.wav"
-		lang := "en"
-		err := b.TTS(ttsReq("Hello world.", "", &lang, dst))
-		Expect(err).ToNot(HaveOccurred())
-		fi, err := os.Stat(dst)
-		Expect(err).ToNot(HaveOccurred())
-		Expect(fi.Size()).To(BeNumerically(">", int64(44)))
-	})
-
-	It("streams audio chunks via TTSStream", func() {
-		b := &OmnivoiceCpp{opts: loadOptions{seed: 42, denoise: true}}
-		results := make(chan []byte, 1024)
-		lang := "en"
-		done := make(chan error, 1)
-		go func() { done <- b.TTSStream(ttsReq("Hello there, streaming test.", "", &lang, ""), results) }()
-
-		var chunks int
-		var first []byte
-		for c := range results {
-			if chunks == 0 {
-				first = c
-			}
-			chunks++
-		}
-		Expect(<-done).ToNot(HaveOccurred())
-		Expect(chunks).To(BeNumerically(">=", 2))
-		Expect(string(first[0:4])).To(Equal("RIFF"))
-		Expect(strings.HasPrefix(string(first[8:12]), "WAVE")).To(BeTrue())
-	})
-})

backend/go/omnivoice-cpp/gomnivoicecpp.go

@@ -1,246 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-	"sync"
-	"unsafe"
-
-	"github.com/ebitengine/purego"
-	"github.com/mudler/LocalAI/pkg/grpc/base"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-)
-
-var (
-	// omni_load(model_path, codec_path, use_fa, clamp_fp16) int
-	CppLoad func(modelPath, codecPath string, useFA, clampFP16 int) int
-	// omni_tts(text, lang, instruct, ref_samples, ref_n, ref_text, seed, denoise, out_n) -> float* (uintptr)
-	CppTTS func(text, lang, instruct string, refSamples unsafe.Pointer, refN int,
-		refText string, seed int64, denoise int, outN unsafe.Pointer) uintptr
-	// omni_tts_stream(text, lang, instruct, ref_samples, ref_n, ref_text, seed, denoise, cb, user) int
-	CppTTSStream func(text, lang, instruct string, refSamples unsafe.Pointer, refN int,
-		refText string, seed int64, denoise int, cb uintptr, user uintptr) int
-	CppPCMFree func(ptr uintptr)
-	CppUnload  func()
-)
-
-type OmnivoiceCpp struct {
-	base.SingleThread
-	opts loadOptions
-	// audioPath is the model-config reference voice (tts.audio_path), used as
-	// the default voice-cloning reference when a request does not set Voice.
-	audioPath string
-}
-
-func (o *OmnivoiceCpp) Load(opts *pb.ModelOptions) error {
-	model := opts.ModelFile
-	if model == "" {
-		model = opts.ModelPath
-	}
-	if !filepath.IsAbs(model) && opts.ModelPath != "" {
-		model = filepath.Join(opts.ModelPath, model)
-	}
-
-	o.opts = parseOptions(opts.Options)
-
-	// Resolve the codec/tokenizer GGUF: explicit option, else auto-discover a
-	// *tokenizer*.gguf sibling of the base model.
-	codec := o.opts.codecPath
-	if codec != "" && !filepath.IsAbs(codec) {
-		codec = filepath.Join(filepath.Dir(model), codec)
-	}
-	if codec == "" {
-		codec = discoverTokenizer(filepath.Dir(model))
-	}
-	if codec == "" {
-		return fmt.Errorf("omnivoice: no codec/tokenizer GGUF found; set option 'tokenizer:<file>'")
-	}
-	o.opts.codecPath = codec
-
-	// tts.audio_path (ModelOptions.AudioPath) is the config-level voice-cloning
-	// reference: a default reference WAV used when a request omits Voice.
-	// Resolved relative to the model directory like the codec.
-	o.audioPath = opts.AudioPath
-	if o.audioPath != "" && !filepath.IsAbs(o.audioPath) {
-		o.audioPath = filepath.Join(filepath.Dir(model), o.audioPath)
-	}
-
-	useFA := boolToInt(o.opts.useFA)
-	clamp := boolToInt(o.opts.clampFP16)
-
-	fmt.Fprintf(os.Stderr, "[omnivoice-cpp] Load model=%s codec=%s use_fa=%d clamp_fp16=%d\n",
-		model, codec, useFA, clamp)
-
-	if rc := CppLoad(model, codec, useFA, clamp); rc != 0 {
-		return fmt.Errorf("omnivoice: failed to load model (rc=%d)", rc)
-	}
-	return nil
-}
-
-// discoverTokenizer returns the first *tokenizer*.gguf in dir, or "".
-func discoverTokenizer(dir string) string {
-	entries, err := os.ReadDir(dir)
-	if err != nil {
-		return ""
-	}
-	for _, e := range entries {
-		name := strings.ToLower(e.Name())
-		if strings.Contains(name, "tokenizer") && strings.HasSuffix(name, ".gguf") {
-			return filepath.Join(dir, e.Name())
-		}
-	}
-	return ""
-}
-
-func boolToInt(b bool) int {
-	if b {
-		return 1
-	}
-	return 0
-}
-
-// refAudio loads the reference WAV (voice cloning) if voice points to a file.
-// Returns nil if no cloning (empty or non-path - voice design uses Instructions).
-func (o *OmnivoiceCpp) refAudio(voice string) ([]float32, error) {
-	v := strings.TrimSpace(voice)
-	if v == "" {
-		return nil, nil
-	}
-	if _, err := os.Stat(v); err != nil {
-		return nil, nil
-	}
-	return readWAVAsFloat(v)
-}
-
-// refAudioFor resolves the cloning reference for a request: the per-request
-// Voice takes precedence, falling back to the model-config audio_path. Empty
-// result means no cloning (voice design via Instructions still applies).
-func (o *OmnivoiceCpp) refAudioFor(req *pb.TTSRequest) ([]float32, error) {
-	voice := strings.TrimSpace(req.Voice)
-	if voice == "" {
-		voice = o.audioPath
-	}
-	return o.refAudio(voice)
-}
-
-func reqParam(req *pb.TTSRequest, key string) string {
-	if req.Params == nil {
-		return ""
-	}
-	return req.Params[key]
-}
-
-func (o *OmnivoiceCpp) seedFor(req *pb.TTSRequest) int64 {
-	if s := reqParam(req, "seed"); s != "" {
-		var n int64
-		if _, err := fmt.Sscan(s, &n); err == nil {
-			return n
-		}
-	}
-	return o.opts.seed
-}
-
-func optStr(p *string) string {
-	if p == nil {
-		return ""
-	}
-	return *p
-}
-
-func (o *OmnivoiceCpp) TTS(req *pb.TTSRequest) error {
-	if req.Dst == "" {
-		return fmt.Errorf("omnivoice: TTS requires a destination path")
-	}
-	lang := normalizeLanguage(optStr(req.Language))
-	instruct := optStr(req.Instructions)
-	refText := reqParam(req, "ref_text")
-	seed := o.seedFor(req)
-
-	ref, err := o.refAudioFor(req)
-	if err != nil {
-		return err
-	}
-	var refPtr unsafe.Pointer
-	if len(ref) > 0 {
-		refPtr = unsafe.Pointer(&ref[0])
-	}
-
-	var n int32
-	ptr := CppTTS(req.Text, lang, instruct, refPtr, len(ref), refText, seed,
-		boolToInt(o.opts.denoise), unsafe.Pointer(&n))
-	runtimeKeepAlive(ref)
-	if ptr == 0 || n <= 0 {
-		return fmt.Errorf("omnivoice: synthesis failed")
-	}
-	defer CppPCMFree(ptr)
-	src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) //nolint:govet // C-allocated PCM, copied out before free
-	out := make([]float32, int(n))
-	copy(out, src)
-	return writeWAV24k(req.Dst, out)
-}
-
-// streamState carries the active TTSStream channel to the single shared C
-// callback. base.SingleThread serializes TTS/TTSStream, so one global slot is
-// safe and avoids leaking a purego callback per request (purego callbacks
-// cannot be freed and are capped).
-var (
-	streamMu     sync.Mutex
-	streamChan   chan []byte
-	streamCbOnce sync.Once
-	streamCbPtr  uintptr
-)
-
-// streamCallback is registered once and forwards each PCM chunk to streamChan.
-func streamCallback(samples *float32, nSamples int32, _ uintptr) uintptr {
-	if nSamples <= 0 || samples == nil || streamChan == nil {
-		return 1 // continue
-	}
-	src := unsafe.Slice(samples, int(nSamples))
-	cp := make([]float32, int(nSamples)) // copy out of C memory before returning
-	copy(cp, src)
-	streamChan <- floatToPCM16LE(cp)
-	return 1 // continue
-}
-
-func (o *OmnivoiceCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
-	defer close(results)
-	if req.Text == "" {
-		return fmt.Errorf("omnivoice: TTSStream requires text")
-	}
-
-	streamCbOnce.Do(func() {
-		streamCbPtr = purego.NewCallback(streamCallback)
-	})
-
-	lang := normalizeLanguage(optStr(req.Language))
-	instruct := optStr(req.Instructions)
-	refText := reqParam(req, "ref_text")
-	seed := o.seedFor(req)
-
-	ref, err := o.refAudioFor(req)
-	if err != nil {
-		return err
-	}
-	var refPtr unsafe.Pointer
-	if len(ref) > 0 {
-		refPtr = unsafe.Pointer(&ref[0])
-	}
-
-	// Emit the WAV header first so the HTTP layer gets a self-describing stream.
-	results <- wavHeader24k()
-
-	streamMu.Lock()
-	streamChan = results
-	rc := CppTTSStream(req.Text, lang, instruct, refPtr, len(ref), refText, seed,
-		boolToInt(o.opts.denoise), streamCbPtr, 0)
-	streamChan = nil
-	streamMu.Unlock()
-	runtimeKeepAlive(ref)
-
-	if rc != 0 {
-		return fmt.Errorf("omnivoice: streaming synthesis failed (rc=%d)", rc)
-	}
-	return nil
-}

backend/go/omnivoice-cpp/gomnivoicecpp_test.go

@@ -1,90 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"encoding/binary"
-	"testing"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func TestOmnivoiceCpp(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "omnivoice-cpp suite")
-}
-
-var _ = Describe("normalizeLanguage", func() {
-	DescribeTable("maps caller language to OmniVoice codes",
-		func(in, want string) {
-			Expect(normalizeLanguage(in)).To(Equal(want))
-		},
-		Entry("empty stays empty", "", ""),
-		Entry("english full name", "English", "en"),
-		Entry("chinese full name", "Chinese", "zh"),
-		Entry("locale suffix stripped", "en-US", "en"),
-		Entry("underscore locale", "zh_CN", "zh"),
-		Entry("already a code", "en", "en"),
-		Entry("unknown passes through normalized", "xx", "xx"),
-	)
-})
-
-var _ = Describe("parseOptions", func() {
-	It("extracts codec, use_fa, clamp_fp16, seed, denoise", func() {
-		o := parseOptions([]string{
-			"tokenizer:tok.gguf",
-			"use_fa:true",
-			"clamp_fp16:true",
-			"seed:7",
-			"denoise:false",
-			"unknown:ignored",
-		})
-		Expect(o.codecPath).To(Equal("tok.gguf"))
-		Expect(o.useFA).To(BeTrue())
-		Expect(o.clampFP16).To(BeTrue())
-		Expect(o.seed).To(Equal(int64(7)))
-		Expect(o.denoise).To(BeFalse())
-	})
-
-	It("accepts codec: as an alias for tokenizer:", func() {
-		o := parseOptions([]string{"codec:c.gguf"})
-		Expect(o.codecPath).To(Equal("c.gguf"))
-	})
-
-	It("defaults seed to -1 and denoise to true", func() {
-		o := parseOptions(nil)
-		Expect(o.seed).To(Equal(int64(-1)))
-		Expect(o.denoise).To(BeTrue())
-	})
-})
-
-var _ = Describe("wavHeader24k", func() {
-	It("emits a 44-byte streaming WAV header at 24 kHz mono 16-bit", func() {
-		h := wavHeader24k()
-		Expect(h).To(HaveLen(44))
-		Expect(string(h[0:4])).To(Equal("RIFF"))
-		Expect(string(h[8:12])).To(Equal("WAVE"))
-		Expect(string(h[12:16])).To(Equal("fmt "))
-		Expect(string(h[36:40])).To(Equal("data"))
-		var sampleRate uint32
-		Expect(binary.Read(bytes.NewReader(h[24:28]), binary.LittleEndian, &sampleRate)).To(Succeed())
-		Expect(sampleRate).To(Equal(uint32(24000)))
-	})
-})
-
-var _ = Describe("floatToPCM16LE", func() {
-	It("clamps and converts float PCM to little-endian int16 bytes", func() {
-		b := floatToPCM16LE([]float32{0, 1.0, -1.0, 2.0, -2.0})
-		Expect(b).To(HaveLen(10)) // 5 samples * 2 bytes
-		read := func(off int) int16 {
-			var v int16
-			_ = binary.Read(bytes.NewReader(b[off:off+2]), binary.LittleEndian, &v)
-			return v
-		}
-		Expect(read(0)).To(Equal(int16(0)))
-		Expect(read(2)).To(Equal(int16(32767)))
-		Expect(read(4)).To(Equal(int16(-32767)))
-		Expect(read(6)).To(Equal(int16(32767)))  // clamped from 2.0
-		Expect(read(8)).To(Equal(int16(-32767))) // clamped from -2.0
-	})
-})

backend/go/omnivoice-cpp/main.go

@@ -1,48 +0,0 @@
-package main
-
-// Note: this is started internally by LocalAI and a server is allocated for each model
-import (
-	"flag"
-	"os"
-
-	"github.com/ebitengine/purego"
-	grpc "github.com/mudler/LocalAI/pkg/grpc"
-)
-
-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
-)
-
-type LibFuncs struct {
-	FuncPtr any
-	Name    string
-}
-
-func main() {
-	libName := os.Getenv("OMNIVOICE_LIBRARY")
-	if libName == "" {
-		libName = "./libgomnivoicecpp-fallback.so"
-	}
-
-	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-	if err != nil {
-		panic(err)
-	}
-
-	libFuncs := []LibFuncs{
-		{&CppLoad, "omni_load"},
-		{&CppTTS, "omni_tts"},
-		{&CppTTSStream, "omni_tts_stream"},
-		{&CppPCMFree, "omni_pcm_free"},
-		{&CppUnload, "omni_unload"},
-	}
-	for _, lf := range libFuncs {
-		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
-	}
-
-	flag.Parse()
-
-	if err := grpc.StartServer(*addr, &OmnivoiceCpp{}); err != nil {
-		panic(err)
-	}
-}

backend/go/omnivoice-cpp/options.go

@@ -1,74 +0,0 @@
-package main
-
-import (
-	"strconv"
-	"strings"
-)
-
-// loadOptions holds the parsed model-level options for OmniVoice.
-type loadOptions struct {
-	codecPath string
-	useFA     bool
-	clampFP16 bool
-	seed      int64
-	denoise   bool
-}
-
-func splitOption(o string) (key, value string, ok bool) {
-	i := strings.Index(o, ":")
-	if i < 0 {
-		return "", "", false
-	}
-	return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true
-}
-
-// parseOptions reads the backend "key:value" option slice. Unknown keys are
-// ignored. Defaults: seed -1 (engine default), denoise true.
-func parseOptions(opts []string) loadOptions {
-	o := loadOptions{seed: -1, denoise: true}
-	for _, oo := range opts {
-		key, value, ok := splitOption(oo)
-		if !ok {
-			continue
-		}
-		switch key {
-		case "tokenizer", "codec":
-			o.codecPath = value
-		case "use_fa":
-			o.useFA = value == "true" || value == "1"
-		case "clamp_fp16":
-			o.clampFP16 = value == "true" || value == "1"
-		case "seed":
-			if n, err := strconv.ParseInt(value, 10, 64); err == nil {
-				o.seed = n
-			}
-		case "denoise":
-			o.denoise = value == "true" || value == "1"
-		}
-	}
-	return o
-}
-
-// languageNameAliases maps full language names to OmniVoice codes. OmniVoice's
-// lang hint accepts "" (auto), "en", "zh" per the upstream convention; other
-// codes pass through and the engine treats unknown hints as auto.
-var languageNameAliases = map[string]string{
-	"english": "en",
-	"chinese": "zh",
-}
-
-// normalizeLanguage lowercases, trims, strips a region/locale suffix, and
-// resolves common full names. Empty stays empty so the engine auto-detects.
-func normalizeLanguage(lang string) string {
-	lang = strings.ToLower(strings.TrimSpace(lang))
-	if lang == "" {
-		return ""
-	}
-	if i := strings.IndexAny(lang, "-_."); i >= 0 {
-		lang = lang[:i]
-	}
-	if code, ok := languageNameAliases[lang]; ok {
-		return code
-	}
-	return lang
-}

backend/go/omnivoice-cpp/package.sh

@@ -1,64 +0,0 @@
-#!/bin/bash
-
-# Script to copy the appropriate libraries based on architecture
-# This script is used in the final stage of the Dockerfile
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-# Create lib directory
-mkdir -p $CURDIR/package/lib
-
-cp -avf $CURDIR/omnivoice-cpp $CURDIR/package/
-cp -fv $CURDIR/libgomnivoicecpp-*.so $CURDIR/package/
-cp -fv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    # x86_64 architecture
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    # ARM64 architecture
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ $(uname -s) = "Darwin" ]; then
-    echo "Detected Darwin"
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/go/omnivoice-cpp/run.sh

@@ -1,52 +0,0 @@
-#!/bin/bash
-set -ex
-
-# Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath $0)")
-
-cd /
-
-echo "CPU info:"
-if [ "$(uname)" != "Darwin" ]; then
-	grep -e "model\sname" /proc/cpuinfo | head -1
-	grep -e "flags" /proc/cpuinfo | head -1
-fi
-
-LIBRARY="$CURDIR/libgomnivoicecpp-fallback.so"
-
-if [ "$(uname)" != "Darwin" ]; then
-	if grep -q -e "\savx\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX    found OK"
-		if [ -e $CURDIR/libgomnivoicecpp-avx.so ]; then
-			LIBRARY="$CURDIR/libgomnivoicecpp-avx.so"
-		fi
-	fi
-
-	if grep -q -e "\savx2\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX2   found OK"
-		if [ -e $CURDIR/libgomnivoicecpp-avx2.so ]; then
-			LIBRARY="$CURDIR/libgomnivoicecpp-avx2.so"
-		fi
-	fi
-
-	# Check avx 512
-	if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX512F found OK"
-		if [ -e $CURDIR/libgomnivoicecpp-avx512.so ]; then
-			LIBRARY="$CURDIR/libgomnivoicecpp-avx512.so"
-		fi
-	fi
-fi
-
-export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
-export OMNIVOICE_LIBRARY=$LIBRARY
-
-# If there is a lib/ld.so, use it
-if [ -f $CURDIR/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using library: $LIBRARY"
-	exec $CURDIR/lib/ld.so $CURDIR/omnivoice-cpp "$@"
-fi
-
-echo "Using library: $LIBRARY"
-exec $CURDIR/omnivoice-cpp "$@"

backend/go/omnivoice-cpp/test.sh

@@ -1,30 +0,0 @@
-#!/bin/bash
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-cd "$CURDIR"
-
-echo "Running omnivoice-cpp backend tests..."
-
-if [ -z "$OMNIVOICE_MODEL" ]; then
-    MODEL_DIR="./omnivoice-models"
-    mkdir -p "$MODEL_DIR"
-    REPO_ID="Serveurperso/OmniVoice-GGUF"
-    BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
-    FILES=( "omnivoice-base-Q4_K_M.gguf" "omnivoice-tokenizer-Q4_K_M.gguf" )
-    for file in "${FILES[@]}"; do
-        dest="${MODEL_DIR}/${file}"
-        if [ -f "${dest}" ]; then
-            echo "  [skip] ${file}"
-        else
-            echo "  [download] ${file}..."
-            curl -L -o "${dest}" "${BASE_URL}/${file}" --progress-bar
-        fi
-    done
-    export OMNIVOICE_MODEL="${MODEL_DIR}/omnivoice-base-Q4_K_M.gguf"
-    export OMNIVOICE_CODEC="${MODEL_DIR}/omnivoice-tokenizer-Q4_K_M.gguf"
-fi
-
-go test -v -timeout 1200s .
-
-echo "All omnivoice-cpp e2e tests passed."

backend/go/parakeet-cpp/Makefile

@@ -1,6 +1,6 @@
 # parakeet-cpp backend Makefile.
 #
-# Upstream pin lives below as PARAKEET_VERSION?=b8012f11e5269126eddb7f4fd02f891a2ccc29b0
+# Upstream pin lives below as PARAKEET_VERSION?=e270af73b94c9a5c37ec516230219ed4580e1db6
 # (.github/bump_deps.sh) can find and update it - matches the
 # whisper.cpp / ds4 / vibevoice-cpp convention.
 #
@@ -15,7 +15,7 @@
 # That's what the L0 smoke test uses. The default target below does the
 # proper clone-at-pin + cmake build so CI doesn't need a side-checkout.
 
-PARAKEET_VERSION?=b8012f11e5269126eddb7f4fd02f891a2ccc29b0
+PARAKEET_VERSION?=e270af73b94c9a5c37ec516230219ed4580e1db6
 PARAKEET_REPO?=https://github.com/mudler/parakeet.cpp
 
 GOCMD?=go
@@ -39,10 +39,7 @@ endif
 # is overwritten back to OFF and the build silently falls back to CPU. Forward the
 # PARAKEET_GGML_* options instead. (openblas is not gated, so -DGGML_BLAS passes through.)
 ifeq ($(BUILD_TYPE),cublas)
-	# GGML_CUDA_GRAPHS is OFF by ggml default; enabling it gives a small free
-	# speedup (~1% measured on GB10, never negative) by capturing/replaying the
-	# CUDA graph. Not gated by parakeet.cpp, so it passes straight through to ggml.
-	CMAKE_ARGS+=-DPARAKEET_GGML_CUDA=ON -DGGML_CUDA_GRAPHS=ON
+	CMAKE_ARGS+=-DPARAKEET_GGML_CUDA=ON
 else ifeq ($(BUILD_TYPE),openblas)
 	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
 else ifeq ($(BUILD_TYPE),hipblas)

backend/go/parakeet-cpp/goparakeetcpp.go

@@ -98,21 +98,17 @@ type transcriptJSON struct {
 }
 
 // streamFeedJSON mirrors the document returned by
-// parakeet_capi_stream_feed_json / parakeet_capi_stream_finalize_json (ABI v5):
+// parakeet_capi_stream_feed_json / parakeet_capi_stream_finalize_json (ABI v4):
 //
-//	{"text":"...","eou":0,"eob":0,"frame_sec":0.080000,
+//	{"text":"...","eou":0,"frame_sec":0.080000,
 //	 "words":[{"w":"...","start":0.480,"end":0.640,"conf":0.9100}, ...]}
 //
 // "text" is the newly-finalized text since the last call; "eou" is 1 when an
-// <EOU> (end of utterance) fired this feed and "eob" is 1 when an <EOB>
-// (backchannel) fired. ABI v4 conflated the two into "eou"; v5 split them, so
-// we read both and treat either as an utterance boundary for segmentation.
-// "words" are the words finalized this call with absolute (stream-relative)
-// start/end seconds.
+// <EOU>/<EOB> fired this feed; "words" are the words finalized this call with
+// absolute (stream-relative) start/end seconds.
 type streamFeedJSON struct {
 	Text     string           `json:"text"`
 	Eou      int              `json:"eou"`
-	Eob      int              `json:"eob"`
 	FrameSec float64          `json:"frame_sec"`
 	Words    []transcriptWord `json:"words"`
 }
@@ -487,10 +483,7 @@ type streamSegmenter struct {
 
 func (s *streamSegmenter) add(doc streamFeedJSON) {
 	s.cur = append(s.cur, doc.Words...)
-	// Close the segment on either turn signal: <EOU> (end of utterance) or
-	// <EOB> (backchannel). ABI v4 reported both via "eou"; v5 split them, so we
-	// OR them here to keep the v4 segmentation boundaries.
-	if doc.Eou != 0 || doc.Eob != 0 {
+	if doc.Eou != 0 {
 		s.flush()
 	}
 }
@@ -678,12 +671,11 @@ func (p *ParakeetCpp) AudioTranscriptionStream(ctx context.Context, opts *pb.Tra
 	return nil
 }
 
-// streamJSON drives the streaming JSON entry points (present since ABI v4): each
-// feed/finalize returns a {text,eou,eob,frame_sec,words} document. The
-// newly-finalized text is emitted as a delta (unchanged streaming contract)
-// while words are accumulated into per-utterance segments (closed on <EOU> or
-// <EOB>) so the closing FinalResult carries timestamped segments. Runs under
-// engineMu (already held by the caller).
+// streamJSON drives the ABI v4 streaming JSON entry points: each feed/finalize
+// returns a {text,eou,frame_sec,words} document. The newly-finalized text is
+// emitted as a delta (unchanged streaming contract) while words are accumulated
+// into per-utterance segments (closed on EOU) so the closing FinalResult carries
+// timestamped segments. Runs under engineMu (already held by the caller).
 func (p *ParakeetCpp) streamJSON(ctx context.Context, stream uintptr, data []float32,
 	duration float32, results chan *pb.TranscriptStreamResponse) error {
 	var (

backend/go/parakeet-cpp/segments_test.go

@@ -124,17 +124,4 @@ var _ = Describe("streaming segment assembly", func() {
 		Expect(acc.segments()).To(HaveLen(1))
 		Expect(acc.segments()[0].Text).To(Equal("hi there"))
 	})
-
-	// ABI v5 split <EOB> (backchannel) out of the "eou" flag into its own "eob"
-	// field; a backchannel must still close the segment as it did in v4.
-	It("closes a segment on EOB (backchannel) too", func() {
-		acc := &streamSegmenter{}
-		acc.add(streamFeedJSON{Text: "uh huh", Eou: 0, Eob: 1, Words: []transcriptWord{
-			{W: "uh", Start: 0.0, End: 0.2}, {W: "huh", Start: 0.2, End: 0.5},
-		}})
-		segs := acc.segments()
-		Expect(segs).To(HaveLen(1))
-		Expect(segs[0].Text).To(Equal("uh huh"))
-		Expect(segs[0].End).To(Equal(secondsToNanos(0.5)))
-	})
 })

backend/go/qwen3-tts-cpp/CMakeLists.txt

@@ -3,36 +3,35 @@ project(goqwen3ttscpp LANGUAGES C CXX)
 set(CMAKE_POSITION_INDEPENDENT_CODE ON)
 set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 
-set(QWENTTS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/qwentts.cpp)
+set(QWEN3TTS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/qwen3-tts.cpp)
 
 # Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
 if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
     set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
 endif()
 
-# Add the upstream project. Its own CMakeLists adds ggml + cpp-httplib + yyjson
-# and builds qwen-core (STATIC, the qt_* impl). EXCLUDE_FROM_ALL keeps its CLI
-# tools / tts-server / tests from building unless referenced.
-add_subdirectory(${QWENTTS_DIR} qwentts EXCLUDE_FROM_ALL)
+# Build ggml from the upstream's submodule FIRST, so that ggml/ggml-base/ggml-cpu
+# CMake targets exist when the upstream project references them by name.
+# The upstream CMakeLists.txt uses target_link_libraries(... ggml ggml-base ggml-cpu)
+# with target_link_directories pointing at a pre-built ggml/build/. By adding ggml
+# as a subdirectory here, CMake resolves those names as targets instead.
+add_subdirectory(${QWEN3TTS_DIR}/ggml ggml EXCLUDE_FROM_ALL)
 
-# Upstream generates version.h into its own CMAKE_CURRENT_BINARY_DIR and adds
-# the top-level ${CMAKE_BINARY_DIR} to qwen-core's include path. Under
-# add_subdirectory those two dirs differ (<build>/qwentts vs <build>), so
-# qwen.cpp cannot find version.h. Point qwen-core at the subproject binary dir
-# where version.h is actually generated. (Fix lives here, never in the fetched
-# upstream checkout.)
-target_include_directories(qwen-core PRIVATE ${CMAKE_BINARY_DIR}/qwentts)
+# Now add the upstream project
+add_subdirectory(${QWEN3TTS_DIR} qwen3tts EXCLUDE_FROM_ALL)
 
 add_library(goqwen3ttscpp MODULE cpp/goqwen3ttscpp.cpp)
-target_link_libraries(goqwen3ttscpp PRIVATE qwen-core)
+target_link_libraries(goqwen3ttscpp PRIVATE qwen3_tts)
 
-target_include_directories(goqwen3ttscpp PRIVATE ${QWENTTS_DIR}/src)
-target_include_directories(goqwen3ttscpp SYSTEM PRIVATE ${QWENTTS_DIR}/ggml/include)
+target_include_directories(goqwen3ttscpp PRIVATE ${QWEN3TTS_DIR}/src)
+target_include_directories(goqwen3ttscpp SYSTEM PRIVATE ${QWEN3TTS_DIR}/ggml/include)
 
-# Link GPU backends if the upstream ggml created them.
-foreach(backend blas cuda metal vulkan sycl)
+# Link GPU backends if available
+foreach(backend blas cuda metal vulkan)
     if(TARGET ggml-${backend})
         target_link_libraries(goqwen3ttscpp PRIVATE ggml-${backend})
+        string(TOUPPER ${backend} BACKEND_UPPER)
+        target_compile_definitions(goqwen3ttscpp PRIVATE QWEN3TTS_HAVE_${BACKEND_UPPER})
         if(backend STREQUAL "cuda")
             find_package(CUDAToolkit QUIET)
             if(CUDAToolkit_FOUND)
@@ -45,8 +44,12 @@ endforeach()
 if(MSVC)
     target_compile_options(goqwen3ttscpp PRIVATE /W4 /wd4100 /wd4505)
 else()
-    target_compile_options(goqwen3ttscpp PRIVATE -Wall -Wextra
-                          -Wno-unused-parameter -Wno-unused-function)
+    target_compile_options(goqwen3ttscpp PRIVATE -Wall -Wextra -Wshadow -Wconversion
+                          -Wno-unused-parameter -Wno-unused-function -Wno-sign-conversion)
+endif()
+
+if(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9.0)
+    target_link_libraries(goqwen3ttscpp PRIVATE stdc++fs)
 endif()
 
 set_property(TARGET goqwen3ttscpp PROPERTY CXX_STANDARD 17)

backend/go/qwen3-tts-cpp/Makefile

@@ -6,9 +6,9 @@ GOCMD?=go
 GO_TAGS?=
 JOBS?=$(shell nproc --ignore=1)
 
-# qwentts.cpp version
-QWEN3TTS_REPO?=https://github.com/ServeurpersoCom/qwentts.cpp
-QWEN3TTS_CPP_VERSION?=0bf4a18b22e8bb8718d95294e9f7f45c0d4270a4
+# qwen3-tts.cpp version
+QWEN3TTS_REPO?=https://github.com/predict-woo/qwen3-tts.cpp
+QWEN3TTS_CPP_VERSION?=136e5d36c17083da0321fd96512dc7b263f94a44
 SO_TARGET?=libgoqwen3ttscpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
@@ -49,9 +49,9 @@ ifeq ($(BUILD_TYPE),sycl_f32)
 		-DCMAKE_CXX_COMPILER=icpx
 endif
 
-sources/qwentts.cpp:
-	mkdir -p sources/qwentts.cpp
-	cd sources/qwentts.cpp && \
+sources/qwen3-tts.cpp:
+	mkdir -p sources/qwen3-tts.cpp
+	cd sources/qwen3-tts.cpp && \
 	git init && \
 	git remote add origin $(QWEN3TTS_REPO) && \
 	git fetch origin && \
@@ -78,7 +78,7 @@ package: qwen3-tts-cpp
 build: package
 
 clean: purge
-	rm -rf libgoqwen3ttscpp*.so package sources/qwentts.cpp qwen3-tts-cpp
+	rm -rf libgoqwen3ttscpp*.so package sources/qwen3-tts.cpp qwen3-tts-cpp
 
 purge:
 	rm -rf build*
@@ -88,24 +88,24 @@ purge:
 
 # Build all variants (Linux only)
 ifeq ($(UNAME_S),Linux)
-libgoqwen3ttscpp-avx.so: sources/qwentts.cpp
+libgoqwen3ttscpp-avx.so: sources/qwen3-tts.cpp
 	$(info ${GREEN}I qwen3-tts-cpp build info:avx${RESET})
 	SO_TARGET=libgoqwen3ttscpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgoqwen3ttscpp-custom
 	rm -rf build-libgoqwen3ttscpp-avx.so
 
-libgoqwen3ttscpp-avx2.so: sources/qwentts.cpp
+libgoqwen3ttscpp-avx2.so: sources/qwen3-tts.cpp
 	$(info ${GREEN}I qwen3-tts-cpp build info:avx2${RESET})
 	SO_TARGET=libgoqwen3ttscpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgoqwen3ttscpp-custom
 	rm -rf build-libgoqwen3ttscpp-avx2.so
 
-libgoqwen3ttscpp-avx512.so: sources/qwentts.cpp
+libgoqwen3ttscpp-avx512.so: sources/qwen3-tts.cpp
 	$(info ${GREEN}I qwen3-tts-cpp build info:avx512${RESET})
 	SO_TARGET=libgoqwen3ttscpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgoqwen3ttscpp-custom
 	rm -rf build-libgoqwen3ttscpp-avx512.so
 endif
 
 # Build fallback variant (all platforms)
-libgoqwen3ttscpp-fallback.so: sources/qwentts.cpp
+libgoqwen3ttscpp-fallback.so: sources/qwen3-tts.cpp
 	$(info ${GREEN}I qwen3-tts-cpp build info:fallback${RESET})
 	SO_TARGET=libgoqwen3ttscpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgoqwen3ttscpp-custom
 	rm -rf build-libgoqwen3ttscpp-fallback.so

backend/go/qwen3-tts-cpp/audio.go

@@ -1,128 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"encoding/binary"
-	"fmt"
-	"os"
-	"runtime"
-
-	"github.com/go-audio/audio"
-	"github.com/go-audio/wav"
-)
-
-const qwen3ttsSampleRate = 24000
-
-// wavHeader24k returns a 44-byte WAV header for a streaming 24 kHz mono 16-bit
-// PCM stream, with placeholder (0xFFFFFFFF) sizes since the total length is
-// unknown up front. Emitted as the first chunk of TTSStream so the HTTP layer
-// receives a self-describing WAV (the gRPC TTSStream path never sets Message,
-// so the backend owns the header - see core/backend/tts.go:ModelTTSStream).
-func wavHeader24k() []byte {
-	var buf bytes.Buffer
-	w := func(v any) { _ = binary.Write(&buf, binary.LittleEndian, v) }
-	buf.WriteString("RIFF")
-	w(uint32(0xFFFFFFFF))
-	buf.WriteString("WAVE")
-	buf.WriteString("fmt ")
-	w(uint32(16))                     // Subchunk1Size
-	w(uint16(1))                      // PCM
-	w(uint16(1))                      // mono
-	w(uint32(qwen3ttsSampleRate))     // sample rate
-	w(uint32(qwen3ttsSampleRate * 2)) // byte rate = SR * blockAlign
-	w(uint16(2))                      // block align (16-bit mono)
-	w(uint16(16))                     // bits per sample
-	buf.WriteString("data")
-	w(uint32(0xFFFFFFFF))
-	return buf.Bytes()
-}
-
-// floatToPCM16LE clamps each sample to [-1,1] and encodes it as little-endian
-// signed 16-bit PCM.
-func floatToPCM16LE(samples []float32) []byte {
-	out := make([]byte, len(samples)*2)
-	for i, s := range samples {
-		if s > 1 {
-			s = 1
-		} else if s < -1 {
-			s = -1
-		}
-		v := int16(s * 32767)
-		out[i*2] = byte(v)
-		out[i*2+1] = byte(v >> 8)
-	}
-	return out
-}
-
-// writeWAV24k writes samples as a finalized 24 kHz mono 16-bit WAV at dst.
-func writeWAV24k(dst string, samples []float32) error {
-	f, err := os.Create(dst)
-	if err != nil {
-		return fmt.Errorf("qwen3-tts: create %q: %w", dst, err)
-	}
-	enc := wav.NewEncoder(f, qwen3ttsSampleRate, 16, 1, 1)
-	ints := make([]int, len(samples))
-	for i, s := range samples {
-		if s > 1 {
-			s = 1
-		} else if s < -1 {
-			s = -1
-		}
-		ints[i] = int(s * 32767)
-	}
-	b := &audio.IntBuffer{
-		Format:         &audio.Format{NumChannels: 1, SampleRate: qwen3ttsSampleRate},
-		Data:           ints,
-		SourceBitDepth: 16,
-	}
-	if err := enc.Write(b); err != nil {
-		_ = enc.Close()
-		_ = f.Close()
-		return fmt.Errorf("qwen3-tts: encode WAV: %w", err)
-	}
-	if err := enc.Close(); err != nil {
-		_ = f.Close()
-		return fmt.Errorf("qwen3-tts: finalize WAV: %w", err)
-	}
-	return f.Close()
-}
-
-// readWAVAsFloat decodes a WAV file (any sample rate/channels) to a mono
-// float32 slice in [-1,1] for use as cloning reference audio. qwentts expects
-// 24 kHz; callers should supply 24 kHz reference clips.
-func readWAVAsFloat(path string) ([]float32, error) {
-	f, err := os.Open(path)
-	if err != nil {
-		return nil, fmt.Errorf("qwen3-tts: open ref %q: %w", path, err)
-	}
-	defer func() { _ = f.Close() }()
-
-	dec := wav.NewDecoder(f)
-	buf, err := dec.FullPCMBuffer()
-	if err != nil {
-		return nil, fmt.Errorf("qwen3-tts: decode ref %q: %w", path, err)
-	}
-	ch := int(buf.Format.NumChannels)
-	if ch < 1 {
-		ch = 1
-	}
-	bitDepth := int(buf.SourceBitDepth)
-	if bitDepth == 0 {
-		bitDepth = 16
-	}
-	scale := float32(int64(1) << uint(bitDepth-1))
-	n := len(buf.Data) / ch
-	out := make([]float32, n)
-	for i := 0; i < n; i++ {
-		var acc int
-		for c := 0; c < ch; c++ {
-			acc += buf.Data[i*ch+c]
-		}
-		out[i] = float32(acc) / float32(ch) / scale
-	}
-	return out, nil
-}
-
-// runtimeKeepAlive prevents the GC from reclaiming the reference-audio slice
-// while its backing pointer is in use across the C call.
-func runtimeKeepAlive(v any) { runtime.KeepAlive(v) }

backend/go/qwen3-tts-cpp/audiopath_test.go

@@ -1,54 +0,0 @@
-package main
-
-import (
-	"path/filepath"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-// These specs pin the voice-selection logic in resolveRequest, in particular
-// the config-level audio_path (tts.audio_path -> ModelOptions.AudioPath) being
-// used as the default voice-cloning reference. No model/C library is needed:
-// resolveRequest only reads the reference WAV via readWAVAsFloat (pure Go).
-var _ = Describe("resolveRequest voice/clone selection", func() {
-	var dir, refWav string
-
-	BeforeEach(func() {
-		dir = GinkgoT().TempDir()
-		refWav = filepath.Join(dir, "ref.wav")
-		// 0.5s of non-silent 24kHz mono audio as a clone reference.
-		samples := make([]float32, qwen3ttsSampleRate/2)
-		for i := range samples {
-			samples[i] = 0.1
-		}
-		Expect(writeWAV24k(refWav, samples)).To(Succeed())
-	})
-
-	It("uses the config audio_path as the clone reference when Voice is empty", func() {
-		q := &Qwen3TtsCpp{audioPath: refWav}
-		_, _, speaker, _, ref, _, err := q.resolveRequest(&pb.TTSRequest{Text: "hi"})
-		Expect(err).ToNot(HaveOccurred())
-		Expect(speaker).To(BeEmpty())
-		Expect(len(ref)).To(Equal(qwen3ttsSampleRate / 2))
-	})
-
-	It("lets a per-request audio Voice override audio_path", func() {
-		other := filepath.Join(dir, "other.wav")
-		Expect(writeWAV24k(other, make([]float32, 100))).To(Succeed())
-		q := &Qwen3TtsCpp{audioPath: refWav}
-		_, _, speaker, _, ref, _, err := q.resolveRequest(&pb.TTSRequest{Text: "hi", Voice: other})
-		Expect(err).ToNot(HaveOccurred())
-		Expect(speaker).To(BeEmpty())
-		Expect(len(ref)).To(Equal(100))
-	})
-
-	It("does not trigger audio_path cloning for a named-speaker Voice", func() {
-		q := &Qwen3TtsCpp{audioPath: refWav}
-		_, _, speaker, _, ref, _, err := q.resolveRequest(&pb.TTSRequest{Text: "hi", Voice: "serena"})
-		Expect(err).ToNot(HaveOccurred())
-		Expect(speaker).To(Equal("serena"))
-		Expect(ref).To(BeNil())
-	})
-})

backend/go/qwen3-tts-cpp/cpp/goqwen3ttscpp.cpp

@@ -1,191 +1,161 @@
 #include "goqwen3ttscpp.h"
 #include "ggml-backend.h"
-#include "qwen.h"
+#include "qwen3_tts.h"
 
+#include <cmath>
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
+#include <string>
 
-static qt_context *g_ctx = nullptr;
+using namespace qwen3_tts;
 
-static void ggml_log_cb(enum ggml_log_level level, const char *log,
-                        void * /*data*/) {
+// Global engine (loaded once, reused across requests)
+static Qwen3TTS *g_engine = nullptr;
+static bool g_loaded = false;
+static int g_threads = 4;
+
+static void ggml_log_cb(enum ggml_log_level level, const char *log, void *data) {
+    const char *level_str;
     if (!log)
         return;
-    const char *lvl = "?????";
     switch (level) {
-    case GGML_LOG_LEVEL_DEBUG: lvl = "DEBUG"; break;
-    case GGML_LOG_LEVEL_INFO:  lvl = "INFO";  break;
-    case GGML_LOG_LEVEL_WARN:  lvl = "WARN";  break;
-    case GGML_LOG_LEVEL_ERROR: lvl = "ERROR"; break;
-    default: break;
+    case GGML_LOG_LEVEL_DEBUG:
+        level_str = "DEBUG";
+        break;
+    case GGML_LOG_LEVEL_INFO:
+        level_str = "INFO";
+        break;
+    case GGML_LOG_LEVEL_WARN:
+        level_str = "WARN";
+        break;
+    case GGML_LOG_LEVEL_ERROR:
+        level_str = "ERROR";
+        break;
+    default:
+        level_str = "?????";
+        break;
     }
-    fprintf(stderr, "[%-5s] %s", lvl, log);
+    fprintf(stderr, "[%-5s] ", level_str);
+    fputs(log, stderr);
     fflush(stderr);
 }
 
-int qt3_load(const char *talker_path, const char *codec_path, int use_fa,
-             int clamp_fp16) {
+// Map language string to language_id token used by the model
+static int language_to_id(const char *lang) {
+    if (!lang || lang[0] == '\0')
+        return 2050; // default: English
+    std::string l(lang);
+    if (l == "en")
+        return 2050;
+    if (l == "ru")
+        return 2069;
+    if (l == "zh")
+        return 2055;
+    if (l == "ja")
+        return 2058;
+    if (l == "ko")
+        return 2064;
+    if (l == "de")
+        return 2053;
+    if (l == "fr")
+        return 2061;
+    if (l == "es")
+        return 2054;
+    if (l == "it")
+        return 2056;
+    if (l == "pt")
+        return 2057;
+    fprintf(stderr, "[qwen3-tts-cpp] Unknown language '%s', defaulting to English\n",
+            lang);
+    return 2050;
+}
+
+int load_model(const char *model_dir, int n_threads) {
     ggml_log_set(ggml_log_cb, nullptr);
     ggml_backend_load_all();
 
-    if (!talker_path || talker_path[0] == '\0') {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: talker_path is required\n");
+    if (n_threads <= 0)
+        n_threads = 4;
+    g_threads = n_threads;
+
+    fprintf(stderr, "[qwen3-tts-cpp] Loading models from %s (threads=%d)\n",
+            model_dir, n_threads);
+
+    g_engine = new Qwen3TTS();
+    if (!g_engine->load_models(model_dir)) {
+        fprintf(stderr, "[qwen3-tts-cpp] FATAL: failed to load models from %s\n",
+                model_dir);
+        delete g_engine;
+        g_engine = nullptr;
         return 1;
     }
-    if (!codec_path || codec_path[0] == '\0') {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: codec_path is required\n");
-        return 2;
-    }
 
-    qt_init_params p;
-    qt_init_default_params(&p);
-    p.talker_path = talker_path;
-    p.codec_path = codec_path;
-    p.use_fa = use_fa != 0;
-    p.clamp_fp16 = clamp_fp16 != 0;
-
-    fprintf(stderr, "[qwen3-tts-cpp] Loading talker=%s codec=%s\n", talker_path,
-            codec_path);
-
-    g_ctx = qt_init(&p);
-    if (!g_ctx) {
-        fprintf(stderr, "[qwen3-tts-cpp] FATAL: qt_init failed: %s\n",
-                qt_last_error());
-        return 3;
-    }
-    fprintf(stderr, "[qwen3-tts-cpp] Model loaded (%s)\n", qt_version());
+    g_loaded = true;
+    fprintf(stderr, "[qwen3-tts-cpp] Models loaded successfully\n");
     return 0;
 }
 
-// Fill a qt_tts_params from the flat wrapper arguments. Unset/zero scalars keep
-// the qt defaults (temperature 0.9, top_k 50, top_p 1.0, rep 1.05, max 2048).
-static void fill_params(qt_tts_params *tp, const char *text, const char *lang,
-                        const char *instruct, const char *speaker,
-                        const float *ref_samples, int ref_n,
-                        const char *ref_text, long long seed, float temperature,
-                        int top_k, float top_p, float repetition_penalty,
-                        int max_new_tokens) {
-    qt_tts_default_params(tp);
-    tp->text = text ? text : "";
-    if (lang && lang[0] != '\0')
-        tp->lang = lang; // else keep default NULL -> auto
-    if (instruct && instruct[0] != '\0')
-        tp->instruct = instruct;
-    if (speaker && speaker[0] != '\0')
-        tp->speaker = speaker;
-    if (ref_samples && ref_n > 0) {
-        tp->ref_audio_24k = ref_samples;
-        tp->ref_n_samples = ref_n;
-        if (ref_text && ref_text[0] != '\0')
-            tp->ref_text = ref_text;
-    }
-    if (seed >= 0)
-        tp->seed = (int64_t)seed; // else default -1 (random)
-    if (temperature > 0.0f)
-        tp->temperature = temperature;
-    if (top_k > 0)
-        tp->top_k = top_k;
-    if (top_p > 0.0f)
-        tp->top_p = top_p;
-    if (repetition_penalty > 0.0f)
-        tp->repetition_penalty = repetition_penalty;
-    if (max_new_tokens > 0)
-        tp->max_new_tokens = max_new_tokens;
-}
-
-float *qt3_tts(const char *text, const char *lang, const char *instruct,
-               const char *speaker, const float *ref_samples, int ref_n,
-               const char *ref_text, long long seed, float temperature,
-               int top_k, float top_p, float repetition_penalty,
-               int max_new_tokens, int *out_n) {
-    if (out_n)
-        *out_n = 0;
-    if (!g_ctx) {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: model not loaded\n");
-        return nullptr;
-    }
-    if (!text || text[0] == '\0') {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: text is required\n");
-        return nullptr;
-    }
-    qt_tts_params tp;
-    fill_params(&tp, text, lang, instruct, speaker, ref_samples, ref_n,
-                ref_text, seed, temperature, top_k, top_p, repetition_penalty,
-                max_new_tokens);
-
-    qt_audio out = {0};
-    enum qt_status rc = qt_synthesize(g_ctx, &tp, &out);
-    if (rc != QT_STATUS_OK || out.n_samples <= 0 || !out.samples) {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: synthesize failed (rc=%d): %s\n",
-                (int)rc, qt_last_error());
-        qt_audio_free(&out);
-        return nullptr;
-    }
-
-    // Copy into a plain malloc buffer the Go side frees via qt3_pcm_free.
-    size_t bytes = (size_t)out.n_samples * sizeof(float);
-    float *buf = (float *)malloc(bytes);
-    if (!buf) {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: malloc(%zu) failed\n", bytes);
-        qt_audio_free(&out);
-        return nullptr;
-    }
-    memcpy(buf, out.samples, bytes);
-    if (out_n)
-        *out_n = out.n_samples;
-    qt_audio_free(&out);
-    return buf;
-}
-
-int qt3_tts_stream(const char *text, const char *lang, const char *instruct,
-                   const char *speaker, const float *ref_samples, int ref_n,
-                   const char *ref_text, long long seed, float temperature,
-                   int top_k, float top_p, float repetition_penalty,
-                   int max_new_tokens, qt3_chunk_cb cb, void *user_data) {
-    if (!g_ctx) {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: model not loaded\n");
+int synthesize(const char *text, const char *ref_audio_path, const char *dst,
+               const char *language, float temperature, float top_p,
+               int top_k, float repetition_penalty, int max_audio_tokens,
+               int n_threads) {
+    if (!g_loaded || !g_engine) {
+        fprintf(stderr, "[qwen3-tts-cpp] ERROR: models not loaded\n");
         return 1;
     }
-    if (!cb) {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: stream callback is null\n");
+
+    if (!text || !dst) {
+        fprintf(stderr, "[qwen3-tts-cpp] ERROR: text and dst are required\n");
         return 2;
     }
-    if (!text || text[0] == '\0') {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: text is required\n");
+
+    tts_params params;
+    params.max_audio_tokens = max_audio_tokens > 0 ? max_audio_tokens : 4096;
+    params.temperature = temperature;
+    params.top_p = top_p;
+    params.top_k = top_k;
+    params.repetition_penalty = repetition_penalty;
+    params.n_threads = n_threads > 0 ? n_threads : g_threads;
+    params.language_id = language_to_id(language);
+
+    fprintf(stderr, "[qwen3-tts-cpp] Synthesizing: text='%.50s%s', lang_id=%d, "
+                    "temp=%.2f, threads=%d\n",
+            text, (strlen(text) > 50 ? "..." : ""), params.language_id,
+            temperature, params.n_threads);
+
+    tts_result result;
+    bool has_ref = ref_audio_path && ref_audio_path[0] != '\0';
+
+    if (has_ref) {
+        fprintf(stderr, "[qwen3-tts-cpp] Voice cloning with ref: %s\n",
+                ref_audio_path);
+        result = g_engine->synthesize_with_voice(text, ref_audio_path, params);
+    } else {
+        result = g_engine->synthesize(text, params);
+    }
+
+    if (!result.success) {
+        fprintf(stderr, "[qwen3-tts-cpp] ERROR: synthesis failed: %s\n",
+                result.error_msg.c_str());
+        return 3;
+    }
+
+    int n_samples = (int)result.audio.size();
+    if (n_samples == 0) {
+        fprintf(stderr, "[qwen3-tts-cpp] ERROR: synthesis produced no samples\n");
         return 4;
     }
-    qt_tts_params tp;
-    fill_params(&tp, text, lang, instruct, speaker, ref_samples, ref_n,
-                ref_text, seed, temperature, top_k, top_p, repetition_penalty,
-                max_new_tokens);
-    // qt_audio_chunk_cb has the identical signature to qt3_chunk_cb
-    // (bool vs int return are ABI-compatible; non-zero == true).
-    tp.on_chunk = (qt_audio_chunk_cb)cb;
-    tp.on_chunk_user_data = user_data;
 
-    qt_audio out = {0}; // stays empty in streaming mode
-    enum qt_status rc = qt_synthesize(g_ctx, &tp, &out);
-    qt_audio_free(&out);
-    if (rc != QT_STATUS_OK && rc != QT_STATUS_CANCELLED) {
-        fprintf(stderr, "[qwen3-tts-cpp] ERROR: stream synth failed (rc=%d): %s\n",
-                (int)rc, qt_last_error());
-        return 3;
+    fprintf(stderr,
+            "[qwen3-tts-cpp] Synthesis done: %d samples (%.2fs @ 24kHz)\n",
+            n_samples, (float)n_samples / 24000.0f);
+
+    if (!save_audio_file(dst, result.audio, result.sample_rate)) {
+        fprintf(stderr, "[qwen3-tts-cpp] ERROR: failed to write %s\n", dst);
+        return 5;
     }
+
+    fprintf(stderr, "[qwen3-tts-cpp] Wrote %s\n", dst);
     return 0;
 }
-
-void qt3_pcm_free(float *p) { free(p); }
-
-void qt3_unload(void) {
-    if (g_ctx) {
-        qt_free(g_ctx);
-        g_ctx = nullptr;
-    }
-}
-
-int qt3_n_speakers(void) { return g_ctx ? qt_n_speakers(g_ctx) : 0; }
-
-const char *qt3_speaker_name(int i) {
-    return g_ctx ? qt_speaker_name(g_ctx, i) : nullptr;
-}

backend/go/qwen3-tts-cpp/cpp/goqwen3ttscpp.h

@@ -1,47 +1,12 @@
 #pragma once
 
+#include <cstddef>
+#include <cstdint>
+
 extern "C" {
-
-// Streaming PCM chunk callback. samples is mono float PCM at 24 kHz, valid
-// only for the duration of the call. Return non-zero to continue, 0 to abort.
-typedef int (*qt3_chunk_cb)(const float *samples, int n_samples,
-                            void *user_data);
-
-// Load the talker + codec/tokenizer GGUFs. use_fa / clamp_fp16 map to
-// qt_init_params (the qt ABI exposes no thread count; ggml uses its own
-// default). Returns 0 on success, non-zero on failure.
-int qt3_load(const char *talker_path, const char *codec_path, int use_fa,
-             int clamp_fp16);
-
-// Synthesize to a malloc'd float PCM buffer (caller frees via qt3_pcm_free).
-// The synthesis mode (base / custom_voice / voice_design) is auto-detected by
-// qt from the talker GGUF; speaker is honoured only for custom_voice, instruct
-// for voice_design / custom_voice, and ref_samples (+ optional ref_text) drive
-// base-mode cloning. qt enforces the rules and we surface qt_last_error() on
-// QT_STATUS_MODE_INVALID. Writes the sample count to *out_n. Returns NULL on
-// failure (out_n set to 0).
-float *qt3_tts(const char *text, const char *lang, const char *instruct,
-               const char *speaker, const float *ref_samples, int ref_n,
-               const char *ref_text, long long seed, float temperature,
-               int top_k, float top_p, float repetition_penalty,
-               int max_new_tokens, int *out_n);
-
-// Streaming synthesis: cb is invoked per PCM chunk as audio is produced. Same
-// param semantics as qt3_tts. Returns 0 on success.
-int qt3_tts_stream(const char *text, const char *lang, const char *instruct,
-                   const char *speaker, const float *ref_samples, int ref_n,
-                   const char *ref_text, long long seed, float temperature,
-                   int top_k, float top_p, float repetition_penalty,
-                   int max_new_tokens, qt3_chunk_cb cb, void *user_data);
-
-// Free a buffer returned by qt3_tts.
-void qt3_pcm_free(float *p);
-
-// Release the qt context.
-void qt3_unload(void);
-
-// Named-speaker introspection (custom_voice models). Returns 0 / NULL when no
-// model is loaded or the index is out of range.
-int qt3_n_speakers(void);
-const char *qt3_speaker_name(int i);
+int load_model(const char *model_dir, int n_threads);
+int synthesize(const char *text, const char *ref_audio_path, const char *dst,
+               const char *language, float temperature, float top_p,
+               int top_k, float repetition_penalty, int max_audio_tokens,
+               int n_threads);
 }

backend/go/qwen3-tts-cpp/e2e_test.go

@@ -1,95 +0,0 @@
-package main
-
-import (
-	"math"
-	"os"
-	"strings"
-
-	"github.com/ebitengine/purego"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func ttsReq(text, voice string, lang *string, dst string) *pb.TTSRequest {
-	return &pb.TTSRequest{Text: text, Voice: voice, Language: lang, Dst: dst}
-}
-
-var _ = Describe("qwen3-tts-cpp e2e", Label("e2e"), func() {
-	var loaded bool
-
-	BeforeEach(func() {
-		modelPath := os.Getenv("QWEN3TTS_MODEL")
-		codecPath := os.Getenv("QWEN3TTS_CODEC")
-		if modelPath == "" || codecPath == "" {
-			Skip("QWEN3TTS_MODEL / QWEN3TTS_CODEC not set; skipping e2e")
-		}
-		if !loaded {
-			lib := os.Getenv("QWEN3TTS_LIBRARY")
-			if lib == "" {
-				lib = "./libgoqwen3ttscpp-fallback.so"
-			}
-			h, err := purego.Dlopen(lib, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-			Expect(err).ToNot(HaveOccurred())
-			purego.RegisterLibFunc(&CppLoad, h, "qt3_load")
-			purego.RegisterLibFunc(&CppTTS, h, "qt3_tts")
-			purego.RegisterLibFunc(&CppTTSStream, h, "qt3_tts_stream")
-			purego.RegisterLibFunc(&CppPCMFree, h, "qt3_pcm_free")
-			purego.RegisterLibFunc(&CppUnload, h, "qt3_unload")
-			Expect(CppLoad(modelPath, codecPath, 1, 0)).To(Equal(0))
-			loaded = true
-		}
-	})
-
-	It("synthesizes a WAV file via TTS", func() {
-		b := &Qwen3TtsCpp{opts: loadOptions{seed: 42, useFA: true}}
-		dst := GinkgoT().TempDir() + "/out.wav"
-		lang := "english"
-		err := b.TTS(ttsReq("Hello world.", "", &lang, dst))
-		Expect(err).ToNot(HaveOccurred())
-		fi, err := os.Stat(dst)
-		Expect(err).ToNot(HaveOccurred())
-		Expect(fi.Size()).To(BeNumerically(">", int64(44)))
-	})
-
-	It("streams audio chunks via TTSStream", func() {
-		b := &Qwen3TtsCpp{opts: loadOptions{seed: 42, useFA: true}}
-		results := make(chan []byte, 1024)
-		lang := "english"
-		done := make(chan error, 1)
-		go func() { done <- b.TTSStream(ttsReq("Hello there, streaming test.", "", &lang, ""), results) }()
-
-		var chunks int
-		var first []byte
-		for c := range results {
-			if chunks == 0 {
-				first = c
-			}
-			chunks++
-		}
-		Expect(<-done).ToNot(HaveOccurred())
-		Expect(chunks).To(BeNumerically(">=", 2))
-		Expect(string(first[0:4])).To(Equal("RIFF"))
-		Expect(strings.HasPrefix(string(first[8:12]), "WAVE")).To(BeTrue())
-	})
-
-	It("clones a voice from the config audio_path reference", func() {
-		// 1s of 24kHz mono audio as a clone reference; the base model carries
-		// a speaker encoder, so audio_path drives x-vector voice cloning.
-		ref := GinkgoT().TempDir() + "/ref.wav"
-		samples := make([]float32, qwen3ttsSampleRate)
-		for i := range samples {
-			samples[i] = float32(0.05 * math.Sin(float64(i)*0.06))
-		}
-		Expect(writeWAV24k(ref, samples)).To(Succeed())
-
-		b := &Qwen3TtsCpp{opts: loadOptions{seed: 42, useFA: true}, audioPath: ref}
-		dst := GinkgoT().TempDir() + "/clone.wav"
-		lang := "english"
-		// Empty Voice -> the config audio_path is used as the clone reference.
-		Expect(b.TTS(ttsReq("Cloned voice test.", "", &lang, dst))).To(Succeed())
-		fi, err := os.Stat(dst)
-		Expect(err).ToNot(HaveOccurred())
-		Expect(fi.Size()).To(BeNumerically(">", int64(44)))
-	})
-})

backend/go/qwen3-tts-cpp/goqwen3ttscpp.go

@@ -5,225 +5,108 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
-	"sync"
-	"unsafe"
 
-	"github.com/ebitengine/purego"
 	"github.com/mudler/LocalAI/pkg/grpc/base"
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 )
 
 var (
-	// qt3_load(talker_path, codec_path, use_fa, clamp_fp16) int
-	CppLoad func(talkerPath, codecPath string, useFA, clampFP16 int) int
-	// qt3_tts(text, lang, instruct, speaker, ref_samples, ref_n, ref_text,
-	//         seed, temperature, top_k, top_p, rep_pen, max_new, out_n) -> float*
-	CppTTS func(text, lang, instruct, speaker string, refSamples unsafe.Pointer,
-		refN int, refText string, seed int64, temperature float32, topK int,
-		topP, repPen float32, maxNew int, outN unsafe.Pointer) uintptr
-	// qt3_tts_stream(..., cb, user) int
-	CppTTSStream func(text, lang, instruct, speaker string, refSamples unsafe.Pointer,
-		refN int, refText string, seed int64, temperature float32, topK int,
-		topP, repPen float32, maxNew int, cb uintptr, user uintptr) int
-	CppPCMFree func(ptr uintptr)
-	CppUnload  func()
+	CppLoadModel  func(modelDir string, nThreads int) int
+	CppSynthesize func(text, refAudioPath, dst, language string,
+		temperature, topP float32, topK int,
+		repetitionPenalty float32, maxAudioTokens, nThreads int) int
 )
 
 type Qwen3TtsCpp struct {
 	base.SingleThread
-	opts loadOptions
-	// audioPath is the model-config reference voice (tts.audio_path), the
-	// default clone reference when a request omits an audio Voice.
-	audioPath string
+	threads int
+}
+
+// languageNameAliases maps common full language names to the canonical
+// two-letter code understood by the C++ language_to_id table.
+var languageNameAliases = map[string]string{
+	"english":    "en",
+	"russian":    "ru",
+	"chinese":    "zh",
+	"japanese":   "ja",
+	"korean":     "ko",
+	"german":     "de",
+	"french":     "fr",
+	"spanish":    "es",
+	"italian":    "it",
+	"portuguese": "pt",
+}
+
+// normalizeLanguage coerces a caller-supplied language into the canonical code
+// the model expects. It lowercases, trims, strips any region/locale suffix
+// (en-US, en_US, ja.JP -> en/ja), and resolves common full names (english -> en).
+// An empty input stays empty so the C++ side applies its English default; an
+// unrecognized value is returned normalized so C++ can log it and default.
+func normalizeLanguage(lang string) string {
+	lang = strings.ToLower(strings.TrimSpace(lang))
+	if lang == "" {
+		return ""
+	}
+
+	// Strip region/locale suffix: keep the segment before the first separator.
+	if i := strings.IndexAny(lang, "-_."); i >= 0 {
+		lang = lang[:i]
+	}
+
+	if code, ok := languageNameAliases[lang]; ok {
+		return code
+	}
+	return lang
 }
 
 func (q *Qwen3TtsCpp) Load(opts *pb.ModelOptions) error {
-	model := opts.ModelFile
-	if model == "" {
-		model = opts.ModelPath
-	}
-	if !filepath.IsAbs(model) && opts.ModelPath != "" {
-		model = filepath.Join(opts.ModelPath, model)
+	// ModelFile is the model directory path (containing GGUF files)
+	modelDir := opts.ModelFile
+	if modelDir == "" {
+		modelDir = opts.ModelPath
 	}
 
-	q.opts = parseOptions(opts.Options)
-
-	// Resolve the codec/tokenizer GGUF: explicit option, else auto-discover a
-	// *tokenizer*.gguf sibling of the talker model.
-	codec := q.opts.codecPath
-	if codec != "" && !filepath.IsAbs(codec) {
-		codec = filepath.Join(filepath.Dir(model), codec)
-	}
-	if codec == "" {
-		codec = discoverTokenizer(filepath.Dir(model))
-	}
-	if codec == "" {
-		return fmt.Errorf("qwen3-tts: no codec/tokenizer GGUF found; set option 'tokenizer:<file>'")
-	}
-	q.opts.codecPath = codec
-
-	q.audioPath = opts.AudioPath
-	if q.audioPath != "" && !filepath.IsAbs(q.audioPath) {
-		q.audioPath = filepath.Join(filepath.Dir(model), q.audioPath)
+	// Resolve relative paths
+	if !filepath.IsAbs(modelDir) && opts.ModelPath != "" {
+		modelDir = filepath.Join(opts.ModelPath, modelDir)
 	}
 
-	useFA := boolToInt(q.opts.useFA)
-	clamp := boolToInt(q.opts.clampFP16)
-
-	fmt.Fprintf(os.Stderr, "[qwen3-tts-cpp] Load talker=%s codec=%s use_fa=%d clamp_fp16=%d\n",
-		model, codec, useFA, clamp)
-
-	if rc := CppLoad(model, codec, useFA, clamp); rc != 0 {
-		return fmt.Errorf("qwen3-tts: failed to load model (rc=%d)", rc)
+	threads := int(opts.Threads)
+	if threads <= 0 {
+		threads = 4
 	}
+	q.threads = threads
+
+	fmt.Fprintf(os.Stderr, "[qwen3-tts-cpp] Loading models from: %s (threads=%d)\n", modelDir, threads)
+
+	if ret := CppLoadModel(modelDir, threads); ret != 0 {
+		return fmt.Errorf("failed to load qwen3-tts model (error code: %d)", ret)
+	}
+
 	return nil
 }
 
-// discoverTokenizer returns the first *tokenizer*.gguf in dir, or "".
-func discoverTokenizer(dir string) string {
-	entries, err := os.ReadDir(dir)
-	if err != nil {
-		return ""
-	}
-	for _, e := range entries {
-		name := strings.ToLower(e.Name())
-		if strings.Contains(name, "tokenizer") && strings.HasSuffix(name, ".gguf") {
-			return filepath.Join(dir, e.Name())
-		}
-	}
-	return ""
-}
-
-func boolToInt(b bool) int {
-	if b {
-		return 1
-	}
-	return 0
-}
-
-func optStr(p *string) string {
-	if p == nil {
-		return ""
-	}
-	return *p
-}
-
-// resolveRequest derives the synthesis inputs from a TTSRequest:
-// language, instruct, speaker, ref-audio samples, ref-text and sampling.
-func (q *Qwen3TtsCpp) resolveRequest(req *pb.TTSRequest) (lang, instruct, speaker, refText string, ref []float32, s sampling, err error) {
-	lang = normalizeLanguage(optStr(req.Language))
-	instruct = optStr(req.Instructions)
-
-	var refPath string
-	speaker, refPath = resolveVoice(req.Voice)
-	if refPath == "" && speaker == "" && q.audioPath != "" {
-		// No per-request voice: fall back to the config clone reference.
-		refPath = q.audioPath
-	}
-	if refPath != "" {
-		ref, err = readWAVAsFloat(refPath)
-		if err != nil {
-			return
-		}
-	}
-
-	if req.Params != nil {
-		refText = req.Params["ref_text"]
-	}
-	s = parseSampling(req.Params, q.opts.seed)
-	return
-}
-
 func (q *Qwen3TtsCpp) TTS(req *pb.TTSRequest) error {
-	if req.Dst == "" {
-		return fmt.Errorf("qwen3-tts: TTS requires a destination path")
-	}
-	if req.Text == "" {
-		return fmt.Errorf("qwen3-tts: TTS requires text")
-	}
-	lang, instruct, speaker, refText, ref, s, err := q.resolveRequest(req)
-	if err != nil {
-		return err
-	}
-	var refPtr unsafe.Pointer
-	if len(ref) > 0 {
-		refPtr = unsafe.Pointer(&ref[0])
+	text := req.Text
+	voice := req.Voice // reference audio path for voice cloning (empty = no cloning)
+	dst := req.Dst
+	language := ""
+	if req.Language != nil {
+		language = normalizeLanguage(*req.Language)
 	}
 
-	var n int32
-	ptr := CppTTS(req.Text, lang, instruct, speaker, refPtr, len(ref), refText,
-		s.seed, s.temperature, s.topK, s.topP, s.repPen, s.maxNew, unsafe.Pointer(&n))
-	runtimeKeepAlive(ref)
-	if ptr == 0 {
-		return fmt.Errorf("qwen3-tts: synthesis failed")
-	}
-	// Register the free as soon as we own a non-null buffer, so the n<=0 guard
-	// below cannot leak it (defensive: the C contract returns NULL on failure).
-	defer CppPCMFree(ptr)
-	if n <= 0 {
-		return fmt.Errorf("qwen3-tts: synthesis produced no samples")
-	}
-	src := unsafe.Slice((*float32)(unsafe.Pointer(ptr)), int(n)) //nolint:govet // C-allocated PCM, copied out before free
-	out := make([]float32, int(n))
-	copy(out, src)
-	return writeWAV24k(req.Dst, out)
-}
+	// Synthesis parameters with sensible defaults
+	temperature := float32(0.9)
+	topP := float32(0.8)
+	topK := 50
+	repetitionPenalty := float32(1.05)
+	maxAudioTokens := 4096
 
-// streamState carries the active TTSStream channel to the single shared C
-// callback. base.SingleThread serializes TTS/TTSStream, so one global slot is
-// safe and avoids leaking a purego callback per request (purego callbacks
-// cannot be freed and are capped).
-var (
-	streamMu     sync.Mutex
-	streamChan   chan []byte
-	streamCbOnce sync.Once
-	streamCbPtr  uintptr
-)
-
-// streamCallback is registered once and forwards each PCM chunk to streamChan.
-func streamCallback(samples *float32, nSamples int32, _ uintptr) uintptr {
-	if nSamples <= 0 || samples == nil || streamChan == nil {
-		return 1 // continue
-	}
-	src := unsafe.Slice(samples, int(nSamples))
-	cp := make([]float32, int(nSamples)) // copy out of C memory before returning
-	copy(cp, src)
-	streamChan <- floatToPCM16LE(cp)
-	return 1 // continue
-}
-
-func (q *Qwen3TtsCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
-	defer close(results)
-	if req.Text == "" {
-		return fmt.Errorf("qwen3-tts: TTSStream requires text")
+	if ret := CppSynthesize(text, voice, dst, language,
+		temperature, topP, topK, repetitionPenalty,
+		maxAudioTokens, q.threads); ret != 0 {
+		return fmt.Errorf("failed to synthesize audio (error code: %d)", ret)
 	}
 
-	streamCbOnce.Do(func() {
-		streamCbPtr = purego.NewCallback(streamCallback)
-	})
-
-	lang, instruct, speaker, refText, ref, s, err := q.resolveRequest(req)
-	if err != nil {
-		return err
-	}
-	var refPtr unsafe.Pointer
-	if len(ref) > 0 {
-		refPtr = unsafe.Pointer(&ref[0])
-	}
-
-	// Emit the WAV header first so the HTTP layer gets a self-describing stream.
-	results <- wavHeader24k()
-
-	streamMu.Lock()
-	streamChan = results
-	rc := CppTTSStream(req.Text, lang, instruct, speaker, refPtr, len(ref), refText,
-		s.seed, s.temperature, s.topK, s.topP, s.repPen, s.maxNew, streamCbPtr, 0)
-	streamChan = nil
-	streamMu.Unlock()
-	runtimeKeepAlive(ref)
-
-	if rc != 0 {
-		return fmt.Errorf("qwen3-tts: streaming synthesis failed (rc=%d)", rc)
-	}
 	return nil
 }

backend/go/qwen3-tts-cpp/language_test.go

@@ -0,0 +1,53 @@
+package main
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestLanguageNormalization(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "qwen3-tts-cpp language normalization")
+}
+
+var _ = Describe("normalizeLanguage", func() {
+	DescribeTable("maps caller input to the canonical model language code",
+		func(input, expected string) {
+			Expect(normalizeLanguage(input)).To(Equal(expected))
+		},
+		// Canonical codes pass through unchanged
+		Entry("canonical en", "en", "en"),
+		Entry("canonical zh", "zh", "zh"),
+		Entry("canonical pt", "pt", "pt"),
+
+		// Case-insensitive
+		Entry("uppercase", "EN", "en"),
+		Entry("mixed case", "Ja", "ja"),
+
+		// Surrounding whitespace
+		Entry("trims whitespace", "  en  ", "en"),
+
+		// Region/locale stripping
+		Entry("BCP-47 region", "en-US", "en"),
+		Entry("underscore region", "en_US", "en"),
+		Entry("dotted locale", "ja.JP", "ja"),
+		Entry("region + case", "ZH-CN", "zh"),
+
+		// Full-name aliases
+		Entry("english name", "english", "en"),
+		Entry("chinese name cased", "Chinese", "zh"),
+		Entry("japanese name", "japanese", "ja"),
+		Entry("russian name", "russian", "ru"),
+		Entry("portuguese name", "portuguese", "pt"),
+
+		// Empty stays empty (C++ applies the English default)
+		Entry("empty", "", ""),
+		Entry("whitespace only", "   ", ""),
+
+		// Unknown values pass through normalized so C++ can log + default
+		Entry("unknown code", "klingon", "klingon"),
+		Entry("unknown with region", "xx-YY", "xx"),
+	)
+})

backend/go/qwen3-tts-cpp/main.go

@@ -19,25 +19,24 @@ type LibFuncs struct {
 }
 
 func main() {
+	// Get library name from environment variable, default to fallback
 	libName := os.Getenv("QWEN3TTS_LIBRARY")
 	if libName == "" {
 		libName = "./libgoqwen3ttscpp-fallback.so"
 	}
 
-	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	gosd, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
 	if err != nil {
 		panic(err)
 	}
 
 	libFuncs := []LibFuncs{
-		{&CppLoad, "qt3_load"},
-		{&CppTTS, "qt3_tts"},
-		{&CppTTSStream, "qt3_tts_stream"},
-		{&CppPCMFree, "qt3_pcm_free"},
-		{&CppUnload, "qt3_unload"},
+		{&CppLoadModel, "load_model"},
+		{&CppSynthesize, "synthesize"},
 	}
+
 	for _, lf := range libFuncs {
-		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
+		purego.RegisterLibFunc(lf.FuncPtr, gosd, lf.Name)
 	}
 
 	flag.Parse()

backend/go/qwen3-tts-cpp/options.go

@@ -1,161 +0,0 @@
-package main
-
-import (
-	"strconv"
-	"strings"
-)
-
-// loadOptions holds the parsed model-level options.
-type loadOptions struct {
-	codecPath string
-	useFA     bool
-	clampFP16 bool
-	seed      int64
-}
-
-// sampling holds per-request generation parameters with qt defaults applied.
-type sampling struct {
-	temperature float32
-	topK        int
-	topP        float32
-	repPen      float32
-	maxNew      int
-	seed        int64
-}
-
-func splitOption(o string) (key, value string, ok bool) {
-	i := strings.Index(o, ":")
-	if i < 0 {
-		return "", "", false
-	}
-	return strings.TrimSpace(o[:i]), strings.TrimSpace(o[i+1:]), true
-}
-
-func parseBool(v string) bool { return v == "true" || v == "1" }
-
-// parseOptions reads the backend "key:value" option slice. Unknown keys are
-// ignored. Defaults: use_fa true (qt default; CPU still uses the F32 chain),
-// seed -1 (engine random).
-func parseOptions(opts []string) loadOptions {
-	o := loadOptions{useFA: true, seed: -1}
-	for _, oo := range opts {
-		key, value, ok := splitOption(oo)
-		if !ok {
-			continue
-		}
-		switch key {
-		case "tokenizer", "codec":
-			o.codecPath = value
-		case "use_fa":
-			o.useFA = parseBool(value)
-		case "clamp_fp16":
-			o.clampFP16 = parseBool(value)
-		case "seed":
-			if n, err := strconv.ParseInt(value, 10, 64); err == nil {
-				o.seed = n
-			}
-		}
-	}
-	return o
-}
-
-// languageAliases maps codes / locales / full names to the upstream qwentts
-// language names. "auto" (and empty) map to "" so the engine auto-detects.
-var languageAliases = map[string]string{
-	"en": "english", "english": "english",
-	"zh": "chinese", "chinese": "chinese", "mandarin": "chinese",
-	"ja": "japanese", "japanese": "japanese",
-	"ko": "korean", "korean": "korean",
-	"de": "german", "german": "german",
-	"fr": "french", "french": "french",
-	"es": "spanish", "spanish": "spanish",
-	"it": "italian", "italian": "italian",
-	"pt": "portuguese", "portuguese": "portuguese",
-	"ru": "russian", "russian": "russian",
-	"auto": "",
-}
-
-// normalizeLanguage lowercases, trims, strips a region/locale suffix
-// (en-US -> en), and resolves to the qwentts language name. Empty stays empty
-// (engine auto-detects); an unknown value passes through normalized.
-func normalizeLanguage(lang string) string {
-	lang = strings.ToLower(strings.TrimSpace(lang))
-	if lang == "" {
-		return ""
-	}
-	if i := strings.IndexAny(lang, "-_."); i >= 0 {
-		lang = lang[:i]
-	}
-	if v, ok := languageAliases[lang]; ok {
-		return v
-	}
-	return lang
-}
-
-var refAudioExts = []string{".wav", ".flac", ".mp3", ".ogg", ".m4a"}
-
-// resolveVoice interprets the request Voice field: a value ending in a known
-// audio extension is a clone-reference path; anything else is a named speaker
-// (custom_voice). Empty input yields no speaker and no reference.
-func resolveVoice(voice string) (speaker, refPath string) {
-	v := strings.TrimSpace(voice)
-	if v == "" {
-		return "", ""
-	}
-	lower := strings.ToLower(v)
-	for _, ext := range refAudioExts {
-		if strings.HasSuffix(lower, ext) {
-			return "", v
-		}
-	}
-	return v, ""
-}
-
-func parseFloat32(v string, def float32) float32 {
-	if v == "" {
-		return def
-	}
-	f, err := strconv.ParseFloat(v, 32)
-	if err != nil {
-		return def
-	}
-	return float32(f)
-}
-
-func parseInt(v string, def int) int {
-	if v == "" {
-		return def
-	}
-	n, err := strconv.Atoi(v)
-	if err != nil {
-		return def
-	}
-	return n
-}
-
-func parseInt64(v string, def int64) int64 {
-	if v == "" {
-		return def
-	}
-	n, err := strconv.ParseInt(v, 10, 64)
-	if err != nil {
-		return def
-	}
-	return n
-}
-
-// parseSampling reads per-request sampling params from the TTSRequest params
-// map, applying qt defaults (matching qt_tts_default_params).
-func parseSampling(params map[string]string, defaultSeed int64) sampling {
-	s := sampling{temperature: 0.9, topK: 50, topP: 1.0, repPen: 1.05, maxNew: 2048, seed: defaultSeed}
-	if params == nil {
-		return s
-	}
-	s.temperature = parseFloat32(params["temperature"], s.temperature)
-	s.topK = parseInt(params["top_k"], s.topK)
-	s.topP = parseFloat32(params["top_p"], s.topP)
-	s.repPen = parseFloat32(params["repetition_penalty"], s.repPen)
-	s.maxNew = parseInt(params["max_new_tokens"], s.maxNew)
-	s.seed = parseInt64(params["seed"], s.seed)
-	return s
-}

backend/go/qwen3-tts-cpp/qwen3ttscpp_test.go

@@ -1,136 +1,173 @@
 package main
 
 import (
-	"bytes"
-	"encoding/binary"
+	"context"
+	"os"
+	"os/exec"
+	"path/filepath"
 	"testing"
+	"time"
 
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials/insecure"
 )
 
-func TestQwen3TtsCpp(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "qwen3-tts-cpp suite")
+const (
+	testAddr    = "localhost:50051"
+	startupWait = 5 * time.Second
+)
+
+func skipIfNoModel(t *testing.T) string {
+	t.Helper()
+	modelDir := os.Getenv("QWEN3TTS_MODEL_DIR")
+	if modelDir == "" {
+		t.Skip("QWEN3TTS_MODEL_DIR not set, skipping test (set to directory with GGUF models)")
+	}
+	if _, err := os.Stat(filepath.Join(modelDir, "qwen3-tts-0.6b-f16.gguf")); os.IsNotExist(err) {
+		t.Skipf("TTS model file not found in %s, skipping", modelDir)
+	}
+	if _, err := os.Stat(filepath.Join(modelDir, "qwen3-tts-tokenizer-f16.gguf")); os.IsNotExist(err) {
+		t.Skipf("Tokenizer model file not found in %s, skipping", modelDir)
+	}
+	return modelDir
 }
 
-var _ = Describe("normalizeLanguage", func() {
-	DescribeTable("maps caller language to qwentts language names",
-		func(in, want string) {
-			Expect(normalizeLanguage(in)).To(Equal(want))
-		},
-		Entry("empty stays empty", "", ""),
-		Entry("auto maps to empty", "auto", ""),
-		Entry("english full name", "English", "english"),
-		Entry("english code", "en", "english"),
-		Entry("locale suffix stripped", "en-US", "english"),
-		Entry("underscore locale", "zh_CN", "chinese"),
-		Entry("mandarin alias", "mandarin", "chinese"),
-		Entry("japanese already full", "japanese", "japanese"),
-		Entry("unknown passes through normalized", "xx", "xx"),
+func startServer(t *testing.T) *exec.Cmd {
+	t.Helper()
+	binary := os.Getenv("QWEN3TTS_BINARY")
+	if binary == "" {
+		binary = "./qwen3-tts-cpp"
+	}
+	if _, err := os.Stat(binary); os.IsNotExist(err) {
+		t.Skipf("Backend binary not found at %s, skipping", binary)
+	}
+	cmd := exec.Command(binary, "--addr", testAddr)
+	cmd.Stdout = os.Stderr
+	cmd.Stderr = os.Stderr
+	if err := cmd.Start(); err != nil {
+		t.Fatalf("Failed to start server: %v", err)
+	}
+	time.Sleep(startupWait)
+	return cmd
+}
+
+func stopServer(cmd *exec.Cmd) {
+	if cmd != nil && cmd.Process != nil {
+		cmd.Process.Kill()
+		cmd.Wait()
+	}
+}
+
+func dialGRPC(t *testing.T) *grpc.ClientConn {
+	t.Helper()
+	conn, err := grpc.Dial(testAddr,
+		grpc.WithTransportCredentials(insecure.NewCredentials()),
+		grpc.WithDefaultCallOptions(
+			grpc.MaxCallRecvMsgSize(50*1024*1024),
+			grpc.MaxCallSendMsgSize(50*1024*1024),
+		),
 	)
-})
+	if err != nil {
+		t.Fatalf("Failed to dial gRPC: %v", err)
+	}
+	return conn
+}
 
-var _ = Describe("resolveVoice", func() {
-	It("treats a bare token as a named speaker", func() {
-		sp, ref := resolveVoice("serena")
-		Expect(sp).To(Equal("serena"))
-		Expect(ref).To(BeEmpty())
-	})
-	It("treats an audio path as a clone reference (case-insensitive ext)", func() {
-		sp, ref := resolveVoice("/x/ref.WAV")
-		Expect(sp).To(BeEmpty())
-		Expect(ref).To(Equal("/x/ref.WAV"))
-	})
-	It("recognizes mp3/flac/ogg/m4a", func() {
-		for _, p := range []string{"a.mp3", "b.flac", "c.ogg", "d.m4a"} {
-			sp, ref := resolveVoice(p)
-			Expect(sp).To(BeEmpty())
-			Expect(ref).To(Equal(p))
-		}
-	})
-	It("returns empty for empty input", func() {
-		sp, ref := resolveVoice("  ")
-		Expect(sp).To(BeEmpty())
-		Expect(ref).To(BeEmpty())
-	})
-})
+func TestServerHealth(t *testing.T) {
+	cmd := startServer(t)
+	defer stopServer(cmd)
 
-var _ = Describe("parseOptions", func() {
-	It("extracts codec, use_fa, clamp_fp16, seed", func() {
-		o := parseOptions([]string{
-			"tokenizer:tok.gguf", "use_fa:false", "clamp_fp16:true",
-			"seed:7", "unknown:ignored",
-		})
-		Expect(o.codecPath).To(Equal("tok.gguf"))
-		Expect(o.useFA).To(BeFalse())
-		Expect(o.clampFP16).To(BeTrue())
-		Expect(o.seed).To(Equal(int64(7)))
-	})
-	It("accepts codec: as an alias for tokenizer:", func() {
-		Expect(parseOptions([]string{"codec:c.gguf"}).codecPath).To(Equal("c.gguf"))
-	})
-	It("defaults use_fa true and seed -1", func() {
-		o := parseOptions(nil)
-		Expect(o.useFA).To(BeTrue())
-		Expect(o.seed).To(Equal(int64(-1)))
-	})
-})
+	conn := dialGRPC(t)
+	defer conn.Close()
 
-var _ = Describe("parseSampling", func() {
-	It("applies qt defaults when params are absent", func() {
-		s := parseSampling(nil, -1)
-		Expect(s.temperature).To(BeNumerically("~", 0.9, 1e-6))
-		Expect(s.topK).To(Equal(50))
-		Expect(s.topP).To(BeNumerically("~", 1.0, 1e-6))
-		Expect(s.repPen).To(BeNumerically("~", 1.05, 1e-6))
-		Expect(s.maxNew).To(Equal(2048))
-		Expect(s.seed).To(Equal(int64(-1)))
-	})
-	It("reads overrides and falls back to default seed", func() {
-		s := parseSampling(map[string]string{
-			"temperature": "0.5", "top_k": "10", "top_p": "0.8",
-			"repetition_penalty": "1.2", "max_new_tokens": "512",
-		}, 99)
-		Expect(s.temperature).To(BeNumerically("~", 0.5, 1e-6))
-		Expect(s.topK).To(Equal(10))
-		Expect(s.topP).To(BeNumerically("~", 0.8, 1e-6))
-		Expect(s.repPen).To(BeNumerically("~", 1.2, 1e-6))
-		Expect(s.maxNew).To(Equal(512))
-		Expect(s.seed).To(Equal(int64(99)))
-	})
-	It("reads an explicit seed override", func() {
-		Expect(parseSampling(map[string]string{"seed": "123"}, -1).seed).To(Equal(int64(123)))
-	})
-})
+	client := pb.NewBackendClient(conn)
+	resp, err := client.Health(context.Background(), &pb.HealthMessage{})
+	if err != nil {
+		t.Fatalf("Health check failed: %v", err)
+	}
+	if string(resp.Message) != "OK" {
+		t.Fatalf("Expected OK, got %s", string(resp.Message))
+	}
+}
 
-var _ = Describe("wavHeader24k", func() {
-	It("emits a 44-byte streaming WAV header at 24 kHz mono 16-bit", func() {
-		h := wavHeader24k()
-		Expect(h).To(HaveLen(44))
-		Expect(string(h[0:4])).To(Equal("RIFF"))
-		Expect(string(h[8:12])).To(Equal("WAVE"))
-		Expect(string(h[12:16])).To(Equal("fmt "))
-		Expect(string(h[36:40])).To(Equal("data"))
-		var sampleRate uint32
-		Expect(binary.Read(bytes.NewReader(h[24:28]), binary.LittleEndian, &sampleRate)).To(Succeed())
-		Expect(sampleRate).To(Equal(uint32(24000)))
-	})
-})
+func TestLoadModel(t *testing.T) {
+	modelDir := skipIfNoModel(t)
+	cmd := startServer(t)
+	defer stopServer(cmd)
 
-var _ = Describe("floatToPCM16LE", func() {
-	It("clamps and converts float PCM to little-endian int16 bytes", func() {
-		b := floatToPCM16LE([]float32{0, 1.0, -1.0, 2.0, -2.0})
-		Expect(b).To(HaveLen(10))
-		read := func(off int) int16 {
-			var v int16
-			_ = binary.Read(bytes.NewReader(b[off:off+2]), binary.LittleEndian, &v)
-			return v
-		}
-		Expect(read(0)).To(Equal(int16(0)))
-		Expect(read(2)).To(Equal(int16(32767)))
-		Expect(read(4)).To(Equal(int16(-32767)))
-		Expect(read(6)).To(Equal(int16(32767)))  // clamped from 2.0
-		Expect(read(8)).To(Equal(int16(-32767))) // clamped from -2.0
+	conn := dialGRPC(t)
+	defer conn.Close()
+
+	client := pb.NewBackendClient(conn)
+
+	resp, err := client.LoadModel(context.Background(), &pb.ModelOptions{
+		ModelFile: modelDir,
+		Threads:   4,
 	})
-})
+	if err != nil {
+		t.Fatalf("LoadModel failed: %v", err)
+	}
+	if !resp.Success {
+		t.Fatalf("LoadModel returned failure: %s", resp.Message)
+	}
+}
+
+func TestTTS(t *testing.T) {
+	modelDir := skipIfNoModel(t)
+
+	tmpDir, err := os.MkdirTemp("", "qwen3tts-test")
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { os.RemoveAll(tmpDir) })
+
+	outputFile := filepath.Join(tmpDir, "output.wav")
+
+	cmd := startServer(t)
+	defer stopServer(cmd)
+
+	conn := dialGRPC(t)
+	defer conn.Close()
+
+	client := pb.NewBackendClient(conn)
+
+	// Load models
+	loadResp, err := client.LoadModel(context.Background(), &pb.ModelOptions{
+		ModelFile: modelDir,
+		Threads:   4,
+	})
+	if err != nil {
+		t.Fatalf("LoadModel failed: %v", err)
+	}
+	if !loadResp.Success {
+		t.Fatalf("LoadModel returned failure: %s", loadResp.Message)
+	}
+
+	// Synthesize speech
+	language := "en"
+	_, err = client.TTS(context.Background(), &pb.TTSRequest{
+		Text:     "Hello, this is a test of the Qwen3 text to speech system.",
+		Dst:      outputFile,
+		Language: &language,
+	})
+	if err != nil {
+		t.Fatalf("TTS failed: %v", err)
+	}
+
+	// Verify output file exists and has content
+	info, err := os.Stat(outputFile)
+	if os.IsNotExist(err) {
+		t.Fatal("Output audio file was not created")
+	}
+	if err != nil {
+		t.Fatalf("Failed to stat output file: %v", err)
+	}
+
+	t.Logf("Output file size: %d bytes", info.Size())
+
+	// WAV header is 44 bytes minimum; any real audio should be much larger
+	if info.Size() < 1000 {
+		t.Errorf("Output file too small (%d bytes), expected real audio data", info.Size())
+	}
+}

backend/go/qwen3-tts-cpp/test.sh

@@ -2,30 +2,51 @@
 set -e
 
 CURDIR=$(dirname "$(realpath $0)")
-cd "$CURDIR"
 
 echo "Running qwen3-tts-cpp backend tests..."
 
-# Auto-download a small model pair only when QWEN3TTS_MODEL is not set.
-if [ -z "$QWEN3TTS_MODEL" ]; then
-    MODEL_DIR="./qwen3-tts-models"
-    mkdir -p "$MODEL_DIR"
-    REPO_ID="Serveurperso/Qwen3-TTS-GGUF"
-    BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
-    FILES=( "qwen-talker-0.6b-base-Q4_K_M.gguf" "qwen-tokenizer-12hz-Q4_K_M.gguf" )
-    for file in "${FILES[@]}"; do
-        dest="${MODEL_DIR}/${file}"
-        if [ -f "${dest}" ]; then
-            echo "  [skip] ${file}"
-        else
-            echo "  [download] ${file}..."
-            curl -L -o "${dest}" "${BASE_URL}/${file}" --progress-bar
-        fi
-    done
-    export QWEN3TTS_MODEL="${MODEL_DIR}/qwen-talker-0.6b-base-Q4_K_M.gguf"
-    export QWEN3TTS_CODEC="${MODEL_DIR}/qwen-tokenizer-12hz-Q4_K_M.gguf"
+# The test requires:
+#   - QWEN3TTS_MODEL_DIR: path to directory containing GGUF model files
+#   - QWEN3TTS_BINARY: path to the qwen3-tts-cpp binary (defaults to ./qwen3-tts-cpp)
+#
+# Tests that require the model will be skipped if QWEN3TTS_MODEL_DIR is not set
+# or the directory does not contain the required model files.
+
+cd "$CURDIR"
+
+# Only auto-download models when QWEN3TTS_MODEL_DIR is not explicitly set
+if [ -z "$QWEN3TTS_MODEL_DIR" ]; then
+    export QWEN3TTS_MODEL_DIR="./qwen3-tts-models"
+
+    if [ ! -d "$QWEN3TTS_MODEL_DIR" ]; then
+        echo "Creating qwen3-tts-models directory for tests..."
+        mkdir -p "$QWEN3TTS_MODEL_DIR"
+        REPO_ID="endo5501/qwen3-tts.cpp"
+        echo "Repository: ${REPO_ID}"
+        echo ""
+
+        # Files to download (smallest model for testing)
+        FILES=(
+            "qwen3-tts-0.6b-f16.gguf"
+            "qwen3-tts-tokenizer-f16.gguf"
+        )
+
+        BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
+
+        for file in "${FILES[@]}"; do
+            dest="${QWEN3TTS_MODEL_DIR}/${file}"
+            if [ -f "${dest}" ]; then
+                echo "  [skip] ${file} (already exists)"
+            else
+                echo "  [download] ${file}..."
+                curl -L -o "${dest}" "${BASE_URL}/${file}" --progress-bar
+                echo "  [done] ${file}"
+            fi
+        done
+    fi
 fi
 
-go test -v -timeout 1200s .
+# Run Go tests
+go test -v -timeout 600s .
 
 echo "All qwen3-tts-cpp tests passed."

backend/go/sherpa-onnx/backend.go

@@ -62,7 +62,7 @@ var (
 	shimVadConfigSetDebug                    func(uintptr, int32)
 	shimCreateVad                            func(uintptr, float32) uintptr
 
-	// TTS (offline, VITS/Piper and Kokoro) config
+	// TTS (offline, VITS) config
 	shimTtsConfigNew                  func() uintptr
 	shimTtsConfigFree                 func(uintptr)
 	shimTtsConfigSetVitsModel         func(uintptr, string)
@@ -76,14 +76,6 @@ var (
 	shimTtsConfigSetDebug             func(uintptr, int32)
 	shimTtsConfigSetProvider          func(uintptr, string)
 	shimTtsConfigSetMaxNumSentences   func(uintptr, int32)
-	shimTtsConfigSetKokoroModel       func(uintptr, string)
-	shimTtsConfigSetKokoroVoices      func(uintptr, string)
-	shimTtsConfigSetKokoroTokens      func(uintptr, string)
-	shimTtsConfigSetKokoroDataDir     func(uintptr, string)
-	shimTtsConfigSetKokoroDictDir     func(uintptr, string)
-	shimTtsConfigSetKokoroLexicon     func(uintptr, string)
-	shimTtsConfigSetKokoroLang        func(uintptr, string)
-	shimTtsConfigSetKokoroLengthScale func(uintptr, float32)
 	shimCreateOfflineTts              func(uintptr) uintptr
 
 	// Offline recognizer config
@@ -109,37 +101,37 @@ var (
 	shimCreateOfflineRecognizer                  func(uintptr) uintptr
 
 	// Online recognizer config
-	shimOnlineRecogConfigNew                        func() uintptr
-	shimOnlineRecogConfigFree                       func(uintptr)
-	shimOnlineRecogConfigSetTransducerEncoder       func(uintptr, string)
-	shimOnlineRecogConfigSetTransducerDecoder       func(uintptr, string)
-	shimOnlineRecogConfigSetTransducerJoiner        func(uintptr, string)
-	shimOnlineRecogConfigSetTokens                  func(uintptr, string)
-	shimOnlineRecogConfigSetNumThreads              func(uintptr, int32)
-	shimOnlineRecogConfigSetDebug                   func(uintptr, int32)
-	shimOnlineRecogConfigSetProvider                func(uintptr, string)
-	shimOnlineRecogConfigSetFeatSampleRate          func(uintptr, int32)
-	shimOnlineRecogConfigSetFeatFeatureDim          func(uintptr, int32)
-	shimOnlineRecogConfigSetDecodingMethod          func(uintptr, string)
-	shimOnlineRecogConfigSetEnableEndpoint          func(uintptr, int32)
+	shimOnlineRecogConfigNew                      func() uintptr
+	shimOnlineRecogConfigFree                     func(uintptr)
+	shimOnlineRecogConfigSetTransducerEncoder     func(uintptr, string)
+	shimOnlineRecogConfigSetTransducerDecoder     func(uintptr, string)
+	shimOnlineRecogConfigSetTransducerJoiner      func(uintptr, string)
+	shimOnlineRecogConfigSetTokens                func(uintptr, string)
+	shimOnlineRecogConfigSetNumThreads            func(uintptr, int32)
+	shimOnlineRecogConfigSetDebug                 func(uintptr, int32)
+	shimOnlineRecogConfigSetProvider              func(uintptr, string)
+	shimOnlineRecogConfigSetFeatSampleRate        func(uintptr, int32)
+	shimOnlineRecogConfigSetFeatFeatureDim        func(uintptr, int32)
+	shimOnlineRecogConfigSetDecodingMethod        func(uintptr, string)
+	shimOnlineRecogConfigSetEnableEndpoint        func(uintptr, int32)
 	shimOnlineRecogConfigSetRule1MinTrailingSilence func(uintptr, float32)
 	shimOnlineRecogConfigSetRule2MinTrailingSilence func(uintptr, float32)
 	shimOnlineRecogConfigSetRule3MinUtteranceLength func(uintptr, float32)
-	shimCreateOnlineRecognizer                      func(uintptr) uintptr
+	shimCreateOnlineRecognizer                    func(uintptr) uintptr
 
 	// Result accessors. Pointer returns use unsafe.Pointer so Go's
 	// vet checker doesn't flag them — the returned memory is C-owned,
 	// not subject to Go GC motion.
-	shimWaveSampleRate           func(uintptr) int32
-	shimWaveNumSamples           func(uintptr) int32
-	shimWaveSamples              func(uintptr) unsafe.Pointer
-	shimOfflineResultText        func(uintptr) unsafe.Pointer
-	shimOnlineResultText         func(uintptr) unsafe.Pointer
-	shimGeneratedAudioSampleRate func(uintptr) int32
-	shimGeneratedAudioN          func(uintptr) int32
-	shimGeneratedAudioSamples    func(uintptr) unsafe.Pointer
-	shimSpeechSegmentStart       func(uintptr) int32
-	shimSpeechSegmentN           func(uintptr) int32
+	shimWaveSampleRate            func(uintptr) int32
+	shimWaveNumSamples            func(uintptr) int32
+	shimWaveSamples               func(uintptr) unsafe.Pointer
+	shimOfflineResultText         func(uintptr) unsafe.Pointer
+	shimOnlineResultText          func(uintptr) unsafe.Pointer
+	shimGeneratedAudioSampleRate  func(uintptr) int32
+	shimGeneratedAudioN           func(uintptr) int32
+	shimGeneratedAudioSamples     func(uintptr) unsafe.Pointer
+	shimSpeechSegmentStart        func(uintptr) int32
+	shimSpeechSegmentN            func(uintptr) int32
 
 	// TTS streaming callback trampoline
 	shimTtsGenerateWithCallback func(tts uintptr, text string, sid int32, speed float32, cb uintptr, ud uintptr) uintptr
@@ -169,13 +161,13 @@ var (
 // pointer returned by the shim or `unsafe.Pointer(&slice[0])` from Go.
 var (
 	// VAD
-	sherpaVadAcceptWaveform    func(vad uintptr, samples unsafe.Pointer, n int32)
-	sherpaVadReset             func(vad uintptr)
-	sherpaVadFlush             func(vad uintptr)
-	sherpaVadEmpty             func(vad uintptr) int32
-	sherpaVadFront             func(vad uintptr) uintptr
-	sherpaVadPop               func(vad uintptr)
-	sherpaDestroySpeechSegment func(seg uintptr)
+	sherpaVadAcceptWaveform       func(vad uintptr, samples unsafe.Pointer, n int32)
+	sherpaVadReset                func(vad uintptr)
+	sherpaVadFlush                func(vad uintptr)
+	sherpaVadEmpty                func(vad uintptr) int32
+	sherpaVadFront                func(vad uintptr) uintptr
+	sherpaVadPop                  func(vad uintptr)
+	sherpaDestroySpeechSegment    func(seg uintptr)
 
 	// Wave IO
 	sherpaReadWave  func(filename string) uintptr
@@ -183,11 +175,11 @@ var (
 	sherpaWriteWave func(samples unsafe.Pointer, n int32, sampleRate int32, filename string) int32
 
 	// Offline ASR
-	sherpaCreateOfflineStream            func(rec uintptr) uintptr
-	sherpaDestroyOfflineStream           func(stream uintptr)
-	sherpaAcceptWaveformOffline          func(stream uintptr, sr int32, samples unsafe.Pointer, n int32)
-	sherpaDecodeOfflineStream            func(rec uintptr, stream uintptr)
-	sherpaGetOfflineStreamResult         func(stream uintptr) uintptr
+	sherpaCreateOfflineStream           func(rec uintptr) uintptr
+	sherpaDestroyOfflineStream          func(stream uintptr)
+	sherpaAcceptWaveformOffline         func(stream uintptr, sr int32, samples unsafe.Pointer, n int32)
+	sherpaDecodeOfflineStream           func(rec uintptr, stream uintptr)
+	sherpaGetOfflineStreamResult        func(stream uintptr) uintptr
 	sherpaDestroyOfflineRecognizerResult func(result uintptr)
 
 	// Online ASR
@@ -203,21 +195,21 @@ var (
 	sherpaOnlineStreamInputFinished     func(stream uintptr)
 
 	// TTS
-	sherpaOfflineTtsGenerate              func(tts uintptr, text string, sid int32, speed float32) uintptr
+	sherpaOfflineTtsGenerate             func(tts uintptr, text string, sid int32, speed float32) uintptr
 	sherpaDestroyOfflineTtsGeneratedAudio func(audio uintptr)
-	sherpaOfflineTtsSampleRate            func(tts uintptr) int32
+	sherpaOfflineTtsSampleRate           func(tts uintptr) int32
 
 	// Offline speaker diarization. Result handle owns the segment-array
 	// pointer returned by ResultSortByStartTime; destroy the segment
 	// array first, then the result, then (at backend Free()) the diarizer.
-	sherpaDestroyOfflineSpeakerDiarization               func(sd uintptr)
-	sherpaOfflineSpeakerDiarizationGetSampleRate         func(sd uintptr) int32
-	sherpaOfflineSpeakerDiarizationProcess               func(sd uintptr, samples unsafe.Pointer, n int32) uintptr
-	sherpaOfflineSpeakerDiarizationResultGetNumSegments  func(result uintptr) int32
-	sherpaOfflineSpeakerDiarizationResultGetNumSpeakers  func(result uintptr) int32
-	sherpaOfflineSpeakerDiarizationResultSortByStartTime func(result uintptr) uintptr
-	sherpaOfflineSpeakerDiarizationDestroySegment        func(segs uintptr)
-	sherpaDestroyOfflineSpeakerDiarizationResult         func(result uintptr)
+	sherpaDestroyOfflineSpeakerDiarization                 func(sd uintptr)
+	sherpaOfflineSpeakerDiarizationGetSampleRate           func(sd uintptr) int32
+	sherpaOfflineSpeakerDiarizationProcess                 func(sd uintptr, samples unsafe.Pointer, n int32) uintptr
+	sherpaOfflineSpeakerDiarizationResultGetNumSegments    func(result uintptr) int32
+	sherpaOfflineSpeakerDiarizationResultGetNumSpeakers    func(result uintptr) int32
+	sherpaOfflineSpeakerDiarizationResultSortByStartTime   func(result uintptr) uintptr
+	sherpaOfflineSpeakerDiarizationDestroySegment          func(segs uintptr)
+	sherpaDestroyOfflineSpeakerDiarizationResult           func(result uintptr)
 )
 
 var (
@@ -286,14 +278,6 @@ func loadSherpaLibsOnce() error {
 		{&shimTtsConfigSetDebug, "sherpa_shim_tts_config_set_debug"},
 		{&shimTtsConfigSetProvider, "sherpa_shim_tts_config_set_provider"},
 		{&shimTtsConfigSetMaxNumSentences, "sherpa_shim_tts_config_set_max_num_sentences"},
-		{&shimTtsConfigSetKokoroModel, "sherpa_shim_tts_config_set_kokoro_model"},
-		{&shimTtsConfigSetKokoroVoices, "sherpa_shim_tts_config_set_kokoro_voices"},
-		{&shimTtsConfigSetKokoroTokens, "sherpa_shim_tts_config_set_kokoro_tokens"},
-		{&shimTtsConfigSetKokoroDataDir, "sherpa_shim_tts_config_set_kokoro_data_dir"},
-		{&shimTtsConfigSetKokoroDictDir, "sherpa_shim_tts_config_set_kokoro_dict_dir"},
-		{&shimTtsConfigSetKokoroLexicon, "sherpa_shim_tts_config_set_kokoro_lexicon"},
-		{&shimTtsConfigSetKokoroLang, "sherpa_shim_tts_config_set_kokoro_lang"},
-		{&shimTtsConfigSetKokoroLengthScale, "sherpa_shim_tts_config_set_kokoro_length_scale"},
 		{&shimCreateOfflineTts, "sherpa_shim_create_offline_tts"},
 
 		{&shimOfflineRecogConfigNew, "sherpa_shim_offline_recog_config_new"},
@@ -704,14 +688,21 @@ func (s *SherpaBackend) loadTTS(opts *pb.ModelOptions) error {
 	cfg := shimTtsConfigNew()
 	defer shimTtsConfigFree(cfg)
 
-	// Kokoro models ship a voices style file alongside the ONNX, whereas
-	// VITS/Piper voices do not. That presence is what tells the two model
-	// families apart, since both arrive as a plain *.onnx in modelDir.
-	if isKokoroModel(modelDir) {
-		s.configureKokoroTTS(cfg, opts, modelFile, modelDir)
-	} else {
-		s.configureVitsTTS(cfg, opts, modelFile, modelDir)
+	shimTtsConfigSetVitsModel(cfg, modelFile)
+
+	if tokensPath := filepath.Join(modelDir, "tokens.txt"); fileExists(tokensPath) {
+		shimTtsConfigSetVitsTokens(cfg, tokensPath)
 	}
+	if lexiconPath := filepath.Join(modelDir, "lexicon.txt"); fileExists(lexiconPath) {
+		shimTtsConfigSetVitsLexicon(cfg, lexiconPath)
+	}
+	if dataDir := filepath.Join(modelDir, "espeak-ng-data"); dirExists(dataDir) {
+		shimTtsConfigSetVitsDataDir(cfg, dataDir)
+	}
+
+	shimTtsConfigSetVitsNoiseScale(cfg, findOptionFloat(opts, optionTtsNoiseScale, 0.667))
+	shimTtsConfigSetVitsNoiseScaleW(cfg, findOptionFloat(opts, optionTtsNoiseScaleW, 0.8))
+	shimTtsConfigSetVitsLengthScale(cfg, findOptionFloat(opts, optionTtsLengthScale, 1.0))
 
 	threads := int32(1)
 	if opts.Threads != 0 {
@@ -732,80 +723,6 @@ func (s *SherpaBackend) loadTTS(opts *pb.ModelOptions) error {
 	return nil
 }
 
-// kokoroVoicesFile is the speaker-style bank that ships with Kokoro models and
-// is absent from VITS/Piper voices; its presence is how loadTTS tells them apart.
-const kokoroVoicesFile = "voices.bin"
-
-// isKokoroModel reports whether modelDir holds a Kokoro model (a voices file
-// next to the ONNX) rather than a VITS/Piper single-speaker model.
-func isKokoroModel(modelDir string) bool {
-	return fileExists(filepath.Join(modelDir, kokoroVoicesFile))
-}
-
-// configureVitsTTS wires a VITS/Piper single-speaker model into cfg: the ONNX
-// plus the optional tokens, lexicon and espeak-ng-data found beside it.
-func (s *SherpaBackend) configureVitsTTS(cfg uintptr, opts *pb.ModelOptions, modelFile, modelDir string) {
-	shimTtsConfigSetVitsModel(cfg, modelFile)
-
-	if tokensPath := filepath.Join(modelDir, "tokens.txt"); fileExists(tokensPath) {
-		shimTtsConfigSetVitsTokens(cfg, tokensPath)
-	}
-	if lexiconPath := filepath.Join(modelDir, "lexicon.txt"); fileExists(lexiconPath) {
-		shimTtsConfigSetVitsLexicon(cfg, lexiconPath)
-	}
-	if dataDir := filepath.Join(modelDir, "espeak-ng-data"); dirExists(dataDir) {
-		shimTtsConfigSetVitsDataDir(cfg, dataDir)
-	}
-
-	shimTtsConfigSetVitsNoiseScale(cfg, findOptionFloat(opts, optionTtsNoiseScale, 0.667))
-	shimTtsConfigSetVitsNoiseScaleW(cfg, findOptionFloat(opts, optionTtsNoiseScaleW, 0.8))
-	shimTtsConfigSetVitsLengthScale(cfg, findOptionFloat(opts, optionTtsLengthScale, 1.0))
-}
-
-// configureKokoroTTS wires a Kokoro model into cfg: the ONNX, its voices bank,
-// tokens, and the optional espeak-ng-data / jieba dict / lexicon assets the
-// multi-lingual packs ship. A language hint comes from the `language=` option.
-func (s *SherpaBackend) configureKokoroTTS(cfg uintptr, opts *pb.ModelOptions, modelFile, modelDir string) {
-	shimTtsConfigSetKokoroModel(cfg, modelFile)
-	shimTtsConfigSetKokoroVoices(cfg, filepath.Join(modelDir, kokoroVoicesFile))
-
-	if tokensPath := filepath.Join(modelDir, "tokens.txt"); fileExists(tokensPath) {
-		shimTtsConfigSetKokoroTokens(cfg, tokensPath)
-	}
-	if dataDir := filepath.Join(modelDir, "espeak-ng-data"); dirExists(dataDir) {
-		shimTtsConfigSetKokoroDataDir(cfg, dataDir)
-	}
-	if dictDir := filepath.Join(modelDir, "dict"); dirExists(dictDir) {
-		shimTtsConfigSetKokoroDictDir(cfg, dictDir)
-	}
-
-	// Multi-lingual Kokoro ships per-language lexicons; the C API takes them as
-	// a single comma-separated list. US and GB English overlap almost entirely,
-	// so pass only one (US preferred) to avoid tens of thousands of "duplicated
-	// word" warnings at load; non-English lexicons (e.g. zh) are additive.
-	var lexicons []string
-	addLexicon := func(name string) {
-		if p := filepath.Join(modelDir, name); fileExists(p) {
-			lexicons = append(lexicons, p)
-		}
-	}
-	if fileExists(filepath.Join(modelDir, "lexicon-us-en.txt")) {
-		addLexicon("lexicon-us-en.txt")
-	} else {
-		addLexicon("lexicon-gb-en.txt")
-	}
-	addLexicon("lexicon-zh.txt")
-	addLexicon("lexicon.txt")
-	if len(lexicons) > 0 {
-		shimTtsConfigSetKokoroLexicon(cfg, strings.Join(lexicons, ","))
-	}
-
-	if lang := findOptionValue(opts, optionLanguage, ""); lang != "" {
-		shimTtsConfigSetKokoroLang(cfg, lang)
-	}
-	shimTtsConfigSetKokoroLengthScale(cfg, findOptionFloat(opts, optionTtsLengthScale, 1.0))
-}
-
 func fileExists(p string) bool {
 	info, err := os.Stat(p)
 	return err == nil && !info.IsDir()
@@ -1335,7 +1252,7 @@ type ttsStreamState struct {
 var (
 	ttsStates      sync.Map // uint64 → *ttsStreamState
 	ttsNextID      atomic.Uint64
-	ttsCallbackPtr uintptr // purego.NewCallback return; registered in loadSherpaLibs
+	ttsCallbackPtr uintptr  // purego.NewCallback return; registered in loadSherpaLibs
 )
 
 // ttsStreamCallback is invoked by sherpa-onnx for each PCM chunk VITS

backend/go/sherpa-onnx/backend_test.go

@@ -124,20 +124,6 @@ var _ = Describe("Sherpa-ONNX", func() {
 			Entry("empty", "", false),
 			Entry("other", "other", false),
 		)
-
-		It("isKokoroModel detects a voices file beside the ONNX", func() {
-			dir, err := os.MkdirTemp("", "sherpa-kokoro-*")
-			Expect(err).NotTo(HaveOccurred())
-			defer func() { _ = os.RemoveAll(dir) }()
-
-			// A bare VITS/Piper directory (ONNX only) is not Kokoro.
-			Expect(os.WriteFile(filepath.Join(dir, "model.onnx"), []byte("x"), 0o600)).To(Succeed())
-			Expect(isKokoroModel(dir)).To(BeFalse())
-
-			// Adding the Kokoro voices bank flips detection on.
-			Expect(os.WriteFile(filepath.Join(dir, kokoroVoicesFile), []byte("x"), 0o600)).To(Succeed())
-			Expect(isKokoroModel(dir)).To(BeTrue())
-		})
 	})
 
 	Context("option parsing", func() {

backend/go/sherpa-onnx/csrc/shim.c

@@ -79,13 +79,6 @@ void sherpa_shim_tts_config_free(void *h) {
     free((char *)c->model.vits.tokens);
     free((char *)c->model.vits.lexicon);
     free((char *)c->model.vits.data_dir);
-    free((char *)c->model.kokoro.model);
-    free((char *)c->model.kokoro.voices);
-    free((char *)c->model.kokoro.tokens);
-    free((char *)c->model.kokoro.data_dir);
-    free((char *)c->model.kokoro.dict_dir);
-    free((char *)c->model.kokoro.lexicon);
-    free((char *)c->model.kokoro.lang);
     free((char *)c->model.provider);
     free(c);
 }
@@ -124,34 +117,6 @@ void sherpa_shim_tts_config_set_max_num_sentences(void *h, int32_t v) {
     ((SherpaOnnxOfflineTtsConfig *)h)->max_num_sentences = v;
 }
 
-// Kokoro multi-speaker / multi-lingual TTS. Distinct ONNX + a voices style
-// file (voices.bin) instead of VITS' single-speaker graph; espeak-ng-data,
-// lexicon and a language hint are optional refinements.
-void sherpa_shim_tts_config_set_kokoro_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.model, v);
-}
-void sherpa_shim_tts_config_set_kokoro_voices(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.voices, v);
-}
-void sherpa_shim_tts_config_set_kokoro_tokens(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.tokens, v);
-}
-void sherpa_shim_tts_config_set_kokoro_data_dir(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.data_dir, v);
-}
-void sherpa_shim_tts_config_set_kokoro_dict_dir(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.dict_dir, v);
-}
-void sherpa_shim_tts_config_set_kokoro_lexicon(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.lexicon, v);
-}
-void sherpa_shim_tts_config_set_kokoro_lang(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.lang, v);
-}
-void sherpa_shim_tts_config_set_kokoro_length_scale(void *h, float v) {
-    ((SherpaOnnxOfflineTtsConfig *)h)->model.kokoro.length_scale = v;
-}
-
 void *sherpa_shim_create_offline_tts(void *h) {
     return (void *)SherpaOnnxCreateOfflineTts(
         (const SherpaOnnxOfflineTtsConfig *)h);

backend/go/sherpa-onnx/csrc/shim.h

@@ -37,7 +37,7 @@ void  sherpa_shim_vad_config_set_provider(void *cfg, const char *v);
 void  sherpa_shim_vad_config_set_debug(void *cfg, int32_t v);
 void *sherpa_shim_create_vad(void *cfg, float buffer_size_seconds);
 
-// --- Offline TTS config (VITS/Piper and Kokoro model families) ---
+// --- Offline TTS config (VITS path — the only TTS family the backend uses) ---
 void *sherpa_shim_tts_config_new(void);
 void  sherpa_shim_tts_config_free(void *cfg);
 void  sherpa_shim_tts_config_set_vits_model(void *cfg, const char *v);
@@ -51,14 +51,6 @@ void  sherpa_shim_tts_config_set_num_threads(void *cfg, int32_t v);
 void  sherpa_shim_tts_config_set_debug(void *cfg, int32_t v);
 void  sherpa_shim_tts_config_set_provider(void *cfg, const char *v);
 void  sherpa_shim_tts_config_set_max_num_sentences(void *cfg, int32_t v);
-void  sherpa_shim_tts_config_set_kokoro_model(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_kokoro_voices(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_kokoro_tokens(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_kokoro_data_dir(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_kokoro_dict_dir(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_kokoro_lexicon(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_kokoro_lang(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_kokoro_length_scale(void *cfg, float v);
 void *sherpa_shim_create_offline_tts(void *cfg);
 
 // --- Offline recognizer config (Whisper / Paraformer / SenseVoice / Omnilingual) ---

backend/go/vibevoice-cpp/CMakeLists.txt

@@ -26,16 +26,8 @@ add_library(govibevoicecpp MODULE cpp/govibevoicecpp.cpp)
 # vv_capi_* symbols (purego dlopens them by name, nothing in our
 # translation unit references them). Force the static archive's
 # entire contents into the MODULE so dlsym finds vv_capi_load etc.
-#
-# Link the `vibevoice` TARGET (not a bare archive path) so CMake builds
-# libvibevoice.a first and tracks the dependency: the upstream project is added
-# with EXCLUDE_FROM_ALL, so without a target-level link there is no rule to
-# build it. Passing only $<TARGET_FILE:vibevoice> as a path on Apple left the
-# build with "No rule to make target 'vibevoice/libvibevoice.a'" (issue #10267).
-# force_load is then applied as a separate link option.
 if(APPLE)
-    target_link_libraries(govibevoicecpp PRIVATE vibevoice)
-    target_link_options(govibevoicecpp PRIVATE "-Wl,-force_load,$<TARGET_FILE:vibevoice>")
+    target_link_libraries(govibevoicecpp PRIVATE -Wl,-force_load $<TARGET_FILE:vibevoice>)
 elseif(MSVC)
     target_link_libraries(govibevoicecpp PRIVATE vibevoice)
     set_property(TARGET govibevoicecpp APPEND PROPERTY LINK_FLAGS "/WHOLEARCHIVE:vibevoice")

backend/go/vibevoice-cpp/Makefile

@@ -94,30 +94,26 @@ purge:
 # Build all variants (Linux only)
 ifeq ($(UNAME_S),Linux)
 libgovibevoicecpp-avx.so: sources/vibevoice.cpp
-	$(MAKE) purge
 	$(info ${GREEN}I vibevoice-cpp build info:avx${RESET})
 	SO_TARGET=libgovibevoicecpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgovibevoicecpp-custom
-	rm -rfv build*
+	rm -rf build-libgovibevoicecpp-avx.so
 
 libgovibevoicecpp-avx2.so: sources/vibevoice.cpp
-	$(MAKE) purge
 	$(info ${GREEN}I vibevoice-cpp build info:avx2${RESET})
 	SO_TARGET=libgovibevoicecpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgovibevoicecpp-custom
-	rm -rfv build*
+	rm -rf build-libgovibevoicecpp-avx2.so
 
 libgovibevoicecpp-avx512.so: sources/vibevoice.cpp
-	$(MAKE) purge
 	$(info ${GREEN}I vibevoice-cpp build info:avx512${RESET})
 	SO_TARGET=libgovibevoicecpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgovibevoicecpp-custom
-	rm -rfv build*
+	rm -rf build-libgovibevoicecpp-avx512.so
 endif
 
 # Build fallback variant (all platforms)
 libgovibevoicecpp-fallback.so: sources/vibevoice.cpp
-	$(MAKE) purge
 	$(info ${GREEN}I vibevoice-cpp build info:fallback${RESET})
 	SO_TARGET=libgovibevoicecpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgovibevoicecpp-custom
-	rm -rfv build*
+	rm -rf build-libgovibevoicecpp-fallback.so
 
 libgovibevoicecpp-custom: CMakeLists.txt cpp/govibevoicecpp.cpp cpp/govibevoicecpp.h
 	mkdir -p build-$(SO_TARGET) && \

backend/index.yaml

@@ -95,6 +95,29 @@
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-ds4"
     metal: "metal-ds4"
     metal-darwin-arm64: "metal-ds4"
+- &dllm
+  name: "dllm"
+  alias: "dllm"
+  license: mit
+  description: |
+    mudler/dllm.cpp - DiffusionGemma block-diffusion LLM inference engine
+    (C++/ggml, GGUF weights). Decodes whole token canvases per diffusion
+    round instead of autoregressive sampling. Runs on CPU and NVIDIA CUDA 13
+    (including Jetson/GB10 L4T targets).
+  urls:
+    - https://github.com/mudler/dllm.cpp
+  tags:
+    - text-to-text
+    - LLM
+    - gguf
+    - diffusion
+    - CPU
+    - CUDA
+  capabilities:
+    default: "cpu-dllm"
+    nvidia: "cuda13-dllm"
+    nvidia-cuda-13: "cuda13-dllm"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-dllm"
 - &whispercpp
   name: "whisper"
   alias: "whisper"
@@ -337,127 +360,6 @@
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-rfdetr-cpp"
     intel: "intel-sycl-f32-rfdetr-cpp"
     vulkan: "vulkan-rfdetr-cpp"
-- &locateanything
-  name: "locate-anything"
-  alias: "locate-anything"
-  license: apache-2.0
-  description: |
-    Open-vocabulary object detection and visual grounding (NVIDIA
-    LocateAnything-3B) in C/C++ using GGML. Loads pre-built GGUF weights
-    and, given an image and a free-form text prompt, returns bounding
-    boxes, class labels, and confidence scores for the referred objects.
-  urls:
-    - https://github.com/mudler/locate-anything.cpp
-    - https://huggingface.co/nvidia/LocateAnything-3B
-  tags:
-    - object-detection
-    - visual-grounding
-    - open-vocabulary
-    - locate-anything
-    - gpu
-    - cpu
-  capabilities:
-    default: "cpu-locate-anything-cpp"
-    nvidia: "cuda12-locate-anything-cpp"
-    nvidia-cuda-12: "cuda12-locate-anything-cpp"
-    nvidia-cuda-13: "cuda13-locate-anything-cpp"
-    nvidia-l4t: "nvidia-l4t-arm64-locate-anything-cpp"
-    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-locate-anything-cpp"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-locate-anything-cpp"
-    intel: "intel-sycl-f32-locate-anything-cpp"
-    vulkan: "vulkan-locate-anything-cpp"
-- !!merge <<: *locateanything
-  name: "locate-anything-development"
-  capabilities:
-    default: "cpu-locate-anything-cpp-development"
-    nvidia: "cuda12-locate-anything-cpp-development"
-    nvidia-cuda-12: "cuda12-locate-anything-cpp-development"
-    nvidia-cuda-13: "cuda13-locate-anything-cpp-development"
-    nvidia-l4t: "nvidia-l4t-arm64-locate-anything-cpp-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-locate-anything-cpp-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-locate-anything-cpp-development"
-    intel: "intel-sycl-f32-locate-anything-cpp-development"
-    vulkan: "vulkan-locate-anything-cpp-development"
-- !!merge <<: *locateanything
-  name: "cpu-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-cpu-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "cpu-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-cpu-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "cuda12-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-12-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "cuda12-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-12-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "cuda13-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-13-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "cuda13-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-13-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "nvidia-l4t-arm64-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-arm64-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "nvidia-l4t-arm64-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-arm64-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "cuda13-nvidia-l4t-arm64-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "cuda13-nvidia-l4t-arm64-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "intel-sycl-f32-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-intel-sycl-f32-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "intel-sycl-f32-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-intel-sycl-f32-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "intel-sycl-f16-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-intel-sycl-f16-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "intel-sycl-f16-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-intel-sycl-f16-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "vulkan-locate-anything-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-vulkan-locate-anything-cpp
-- !!merge <<: *locateanything
-  name: "vulkan-locate-anything-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-locate-anything-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-vulkan-locate-anything-cpp
 - &vllm
   name: "vllm"
   license: apache-2.0
@@ -547,9 +449,12 @@
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vllm-omni"
 - &mlx
   name: "mlx"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx"
   icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
   urls:
     - https://github.com/ml-explore/mlx-lm
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-mlx
   license: MIT
   description: |
       Run LLMs with MLX
@@ -568,9 +473,12 @@
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx"
 - &mlx-vlm
   name: "mlx-vlm"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-vlm"
   icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
   urls:
     - https://github.com/Blaizzy/mlx-vlm
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-mlx-vlm
   license: MIT
   description: |
       Run Vision-Language Models with MLX
@@ -591,9 +499,12 @@
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx-vlm"
 - &mlx-audio
   name: "mlx-audio"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-audio"
   icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
   urls:
     - https://github.com/Blaizzy/mlx-audio
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-mlx-audio
   license: MIT
   description: |
       Run Audio Models with MLX
@@ -614,9 +525,12 @@
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx-audio"
 - &mlx-distributed
   name: "mlx-distributed"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-distributed"
   icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
   urls:
     - https://github.com/ml-explore/mlx-lm
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-mlx-distributed
   license: MIT
   description: |
       Run distributed LLM inference with MLX across multiple Apple Silicon Macs
@@ -712,7 +626,7 @@
     default: "cpu-diffusers"
     nvidia-cuda-13: "cuda13-diffusers"
     nvidia-cuda-12: "cuda12-diffusers"
-    nvidia-l4t-cuda-12: "nvidia-l4t-diffusers"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-diffusers"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-diffusers"
 - &ace-step
   name: "ace-step"
@@ -768,17 +682,14 @@
 - &qwen3ttscpp
   name: "qwen3-tts-cpp"
   description: |
-    Qwen3-TTS C++ backend using GGML (qwentts.cpp). Native C++ text-to-speech
-    with streaming output, named speakers, voice design, and zero-shot voice
-    cloning. 24kHz mono, 11 languages with Mandarin dialects. 0.6B and 1.7B
-    models in Q8_0 / Q4_K_M.
+    Qwen3-TTS C++ backend using GGML. Native C++ text-to-speech with voice cloning support.
+    Generates 24kHz mono audio from text with optional reference audio for voice cloning via ECAPA-TDNN speaker embeddings.
   urls:
-    - https://github.com/ServeurpersoCom/qwentts.cpp
+    - https://github.com/predict-woo/qwen3-tts.cpp
   tags:
     - text-to-speech
     - tts
     - voice-cloning
-    - streaming
   alias: "qwen3-tts-cpp"
   capabilities:
     default: "cpu-qwen3-tts-cpp"
@@ -792,33 +703,6 @@
     nvidia-l4t: "nvidia-l4t-arm64-qwen3-tts-cpp"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-qwen3-tts-cpp"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-qwen3-tts-cpp"
-- &omnivoicecpp
-  name: "omnivoice-cpp"
-  description: |
-    OmniVoice C++ backend using GGML. Native text-to-speech with voice cloning
-    (reference audio + transcript) and voice design (attribute keywords: gender,
-    age, pitch, style, volume, emotion). 24kHz mono output, 646 languages.
-    Supports streaming synthesis.
-  urls:
-    - https://github.com/ServeurpersoCom/omnivoice.cpp
-  tags:
-    - text-to-speech
-    - tts
-    - voice-cloning
-    - voice-design
-  alias: "omnivoice-cpp"
-  capabilities:
-    default: "cpu-omnivoice-cpp"
-    nvidia: "cuda12-omnivoice-cpp"
-    nvidia-cuda-13: "cuda13-omnivoice-cpp"
-    nvidia-cuda-12: "cuda12-omnivoice-cpp"
-    intel: "intel-sycl-f16-omnivoice-cpp"
-    metal: "metal-omnivoice-cpp"
-    amd: "rocm-omnivoice-cpp"
-    vulkan: "vulkan-omnivoice-cpp"
-    nvidia-l4t: "nvidia-l4t-arm64-omnivoice-cpp"
-    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-omnivoice-cpp"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-omnivoice-cpp"
 - &vibevoicecpp
   name: "vibevoice-cpp"
   description: |
@@ -964,7 +848,7 @@
     metal: "metal-kokoro"
     nvidia-cuda-13: "cuda13-kokoro"
     nvidia-cuda-12: "cuda12-kokoro"
-    nvidia-l4t-cuda-12: "nvidia-l4t-kokoro"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-kokoro"
 - &kokoros
   icon: https://avatars.githubusercontent.com/u/166769057?v=4
   description: |
@@ -1007,6 +891,7 @@
     intel: "intel-coqui"
     amd: "rocm-coqui"
     metal: "metal-coqui"
+    nvidia-cuda-13: "cuda13-coqui"
     nvidia-cuda-12: "cuda12-coqui"
   icon: https://avatars.githubusercontent.com/u/1338804?s=200&v=4
 - &outetts
@@ -1256,27 +1141,27 @@
   icon: https://avatars.githubusercontent.com/u/151010778?s=200&v=4
 - &piper
   name: "piper"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-piper"
   icon: https://github.com/OHF-Voice/piper1-gpl/raw/main/etc/logo.png
   urls:
     - https://github.com/rhasspy/piper
     - https://github.com/mudler/go-piper
+  mirrors:
+    - localai/localai-backends:latest-piper
   license: MIT
   description: |
      A fast, local neural text to speech system
   tags:
     - text-to-speech
     - TTS
-  capabilities:
-    default: "cpu-piper"
-    metal: "metal-piper"
 - &opus
   name: "opus"
   alias: "opus"
-  capabilities:
-    default: "cpu-opus"
-    metal: "metal-opus"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-opus"
   urls:
     - https://opus-codec.org/
+  mirrors:
+    - localai/localai-backends:latest-cpu-opus
   license: BSD-3-Clause
   description: |
     Opus audio codec backend for encoding and decoding audio.
@@ -1286,19 +1171,15 @@
     - opus
     - WebRTC
     - realtime
-- !!merge <<: *opus
-  name: "opus-development"
-  capabilities:
-    default: "cpu-opus-development"
-    metal: "metal-opus-development"
+    - CPU
 - &silero-vad
   name: "silero-vad"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-silero-vad"
   icon: https://user-images.githubusercontent.com/12515440/89997349-b3523080-dc94-11ea-9906-ca2e8bc50535.png
   urls:
     - https://github.com/snakers4/silero-vad
-  capabilities:
-    default: "cpu-silero-vad"
-    metal: "metal-silero-vad"
+  mirrors:
+    - localai/localai-backends:latest-cpu-silero-vad
   description: |
     Silero VAD: pre-trained enterprise-grade Voice Activity Detector.
     Silero VAD is a voice activity detection model that can be used to detect whether a given audio contains speech or not.
@@ -1309,6 +1190,9 @@
     - CPU
 - &local-store
   name: "local-store"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-local-store"
+  mirrors:
+    - localai/localai-backends:latest-cpu-local-store
   urls:
     - https://github.com/mudler/LocalAI
   description: |
@@ -1319,11 +1203,11 @@
     - open-source
     - CPU
   license: MIT
-  capabilities:
-    default: "cpu-local-store"
-    metal: "metal-local-store"
 - &kitten-tts
   name: "kitten-tts"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-kitten-tts"
+  mirrors:
+    - localai/localai-backends:latest-kitten-tts
   urls:
     - https://github.com/KittenML/KittenTTS
   description: |
@@ -1332,9 +1216,6 @@
     - text-to-speech
     - TTS
   license: apache-2.0
-  capabilities:
-    default: "cpu-kitten-tts"
-    metal: "metal-kitten-tts"
 - &neutts
   name: "neutts"
   urls:
@@ -1367,7 +1248,6 @@
     default: "cpu-sherpa-onnx"
     nvidia: "cuda12-sherpa-onnx"
     nvidia-cuda-12: "cuda12-sherpa-onnx"
-    metal: "metal-sherpa-onnx"
 - !!merge <<: *neutts
   name: "neutts-development"
   capabilities:
@@ -1415,6 +1295,13 @@
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-ds4-development"
     metal: "metal-ds4-development"
     metal-darwin-arm64: "metal-ds4-development"
+- !!merge <<: *dllm
+  name: "dllm-development"
+  capabilities:
+    default: "cpu-dllm-development"
+    nvidia: "cuda13-dllm-development"
+    nvidia-cuda-13: "cuda13-dllm-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-dllm-development"
 - !!merge <<: *stablediffusionggml
   name: "stablediffusion-ggml-development"
   capabilities:
@@ -1460,89 +1347,25 @@
   mirrors:
     - localai/localai-backends:master-gpu-rocm-hipblas-neutts
 - !!merge <<: *mlx
-  name: "metal-mlx"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-mlx
-- !!merge <<: *mlx
-  name: "metal-mlx-development"
+  name: "mlx-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-mlx
-- !!merge <<: *mlx
-  name: "mlx-development"
-  capabilities:
-    default: "cpu-mlx-development"
-    nvidia: "cuda12-mlx-development"
-    metal: "metal-mlx-development"
-    nvidia-cuda-12: "cuda12-mlx-development"
-    nvidia-cuda-13: "cuda13-mlx-development"
-    nvidia-l4t: "nvidia-l4t-mlx-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-mlx-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx-development"
 - !!merge <<: *mlx-vlm
-  name: "metal-mlx-vlm"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-vlm"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-mlx-vlm
-- !!merge <<: *mlx-vlm
-  name: "metal-mlx-vlm-development"
+  name: "mlx-vlm-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx-vlm"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-mlx-vlm
-- !!merge <<: *mlx-vlm
-  name: "mlx-vlm-development"
-  capabilities:
-    default: "cpu-mlx-vlm-development"
-    nvidia: "cuda12-mlx-vlm-development"
-    metal: "metal-mlx-vlm-development"
-    nvidia-cuda-12: "cuda12-mlx-vlm-development"
-    nvidia-cuda-13: "cuda13-mlx-vlm-development"
-    nvidia-l4t: "nvidia-l4t-mlx-vlm-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-mlx-vlm-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx-vlm-development"
 - !!merge <<: *mlx-audio
-  name: "metal-mlx-audio"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-audio"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-mlx-audio
-- !!merge <<: *mlx-audio
-  name: "metal-mlx-audio-development"
+  name: "mlx-audio-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx-audio"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-mlx-audio
-- !!merge <<: *mlx-audio
-  name: "mlx-audio-development"
-  capabilities:
-    default: "cpu-mlx-audio-development"
-    nvidia: "cuda12-mlx-audio-development"
-    metal: "metal-mlx-audio-development"
-    nvidia-cuda-12: "cuda12-mlx-audio-development"
-    nvidia-cuda-13: "cuda13-mlx-audio-development"
-    nvidia-l4t: "nvidia-l4t-mlx-audio-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-mlx-audio-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx-audio-development"
 - !!merge <<: *mlx-distributed
-  name: "metal-mlx-distributed"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-distributed"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-mlx-distributed
-- !!merge <<: *mlx-distributed
-  name: "metal-mlx-distributed-development"
+  name: "mlx-distributed-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx-distributed"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-mlx-distributed
-- !!merge <<: *mlx-distributed
-  name: "mlx-distributed-development"
-  capabilities:
-    default: "cpu-mlx-distributed-development"
-    nvidia: "cuda12-mlx-distributed-development"
-    metal: "metal-mlx-distributed-development"
-    nvidia-cuda-12: "cuda12-mlx-distributed-development"
-    nvidia-cuda-13: "cuda13-mlx-distributed-development"
-    nvidia-l4t: "nvidia-l4t-mlx-distributed-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-mlx-distributed-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx-distributed-development"
 ## mlx
 - !!merge <<: *mlx
   name: "cpu-mlx"
@@ -1748,20 +1571,10 @@
   mirrors:
     - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-mlx-distributed
 - !!merge <<: *kitten-tts
-  name: "cpu-kitten-tts"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-kitten-tts"
-  mirrors:
-    - localai/localai-backends:latest-kitten-tts
-- !!merge <<: *kitten-tts
-  name: "cpu-kitten-tts-development"
+  name: "kitten-tts-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-kitten-tts"
   mirrors:
     - localai/localai-backends:master-kitten-tts
-- !!merge <<: *kitten-tts
-  name: "kitten-tts-development"
-  capabilities:
-    default: "cpu-kitten-tts-development"
-    metal: "metal-kitten-tts-development"
 - !!merge <<: *kitten-tts
   name: "metal-kitten-tts"
   uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-kitten-tts"
@@ -1773,23 +1586,10 @@
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-kitten-tts
 - !!merge <<: *local-store
-  name: "cpu-local-store"
-  alias: "local-store"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-local-store"
-  mirrors:
-    - localai/localai-backends:latest-cpu-local-store
-- !!merge <<: *local-store
-  name: "cpu-local-store-development"
-  alias: "local-store"
+  name: "local-store-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-cpu-local-store"
   mirrors:
     - localai/localai-backends:master-cpu-local-store
-- !!merge <<: *local-store
-  name: "local-store-development"
-  alias: "local-store"
-  capabilities:
-    default: "cpu-local-store-development"
-    metal: "metal-local-store-development"
 - !!merge <<: *local-store
   name: "metal-local-store"
   uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-local-store"
@@ -1797,17 +1597,11 @@
     - localai/localai-backends:latest-metal-darwin-arm64-local-store
 - !!merge <<: *local-store
   name: "metal-local-store-development"
-  alias: "local-store"
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-local-store"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-local-store
 - !!merge <<: *opus
-  name: "cpu-opus"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-opus"
-  mirrors:
-    - localai/localai-backends:latest-cpu-opus
-- !!merge <<: *opus
-  name: "cpu-opus-development"
+  name: "opus-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-cpu-opus"
   mirrors:
     - localai/localai-backends:master-cpu-opus
@@ -1822,20 +1616,10 @@
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-opus
 - !!merge <<: *silero-vad
-  name: "cpu-silero-vad"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-silero-vad"
-  mirrors:
-    - localai/localai-backends:latest-cpu-silero-vad
-- !!merge <<: *silero-vad
-  name: "cpu-silero-vad-development"
+  name: "silero-vad-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-cpu-silero-vad"
   mirrors:
     - localai/localai-backends:master-cpu-silero-vad
-- !!merge <<: *silero-vad
-  name: "silero-vad-development"
-  capabilities:
-    default: "cpu-silero-vad-development"
-    metal: "metal-silero-vad-development"
 - !!merge <<: *silero-vad
   name: "metal-silero-vad"
   uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-silero-vad"
@@ -1847,20 +1631,10 @@
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-silero-vad
 - !!merge <<: *piper
-  name: "cpu-piper"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-piper"
-  mirrors:
-    - localai/localai-backends:latest-piper
-- !!merge <<: *piper
-  name: "cpu-piper-development"
+  name: "piper-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-piper"
   mirrors:
     - localai/localai-backends:master-piper
-- !!merge <<: *piper
-  name: "piper-development"
-  capabilities:
-    default: "cpu-piper-development"
-    metal: "metal-piper-development"
 - !!merge <<: *piper
   name: "metal-piper"
   uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-piper"
@@ -2115,6 +1889,37 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-ds4"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-ds4
+## dllm
+- !!merge <<: *dllm
+  name: "cpu-dllm"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-dllm"
+  mirrors:
+    - localai/localai-backends:latest-cpu-dllm
+- !!merge <<: *dllm
+  name: "cpu-dllm-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-dllm"
+  mirrors:
+    - localai/localai-backends:master-cpu-dllm
+- !!merge <<: *dllm
+  name: "cuda13-dllm"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-dllm"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-dllm
+- !!merge <<: *dllm
+  name: "cuda13-dllm-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-dllm"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-dllm
+- !!merge <<: *dllm
+  name: "cuda13-nvidia-l4t-arm64-dllm"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-dllm"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-dllm
+- !!merge <<: *dllm
+  name: "cuda13-nvidia-l4t-arm64-dllm-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-dllm"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-dllm
 ## whisper
 - !!merge <<: *whispercpp
   name: "whisper-development"
@@ -3503,121 +3308,6 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-qwen3-tts-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-qwen3-tts-cpp
-## omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "omnivoice-cpp-development"
-  capabilities:
-    default: "cpu-omnivoice-cpp-development"
-    nvidia: "cuda12-omnivoice-cpp-development"
-    nvidia-cuda-13: "cuda13-omnivoice-cpp-development"
-    nvidia-cuda-12: "cuda12-omnivoice-cpp-development"
-    intel: "intel-sycl-f16-omnivoice-cpp-development"
-    metal: "metal-omnivoice-cpp-development"
-    amd: "rocm-omnivoice-cpp-development"
-    vulkan: "vulkan-omnivoice-cpp-development"
-    nvidia-l4t: "nvidia-l4t-arm64-omnivoice-cpp-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-omnivoice-cpp-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-omnivoice-cpp-development"
-- !!merge <<: *omnivoicecpp
-  name: "nvidia-l4t-arm64-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-arm64-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "nvidia-l4t-arm64-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-arm64-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cuda13-nvidia-l4t-arm64-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cuda13-nvidia-l4t-arm64-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cpu-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-cpu-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "metal-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "metal-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-metal-darwin-arm64-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cpu-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-cpu-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cuda12-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-12-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "rocm-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-rocm-hipblas-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "intel-sycl-f32-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-intel-sycl-f32-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "intel-sycl-f16-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-intel-sycl-f16-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "vulkan-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-vulkan-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "vulkan-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-vulkan-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cuda12-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-12-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "rocm-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-rocm-hipblas-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "intel-sycl-f32-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-intel-sycl-f32-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "intel-sycl-f16-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-intel-sycl-f16-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cuda13-omnivoice-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-13-omnivoice-cpp
-- !!merge <<: *omnivoicecpp
-  name: "cuda13-omnivoice-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-omnivoice-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-13-omnivoice-cpp
 ## vibevoice-cpp
 - !!merge <<: *vibevoicecpp
   name: "nvidia-l4t-arm64-vibevoice-cpp"
@@ -4948,24 +4638,24 @@
     - localai/localai-backends:master-cpu-trl
 - !!merge <<: *trl
   name: "cuda12-trl"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-trl"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cublas-cuda12-trl"
   mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-12-trl
+    - localai/localai-backends:latest-cublas-cuda12-trl
 - !!merge <<: *trl
   name: "cuda12-trl-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-trl"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cublas-cuda12-trl"
   mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-12-trl
+    - localai/localai-backends:master-cublas-cuda12-trl
 - !!merge <<: *trl
   name: "cuda13-trl"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-trl"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cublas-cuda13-trl"
   mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-13-trl
+    - localai/localai-backends:latest-cublas-cuda13-trl
 - !!merge <<: *trl
   name: "cuda13-trl-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-trl"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cublas-cuda13-trl"
   mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-13-trl
+    - localai/localai-backends:master-cublas-cuda13-trl
 ## llama.cpp quantization backend
 - &llama-cpp-quantization
   name: "llama-cpp-quantization"
@@ -5056,14 +4746,12 @@
     default: "cpu-speaker-recognition"
     nvidia: "cuda12-speaker-recognition"
     nvidia-cuda-12: "cuda12-speaker-recognition"
-    metal: "metal-speaker-recognition"
 - !!merge <<: *speakerrecognition
   name: "speaker-recognition-development"
   capabilities:
     default: "cpu-speaker-recognition-development"
     nvidia: "cuda12-speaker-recognition-development"
     nvidia-cuda-12: "cuda12-speaker-recognition-development"
-    metal: "metal-speaker-recognition-development"
 - !!merge <<: *speakerrecognition
   name: "cpu-speaker-recognition"
   uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-speaker-recognition"
@@ -5084,16 +4772,6 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-speaker-recognition"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-12-speaker-recognition
-- !!merge <<: *speakerrecognition
-  name: "metal-speaker-recognition"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-speaker-recognition"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-speaker-recognition
-- !!merge <<: *speakerrecognition
-  name: "metal-speaker-recognition-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-speaker-recognition"
-  mirrors:
-    - localai/localai-backends:master-metal-darwin-arm64-speaker-recognition
 ## sherpa-onnx
 - !!merge <<: *sherpa-onnx
   name: "sherpa-onnx-development"
@@ -5101,7 +4779,6 @@
     default: "cpu-sherpa-onnx-development"
     nvidia: "cuda12-sherpa-onnx-development"
     nvidia-cuda-12: "cuda12-sherpa-onnx-development"
-    metal: "metal-sherpa-onnx-development"
 - !!merge <<: *sherpa-onnx
   name: "cpu-sherpa-onnx"
   uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-sherpa-onnx"
@@ -5122,13 +4799,3 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-sherpa-onnx"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-12-sherpa-onnx
-- !!merge <<: *sherpa-onnx
-  name: "metal-sherpa-onnx"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-sherpa-onnx"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-sherpa-onnx
-- !!merge <<: *sherpa-onnx
-  name: "metal-sherpa-onnx-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-sherpa-onnx"
-  mirrors:
-    - localai/localai-backends:master-metal-darwin-arm64-sherpa-onnx

backend/python/mlx/backend.py

@@ -407,24 +407,6 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         if not request.Prompt and request.UseTokenizerTemplate and request.Messages:
             messages = messages_to_dicts(request.Messages)
 
-            # The mlx-lm tokenizer only carries a text-LM chat template. A
-            # vision-language checkpoint (e.g. gemma-4 E4B) loaded here has no
-            # usable template, so apply_chat_template silently passes the raw
-            # text through and the model just echoes/loops (issue #10269).
-            # Warn loudly so the misroute is visible; such models belong on the
-            # mlx-vlm backend.
-            chat_template = getattr(self.tokenizer, "chat_template", None)
-            if not chat_template:
-                underlying = getattr(self.tokenizer, "_tokenizer", None)
-                chat_template = getattr(underlying, "chat_template", None)
-            if not chat_template:
-                print(
-                    "WARNING: this model has no chat template; output may be "
-                    "degenerate. Vision-language models (e.g. gemma-4 E4B) must "
-                    "use the 'mlx-vlm' backend instead of 'mlx'.",
-                    file=sys.stderr,
-                )
-
             kwargs = {"tokenize": False, "add_generation_prompt": True}
             if request.Tools:
                 try:

backend/python/neutts/requirements-cpu.txt

@@ -1,7 +1,6 @@
 --extra-index-url https://download.pytorch.org/whl/cpu
 accelerate
 torch==2.8.0
-torchaudio==2.8.0
 transformers==4.56.1
 librosa==0.11.0
 neucodec>=0.0.4

backend/python/neutts/requirements-cublas12.txt

@@ -3,7 +3,6 @@ neucodec>=0.0.4
 phonemizer==3.3.0
 soundfile==0.13.1
 torch==2.8.0
-torchaudio==2.8.0
 transformers==4.56.1
 resemble-perth==1.0.1
 accelerate

backend/python/speaker-recognition/requirements-mps.txt

@@ -1,5 +0,0 @@
-torch
-torchaudio
-speechbrain
-transformers
-onnxruntime

backend/python/vllm/backend.py

@@ -150,24 +150,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 d["reasoning_content"] = msg.reasoning_content
             if msg.tool_calls:
                 try:
-                    tool_calls = json.loads(msg.tool_calls)
+                    d["tool_calls"] = json.loads(msg.tool_calls)
                 except json.JSONDecodeError:
                     pass
-                else:
-                    # OpenAI wire format carries function.arguments as a
-                    # JSON-encoded string, but chat templates (e.g. Qwen3)
-                    # iterate over it as a mapping. vLLM's own OpenAI server
-                    # parses arguments before applying the template, so do
-                    # the same here.
-                    if isinstance(tool_calls, list):
-                        for tc in tool_calls:
-                            func = tc.get("function") if isinstance(tc, dict) else None
-                            if isinstance(func, dict) and isinstance(func.get("arguments"), str):
-                                try:
-                                    func["arguments"] = json.loads(func["arguments"])
-                                except json.JSONDecodeError:
-                                    pass
-                    d["tool_calls"] = tool_calls
             result.append(d)
         return result
 

backend/python/vllm/requirements-cublas13-after.txt

@@ -3,5 +3,5 @@
 # on a cu130 host. Pull the cu130-flavoured wheel from vLLM's per-tag index
 # instead — the cublas13 case in install.sh adds --index-strategy=unsafe-best-match
 # so uv consults this index alongside PyPI.
---extra-index-url https://wheels.vllm.ai/0.23.0/cu130
-vllm==0.23.0
+--extra-index-url https://wheels.vllm.ai/0.22.1/cu130
+vllm==0.22.1

core/application/distributed.go

@@ -161,21 +161,6 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 	}
 	xlog.Info("Node registry initialized")
 
-	// Seed declarative per-model scheduling config (LOCALAI_MODEL_SCHEDULING /
-	// LOCALAI_MODEL_SCHEDULING_CONFIG). Authoritative: overwrites matching models
-	// on every boot. Runs before the reconciler starts so the first tick already
-	// sees the desired state. Models not listed are left untouched.
-	if cfg.Distributed.ModelSchedulingJSON != "" || cfg.Distributed.ModelSchedulingConfigPath != "" {
-		schedConfigs, err := nodes.ParseSchedulingSeed(cfg.Distributed.ModelSchedulingJSON, cfg.Distributed.ModelSchedulingConfigPath)
-		if err != nil {
-			return nil, fmt.Errorf("parsing declarative model scheduling config: %w", err)
-		}
-		if err := registry.SeedModelScheduling(context.Background(), schedConfigs); err != nil {
-			return nil, fmt.Errorf("seeding declarative model scheduling config: %w", err)
-		}
-		xlog.Info("Applied declarative model scheduling config", "models", len(schedConfigs))
-	}
-
 	// Collect SmartRouter option values; the router itself is created after all
 	// dependencies (including FileStager and Unloader) are ready.
 	var routerAuthToken string

core/application/mitm.go

@@ -11,29 +11,6 @@ import (
 	"github.com/mudler/xlog"
 )
 
-// startMITMIfConfigured brings up the cloudproxy MITM listener when an
-// address is configured, treating any startup failure as non-fatal.
-//
-// The listener is opt-in middleware whose address is persisted in runtime
-// settings (/api/settings → runtime_settings.json) and replayed on every
-// boot. A bad value — e.g. a host the process can't bind, like a LAN IP
-// inside a container — must NOT abort the whole server: doing so crash-loops
-// with no way out, because the Settings UI used to correct the address can't
-// load if startup never completes. So on failure we log loudly and carry on;
-// the admin fixes the address via /api/settings, which calls RestartMITM.
-func startMITMIfConfigured(app *Application, options *config.ApplicationConfig) {
-	if options.MITMListen == "" {
-		return
-	}
-	if err := startMITMProxy(app, options); err != nil {
-		xlog.Error("mitm: cloudproxy listener failed to start — continuing without it",
-			"listen", options.MITMListen,
-			"error", err,
-			"hint", "fix the address via Settings (e.g. \":8082\" to bind all interfaces) and the listener will restart",
-		)
-	}
-}
-
 func startMITMProxy(app *Application, options *config.ApplicationConfig) error {
 	app.mitmMutex.Lock()
 	defer app.mitmMutex.Unlock()

core/application/mitm_test.go

@@ -1,58 +0,0 @@
-package application
-
-import (
-	"github.com/mudler/LocalAI/core/config"
-	"github.com/mudler/LocalAI/pkg/system"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-// minimal Application wired enough for startMITMProxy: an empty model
-// config loader (no host claims), CA written under a temp DataPath.
-func newMITMTestApp(dataPath string) (*Application, *config.ApplicationConfig) {
-	state, err := system.GetSystemState()
-	Expect(err).NotTo(HaveOccurred())
-	state.Model.ModelsPath = dataPath
-	opts := config.NewApplicationConfig(
-		config.WithSystemState(state),
-		config.WithDataPath(dataPath),
-	)
-	return newApplication(opts), opts
-}
-
-var _ = Describe("startMITMIfConfigured", func() {
-	It("does nothing when no listen address is configured", func() {
-		app, opts := newMITMTestApp(GinkgoT().TempDir())
-		opts.MITMListen = ""
-
-		Expect(func() { startMITMIfConfigured(app, opts) }).NotTo(Panic())
-		Expect(app.mitmServer.Load()).To(BeNil(), "no listener should be stored when disabled")
-	})
-
-	// Regression: a persisted-but-unbindable MITM address (e.g. a LAN host
-	// inside a container) must not abort startup. startMITMIfConfigured
-	// swallows the bind error so the rest of LocalAI still comes up and the
-	// admin can fix the address via the Settings UI.
-	It("logs and continues when the listen address cannot be bound", func() {
-		app, opts := newMITMTestApp(GinkgoT().TempDir())
-		// 192.0.2.1 is TEST-NET-1 (RFC 5737): guaranteed not assigned to any
-		// local interface, so bind fails deterministically without DNS.
-		opts.MITMListen = "192.0.2.1:8082"
-
-		Expect(func() { startMITMIfConfigured(app, opts) }).NotTo(Panic())
-		Expect(app.mitmServer.Load()).To(BeNil(), "failed listener must not be stored")
-	})
-
-	It("starts and stores the listener on a bindable address", func() {
-		app, opts := newMITMTestApp(GinkgoT().TempDir())
-		opts.MITMListen = "127.0.0.1:0" // OS-assigned free port
-
-		startMITMIfConfigured(app, opts)
-
-		srv := app.mitmServer.Load()
-		Expect(srv).NotTo(BeNil(), "listener should be stored on success")
-		DeferCleanup(srv.Stop)
-		Expect(srv.Addr()).NotTo(BeEmpty())
-	})
-})

core/application/router_factories.go

@@ -1,120 +1,63 @@
 package application
 
 import (
-	"context"
-	"fmt"
-
 	"github.com/mudler/LocalAI/core/backend"
 	"github.com/mudler/LocalAI/core/config"
 )
 
-// adapterConfig resolves a model name to its runtime ModelConfig, or nil when
-// unknown. LoadModelConfigFileByNameDefaultOptions never returns nil — for an
-// unknown name it returns a defaults-filled stub with an empty Name (the YAML
-// `name:` field is required by Validate), which is how we tell the two apart.
+// adapterConfig resolves a model name to its runtime ModelConfig, or
+// nil when the name is unknown. Shared by the router-facing factories
+// below and by ModelConfigLookup.
 func (a *Application) adapterConfig(modelName string) *config.ModelConfig {
 	cfg, err := a.backendLoader.LoadModelConfigFileByNameDefaultOptions(modelName, a.applicationConfig)
-	if err != nil || cfg == nil || cfg.Name == "" {
+	if err != nil || cfg == nil {
 		return nil
 	}
 	return cfg
 }
 
-// ModelConfigLookup is the lookup the router middleware's classifier validator
-// uses to confirm classifier_model declares FLAG_SCORE before binding it.
+// ModelConfigLookup is the lookup function the router middleware's
+// classifier validator uses to confirm classifier_model declares
+// FLAG_SCORE before binding it.
 func (a *Application) ModelConfigLookup() func(modelName string) *config.ModelConfig {
 	return a.adapterConfig
 }
 
-// The router-facing factories below (Scorer, Embedder, Reranker, TokenCounter)
-// bind a model NAME at construction and re-resolve the CONFIG on every call.
-// Capturing the config at construction would bake in whatever state
-// adapterConfig saw first — including a stub returned before the YAML reached
-// bcl.configs (e.g. /import-model or gallery install racing startup). The
-// classifier registry caches factories by router-config fingerprint, so a
-// once-stale capture stays stale until the router config is edited.
-
+// Scorer returns a backend.Scorer bound to the named model, or nil
+// when the model is unknown. Used as a method value (app.Scorer) by
+// router.ClassifierDeps — no factory-of-factory wrapper needed.
 func (a *Application) Scorer(modelName string) backend.Scorer {
-	if a.adapterConfig(modelName) == nil {
-		return nil
-	}
-	return &lazyScorer{app: a, modelName: modelName}
-}
-
-type lazyScorer struct {
-	app       *Application
-	modelName string
-}
-
-func (l *lazyScorer) Score(ctx context.Context, prompt string, candidates []string) ([]backend.CandidateScore, error) {
-	cfg := l.app.adapterConfig(l.modelName)
+	cfg := a.adapterConfig(modelName)
 	if cfg == nil {
-		return nil, fmt.Errorf("scorer: model %q no longer available", l.modelName)
-	}
-	return backend.NewScorer(l.app.modelLoader, *cfg, l.app.applicationConfig).Score(ctx, prompt, candidates)
-}
-
-// TokenCounter returns a func so the middleware's literal field type accepts
-// it as a method value without importing core/http/middleware from here.
-func (a *Application) TokenCounter(modelName string) func(string) (int, error) {
-	if a.adapterConfig(modelName) == nil {
 		return nil
 	}
-	return func(text string) (int, error) {
-		cfg := a.adapterConfig(modelName)
-		if cfg == nil {
-			return 0, fmt.Errorf("token counter: model %q no longer available", modelName)
-		}
-		resp, err := backend.ModelTokenize(text, a.modelLoader, *cfg, a.applicationConfig)
-		if err != nil {
-			return 0, err
-		}
-		return len(resp.Tokens), nil
-	}
+	return backend.NewScorer(a.modelLoader, *cfg, a.applicationConfig)
 }
 
+// Reranker returns a backend.Reranker bound to the named model, or
+// nil when unknown. The reranker model's `type:` (e.g. "colbert")
+// selects the scoring head inside the rerankers backend.
 func (a *Application) Reranker(modelName string) backend.Reranker {
-	if a.adapterConfig(modelName) == nil {
+	cfg := a.adapterConfig(modelName)
+	if cfg == nil {
 		return nil
 	}
-	return &lazyReranker{app: a, modelName: modelName}
-}
-
-type lazyReranker struct {
-	app       *Application
-	modelName string
-}
-
-func (l *lazyReranker) Rerank(ctx context.Context, query string, documents []string) ([]backend.RerankResult, error) {
-	cfg := l.app.adapterConfig(l.modelName)
-	if cfg == nil {
-		return nil, fmt.Errorf("reranker: model %q no longer available", l.modelName)
-	}
-	return backend.NewReranker(l.app.modelLoader, *cfg, l.app.applicationConfig).Rerank(ctx, query, documents)
+	return backend.NewReranker(a.modelLoader, *cfg, a.applicationConfig)
 }
 
+// Embedder returns a backend.Embedder bound to the named model, or
+// nil when unknown. Used by the router's L2 embedding cache.
 func (a *Application) Embedder(modelName string) backend.Embedder {
-	if a.adapterConfig(modelName) == nil {
+	cfg := a.adapterConfig(modelName)
+	if cfg == nil {
 		return nil
 	}
-	return &lazyEmbedder{app: a, modelName: modelName}
+	return backend.NewEmbedder(a.modelLoader, *cfg, a.applicationConfig)
 }
 
-type lazyEmbedder struct {
-	app       *Application
-	modelName string
-}
-
-func (l *lazyEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
-	cfg := l.app.adapterConfig(l.modelName)
-	if cfg == nil {
-		return nil, fmt.Errorf("embedder: model %q no longer available", l.modelName)
-	}
-	return backend.NewEmbedder(l.app.modelLoader, *cfg, l.app.applicationConfig).Embed(ctx, text)
-}
-
-// VectorStore takes a store name, not a model name — no adapterConfig, no
-// staleness to avoid.
+// VectorStore returns a backend.VectorStore for the named collection,
+// or nil when the name is empty. Each router model gets its own
+// backend process via the model loader's cache keyed by storeName.
 func (a *Application) VectorStore(storeName string) backend.VectorStore {
 	return backend.NewVectorStore(a.modelLoader, a.applicationConfig, storeName)
 }

core/application/router_factories_test.go

@@ -1,155 +0,0 @@
-package application
-
-import (
-	"context"
-	"os"
-	"path/filepath"
-
-	"github.com/mudler/LocalAI/core/config"
-	"github.com/mudler/LocalAI/pkg/model"
-	"github.com/mudler/LocalAI/pkg/system"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-// Regression: the router-facing factories used to capture
-// *config.ModelConfig at construction. A gallery install that raced
-// startup left a stub (Backend="") bound for the lifetime of the
-// classifier registry's cache entry, bypassing the user's `backend:`
-// config. These specs pin the lazy re-resolve.
-var _ = Describe("router_factories lazy config resolution", func() {
-	var (
-		tmpDir string
-		app    *Application
-	)
-
-	BeforeEach(func() {
-		var err error
-		tmpDir, err = os.MkdirTemp("", "router-factories-*")
-		Expect(err).NotTo(HaveOccurred())
-
-		appCfg := &config.ApplicationConfig{
-			Context:     context.Background(),
-			SystemState: &system.SystemState{Model: system.Model{ModelsPath: tmpDir}},
-		}
-		app = &Application{
-			backendLoader:     config.NewModelConfigLoader(tmpDir),
-			modelLoader:       model.NewModelLoader(appCfg.SystemState),
-			applicationConfig: appCfg,
-		}
-	})
-
-	AfterEach(func() {
-		_ = os.RemoveAll(tmpDir)
-	})
-
-	// writeCfg seeds both the on-disk YAML and the in-memory cache —
-	// removing only the cache would fall through to file-read.
-	writeCfg := func(name, backend string) {
-		yaml := "name: " + name + "\nbackend: " + backend + "\nparameters:\n  model: " + name + ".bin\n"
-		Expect(os.WriteFile(filepath.Join(tmpDir, name+".yaml"), []byte(yaml), 0644)).To(Succeed())
-		Expect(app.backendLoader.LoadModelConfigsFromPath(tmpDir)).To(Succeed())
-		cfg, ok := app.backendLoader.GetModelConfig(name)
-		Expect(ok).To(BeTrue(), "config must be loaded before the spec runs")
-		Expect(cfg.Backend).To(Equal(backend))
-	}
-
-	// removeCfg purges both the cache and the YAML so LoadModelConfigFileByName
-	// returns the empty-stub case and adapterConfig returns nil.
-	removeCfg := func(name string) {
-		app.backendLoader.RemoveModelConfig(name)
-		Expect(os.Remove(filepath.Join(tmpDir, name+".yaml"))).To(Succeed())
-	}
-
-	Context("Embedder", func() {
-		It("returns nil at construction for an unknown model", func() {
-			Expect(app.Embedder("missing")).To(BeNil())
-		})
-
-		It("re-resolves the model config on each Embed call", func() {
-			writeCfg("emb-test", "llama-cpp")
-			emb := app.Embedder("emb-test")
-			Expect(emb).NotTo(BeNil())
-
-			// The factory must hold the NAME, not a captured config —
-			// otherwise stale captures survive cache invalidation.
-			lazy, ok := emb.(*lazyEmbedder)
-			Expect(ok).To(BeTrue(), "Embedder must return *lazyEmbedder")
-			Expect(lazy.modelName).To(Equal("emb-test"))
-
-			// Mutate the cached config. A lazy implementation sees the
-			// update on the next adapterConfig call; a captured-at-
-			// construction implementation would still see "llama-cpp".
-			app.backendLoader.UpdateModelConfig("emb-test", func(c *config.ModelConfig) {
-				c.Backend = "rerankers"
-			})
-			Expect(lazy.app.adapterConfig("emb-test").Backend).To(Equal("rerankers"))
-
-			// Remove the config entirely → Embed must surface the disappearance.
-			removeCfg("emb-test")
-			_, err := emb.Embed(context.Background(), "anything")
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("no longer available"))
-		})
-	})
-
-	Context("Scorer", func() {
-		It("returns nil at construction for an unknown model", func() {
-			Expect(app.Scorer("missing")).To(BeNil())
-		})
-
-		It("re-resolves the model config on each Score call", func() {
-			writeCfg("score-test", "llama-cpp")
-			sc := app.Scorer("score-test")
-			Expect(sc).NotTo(BeNil())
-
-			lazy, ok := sc.(*lazyScorer)
-			Expect(ok).To(BeTrue(), "Scorer must return *lazyScorer")
-			Expect(lazy.modelName).To(Equal("score-test"))
-
-			removeCfg("score-test")
-			_, err := sc.Score(context.Background(), "prompt", []string{"a"})
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("no longer available"))
-		})
-	})
-
-	Context("Reranker", func() {
-		It("returns nil at construction for an unknown model", func() {
-			Expect(app.Reranker("missing")).To(BeNil())
-		})
-
-		It("re-resolves the model config on each Rerank call", func() {
-			writeCfg("rerank-test", "rerankers")
-			rr := app.Reranker("rerank-test")
-			Expect(rr).NotTo(BeNil())
-
-			lazy, ok := rr.(*lazyReranker)
-			Expect(ok).To(BeTrue(), "Reranker must return *lazyReranker")
-			Expect(lazy.modelName).To(Equal("rerank-test"))
-
-			removeCfg("rerank-test")
-			_, err := rr.Rerank(context.Background(), "q", []string{"d"})
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("no longer available"))
-		})
-	})
-
-	Context("TokenCounter", func() {
-		It("returns nil at construction for an unknown model", func() {
-			Expect(app.TokenCounter("missing")).To(BeNil())
-		})
-
-		It("re-resolves the model config on each call", func() {
-			writeCfg("tok-test", "llama-cpp")
-			tc := app.TokenCounter("tok-test")
-			Expect(tc).NotTo(BeNil())
-
-			removeCfg("tok-test")
-			_, err := tc("anything")
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("no longer available"))
-		})
-	})
-})

core/application/startup.go

@@ -462,7 +462,11 @@ func New(opts ...config.AppOption) (*Application, error) {
 	// traffic doesn't need a parallel config for MITM traffic.
 	// Runs after loadRuntimeSettingsFromFile so a listener configured
 	// via /api/settings is brought back up across restarts.
-	startMITMIfConfigured(application, options)
+	if options.MITMListen != "" {
+		if err := startMITMProxy(application, options); err != nil {
+			return nil, fmt.Errorf("mitm: startup: %w", err)
+		}
+	}
 
 	application.ModelLoader().SetBackendLoggingEnabled(options.EnableBackendLogging)
 

core/backend/embeddings.go

@@ -100,13 +100,8 @@ func ModelEmbedding(ctx context.Context, s string, tokens []int, loader *model.M
 		trace.InitBackendTracingIfEnabled(appConfig.TracingMaxItems, appConfig.TracingMaxBodyBytes)
 
 		traceData := map[string]any{
-			"input_text": trace.TruncateString(s, 1000),
-		}
-		// Only present for token-mode callers (pre-tokenized override);
-		// emitting "0" alongside input_text would read as "consumed zero
-		// tokens", which is wrong.
-		if len(tokens) > 0 {
-			traceData["input_tokens_count"] = len(tokens)
+			"input_text":         trace.TruncateString(s, 1000),
+			"input_tokens_count": len(tokens),
 		}
 
 		startTime := time.Now()

core/backend/options.go

@@ -87,47 +87,11 @@ func getSeed(c config.ModelConfig) int32 {
 	return seed
 }
 
-// DefaultContextSize and DefaultBatchSize are the backend's fallbacks when a
-// model config leaves them unset. Exported so callers that must respect the
-// effective decode window — notably the router's prompt trimmer — resolve the
-// same numbers grpcModelOpts does instead of guessing.
-const (
-	DefaultContextSize = 4096
-	DefaultBatchSize   = 512
-)
-
-// EffectiveContextSize is the context window the backend will run with: the
-// configured value, or DefaultContextSize when unset.
-func EffectiveContextSize(c config.ModelConfig) int {
-	if c.ContextSize != nil {
-		return *c.ContextSize
-	}
-	return DefaultContextSize
-}
-
-// EffectiveBatchSize is the single-decode batch the backend will run with.
-// Score, embedding and rerank all process the whole input in one pass: score
-// decodes prompt+candidate (asserts n_tokens <= n_batch), and embedding/rerank
-// pool over the full sequence in one physical batch (n_ubatch). So the batch
-// is sized to the context — anything that fits the context fits one pass,
-// avoiding both the GGML_ASSERT crash and the "input is too large to process"
-// error. Explicit `batch:` always wins.
-func EffectiveBatchSize(c config.ModelConfig) int {
-	if c.Batch != 0 {
-		return c.Batch
-	}
-	singlePass := c.HasUsecases(config.FLAG_SCORE) ||
-		c.HasUsecases(config.FLAG_EMBEDDINGS) ||
-		c.HasUsecases(config.FLAG_RERANK)
-	if ctx := EffectiveContextSize(c); singlePass && ctx > DefaultBatchSize {
-		return ctx
-	}
-	return DefaultBatchSize
-}
-
 func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
-	ctxSize := EffectiveContextSize(c)
-	b := EffectiveBatchSize(c)
+	b := 512
+	if c.Batch != 0 {
+		b = c.Batch
+	}
 
 	flashAttention := "auto"
 
@@ -170,6 +134,11 @@ func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
 		}
 	}
 
+	ctxSize := 4096
+	if c.ContextSize != nil {
+		ctxSize = *c.ContextSize
+	}
+
 	mmlock := false
 	if c.MMlock != nil {
 		mmlock = *c.MMlock
@@ -307,19 +276,11 @@ func gRPCPredictOpts(c config.ModelConfig, modelPath string) *pb.PredictOptions
 		}
 	}
 
-	// TopK may be nil after SetDefaults for backends that don't use llama.cpp's
-	// top_k=40 default (issue #6632, e.g. mlx). proto3 int32 can't be unset, so
-	// send 0 — the value mlx actually wants (top-k disabled).
-	var topK int32
-	if c.TopK != nil {
-		topK = int32(*c.TopK)
-	}
-
 	pbOpts := &pb.PredictOptions{
 		Temperature:         float32(*c.Temperature),
 		TopP:                float32(*c.TopP),
 		NDraft:              c.NDraft,
-		TopK:                topK,
+		TopK:                int32(*c.TopK),
 		MinP:                float32(*c.MinP),
 		Tokens:              int32(*c.Maxtokens),
 		Threads:             int32(*c.Threads),

core/backend/options_internal_test.go

@@ -97,67 +97,3 @@ var _ = Describe("gRPCPredictOpts reasoning_effort metadata", func() {
 		Expect(opts.Metadata).ToNot(HaveKey("reasoning_effort"))
 	})
 })
-
-var _ = Describe("grpcModelOpts NBatch", func() {
-	scoreUsecase := config.FLAG_SCORE
-	threads := 1
-	ctx := 4096
-
-	It("defaults to 512 for an ordinary model", func() {
-		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}}
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.NBatch).To(BeEquivalentTo(512))
-	})
-
-	It("sizes the batch to the context window for score models", func() {
-		// Score models decode the whole prompt+candidate in one
-		// llama_decode; n_batch must cover it or the backend aborts.
-		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &scoreUsecase}
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.NBatch).To(BeEquivalentTo(4096))
-	})
-
-	It("keeps an explicit batch over the score default", func() {
-		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &scoreUsecase}
-		cfg.Batch = 1024
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.NBatch).To(BeEquivalentTo(1024))
-	})
-
-	It("sizes the batch to the context window for embedding models", func() {
-		// Embedding/rerank pool over the whole sequence in one physical batch
-		// (n_ubatch); without this the input is capped at the 512 default and
-		// the backend returns "input is too large to process".
-		embeddings := true
-		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}}
-		cfg.Embeddings = &embeddings
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.NBatch).To(BeEquivalentTo(4096))
-	})
-
-	It("sizes the batch to the context window for rerank models", func() {
-		reranking := true
-		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}}
-		cfg.Reranking = &reranking
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.NBatch).To(BeEquivalentTo(4096))
-	})
-
-	It("does not raise the batch when a score model's context is below the default", func() {
-		small := 256
-		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &small}, KnownUsecases: &scoreUsecase}
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.NBatch).To(BeEquivalentTo(512))
-	})
-
-	It("sizes the batch to the effective 4096 default for a score model with no explicit context_size", func() {
-		// The crash case: the backend defaults n_ctx to 4096, so n_batch must
-		// follow even when context_size is unset — otherwise n_batch stays 512
-		// against a 4096 window and the score decode hits the GGML_ASSERT.
-		cfg := config.ModelConfig{Threads: &threads, KnownUsecases: &scoreUsecase}
-		Expect(cfg.ContextSize).To(BeNil())
-		opts := grpcModelOpts(cfg, "/tmp/models")
-		Expect(opts.NBatch).To(BeEquivalentTo(4096))
-		Expect(opts.ContextSize).To(BeEquivalentTo(4096), "n_batch must match the effective n_ctx the backend receives")
-	})
-})

core/backend/stores.go

@@ -3,10 +3,9 @@ package backend
 import (
 	"context"
 	"fmt"
-	"time"
+	"strings"
 
 	"github.com/mudler/LocalAI/core/config"
-	"github.com/mudler/LocalAI/core/trace"
 
 	"github.com/mudler/LocalAI/pkg/grpc"
 	"github.com/mudler/LocalAI/pkg/model"
@@ -40,85 +39,34 @@ func (s *localVectorStore) backend(_ context.Context) (grpc.Backend, error) {
 	return StoreBackend(s.loader, s.appConfig, s.storeName, "")
 }
 
-func (s *localVectorStore) Search(ctx context.Context, vec []float32) (sim float64, payload []byte, ok bool, err error) {
-	start := time.Now()
-	outcome := "hit"
-	defer func() {
-		s.recordTrace(start, "search", len(vec), sim, outcome, err)
-	}()
-	be, berr := s.backend(ctx)
-	if berr != nil {
-		outcome = "backend_load_error"
-		return 0, nil, false, fmt.Errorf("vector store load: %w", berr)
+func (s *localVectorStore) Search(ctx context.Context, vec []float32) (float64, []byte, bool, error) {
+	be, err := s.backend(ctx)
+	if err != nil {
+		return 0, nil, false, fmt.Errorf("vector store load: %w", err)
 	}
-	_, values, similarities, ferr := store.Find(ctx, be, vec, 1)
-	if ferr != nil {
-		outcome = "find_error"
-		return 0, nil, false, fmt.Errorf("vector store find: %w", ferr)
+	_, values, similarities, err := store.Find(ctx, be, vec, 1)
+	if err != nil {
+		// local-store's Find returns "existing length is -1" before
+		// any keys are inserted. Surface that as a clean miss so the
+		// cache layer treats it as an empty store and proceeds to
+		// Insert rather than skipping.
+		if strings.Contains(err.Error(), "existing length is -1") {
+			return 0, nil, false, nil
+		}
+		return 0, nil, false, fmt.Errorf("vector store find: %w", err)
 	}
 	if len(values) == 0 || len(similarities) == 0 {
-		outcome = "miss"
 		return 0, nil, false, nil
 	}
 	return float64(similarities[0]), values[0], true, nil
 }
 
-func (s *localVectorStore) Insert(ctx context.Context, vec []float32, payload []byte) (err error) {
-	start := time.Now()
-	outcome := "ok"
-	defer func() {
-		s.recordTrace(start, "insert", len(vec), 0, outcome, err)
-	}()
-	be, berr := s.backend(ctx)
-	if berr != nil {
-		outcome = "backend_load_error"
-		return fmt.Errorf("vector store load: %w", berr)
-	}
-	if serr := store.SetSingle(ctx, be, vec, payload); serr != nil {
-		outcome = "insert_error"
-		return serr
-	}
-	return nil
-}
-
-// recordTrace surfaces vector-store calls in /api/backend-traces, including
-// the backend-load-failure path that otherwise vanishes into an xlog.Warn.
-// modelName uses the store namespace (e.g. "router-cache-smart-router") so
-// admins can tell which router's cache misbehaved; the backend is always
-// "local-store" and can't disambiguate.
-func (s *localVectorStore) recordTrace(start time.Time, op string, vecDim int, sim float64, outcome string, err error) {
-	if s.appConfig == nil || !s.appConfig.EnableTracing {
-		return
-	}
-	trace.InitBackendTracingIfEnabled(s.appConfig.TracingMaxItems, s.appConfig.TracingMaxBodyBytes)
-	errStr := ""
+func (s *localVectorStore) Insert(ctx context.Context, vec []float32, payload []byte) error {
+	be, err := s.backend(ctx)
 	if err != nil {
-		errStr = err.Error()
+		return fmt.Errorf("vector store load: %w", err)
 	}
-	summary := op + " " + outcome
-	if op == "search" && outcome == "hit" {
-		summary = fmt.Sprintf("search hit (sim=%.3f)", sim)
-	}
-	data := map[string]any{
-		"op":         op,
-		"outcome":    outcome,
-		"vector_dim": vecDim,
-	}
-	// Only include similarity for a real neighbor — miss/empty_store would
-	// otherwise render "similarity: 0" and read as a measured value.
-	if op == "search" && outcome == "hit" {
-		data["similarity"] = sim
-	}
-	trace.RecordBackendTrace(trace.BackendTrace{
-		Timestamp: start,
-		Duration:  time.Since(start),
-		Type:      trace.BackendTraceVectorStore,
-		ModelName: s.storeName,
-		Backend:   model.LocalStoreBackend,
-		Summary:   summary,
-		Error:     errStr,
-		Data:      data,
-	})
+	return store.SetSingle(ctx, be, vec, payload)
 }
 
 func StoreBackend(sl *model.ModelLoader, appConfig *config.ApplicationConfig, storeName string, backend string) (grpc.Backend, error) {

core/backend/stores_test.go

@@ -1,88 +0,0 @@
-package backend
-
-import (
-	"context"
-
-	"github.com/mudler/LocalAI/core/config"
-	"github.com/mudler/LocalAI/core/trace"
-	"github.com/mudler/LocalAI/pkg/model"
-	"github.com/mudler/LocalAI/pkg/system"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-// findVectorStoreTrace returns the most recent vector_store trace whose
-// model_name matches storeName, or nil if none was recorded. Used by
-// the specs below to assert the trace landed without relying on
-// ring-buffer ordering across other tests in the suite.
-func findVectorStoreTrace(storeName string) *trace.BackendTrace {
-	traces := trace.GetBackendTraces()
-	for i := range traces {
-		bt := &traces[i]
-		if bt.Type == trace.BackendTraceVectorStore && bt.ModelName == storeName {
-			return bt
-		}
-	}
-	return nil
-}
-
-var _ = Describe("localVectorStore tracing", func() {
-	// Pin the trace surface admins read from /api/backend-traces.
-	// The original failure mode that motivated these specs — the
-	// local-store backend not installed — was silent on every surface
-	// except a per-call xlog.Warn. With tracing wired in, the row
-	// appears next to the embedder/score traces for the same request.
-	BeforeEach(func() {
-		trace.ClearBackendTraces()
-	})
-
-	It("records a vector_store trace with outcome=backend_load_error when the backend can't be loaded", func() {
-		// nil ModelLoader → s.backend → StoreBackend → panics on load.
-		// Use a real-but-empty loader so the failure surfaces as an
-		// error instead, exercising the load-failure trace path the
-		// admin would hit when local-store isn't installed.
-		appCfg := &config.ApplicationConfig{
-			EnableTracing:       true,
-			TracingMaxItems:     16,
-			TracingMaxBodyBytes: 1024,
-		}
-		s := &localVectorStore{
-			loader:    model.NewModelLoader(&system.SystemState{}),
-			appConfig: appCfg,
-			storeName: "router-cache-test",
-		}
-
-		// Search must surface the error AND record a trace describing it.
-		_, _, _, err := s.Search(context.Background(), []float32{0.1, 0.2, 0.3})
-		Expect(err).To(HaveOccurred())
-
-		Eventually(func() *trace.BackendTrace {
-			return findVectorStoreTrace("router-cache-test")
-		}).ShouldNot(BeNil())
-
-		bt := findVectorStoreTrace("router-cache-test")
-		Expect(bt.Backend).To(Equal(model.LocalStoreBackend))
-		Expect(bt.Data["op"]).To(Equal("search"))
-		Expect(bt.Data["outcome"]).To(Equal("backend_load_error"))
-		Expect(bt.Data["vector_dim"]).To(Equal(3))
-		// Error is the wrapped "vector store load: …" surfaced to the caller.
-		Expect(bt.Error).To(ContainSubstring("vector store load"))
-	})
-
-	It("does not record a trace when tracing is disabled", func() {
-		// Opt-out path: appConfig.EnableTracing=false must short-circuit
-		// before InitBackendTracingIfEnabled, so a workload with tracing
-		// turned off doesn't pay the channel-send cost per cache call.
-		appCfg := &config.ApplicationConfig{EnableTracing: false}
-		s := &localVectorStore{
-			loader:    model.NewModelLoader(&system.SystemState{}),
-			appConfig: appCfg,
-			storeName: "router-cache-disabled",
-		}
-		_, _, _, _ = s.Search(context.Background(), []float32{1})
-		Consistently(func() *trace.BackendTrace {
-			return findVectorStoreTrace("router-cache-disabled")
-		}).Should(BeNil())
-	})
-})

core/backend/tokenize.go

@@ -7,23 +7,9 @@ import (
 	"github.com/mudler/LocalAI/core/schema"
 	"github.com/mudler/LocalAI/core/trace"
 	"github.com/mudler/LocalAI/pkg/grpc"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 	"github.com/mudler/LocalAI/pkg/model"
 )
 
-// tokenizeTokenCount returns the number of tokens in a backend response,
-// treating a nil response as zero. The gRPC client returns (nil, err) on
-// failure, and the tracing block below runs before that error is returned —
-// so the count must be read nil-safely here. Reading resp.Tokens on a nil
-// resp previously panicked the whole HTTP handler when tracing was enabled
-// (e.g. a transient tokenize failure during router probe-budget sizing).
-func tokenizeTokenCount(resp *pb.TokenizationResponse) int {
-	if resp == nil {
-		return 0
-	}
-	return len(resp.Tokens)
-}
-
 func ModelTokenize(s string, loader *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig) (schema.TokenizeResponse, error) {
 
 	var inferenceModel grpc.Backend
@@ -54,7 +40,10 @@ func ModelTokenize(s string, loader *model.ModelLoader, modelConfig config.Model
 			errStr = err.Error()
 		}
 
-		tokenCount := tokenizeTokenCount(resp)
+		tokenCount := 0
+		if resp.Tokens != nil {
+			tokenCount = len(resp.Tokens)
+		}
 
 		trace.RecordBackendTrace(trace.BackendTrace{
 			Timestamp: startTime,
@@ -75,8 +64,8 @@ func ModelTokenize(s string, loader *model.ModelLoader, modelConfig config.Model
 		return schema.TokenizeResponse{}, err
 	}
 
-	if resp == nil || resp.Tokens == nil {
-		return schema.TokenizeResponse{Tokens: make([]int32, 0)}, nil
+	if resp.Tokens == nil {
+		resp.Tokens = make([]int32, 0)
 	}
 
 	return schema.TokenizeResponse{

core/backend/tokenize_test.go

@@ -1,27 +0,0 @@
-package backend
-
-import (
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("tokenizeTokenCount", func() {
-	// Regression: the gRPC client returns (nil, err) when a tokenize call
-	// fails, and ModelTokenize's tracing block reads the token count before
-	// the error is returned. Dereferencing a nil response there panicked the
-	// HTTP handler (nil pointer dereference) — e.g. a transient tokenize
-	// failure while the router sized its probe-token budget.
-	It("returns zero for a nil response instead of panicking", func() {
-		Expect(tokenizeTokenCount(nil)).To(Equal(0))
-	})
-
-	It("returns zero when the response carries no tokens", func() {
-		Expect(tokenizeTokenCount(&pb.TokenizationResponse{})).To(Equal(0))
-	})
-
-	It("counts the tokens present on the response", func() {
-		Expect(tokenizeTokenCount(&pb.TokenizationResponse{Tokens: []int32{1, 2, 3}})).To(Equal(3))
-	})
-})

core/cli/run.go

@@ -172,8 +172,6 @@ type RunCMD struct {
 	NatsTLSCert               string `env:"LOCALAI_NATS_TLS_CERT" type:"existingfile" help:"Client certificate for NATS mTLS" group:"distributed"`
 	NatsTLSKey                string `env:"LOCALAI_NATS_TLS_KEY" type:"existingfile" help:"Client private key for NATS mTLS" group:"distributed"`
 	ExposeNodeHeader          bool   `env:"LOCALAI_EXPOSE_NODE_HEADER" default:"false" help:"Set the X-LocalAI-Node response header on inference responses (OpenAI chat/completions/embeddings, Anthropic /v1/messages, Ollama /api/chat,/api/generate,/api/embed) with the ID of the worker that served the request. Disabled by default: the node ID reveals internal topology and should not be exposed on a public endpoint. Best-effort: under heavy concurrency the header may reflect a recent routing decision rather than this exact request's." group:"distributed"`
-	ModelScheduling           string `env:"LOCALAI_MODEL_SCHEDULING" help:"Declarative per-model scheduling config applied at startup (inline JSON list of {model_name,node_selector,min_replicas,max_replicas,replicas:\"all\"}). Authoritative: overwrites matching models on every boot. Distributed mode only." group:"distributed"`
-	ModelSchedulingConfig     string `env:"LOCALAI_MODEL_SCHEDULING_CONFIG" help:"Path to a YAML file with the same per-model scheduling list as LOCALAI_MODEL_SCHEDULING. Distributed mode only." group:"distributed"`
 
 	Version bool
 
@@ -349,15 +347,6 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	if r.ExposeNodeHeader {
 		opts = append(opts, config.WithExposeNodeHeader(true))
 	}
-	if r.ModelScheduling != "" {
-		opts = append(opts, config.WithModelSchedulingJSON(r.ModelScheduling))
-	}
-	if r.ModelSchedulingConfig != "" {
-		opts = append(opts, config.WithModelSchedulingConfigPath(r.ModelSchedulingConfig))
-	}
-	if !r.Distributed && (r.ModelScheduling != "" || r.ModelSchedulingConfig != "") {
-		xlog.Warn("LOCALAI_MODEL_SCHEDULING / LOCALAI_MODEL_SCHEDULING_CONFIG is set but distributed mode is disabled (LOCALAI_DISTRIBUTED=false) - ignoring")
-	}
 
 	if r.DisableMetricsEndpoint {
 		opts = append(opts, config.DisableMetricsEndpoint)

core/config/application_config.go

@@ -65,7 +65,7 @@ type ApplicationConfig struct {
 	//
 	//   patterns:
 	//     - id: email
-	//       action: allow            # downgrade default mask -> allow (log only)
+	//       action: route_local      # downgrade default mask -> route_local
 	//     - id: ssn
 	//       action: block            # upgrade default mask -> block
 	//

core/config/backend_capabilities.go

@@ -396,10 +396,10 @@ var BackendCapabilities = map[string]BackendCapability{
 		Description:      "Qwen TTS",
 	},
 	"qwen3-tts-cpp": {
-		GRPCMethods:      []GRPCMethod{MethodTTS, MethodTTSStream},
+		GRPCMethods:      []GRPCMethod{MethodTTS},
 		PossibleUsecases: []string{UsecaseTTS},
 		DefaultUsecases:  []string{UsecaseTTS},
-		Description:      "Qwen3 TTS C++ - text-to-speech with streaming, named speakers, voice design and cloning (qwentts.cpp / GGML)",
+		Description:      "Qwen3 TTS C++ — text-to-speech, C++ backend",
 	},
 	"faster-qwen3-tts": {
 		GRPCMethods:      []GRPCMethod{MethodTTS},
@@ -517,33 +517,6 @@ func NormalizeBackendName(backend string) string {
 	return strings.ReplaceAll(backend, ".", "-")
 }
 
-// nonLlamaSamplerBackends lists backends whose native sampler defaults differ
-// from llama.cpp's, so LocalAI must NOT inject llama.cpp's top_k=40 default for
-// them (issue #6632). mlx_lm's intended default is top_k=0 (disabled) and mlx
-// does not remap 0->40, so shipping 40 silently changes sampling for clients
-// that omit top_k. Leaving TopK nil lets the wire value default to 0.
-//
-// This is intentionally a small allow-list of KNOWN non-llama backends: empty
-// and unknown backends fall through to the llama.cpp default to preserve the
-// GGUF auto-detect path's behavior.
-var nonLlamaSamplerBackends = map[string]struct{}{
-	"mlx":             {},
-	"mlx-vlm":         {},
-	"mlx-distributed": {},
-}
-
-// UsesLlamaSamplerDefaults reports whether a backend should receive llama.cpp's
-// sampler defaults (e.g. top_k=40). Empty/unknown backends return true so the
-// GGUF auto-detect path (which resolves to llama.cpp) keeps today's behavior;
-// only the known non-llama backends in nonLlamaSamplerBackends return false.
-func UsesLlamaSamplerDefaults(backend string) bool {
-	if backend == "" {
-		return true
-	}
-	_, isNonLlama := nonLlamaSamplerBackends[NormalizeBackendName(backend)]
-	return !isNonLlama
-}
-
 // GetBackendCapability returns the capability info for a backend, or nil if unknown.
 // Handles backend name normalization.
 func GetBackendCapability(backend string) *BackendCapability {

core/config/distributed_config.go

@@ -84,12 +84,6 @@ type DistributedConfig struct {
 	// drives the background eviction cadence (eviction runs every TTL/2). Zero
 	// means use the prefixcache package default (5m).
 	PrefixCacheTTL time.Duration
-	// ModelSchedulingJSON is an inline JSON list of per-model scheduling configs
-	// applied authoritatively at startup (LOCALAI_MODEL_SCHEDULING).
-	ModelSchedulingJSON string
-	// ModelSchedulingConfigPath is a path to a YAML file with the same list
-	// (LOCALAI_MODEL_SCHEDULING_CONFIG).
-	ModelSchedulingConfigPath string
 }
 
 // Validate checks that the distributed configuration is internally consistent.
@@ -296,21 +290,6 @@ func WithPrefixCacheTTL(d time.Duration) AppOption {
 	}
 }
 
-// WithModelSchedulingJSON sets the inline-JSON declarative scheduling config.
-func WithModelSchedulingJSON(s string) AppOption {
-	return func(o *ApplicationConfig) {
-		o.Distributed.ModelSchedulingJSON = s
-	}
-}
-
-// WithModelSchedulingConfigPath sets the path to a YAML declarative scheduling
-// config file.
-func WithModelSchedulingConfigPath(path string) AppOption {
-	return func(o *ApplicationConfig) {
-		o.Distributed.ModelSchedulingConfigPath = path
-	}
-}
-
 // Flag names for distributed timeout / interval configuration. These are
 // the kebab-case identifiers kong derives from the matching RunCMD struct
 // fields; they appear in Validate error messages and any other operator-

core/config/meta/build.go

@@ -93,9 +93,6 @@ func applyOverride(f *FieldMeta, o FieldMetaOverride) {
 	if o.Component != "" {
 		f.Component = o.Component
 	}
-	if o.Language != "" {
-		f.Language = o.Language
-	}
 	if o.Placeholder != "" {
 		f.Placeholder = o.Placeholder
 	}