fix(distributed): per-replica backend logs (store aggregation + UI)

The multi-replica refactor (PR #9583) changed the worker's process key
from `modelID` to `modelID#replicaIndex`, but the BackendLogStore kept
the bare-modelID lookup. Result: every distributed deployment lost
backend logs in the Nodes UI — single-replica too, since even the
default capacity of 1 produces a `#0` suffix.

Two changes wired together:

* pkg/model: BackendLogStore.GetLines/Subscribe now treat a modelID
  without `#` as a model prefix and merge across all `modelID#N` replica
  buffers (timestamp-sorted for GetLines; fan-in for Subscribe). Calls
  with a full `modelID#N` key resolve exactly. ListModels strips
  replica suffixes and deduplicates so the listing surfaces one entry
  per loaded model.

* react-ui: per-replica log streams as the default. Loaded Models
  table disambiguates each row with a `rep N` pill (only when the node
  hosts >1 replica of a model). Each row's "View logs" link routes to
  the per-replica process key so operators see only that replica's
  output. The logs page renders the replica context as a chip in the
  title and surfaces a segmented control — `Replica 0 / 1 / … / All
  merged` — when the model has multiple replicas; the merged segment
  uses the bare-modelID URL (delegating to the store's prefix
  aggregation) for the side-by-side comparison case. Single-replica
  deployments see no extra UI.

Tests added first (TDD): the regression set in
backend_log_store_test.go reproduces the bug at the exact failure
point — GetLines/ListModels/Subscribe assertions all fail against the
broken code, all pass against the fix. TestSubscribe_PerReplicaFilter
pins the exact-key path so a future change can't silently break it.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude-code:opus-4-7 [Edit] [Skill:critique] [Skill:audit] [Skill:polish] [Skill:distill]

.agents/adding-backends.md

@@ -8,6 +8,7 @@ Create the backend directory under the appropriate location:
 - **Python backends**: `backend/python/<backend-name>/`
 - **Go backends**: `backend/go/<backend-name>/`
 - **C++ backends**: `backend/cpp/<backend-name>/`
+- **Rust backends**: `backend/rust/<backend-name>/`
 
 For Python backends, you'll typically need:
 - `backend.py` - Main gRPC server implementation
@@ -18,9 +19,22 @@ For Python backends, you'll typically need:
 - `run.sh` - Runtime script
 - `test.py` / `test.sh` - Test files
 
+For Rust backends, you'll typically need (see `backend/rust/kokoros/` as a reference):
+- `Cargo.toml` - Crate manifest; depend on the upstream project as a submodule under `sources/`
+- `build.rs` - Invokes `tonic_build` to generate gRPC stubs from `backend/backend.proto` (use the `BACKEND_PROTO_PATH` env var so the Makefile can inject the canonical copy)
+- `src/` - The gRPC server implementation (implement `Backend` via `tonic`)
+- `Makefile` - Copies `backend.proto` into the crate, runs `cargo build --release`, then `package.sh`
+- `package.sh` - Uses `ldd` to bundle the binary's dynamic deps and `ld.so` into `package/lib/`
+- `run.sh` - Sets `LD_LIBRARY_PATH`/`SSL_CERT_DIR` and execs the binary via the bundled `lib/ld.so`
+- `sources/<UpstreamProject>/` - Git submodule with the upstream Rust crate
+
 ## 2. Add Build Configurations to `.github/workflows/backend.yml`
 
-Add build matrix entries for each platform/GPU type you want to support. Look at similar backends (e.g., `chatterbox`, `faster-whisper`) for reference.
+Add build matrix entries for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
+
+**Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
+
+If you add a new language bucket, `scripts/changed-backends.js` also needs a branch in `inferBackendPath` so PR change-detection routes file edits correctly.
 
 **Placement in file:**
 - CPU builds: Add after other CPU builds (e.g., after `cpu-chatterbox`)
@@ -29,7 +43,7 @@ Add build matrix entries for each platform/GPU type you want to support. Look at
 
 **Additional build types you may need:**
 - ROCm/HIP: Use `build-type: 'hipblas'` with `base-image: "rocm/dev-ubuntu-24.04:7.2.1"`
-- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"`
+- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"`
 - L4T (ARM): Use `build-type: 'l4t'` with `platforms: 'linux/arm64'` and `runs-on: 'ubuntu-24.04-arm'`
 
 ## 3. Add Backend Metadata to `backend/index.yaml`
@@ -56,24 +70,28 @@ Add `backends/<backend-name>` to the `.NOTPARALLEL` line (around line 2) to prev
 
 **Step 4b: Add to `prepare-test-extra`**
 
-Add the backend to the `prepare-test-extra` target (around line 312) to prepare it for testing:
+Add the backend to the `prepare-test-extra` target to prepare it for testing. Use the path matching your language bucket (`backend/python/`, `backend/go/`, `backend/rust/`, …):
 
 ```makefile
 prepare-test-extra: protogen-python
 	...
-	$(MAKE) -C backend/python/<backend-name>
+	$(MAKE) -C backend/<lang>/<backend-name>
 ```
 
+For Rust backends the target is usually the crate build target itself (e.g. `$(MAKE) -C backend/rust/<backend-name> <backend-name>-grpc`) so the binary is in place before `test` runs.
+
 **Step 4c: Add to `test-extra`**
 
-Add the backend to the `test-extra` target (around line 319) to run its tests:
+Add the backend to the `test-extra` target to run its tests — applies to Go and Rust backends too, not only Python:
 
 ```makefile
 test-extra: prepare-test-extra
 	...
-	$(MAKE) -C backend/python/<backend-name> test
+	$(MAKE) -C backend/<lang>/<backend-name> test
 ```
 
+Each backend's own `Makefile` should define a `test` target so this line works regardless of language. Integration tests that need large model downloads should be gated behind an env var (see `backend/rust/kokoros/`'s `KOKOROS_MODEL_PATH` pattern) so CI only runs unit tests.
+
 **Step 4d: Add Backend Definition**
 
 Add a backend definition variable in the backend definitions section (around line 428-457). The format depends on the backend type:
@@ -93,6 +111,13 @@ BACKEND_<BACKEND_NAME> = <backend-name>|python|./backend|false|true
 BACKEND_<BACKEND_NAME> = <backend-name>|golang|.|false|true
 ```
 
+**For Rust backends**:
+```makefile
+BACKEND_<BACKEND_NAME> = <backend-name>|rust|.|false|true
+```
+
+The language field (`python`/`golang`/`rust`/…) must match a `backend/Dockerfile.<lang>` file.
+
 **Step 4e: Generate Docker Build Target**
 
 Add an eval call to generate the docker-build target (around line 480-501):
@@ -153,6 +178,29 @@ ls /tmp/check    # expect the bundled .so files + symlinks
 
 Then boot it inside a fresh `ubuntu:24.04` (which intentionally does *not* have the lib installed) to confirm it actually loads from the backend dir.
 
+## Importer integration
+
+When you add a new backend, you MUST also make it importable via the model import form (`/import-model`). The import form dropdown is sourced dynamically from `GET /backends/known` — it reads the importer registry at `core/gallery/importers/importers.go`, so the steps below are the ONLY way to make your backend show up.
+
+Required steps:
+
+1. **If your backend has unambiguous detection signals** (unique file extension, HF `pipeline_tag`, unique repo name pattern, unique artefact like `modules.json`):
+   - Create an importer file at `core/gallery/importers/<backend>.go` following the Match/Import pattern in `llama-cpp.go`.
+   - Register it in `importers.go:defaultImporters` in **specificity order** — more specific detectors must appear BEFORE more generic ones (e.g. `sentencetransformers` before `transformers`, `stablediffusion-ggml` before `llama-cpp`, `vllm-omni` before `vllm`). First match wins.
+2. **If your backend is a drop-in replacement** (same artefacts as another backend, e.g. `ik-llama-cpp` and `turboquant` both consume GGUF the same way `llama-cpp` does):
+   - Do NOT create a new importer. Extend the existing importer's `Import()` to swap the emitted `backend:` field when `preferences.backend` matches. See `llama-cpp.go` for the pattern.
+3. **If your backend has no reliable auto-detect signal** (preference-only — e.g. `sglang`, `tinygrad`, `whisperx`):
+   - Do NOT create an importer. Instead add the backend name to the curated pref-only slice in `core/http/endpoints/localai/backend.go` that feeds `/backends/known`. A single line addition.
+4. **Always** add a table-driven test in `core/gallery/importers/importers_test.go` (Ginkgo/Gomega):
+   - Use a real public HuggingFace repo URI as the test fixture (existing tests already hit the live HF API — follow that pattern).
+   - Cover detection (auto-match without preferences), preference-override (explicit `backend:` in preferences wins), and — if the backend's modality has a common `pipeline_tag` but ambiguous artefacts — an ambiguity test asserting `errors.Is(err, importers.ErrAmbiguousImport)`.
+
+Rules of thumb:
+
+- When in doubt, lean pref-only. A wrong auto-detect is worse than a forced preference.
+- Never silently emit a modality mismatch (e.g. emit `llama-cpp` for a TTS repo because `.gguf` is present). Return `ErrAmbiguousImport` instead.
+- Registration order is the single most common source of bugs. Check by running `go test ./core/gallery/importers/...` — the existing suite will fail if you've shadowed a pre-existing detector.
+
 ## 6. Example: Adding a Python Backend
 
 For reference, when `moonshine` was added:

.agents/ci-caching.md

@@ -0,0 +1,111 @@
+# CI Build Caching
+
+Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache. This file explains how that cache is laid out, what invalidates it, and how to bypass it.
+
+## Cache layout
+
+- **Cache registry**: `quay.io/go-skynet/ci-cache`
+- **One tag per matrix entry**, derived from the existing `tag-suffix`:
+  - Backend builds (`backend_build.yml`): `cache<tag-suffix>`
+    - e.g. `cache-gpu-nvidia-cuda-12-llama-cpp`, `cache-cpu-vllm`, `cache-nvidia-l4t-cuda-13-arm64-vllm`
+  - Root image builds (`image_build.yml`): `cache-localai<tag-suffix>`
+    - e.g. `cache-localai-gpu-nvidia-cuda-12`, `cache-localai-gpu-vulkan`
+- Each tag stores a multi-arch BuildKit cache manifest (`mode=max`), so every intermediate stage is re-usable, not just the final image.
+
+## Read/write semantics
+
+| Trigger | `cache-from` | `cache-to` |
+|---|---|---|
+| `push` to `master` / tag | yes | yes (`mode=max,ignore-error=true`) |
+| `pull_request` | yes | **no** |
+
+PR builds read master's warm cache but never write — this prevents PRs from polluting the shared cache with their experimental state. After merge, the master build for that matrix entry refreshes the cache.
+
+`ignore-error=true` on the write side means a transient quay push failure does not fail the build; the next master push retries.
+
+## Self-warming, no separate populator
+
+There is no cron job that pre-warms the cache. The production builds *are* the populator. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`, Python wheel installs, etc.).
+
+Historically there was a `generate_grpc_cache.yaml` cron that targeted a `grpc` stage in the root Dockerfile. That stage was removed in July 2025 and the cron silently failed every night for 9 months without writing anything. It was deleted along with the registry-cache rollout.
+
+## The `DEPS_REFRESH` cache-buster (Python backends)
+
+Every Python backend goes through the shared `backend/Dockerfile.python`, which ends with:
+
+```dockerfile
+ARG DEPS_REFRESH=initial
+RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
+```
+
+Most Python backends ship `requirements*.txt` files that **do not pin every transitive dep** (`torch`, `transformers`, `vllm`, `diffusers`, etc. are listed without a `==` pin, or with `>=` lower bounds only). With a warm BuildKit cache, the `make` layer hashes only on Dockerfile instructions + COPYed source — not on what `pip install` resolves at runtime. So a warm cache would ship the *first* version of `vllm` ever cached and never pick up upstream releases.
+
+`DEPS_REFRESH` defends against that:
+
+- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W17`) before each build and passes it as a build-arg.
+- The `RUN ... make` layer's BuildKit hash now includes that string, so the layer invalidates **at most once per week**, automatically picking up newer wheels.
+- Within a week, builds stay warm.
+
+This applies only to `Dockerfile.python` because:
+- Go (`Dockerfile.golang`) pins versions in `go.mod` / `go.sum`.
+- Rust (`Dockerfile.rust`) pins via `Cargo.lock`.
+- C++ backends (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) clone gRPC at a pinned tag (`v1.65.0`) and llama.cpp at a pinned commit; their inputs don't drift between rebuilds.
+
+### Adjusting the cadence
+
+If you need a faster refresh (e.g. while debugging an upstream flake), bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`). If you need a one-shot rebuild for a specific backend without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
+
+## Manually evicting cache
+
+To force a fully cold build for one backend or the whole image:
+
+```bash
+# Delete a single tag (requires quay credentials with admin on the repo)
+curl -X DELETE \
+  -H "Authorization: Bearer ${QUAY_TOKEN}" \
+  https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm
+
+# List all tags
+curl -s -H "Authorization: Bearer ${QUAY_TOKEN}" \
+  "https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/?limit=100" | jq '.tags[].name'
+```
+
+Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry so a stale tag never bleeds into a different build.
+
+## What the cache **does not** cover
+
+- The "Free Disk Space" / "Release space from worker" steps run on every job — these reclaim ~6 GB on `ubuntu-latest` runners. They are runner-state cleanup, not Docker, and BuildKit caches don't apply.
+- Intermediate artifacts of `Build and push (PR)` are not pushed anywhere — PRs only build for verification.
+- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
+
+## Darwin native caches
+
+`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
+
+| Cache | Path(s) | Key | Scope |
+|---|---|---|---|
+| Go modules + build | `~/go/pkg/mod`, `~/Library/Caches/go-build` | `go.sum` (managed by `actions/setup-go@v5` `cache: true`) | All darwin jobs |
+| Homebrew | `~/Library/Caches/Homebrew/downloads`, selected `/opt/homebrew/Cellar/*` | hash of `backend_build_darwin.yml` | All darwin jobs |
+| ccache (llama.cpp CMake) | `~/Library/Caches/ccache` | pinned `LLAMA_VERSION` from `backend/cpp/llama-cpp/Makefile` | `inputs.backend == 'llama-cpp'` only |
+| Python wheels (uv + pip) | `~/Library/Caches/pip`, `~/Library/Caches/uv` | `inputs.backend` + ISO week (`+%Y-W%V`) + hash of that backend's `requirements*.txt` | `inputs.lang == 'python'` only |
+
+Read/write semantics match the BuildKit cache: `actions/cache/restore` runs every time, `actions/cache/save` is gated on `github.event_name != 'pull_request'`. PRs read master's warm cache but never write back.
+
+The Python wheel cache uses the same ISO-week cache-buster as the Linux `DEPS_REFRESH` build-arg — same problem (unpinned `torch`/`mlx`/`diffusers`/`transformers` resolve to fresh wheels weekly), same ~one-cold-rebuild-per-week solution.
+
+The brew Cellar cache requires `HOMEBREW_NO_AUTO_UPDATE=1` and `HOMEBREW_NO_INSTALL_CLEANUP=1` (set as job-level env). Without those, `brew install` would mutate the very directories that were just restored, defeating the cache.
+
+For ccache, the workflow exports `CMAKE_ARGS=… -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache` via `$GITHUB_ENV` before running `make build-darwin-go-backend`. The Makefile in `backend/cpp/llama-cpp/` already forwards `CMAKE_ARGS` through to each variant build (`fallback`, `grpc`, `rpc-server`), so no script changes are needed. The three variants share most TUs, so ccache dedupes object files across them.
+
+### Cache budget on Darwin
+
+GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~800 MB Go cache + ~2 GB brew Cellar + up to 2 GB ccache + ~1.5 GB × 5 python backends. If the cap is hit, prefer collapsing the per-backend Python keys into a shared `pyenv-darwin-shared-<week>` key (accepts more cross-backend churn for a smaller footprint) before reducing other caches.
+
+## Touching the cache pipeline
+
+When changing `image_build.yml`, `backend_build.yml`, or any of the `backend/Dockerfile.*` files:
+
+1. **Don't drop `DEPS_REFRESH=...` from the build-args** without a replacement strategy (lockfiles, pinned requirements). Otherwise master will silently freeze on whichever versions were cached at the time.
+2. **Keep `tag-suffix` unique per matrix entry** — it's the cache namespace. Two matrix entries sharing a tag-suffix would clobber each other's cache.
+3. **Keep `cache-to` gated on `github.event_name != 'pull_request'`** — PRs must not write.
+4. **Keep `ignore-error=true` on `cache-to`** — quay registry hiccups must not fail builds.

.agents/coding-style.md

@@ -42,6 +42,12 @@ trim_trailing_whitespace = false
 
 Use `github.com/mudler/xlog` for logging which has the same API as slog.
 
+## Go tests
+
+All Go tests — including backend tests — must use [Ginkgo](https://onsi.github.io/ginkgo/) (v2) with Gomega matchers, not the stdlib `testing` package with `t.Run` / `t.Errorf`. A test file should register a suite with `RegisterFailHandler(Fail)` in a `TestXxx(t *testing.T)` bootstrap and use `Describe`/`Context`/`It` blocks for the actual cases. Look at any existing `*_test.go` under `core/` or `pkg/` for a template.
+
+Do not mix styles within a package. If you are extending tests in a package that already uses Ginkgo, keep using Ginkgo. If you find stdlib-style Go tests in the tree, treat them as tech debt to be migrated rather than as a pattern to follow.
+
 ## Documentation
 
 The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.

.github/workflows/backend.yml

@@ -141,7 +141,7 @@ jobs:
           - build-type: ''
             cuda-major-version: ""
             cuda-minor-version: ""
-            platforms: 'linux/amd64'
+            platforms: 'linux/amd64,linux/arm64'
             tag-latest: 'auto'
             tag-suffix: '-cpu-whisperx'
             runs-on: 'ubuntu-latest'
@@ -154,7 +154,7 @@ jobs:
           - build-type: ''
             cuda-major-version: ""
             cuda-minor-version: ""
-            platforms: 'linux/amd64'
+            platforms: 'linux/amd64,linux/arm64'
             tag-latest: 'auto'
             tag-suffix: '-cpu-faster-whisper'
             runs-on: 'ubuntu-latest'
@@ -724,6 +724,19 @@ jobs:
             dockerfile: "./backend/Dockerfile.python"
             context: "./"
             ubuntu-version: '2404'
+          - build-type: 'cublas'
+            cuda-major-version: "12"
+            cuda-minor-version: "8"
+            platforms: 'linux/amd64'
+            tag-latest: 'auto'
+            tag-suffix: '-gpu-nvidia-cuda-12-speaker-recognition'
+            runs-on: 'ubuntu-latest'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            backend: "speaker-recognition"
+            dockerfile: "./backend/Dockerfile.python"
+            context: "./"
+            ubuntu-version: '2404'
           - build-type: 'cublas'
             cuda-major-version: "12"
             cuda-minor-version: "8"
@@ -907,6 +920,32 @@ jobs:
             backend: "turboquant"
             dockerfile: "./backend/Dockerfile.turboquant"
             context: "./"
+          - build-type: 'cublas'
+            cuda-major-version: "13"
+            cuda-minor-version: "0"
+            platforms: 'linux/amd64'
+            tag-latest: 'auto'
+            tag-suffix: '-gpu-nvidia-cuda-13-vllm'
+            runs-on: 'arc-runner-set'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            backend: "vllm"
+            dockerfile: "./backend/Dockerfile.python"
+            context: "./"
+            ubuntu-version: '2404'
+          - build-type: 'cublas'
+            cuda-major-version: "13"
+            cuda-minor-version: "0"
+            platforms: 'linux/amd64'
+            tag-latest: 'auto'
+            tag-suffix: '-gpu-nvidia-cuda-13-vllm-omni'
+            runs-on: 'arc-runner-set'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            backend: "vllm-omni"
+            dockerfile: "./backend/Dockerfile.python"
+            context: "./"
+            ubuntu-version: '2404'
           - build-type: 'cublas'
             cuda-major-version: "13"
             cuda-minor-version: "0"
@@ -1063,6 +1102,45 @@ jobs:
             backend: "diffusers"
             dockerfile: "./backend/Dockerfile.python"
             context: "./"
+          - build-type: 'l4t'
+            cuda-major-version: "13"
+            cuda-minor-version: "0"
+            platforms: 'linux/arm64'
+            tag-latest: 'auto'
+            tag-suffix: '-nvidia-l4t-cuda-13-arm64-vllm'
+            runs-on: 'ubuntu-24.04-arm'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            ubuntu-version: '2404'
+            backend: "vllm"
+            dockerfile: "./backend/Dockerfile.python"
+            context: "./"
+          - build-type: 'l4t'
+            cuda-major-version: "13"
+            cuda-minor-version: "0"
+            platforms: 'linux/arm64'
+            tag-latest: 'auto'
+            tag-suffix: '-nvidia-l4t-cuda-13-arm64-vllm-omni'
+            runs-on: 'ubuntu-24.04-arm'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            ubuntu-version: '2404'
+            backend: "vllm-omni"
+            dockerfile: "./backend/Dockerfile.python"
+            context: "./"
+          - build-type: 'l4t'
+            cuda-major-version: "13"
+            cuda-minor-version: "0"
+            platforms: 'linux/arm64'
+            tag-latest: 'auto'
+            tag-suffix: '-nvidia-l4t-cuda-13-arm64-sglang'
+            runs-on: 'ubuntu-24.04-arm'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            ubuntu-version: '2404'
+            backend: "sglang"
+            dockerfile: "./backend/Dockerfile.python"
+            context: "./"
           - build-type: 'l4t'
             cuda-major-version: "13"
             cuda-minor-version: "0"
@@ -1658,7 +1736,7 @@ jobs:
             tag-latest: 'auto'
             tag-suffix: '-gpu-intel-rerankers'
             runs-on: 'ubuntu-latest'
-            base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+            base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
             skip-drivers: 'false'
             backend: "rerankers"
             dockerfile: "./backend/Dockerfile.python"
@@ -1671,7 +1749,7 @@ jobs:
             tag-latest: 'auto'
             tag-suffix: '-gpu-intel-sycl-f32-llama-cpp'
             runs-on: 'ubuntu-latest'
-            base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+            base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
             skip-drivers: 'false'
             backend: "llama-cpp"
             dockerfile: "./backend/Dockerfile.llama-cpp"
@@ -2653,6 +2731,20 @@ jobs:
             dockerfile: "./backend/Dockerfile.python"
             context: "./"
             ubuntu-version: '2404'
+          # speaker-recognition (voice/speaker biometrics)
+          - build-type: ''
+            cuda-major-version: ""
+            cuda-minor-version: ""
+            platforms: 'linux/amd64,linux/arm64'
+            tag-latest: 'auto'
+            tag-suffix: '-cpu-speaker-recognition'
+            runs-on: 'ubuntu-latest'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            backend: "speaker-recognition"
+            dockerfile: "./backend/Dockerfile.python"
+            context: "./"
+            ubuntu-version: '2404'
           - build-type: 'intel'
             cuda-major-version: ""
             cuda-minor-version: ""
@@ -2850,6 +2942,49 @@ jobs:
             dockerfile: "./backend/Dockerfile.python"
             context: "./"
             ubuntu-version: '2404'
+          # sherpa-onnx CPU
+          - build-type: ''
+            cuda-major-version: ""
+            cuda-minor-version: ""
+            platforms: 'linux/amd64,linux/arm64'
+            tag-latest: 'auto'
+            tag-suffix: '-cpu-sherpa-onnx'
+            runs-on: 'ubuntu-latest'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            backend: "sherpa-onnx"
+            dockerfile: "./backend/Dockerfile.golang"
+            context: "./"
+            ubuntu-version: '2404'
+          # sherpa-onnx CUDA 12
+          - build-type: 'cublas'
+            cuda-major-version: "12"
+            cuda-minor-version: "8"
+            platforms: 'linux/amd64'
+            tag-latest: 'auto'
+            tag-suffix: '-gpu-nvidia-cuda-12-sherpa-onnx'
+            runs-on: 'ubuntu-latest'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            backend: "sherpa-onnx"
+            dockerfile: "./backend/Dockerfile.golang"
+            context: "./"
+            ubuntu-version: '2404'
+          # sherpa-onnx CUDA 13 — requires onnxruntime 1.24.x+ for the
+          # gpu_cuda13 tarball; sherpa-onnx SHERPA_COMMIT pins to v1.12.39.
+          - build-type: 'cublas'
+            cuda-major-version: "13"
+            cuda-minor-version: "0"
+            platforms: 'linux/amd64'
+            tag-latest: 'auto'
+            tag-suffix: '-gpu-nvidia-cuda-13-sherpa-onnx'
+            runs-on: 'ubuntu-latest'
+            base-image: "ubuntu:24.04"
+            skip-drivers: 'false'
+            backend: "sherpa-onnx"
+            dockerfile: "./backend/Dockerfile.golang"
+            context: "./"
+            ubuntu-version: '2404'
   backend-jobs-darwin:
     uses: ./.github/workflows/backend_build_darwin.yml
     strategy:

.github/workflows/backend_build.yml

@@ -108,6 +108,8 @@ jobs:
 
       - name: Checkout
         uses: actions/checkout@v6
+        with:
+          submodules: true
 
       - name: Release space from worker
         if: inputs.runs-on == 'ubuntu-latest'
@@ -206,6 +208,15 @@ jobs:
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
+      # Weekly cache-buster for the per-backend `make` step. Most Python
+      # backends list unpinned deps (torch, transformers, vllm, ...), so a
+      # warm cache freezes upstream versions indefinitely. Rolling this
+      # weekly forces a re-resolve of the install layer at most once per
+      # week, picking up newer wheels without a full cold rebuild.
+      - name: Compute deps refresh key
+        id: deps_refresh
+        run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
+
       - name: Build and push
         uses: docker/build-push-action@v7
         if: github.event_name != 'pull_request'
@@ -220,9 +231,11 @@ jobs:
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
+            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=gha
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}
+          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }},mode=max,ignore-error=true
           platforms: ${{ inputs.platforms }}
           push: ${{ github.event_name != 'pull_request' }}
           tags: ${{ steps.meta.outputs.tags }}
@@ -242,9 +255,10 @@ jobs:
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
+            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=gha
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}
           platforms: ${{ inputs.platforms }}
           push: ${{ env.quay_username != '' }}
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/backend_build_darwin.yml

@@ -48,6 +48,13 @@ jobs:
     strategy:
       matrix:
         go-version: ['${{ inputs.go-version }}']
+    env:
+      # Keep the brew Cellar stable across cache restores. Without these,
+      # `brew install` would auto-update brew itself and re-link formulas,
+      # mutating the very paths the cache just restored.
+      HOMEBREW_NO_AUTO_UPDATE: '1'
+      HOMEBREW_NO_INSTALL_CLEANUP: '1'
+      HOMEBREW_NO_ANALYTICS: '1'
     steps:
       - name: Clone
         uses: actions/checkout@v6
@@ -58,21 +65,141 @@ jobs:
         uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go-version }}
-          cache: false
+          # Caches ~/go/pkg/mod and ~/Library/Caches/go-build keyed on go.sum.
+          # Shared across every darwin matrix entry — first job in a run warms
+          # it, the rest hit warm.
+          cache: true
 
       # You can test your matrix by printing the current Go version
       - name: Display Go version
         run: go version
 
+      # ---- Homebrew cache ----
+      # macOS runners have no Docker daemon, so the BuildKit registry cache used
+      # for Linux backend images (see .agents/ci-caching.md) doesn't apply here.
+      # We cache the brew downloads + Cellar entries for the formulas we install
+      # below. Read on every run, write only on master/tag pushes — same policy
+      # as the Linux registry cache.
+      - name: Restore Homebrew cache
+        id: brew-cache
+        uses: actions/cache/restore@v4
+        with:
+          path: |
+            ~/Library/Caches/Homebrew/downloads
+            /opt/homebrew/Cellar/protobuf
+            /opt/homebrew/Cellar/grpc
+            /opt/homebrew/Cellar/protoc-gen-go
+            /opt/homebrew/Cellar/protoc-gen-go-grpc
+            /opt/homebrew/Cellar/libomp
+            /opt/homebrew/Cellar/llvm
+            /opt/homebrew/Cellar/ccache
+          key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
+
       - name: Dependencies
         run: |
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm
+          # ccache is always installed (used by the llama-cpp variant build) so
+          # the brew cache content stays stable across every backend in the
+          # matrix — they all share one cache key.
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache
+
+      - name: Save Homebrew cache
+        if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v4
+        with:
+          path: |
+            ~/Library/Caches/Homebrew/downloads
+            /opt/homebrew/Cellar/protobuf
+            /opt/homebrew/Cellar/grpc
+            /opt/homebrew/Cellar/protoc-gen-go
+            /opt/homebrew/Cellar/protoc-gen-go-grpc
+            /opt/homebrew/Cellar/libomp
+            /opt/homebrew/Cellar/llvm
+            /opt/homebrew/Cellar/ccache
+          key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
+
+      # ---- ccache for llama.cpp CMake builds ----
+      # Three CMake variants (fallback, grpc, rpc-server) compile the same
+      # llama.cpp source tree with overlapping flags — ccache dedupes object
+      # files across them. Key on the pinned LLAMA_VERSION so a pin bump
+      # invalidates cleanly; restore-keys fall back to the latest entry for the
+      # same pin so unchanged TUs stay warm even when the cache is fresh.
+      - name: Compute llama.cpp version
+        if: inputs.backend == 'llama-cpp'
+        id: llama-version
+        run: |
+          version=$(grep '^LLAMA_VERSION' backend/cpp/llama-cpp/Makefile | head -1 | cut -d= -f2 | cut -d'?' -f1 | tr -d ' ')
+          echo "version=${version}" >> "$GITHUB_OUTPUT"
+
+      - name: Restore ccache
+        if: inputs.backend == 'llama-cpp'
+        id: ccache-cache
+        uses: actions/cache/restore@v4
+        with:
+          path: ~/Library/Caches/ccache
+          key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
+          restore-keys: |
+            ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-
+
+      - name: Configure ccache
+        if: inputs.backend == 'llama-cpp'
+        run: |
+          mkdir -p "$HOME/Library/Caches/ccache"
+          ccache -M 2G
+          ccache -z
+          # llama-cpp-darwin.sh reads CMAKE_ARGS / CCACHE_DIR from env.
+          {
+            echo "CMAKE_ARGS=${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache"
+            echo "CCACHE_DIR=$HOME/Library/Caches/ccache"
+          } >> "$GITHUB_ENV"
+
+      # ---- Python wheel cache (uv + pip) ----
+      # Mirrors the Linux DEPS_REFRESH cadence (see .agents/ci-caching.md): the
+      # ISO-week segment of the cache key forces at most one cold rebuild per
+      # backend per week, automatically picking up newer wheels for unpinned
+      # deps (torch, mlx, diffusers, …). Restore-keys fall back to the most
+      # recent build of the same backend so off-week PRs still hit warm.
+      - name: Compute weekly cache bucket
+        if: inputs.lang == 'python'
+        id: weekly
+        run: echo "bucket=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
+
+      - name: Restore Python wheel cache
+        if: inputs.lang == 'python'
+        id: pyenv-cache
+        uses: actions/cache/restore@v4
+        with:
+          path: |
+            ~/Library/Caches/pip
+            ~/Library/Caches/uv
+          key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
+          restore-keys: |
+            pyenv-darwin-${{ inputs.backend }}-
 
       - name: Build ${{ inputs.backend }}-darwin
         run: |
           make protogen-go
           BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
 
+      - name: ccache stats
+        if: inputs.backend == 'llama-cpp'
+        run: ccache -s
+
+      - name: Save ccache
+        if: inputs.backend == 'llama-cpp' && github.event_name != 'pull_request'
+        uses: actions/cache/save@v4
+        with:
+          path: ~/Library/Caches/ccache
+          key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
+
+      - name: Save Python wheel cache
+        if: inputs.lang == 'python' && github.event_name != 'pull_request' && steps.pyenv-cache.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v4
+        with:
+          path: |
+            ~/Library/Caches/pip
+            ~/Library/Caches/uv
+          key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
+
       - name: Upload ${{ inputs.backend }}.tar
         uses: actions/upload-artifact@v7
         with:

.github/workflows/gallery-agent.yaml

@@ -2,7 +2,7 @@ name: Gallery Agent
 on:
 
   schedule:
-    - cron: '0 */3 * * *'  # Run every 4 hours
+    - cron: '0 */12 * * *'  # Run every 4 hours
   workflow_dispatch:
     inputs:
       search_term:

.github/workflows/generate_grpc_cache.yaml

@@ -1,96 +0,0 @@
-name: 'generate and publish GRPC docker caches'
-
-on:
-  workflow_dispatch:
-
-  schedule:
-    # daily at midnight
-    - cron: '0 0 * * *'
-
-concurrency:
-  group: grpc-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
-
-jobs:
-  generate_caches:
-    if: github.repository == 'mudler/LocalAI'
-    strategy:
-      matrix:
-        include:
-          - grpc-base-image: ubuntu:24.04
-            runs-on: 'ubuntu-latest'
-            platforms: 'linux/amd64,linux/arm64'
-    runs-on: ${{matrix.runs-on}}
-    steps:
-      - name: Release space from worker
-        if: matrix.runs-on == 'ubuntu-latest'
-        run: |
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          df -h
-          echo
-          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
-          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
-          sudo rm -rf /usr/local/lib/android
-          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-          sudo rm -rf /usr/share/dotnet
-          sudo apt-get remove -y '^mono-.*' || true
-          sudo apt-get remove -y '^ghc-.*' || true
-          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-          sudo apt-get remove -y 'php.*' || true
-          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-          sudo apt-get remove -y '^google-.*' || true
-          sudo apt-get remove -y azure-cli || true
-          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-          sudo apt-get remove -y '^gfortran-.*' || true
-          sudo apt-get remove -y microsoft-edge-stable || true
-          sudo apt-get remove -y firefox || true
-          sudo apt-get remove -y powershell || true
-          sudo apt-get remove -y r-base-core || true
-          sudo apt-get autoremove -y
-          sudo apt-get clean
-          echo
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          sudo rm -rfv build || true
-          sudo rm -rf /usr/share/dotnet || true
-          sudo rm -rf /opt/ghc || true
-          sudo rm -rf "/usr/local/share/boost" || true
-          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
-          df -h
-
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@master
-        with:
-          platforms: all
-
-      - name: Set up Docker Buildx
-        id: buildx
-        uses: docker/setup-buildx-action@master
-
-      - name: Checkout
-        uses: actions/checkout@v6
-
-      - name: Cache GRPC
-        uses: docker/build-push-action@v7
-        with:
-          builder: ${{ steps.buildx.outputs.name }}
-          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
-          # This means that even the MAKEFLAGS have to be an EXACT match.
-          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
-          build-args: |
-            GRPC_BASE_IMAGE=${{ matrix.grpc-base-image }}
-            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
-            GRPC_VERSION=v1.65.0
-          context: .
-          file: ./Dockerfile
-          cache-to: type=gha,ignore-error=true
-          cache-from: type=gha
-          target: grpc
-          platforms: ${{ matrix.platforms }}
-          push: false

.github/workflows/generate_intel_image.yaml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       matrix:
         include:
-          - base-image: intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04
+          - base-image: intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04
             runs-on: 'arc-runner-set'
             platforms: 'linux/amd64'
     runs-on: ${{matrix.runs-on}}

.github/workflows/image-pr.yml

@@ -20,7 +20,6 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
-        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
       secrets:
@@ -60,15 +59,13 @@
               tag-latest: 'false'
               tag-suffix: '-hipblas'
               base-image: "rocm/dev-ubuntu-24.04:7.2.1"
-              grpc-base-image: "ubuntu:24.04"
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'sycl'
               platforms: 'linux/amd64'
               tag-latest: 'false'
-              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-              grpc-base-image: "ubuntu:24.04"
+              base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
               tag-suffix: 'sycl'
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"

.github/workflows/image.yml

@@ -25,7 +25,6 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
-        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
         ubuntu-codename: ${{ matrix.ubuntu-codename }}
@@ -42,12 +41,11 @@
               tag-latest: 'auto'
               tag-suffix: '-gpu-hipblas'
               base-image: "rocm/dev-ubuntu-24.04:7.2.1"
-              grpc-base-image: "ubuntu:24.04"
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-  
+
     core-image-build:
       if: github.repository == 'mudler/LocalAI'
       uses: ./.github/workflows/image_build.yml
@@ -60,7 +58,6 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
-        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         skip-drivers: ${{ matrix.skip-drivers }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
@@ -121,8 +118,7 @@
             - build-type: 'intel'
               platforms: 'linux/amd64'
               tag-latest: 'auto'
-              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-              grpc-base-image: "ubuntu:24.04"
+              base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
               tag-suffix: '-gpu-intel'
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
@@ -141,7 +137,6 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
-        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         skip-drivers: ${{ matrix.skip-drivers }}
         ubuntu-version: ${{ matrix.ubuntu-version }}

.github/workflows/image_build.yml

@@ -8,11 +8,6 @@ on:
         description: 'Base image'
         required: true
         type: string
-      grpc-base-image:
-        description: 'GRPC Base image, must be a compatible image with base-image'
-        required: false
-        default: ''
-        type: string
       build-type:
         description: 'Build type'
         default: ''
@@ -201,25 +196,19 @@ jobs:
         if: github.event_name != 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
-          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
-          # This means that even the MAKEFLAGS have to be an EXACT match.
-          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
-          # This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
           build-args: |
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
-            GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
-            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
-            GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
           context: .
           file: ./Dockerfile
-          cache-from: type=gha
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
+          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }},mode=max,ignore-error=true
           platforms: ${{ inputs.platforms }}
           push: ${{ github.event_name != 'pull_request' }}
           tags: ${{ steps.meta.outputs.tags }}
@@ -230,25 +219,18 @@ jobs:
         if: github.event_name == 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
-          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
-          # This means that even the MAKEFLAGS have to be an EXACT match.
-          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
-          # This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
           build-args: |
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
-            GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
-            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
-            GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
           context: .
           file: ./Dockerfile
-          cache-from: type=gha
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
           platforms: ${{ inputs.platforms }}
           #push: true
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/test-extra.yml

@@ -39,6 +39,8 @@ jobs:
       voxtral: ${{ steps.detect.outputs.voxtral }}
       kokoros: ${{ steps.detect.outputs.kokoros }}
       insightface: ${{ steps.detect.outputs.insightface }}
+      speaker-recognition: ${{ steps.detect.outputs.speaker-recognition }}
+      sherpa-onnx: ${{ steps.detect.outputs.sherpa-onnx }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -505,6 +507,72 @@ jobs:
       - name: Build llama-cpp backend image and run audio transcription gRPC e2e tests
         run: |
           make test-extra-backend-llama-cpp-transcription
+  # Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked LLM.
+  # Builds the sherpa-onnx Docker image, extracts the rootfs so the e2e suite
+  # can discover the backend binary + shared libs, downloads the three model
+  # bundles (silero-vad, omnilingual-asr, vits-ljs) and drives the realtime
+  # websocket spec end-to-end.
+  tests-sherpa-onnx-realtime:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25.4'
+      - name: Setup Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: '22'
+      - name: Build sherpa-onnx backend image and run realtime e2e tests
+        run: |
+          make test-extra-e2e-realtime-sherpa
+  # Streaming ASR via the sherpa-onnx online recognizer (zipformer
+  # transducer). Exercises both AudioTranscription (buffered) and
+  # AudioTranscriptionStream (real-time deltas) on the e2e-backends
+  # harness.
+  tests-sherpa-onnx-grpc-transcription:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25.4'
+      - name: Build sherpa-onnx backend image and run streaming ASR gRPC e2e tests
+        run: |
+          make test-extra-backend-sherpa-onnx-transcription
+  # VITS TTS via the sherpa-onnx backend. Drives both TTS (file write) and
+  # TTSStream (PCM chunks) on the e2e-backends harness.
+  tests-sherpa-onnx-grpc-tts:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25.4'
+      - name: Build sherpa-onnx backend image and run TTS gRPC e2e tests
+        run: |
+          make test-extra-backend-sherpa-onnx-tts
   tests-ik-llama-cpp-grpc:
     needs: detect-changes
     if: needs.detect-changes.outputs.ik-llama-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
@@ -778,3 +846,29 @@ jobs:
       - name: Build insightface backend image and run both model configurations
         run: |
           make test-extra-backend-insightface-all
+  tests-speaker-recognition-grpc:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.speaker-recognition == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends \
+              make build-essential curl ca-certificates git tar
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.0'
+      - name: Free disk space
+        run: |
+          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
+          df -h
+      - name: Build speaker-recognition backend image and run the ECAPA-TDNN configuration
+        run: |
+          make test-extra-backend-speaker-recognition-all

.github/workflows/test.yml

@@ -9,9 +9,6 @@ on:
     tags:
       - '*'
 
-env:
-  GRPC_VERSION: v1.65.0
-
 concurrency:
   group: ci-tests-${{ github.head_ref || github.ref }}-${{ github.repository }}
   cancel-in-progress: true
@@ -195,7 +192,7 @@ jobs:
         run: go version
       - name: Dependencies
         run: |
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus ffmpeg
           pip install --user --no-cache-dir grpcio-tools grpcio
       - name: Setup Node.js
         uses: actions/setup-node@v6

AGENTS.md

@@ -19,7 +19,8 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 |------|-------------|
 | [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md) | Policy for AI-assisted contributions — licensing, DCO, attribution |
 | [.agents/building-and-testing.md](.agents/building-and-testing.md) | Building the project, running tests, Docker builds for specific platforms |
-| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist |
+| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, manual eviction |
+| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist, including importer integration (the `/import-model` dropdown is server-driven from `GET /backends/known`) |
 | [.agents/coding-style.md](.agents/coding-style.md) | Code style, editorconfig, logging, documentation conventions |
 | [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
 | [.agents/vllm-backend.md](.agents/vllm-backend.md) | Working on the vLLM / vLLM-omni backends — native parsers, ChatDelta, CPU build, libnuma packaging, backend hooks |

Dockerfile

@@ -1,5 +1,4 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 ARG INTEL_BASE_IMAGE=${BASE_IMAGE}
 ARG UBUNTU_CODENAME=noble
 
@@ -149,6 +148,7 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
+            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/tinygrad
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/tinygrad backends/sherpa-onnx
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -394,7 +394,13 @@ protoc:
 .PHONY: protogen-go
 protogen-go: protoc install-go-tools
 	mkdir -p pkg/grpc/proto
-	./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
+	# install-go-tools writes protoc-gen-go and protoc-gen-go-grpc into
+	# $(shell go env GOPATH)/bin, which isn't on every dev's PATH. protoc
+	# resolves its code-gen plugins via PATH, so without this prefix the
+	# generate step fails with "protoc-gen-go: program not found". Prepend
+	# GOPATH/bin so the freshly-installed plugins win without requiring a
+	# shell-profile change.
+	PATH="$$(go env GOPATH)/bin:$$PATH" ./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
     backend/backend.proto
 
 core/config/inference_defaults.json: ## Fetch inference defaults from unsloth (only if missing)
@@ -435,6 +441,7 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/trl
 	$(MAKE) -C backend/python/tinygrad
 	$(MAKE) -C backend/python/insightface
+	$(MAKE) -C backend/python/speaker-recognition
 	$(MAKE) -C backend/rust/kokoros kokoros-grpc
 
 test-extra: prepare-test-extra
@@ -459,6 +466,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/trl test
 	$(MAKE) -C backend/python/tinygrad test
 	$(MAKE) -C backend/python/insightface test
+	$(MAKE) -C backend/python/speaker-recognition test
 	$(MAKE) -C backend/rust/kokoros test
 
 ##
@@ -621,6 +629,11 @@ test-extra-backend-tinygrad-all: \
 FACE_IMAGE_1_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
 FACE_IMAGE_2_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
 FACE_IMAGE_3_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/mask_white.jpg
+## Known spoof fixture used by the face_antispoof e2e cap. This is
+## upstream's own `image_F2.jpg` (Silent-Face repo, via yakhyo mirror)
+## — verified to classify as is_real=false with score < 0.05 on the
+## MiniFASNetV2 + MiniFASNetV1SE ensemble.
+FACE_SPOOF_IMAGE_URL ?= https://github.com/yakhyo/face-anti-spoofing/raw/main/assets/image_F2.jpg
 
 ## Host-side cache for the OpenCV Zoo face ONNX files used by the
 ## opencv e2e target. The backend image no longer bakes model weights —
@@ -644,6 +657,15 @@ INSIGHTFACE_BUFFALO_SC_DIR := /tmp/localai-insightface-buffalo-sc-cache
 INSIGHTFACE_BUFFALO_SC_URL := https://github.com/deepinsight/insightface/releases/download/v0.7/buffalo_sc.zip
 INSIGHTFACE_BUFFALO_SC_SHA := 57d31b56b6ffa911c8a73cfc1707c73cab76efe7f13b675a05223bf42de47c72
 
+## Silent-Face antispoofing (MiniFASNetV2 + MiniFASNetV1SE) — shared
+## between the buffalo_sc and opencv e2e targets. Both ONNX files are
+## ~1.7MB, Apache 2.0. URLs + SHAs mirror the gallery entries.
+INSIGHTFACE_ANTISPOOF_DIR := /tmp/localai-insightface-antispoof-cache
+INSIGHTFACE_ANTISPOOF_V2_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV2.onnx
+INSIGHTFACE_ANTISPOOF_V2_SHA := b32929adc2d9c34b9486f8c4c7bc97c1b69bc0ea9befefc380e4faae4e463907
+INSIGHTFACE_ANTISPOOF_V1SE_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV1SE.onnx
+INSIGHTFACE_ANTISPOOF_V1SE_SHA := ebab7f90c7833fbccd46d3a555410e78d969db5438e169b6524be444862b3676
+
 .PHONY: insightface-opencv-models
 insightface-opencv-models:
 	@mkdir -p $(INSIGHTFACE_OPENCV_DIR)
@@ -658,6 +680,20 @@ insightface-opencv-models:
 		echo "$(INSIGHTFACE_OPENCV_SFACE_SHA)  $(INSIGHTFACE_OPENCV_DIR)/sface.onnx" | sha256sum -c; \
 	fi
 
+.PHONY: insightface-antispoof-models
+insightface-antispoof-models:
+	@mkdir -p $(INSIGHTFACE_ANTISPOOF_DIR)
+	@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V2_SHA)" ]; then \
+		echo "Fetching MiniFASNetV2..."; \
+		curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx $(INSIGHTFACE_ANTISPOOF_V2_URL); \
+		echo "$(INSIGHTFACE_ANTISPOOF_V2_SHA)  $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx" | sha256sum -c; \
+	fi
+	@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA)" ]; then \
+		echo "Fetching MiniFASNetV1SE..."; \
+		curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx $(INSIGHTFACE_ANTISPOOF_V1SE_URL); \
+		echo "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA)  $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx" | sha256sum -c; \
+	fi
+
 .PHONY: insightface-buffalo-sc-models
 insightface-buffalo-sc-models:
 	@mkdir -p $(INSIGHTFACE_BUFFALO_SC_DIR)
@@ -680,14 +716,15 @@ insightface-buffalo-sc-models:
 ## the e2e suite drives LoadModel directly without going through
 ## LocalAI's gallery flow (which is what would normally populate
 ## ModelPath and in turn the engine's `_model_dir` option).
-test-extra-backend-insightface-buffalo-sc: docker-build-insightface insightface-buffalo-sc-models
+test-extra-backend-insightface-buffalo-sc: docker-build-insightface insightface-buffalo-sc-models insightface-antispoof-models
 	BACKEND_IMAGE=local-ai-backend:insightface \
 	BACKEND_TEST_MODEL_NAME=insightface-buffalo-sc \
-	BACKEND_TEST_OPTIONS=engine:insightface,model_pack:buffalo_sc,root:$(INSIGHTFACE_BUFFALO_SC_DIR) \
-	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify \
+	BACKEND_TEST_OPTIONS=engine:insightface,model_pack:buffalo_sc,root:$(INSIGHTFACE_BUFFALO_SC_DIR),antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
+	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
 	BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
 	BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
 	BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
+	BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
 	BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
 	$(MAKE) test-extra-backend
 
@@ -696,14 +733,15 @@ test-extra-backend-insightface-buffalo-sc: docker-build-insightface insightface-
 ## pre-fetched on the host via the insightface-opencv-models target and
 ## passed as absolute paths, since the e2e suite drives LoadModel
 ## directly without going through LocalAI's gallery flow.
-test-extra-backend-insightface-opencv: docker-build-insightface insightface-opencv-models
+test-extra-backend-insightface-opencv: docker-build-insightface insightface-opencv-models insightface-antispoof-models
 	BACKEND_IMAGE=local-ai-backend:insightface \
 	BACKEND_TEST_MODEL_NAME=insightface-opencv \
-	BACKEND_TEST_OPTIONS=engine:onnx_direct,detector_onnx:$(INSIGHTFACE_OPENCV_DIR)/yunet.onnx,recognizer_onnx:$(INSIGHTFACE_OPENCV_DIR)/sface.onnx \
-	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify \
+	BACKEND_TEST_OPTIONS=engine:onnx_direct,detector_onnx:$(INSIGHTFACE_OPENCV_DIR)/yunet.onnx,recognizer_onnx:$(INSIGHTFACE_OPENCV_DIR)/sface.onnx,antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
+	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
 	BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
 	BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
 	BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
+	BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
 	BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
 	$(MAKE) test-extra-backend
 
@@ -713,6 +751,79 @@ test-extra-backend-insightface-all: \
 	test-extra-backend-insightface-buffalo-sc \
 	test-extra-backend-insightface-opencv
 
+## speaker-recognition — voice (speaker) biometrics.
+##
+## Audio fixtures default to the speechbrain test samples served
+## straight from their GitHub repo — public, no auth needed, and they
+## ship as 16kHz mono WAV/FLAC which is exactly what the engine wants.
+## example{1,2,5} are three different speakers; the suite treats
+## example1 as the "same-image twin" probe (verify(clip, clip) must
+## return distance≈0) and the other two as cross-speaker ceilings.
+## Override with BACKEND_TEST_VOICE_AUDIO_{1,2,3}_FILE for offline runs.
+VOICE_AUDIO_1_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example1.wav
+VOICE_AUDIO_2_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example2.flac
+VOICE_AUDIO_3_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example5.wav
+
+## ECAPA-TDNN via SpeechBrain — default CI configuration. Auto-downloads
+## the checkpoint from HuggingFace on first LoadModel (bundled in the
+## backend image pip install). 192-d embeddings, cosine-distance based.
+## The e2e suite drives LoadModel directly so we don't rely on LocalAI's
+## gallery flow here.
+test-extra-backend-speaker-recognition-ecapa: docker-build-speaker-recognition
+	BACKEND_IMAGE=local-ai-backend:speaker-recognition \
+	BACKEND_TEST_MODEL_NAME=speechbrain/spkrec-ecapa-voxceleb \
+	BACKEND_TEST_OPTIONS=engine:speechbrain,source:speechbrain/spkrec-ecapa-voxceleb \
+	BACKEND_TEST_CAPS=health,load,voice_embed,voice_verify \
+	BACKEND_TEST_VOICE_AUDIO_1_URL=$(VOICE_AUDIO_1_URL) \
+	BACKEND_TEST_VOICE_AUDIO_2_URL=$(VOICE_AUDIO_2_URL) \
+	BACKEND_TEST_VOICE_AUDIO_3_URL=$(VOICE_AUDIO_3_URL) \
+	BACKEND_TEST_VOICE_VERIFY_DISTANCE_CEILING=0.4 \
+	$(MAKE) test-extra-backend
+
+## Aggregate — today there's only one voice config; the target exists
+## so the CI workflow matches the insightface-all naming convention and
+## can grow to include WeSpeaker / 3D-Speaker later.
+test-extra-backend-speaker-recognition-all: \
+	test-extra-backend-speaker-recognition-ecapa
+
+## Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked
+## LLM. Extracts the sherpa-onnx Docker image rootfs, downloads the three
+## gallery-referenced model bundles (silero-vad, omnilingual-asr, vits-ljs),
+## writes the corresponding model config YAMLs, and runs the realtime
+## websocket spec in tests/e2e with REALTIME_* env vars wiring the sherpa
+## slots into the pipeline. The LLM slot stays on the in-repo mock-backend
+## registered unconditionally by tests/e2e/e2e_suite_test.go. See
+## tests/e2e/run-realtime-sherpa.sh for the full orchestration.
+test-extra-e2e-realtime-sherpa: build-mock-backend docker-build-sherpa-onnx protogen-go react-ui
+	bash tests/e2e/run-realtime-sherpa.sh
+
+## Streaming ASR via the sherpa-onnx online recognizer. Uses the streaming
+## zipformer English model (encoder/decoder/joiner int8 + tokens) from the
+## sherpa-onnx gallery entry. Drives both AudioTranscription and
+## AudioTranscriptionStream via the e2e-backends gRPC harness; streaming
+## emits real partial deltas during decode. Each file is renamed on download
+## to the shape sherpa-onnx's online loader expects (encoder.int8.onnx etc.).
+test-extra-backend-sherpa-onnx-transcription: docker-build-sherpa-onnx
+	BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
+	BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#encoder.int8.onnx' \
+	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/decoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#decoder.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/joiner-epoch-99-avg-1-chunk-16-left-128.int8.onnx#joiner.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/tokens.txt' \
+	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
+	BACKEND_TEST_CAPS=health,load,transcription \
+	BACKEND_TEST_OPTIONS=subtype=online \
+	$(MAKE) test-extra-backend
+
+## VITS TTS via the sherpa-onnx backend. Pulls the individual files from
+## HuggingFace (the vits-ljs release tarball lives on the k2-fsa github
+## but is also mirrored as discrete files on HF). Exercises both
+## TTS (write-to-file) and TTSStream (PCM chunks + WAV header) via the
+## e2e-backends gRPC harness.
+test-extra-backend-sherpa-onnx-tts: docker-build-sherpa-onnx
+	BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
+	BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/vits-ljs.onnx#vits-ljs.onnx' \
+	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/tokens.txt|https://huggingface.co/csukuangfj/vits-ljs/resolve/main/lexicon.txt' \
+	BACKEND_TEST_CAPS=health,load,tts \
+	$(MAKE) test-extra-backend
+
 ## sglang mirrors the vllm setup: HuggingFace model id, same tiny Qwen,
 ## tool-call extraction via sglang's native qwen parser. CPU builds use
 ## sglang's upstream pyproject_cpu.toml recipe (see backend/python/sglang/install.sh).
@@ -772,7 +883,7 @@ docker-cuda12:
 
 docker-image-intel:
 	docker build \
-		--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04 \
+		--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04 \
 		--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
@@ -850,6 +961,7 @@ BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
 BACKEND_OPUS = opus|golang|.|false|true
+BACKEND_SHERPA_ONNX = sherpa-onnx|golang|.|false|true
 
 # Python backends with root context
 BACKEND_RERANKERS = rerankers|python|.|false|true
@@ -859,6 +971,7 @@ BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
 BACKEND_COQUI = coqui|python|.|false|true
 BACKEND_RFDETR = rfdetr|python|.|false|true
 BACKEND_INSIGHTFACE = insightface|python|.|false|true
+BACKEND_SPEAKER_RECOGNITION = speaker-recognition|python|.|false|true
 BACKEND_KITTEN_TTS = kitten-tts|python|.|false|true
 BACKEND_NEUTTS = neutts|python|.|false|true
 BACKEND_KOKORO = kokoro|python|.|false|true
@@ -931,6 +1044,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_INSIGHTFACE)))
+$(eval $(call generate-docker-build-target,$(BACKEND_SPEAKER_RECOGNITION)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KITTEN_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
@@ -960,12 +1074,13 @@ $(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP_QUANTIZATION)))
 $(eval $(call generate-docker-build-target,$(BACKEND_TINYGRAD)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KOKOROS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_SAM3_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX)))
 
 # Pattern rule for docker-save targets
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-insightface
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx
 
 ########################################################
 ### Mock Backend for E2E Tests

README.md

@@ -149,6 +149,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
 
 ## Latest News
 
+- **April 2026**: [Voice recognition](https://github.com/mudler/LocalAI/pull/9500), [Face recognition, identification & liveness detection](https://github.com/mudler/LocalAI/pull/9480), [Ollama API compatibility](https://github.com/mudler/LocalAI/pull/9284), [Video generation in stable-diffusion.ggml](https://github.com/mudler/LocalAI/pull/9420), [Backend versioning with auto-upgrade](https://github.com/mudler/LocalAI/pull/9315), [Pin models & load-on-demand toggle](https://github.com/mudler/LocalAI/pull/9309), [Universal model importer](https://github.com/mudler/LocalAI/pull/9466), new backends: [sglang](https://github.com/mudler/LocalAI/pull/9359), [ik-llama-cpp](https://github.com/mudler/LocalAI/pull/9326), [TurboQuant](https://github.com/mudler/LocalAI/pull/9355), [sam.cpp](https://github.com/mudler/LocalAI/pull/9288), [Kokoros](https://github.com/mudler/LocalAI/pull/9212), [qwen3tts.cpp](https://github.com/mudler/LocalAI/pull/9316), [tinygrad multimodal](https://github.com/mudler/LocalAI/pull/9364)
 - **March 2026**: [Agent management](https://github.com/mudler/LocalAI/pull/8820), [New React UI](https://github.com/mudler/LocalAI/pull/8772), [WebRTC](https://github.com/mudler/LocalAI/pull/8790), [MLX-distributed via P2P and RDMA](https://github.com/mudler/LocalAI/pull/8801), [MCP Apps, MCP Client-side](https://github.com/mudler/LocalAI/pull/8947)
 - **February 2026**: [Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245), [ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396)
 - **January 2026**: **LocalAI 3.10.0** — Anthropic API support, Open Responses API, video & image generation (LTX-2), unified GPU backends, tool streaming, Moonshine, Pocket-TTS. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0)

backend/Dockerfile.golang

@@ -147,6 +147,7 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
+            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \

backend/Dockerfile.ik-llama-cpp

@@ -204,6 +204,7 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
+            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \

backend/Dockerfile.llama-cpp

@@ -206,6 +206,7 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
+            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \

backend/Dockerfile.python

@@ -162,6 +162,7 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
+            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \
@@ -202,6 +203,13 @@ COPY scripts/build/package-gpu-libs.sh /package-gpu-libs.sh
 ARG FROM_SOURCE=""
 ENV FROM_SOURCE=${FROM_SOURCE}
 
+# Cache-buster for the per-backend `make` step. Most Python backends list
+# unpinned deps (torch, transformers, vllm, ...), so a warm registry cache
+# would otherwise freeze upstream versions indefinitely. CI passes a value
+# that rolls weekly so the install layer is rebuilt at most once per week
+# and picks up newer wheels from PyPI / nightly indexes.
+ARG DEPS_REFRESH=initial
+
 RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
 
 # Package GPU libraries into the backend's lib directory
@@ -216,4 +224,4 @@ RUN if [ -f "/${BACKEND}/package.sh" ]; then \
 
 FROM scratch
 ARG BACKEND=rerankers
-COPY --from=builder /${BACKEND}/ /
+COPY --from=builder /${BACKEND}/ /

backend/Dockerfile.turboquant

@@ -204,6 +204,7 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
+            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \

backend/backend.proto

@@ -26,6 +26,9 @@ service Backend {
   rpc Detect(DetectOptions) returns (DetectResponse) {}
   rpc FaceVerify(FaceVerifyRequest) returns (FaceVerifyResponse) {}
   rpc FaceAnalyze(FaceAnalyzeRequest) returns (FaceAnalyzeResponse) {}
+  rpc VoiceVerify(VoiceVerifyRequest) returns (VoiceVerifyResponse) {}
+  rpc VoiceAnalyze(VoiceAnalyzeRequest) returns (VoiceAnalyzeResponse) {}
+  rpc VoiceEmbed(VoiceEmbedRequest) returns (VoiceEmbedResponse) {}
 
   rpc StoresSet(StoresSetOptions) returns (Result) {}
   rpc StoresDelete(StoresDeleteOptions) returns (Result) {}
@@ -490,7 +493,7 @@ message FaceVerifyRequest {
   string img1 = 1;              // base64-encoded image
   string img2 = 2;              // base64-encoded image
   float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
-  bool   anti_spoofing = 4;     // reserved for future MiniFASNet bolt-on
+  bool   anti_spoofing = 4;     // run MiniFASNet liveness on each image; failed liveness forces verified=false
 }
 
 message FaceVerifyResponse {
@@ -502,6 +505,10 @@ message FaceVerifyResponse {
   FacialArea img1_area = 6;
   FacialArea img2_area = 7;
   float      processing_time_ms = 8;
+  bool       img1_is_real = 9;          // anti-spoofing result when enabled
+  float      img1_antispoof_score = 10;
+  bool       img2_is_real = 11;
+  float      img2_antispoof_score = 12;
 }
 
 message FaceAnalyzeRequest {
@@ -528,6 +535,57 @@ message FaceAnalyzeResponse {
   repeated FaceAnalysis faces = 1;
 }
 
+// --- Voice (speaker) recognition messages ---
+//
+// Analogous to the Face* messages above, but for speaker biometrics.
+// Audio fields accept a filesystem path (same convention as
+// TranscriptRequest.dst). The HTTP layer materialises base64 / URL /
+// data-URI inputs to a temp file before calling the gRPC backend.
+
+message VoiceVerifyRequest {
+  string audio1 = 1;            // path to first audio clip
+  string audio2 = 2;            // path to second audio clip
+  float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
+  bool   anti_spoofing = 4;     // reserved for future AASIST bolt-on
+}
+
+message VoiceVerifyResponse {
+  bool   verified = 1;
+  float  distance = 2;          // 1 - cosine_similarity
+  float  threshold = 3;
+  float  confidence = 4;        // 0-100
+  string model = 5;             // e.g. "speechbrain/spkrec-ecapa-voxceleb"
+  float  processing_time_ms = 6;
+}
+
+message VoiceAnalyzeRequest {
+  string          audio = 1;        // path to audio clip
+  repeated string actions = 2;      // subset of ["age","gender","emotion"]; empty = all-supported
+}
+
+message VoiceAnalysis {
+  float              start = 1;          // segment start time in seconds (0 if single-utterance)
+  float              end = 2;            // segment end time in seconds
+  float              age = 3;
+  string             dominant_gender = 4;
+  map<string, float> gender = 5;
+  string             dominant_emotion = 6;
+  map<string, float> emotion = 7;
+}
+
+message VoiceAnalyzeResponse {
+  repeated VoiceAnalysis segments = 1;
+}
+
+message VoiceEmbedRequest {
+  string audio = 1;              // path to audio clip
+}
+
+message VoiceEmbedResponse {
+  repeated float embedding = 1;
+  string         model = 2;
+}
+
 message ToolFormatMarkers {
   string format_type = 1;           // "json_native", "tag_with_json", "tag_with_tagged"
 

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=d4824131580b94ffa7b0e91c955e2b237c2fe16e
+IK_LLAMA_VERSION?=3a945af45d45936341a45bbf7deda56776a4af26
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/ik-llama-cpp/grpc-server.cpp

@@ -686,7 +686,16 @@ struct llama_server_context
         slot->sparams.mirostat_eta      = json_value(data, "mirostat_eta",      default_sparams.mirostat_eta);
         slot->params.n_keep             = json_value(data, "n_keep",            slot->params.n_keep);
         slot->sparams.seed               = json_value(data, "seed",              default_sparams.seed);
-        slot->sparams.grammar           = json_value(data, "grammar",           default_sparams.grammar);
+        {
+            // upstream changed common_params_sampling::grammar from std::string to
+            // the common_grammar struct (type + grammar). The incoming JSON still
+            // carries a plain string, so build the user-provided grammar here and
+            // fall back to the server default when the request omits it.
+            std::string grammar_str = json_value(data, "grammar", std::string());
+            slot->sparams.grammar = grammar_str.empty()
+                ? default_sparams.grammar
+                : common_grammar{COMMON_GRAMMAR_TYPE_USER, std::move(grammar_str)};
+        }
         slot->sparams.n_probs           = json_value(data, "n_probs",           default_sparams.n_probs);
         slot->sparams.min_keep          = json_value(data, "min_keep",          default_sparams.min_keep);
         slot->sparams.grammar_triggers = grammar_triggers;
@@ -1232,7 +1241,7 @@ struct llama_server_context
              //      {"logit_bias",        slot.sparams.logit_bias},
             {"n_probs",           slot.sparams.n_probs},
             {"min_keep",          slot.sparams.min_keep},
-            {"grammar",           slot.sparams.grammar},
+            {"grammar",           slot.sparams.grammar.grammar},
             {"samplers",          samplers}
         };
     }

backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch

@@ -0,0 +1,11 @@
+--- a/examples/llava/clip.cpp
++++ b/examples/llava/clip.cpp
+@@ -2494,7 +2494,7 @@
+             }
+             new_data = work.data();
+
+-            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr);
++            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr, nullptr);
+         } else {
+             new_type = cur->type;
+             new_data = cur->data;

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=5a4cd6741fc33227cdacb329f355ab21f8481de2
+LLAMA_VERSION?=f53577432541bb9edc1588c4ef45c66bf07e4468
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/grpc-server.cpp

@@ -10,6 +10,14 @@
 #include "server-task.cpp"
 #include "server-queue.cpp"
 #include "server-common.cpp"
+// server-chat.cpp exists only in llama.cpp after the upstream refactor that
+// split OAI/Anthropic/Responses/transcription conversion helpers out of
+// server-common.cpp. When present, server-context.cpp and server-task.cpp
+// above call into it, so we must pull its definitions into this TU or the
+// link fails. __has_include keeps the source compatible with older pins.
+#if __has_include("server-chat.cpp")
+#include "server-chat.cpp"
+#endif
 #include "server-context.cpp"
 
 // LocalAI
@@ -634,6 +642,21 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
                 params.no_op_offload = false;
             }
+        } else if (!strcmp(optname, "split_mode") || !strcmp(optname, "sm")) {
+            // Accepts: none | layer | row | tensor (the latter requires a llama.cpp build
+            // that includes ggml-org/llama.cpp#19378, FlashAttention enabled, and KV-cache
+            // quantization disabled).
+            if (optval != NULL) {
+                if (optval_str == "none") {
+                    params.split_mode = LLAMA_SPLIT_MODE_NONE;
+                } else if (optval_str == "layer") {
+                    params.split_mode = LLAMA_SPLIT_MODE_LAYER;
+                } else if (optval_str == "row") {
+                    params.split_mode = LLAMA_SPLIT_MODE_ROW;
+                } else if (optval_str == "tensor") {
+                    params.split_mode = LLAMA_SPLIT_MODE_TENSOR;
+                }
+            }
         } else if (!strcmp(optname, "kv_unified") || !strcmp(optname, "unified_kv")) {
             if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
                 params.kv_unified = true;

backend/cpp/turboquant/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-TURBOQUANT_VERSION?=627ebbc6e27727bd4f65422d8aa60b13404993c8
+TURBOQUANT_VERSION?=11a241d0db78a68e0a5b99fe6f36de6683100f6a
 LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
 
 CMAKE_ARGS?=

backend/go/local-store/store.go

@@ -4,7 +4,6 @@ package main
 // It is meant to be used by the main executable that is the server for the specific backend type (falcon, gpt3, etc)
 import (
 	"container/heap"
-	"errors"
 	"fmt"
 	"math"
 	"slices"
@@ -100,9 +99,16 @@ func sortIntoKeySlicese(keys []*pb.StoresKey) [][]float32 {
 }
 
 func (s *Store) Load(opts *pb.ModelOptions) error {
-	if opts.Model != "" {
-		return errors.New("not implemented")
-	}
+	// local-store is an in-memory vector store with no on-disk artefact to
+	// load — opts.Model is just a namespace identifier. The old `!= ""` guard
+	// rejected any non-empty model name with "not implemented", which broke
+	// callers that pass a namespace to isolate embedding spaces (face vs.
+	// voice biometrics both go through local-store but need distinct stores
+	// so ArcFace 512-D and ECAPA-TDNN 192-D don't collide). Namespace
+	// isolation is already handled upstream: ModelLoader spawns a fresh
+	// local-store process per (backend, model) tuple, so each namespace is
+	// its own Store{} instance. Nothing to do here beyond accepting the load.
+	_ = opts
 	return nil
 }
 

backend/go/sherpa-onnx/.gitignore

@@ -0,0 +1,11 @@
+.cache/
+sources/
+build*/
+package/
+backend-assets/
+sherpa-onnx
+*.so
+compile_commands.json
+sherpa-onnx-whisper-*
+vits-ljs/
+streaming-zipformer-en/

backend/go/sherpa-onnx/Makefile

@@ -0,0 +1,120 @@
+CURRENT_DIR=$(abspath ./)
+GOCMD=go
+
+ONNX_VERSION?=1.24.4
+# v1.12.39 — includes upstream's onnxruntime 1.24.4 bump (#3501). Earlier
+# pinned commits only support onnxruntime 1.23.2, which has no CUDA 13
+# pre-built tarball, blocking the -gpu-nvidia-cuda-13 build matrix entry.
+SHERPA_COMMIT?=7288d15e3e31a7bd589b2ba88828d521e7a6b140
+ONNX_ARCH?=x64
+ONNX_OS?=linux
+
+ifneq (,$(findstring aarch64,$(shell uname -m)))
+	ONNX_ARCH=aarch64
+endif
+
+ifeq ($(OS),Darwin)
+	ONNX_OS=osx
+	ifneq (,$(findstring aarch64,$(shell uname -m)))
+		ONNX_ARCH=arm64
+	else ifneq (,$(findstring arm64,$(shell uname -m)))
+		ONNX_ARCH=arm64
+	else
+		ONNX_ARCH=x86_64
+	endif
+endif
+
+# Upstream onnxruntime ships CUDA 12 and CUDA 13 variants under different
+# names: -gpu-<ver>.tgz for CUDA 12, -gpu_cuda13-<ver>.tgz for CUDA 13
+# (note underscore vs dash). CUDA 13 tarballs only exist from 1.24.x onward.
+ifeq ($(BUILD_TYPE),cublas)
+	SHERPA_GPU=ON
+	ONNX_PROVIDER=cuda
+	ifeq ($(CUDA_MAJOR_VERSION),13)
+		ONNX_VARIANT=-gpu_cuda13
+	else
+		ONNX_VARIANT=-gpu
+	endif
+else
+	ONNX_VARIANT=
+	SHERPA_GPU=OFF
+	ONNX_PROVIDER=cpu
+endif
+
+JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+sources/onnxruntime:
+	mkdir -p sources/onnxruntime
+	curl -L https://github.com/microsoft/onnxruntime/releases/download/v$(ONNX_VERSION)/onnxruntime-$(ONNX_OS)-$(ONNX_ARCH)$(ONNX_VARIANT)-$(ONNX_VERSION).tgz \
+	  -o sources/onnxruntime/onnxruntime.tgz
+	cd sources/onnxruntime && tar -xf onnxruntime.tgz --strip-components=1 && rm onnxruntime.tgz
+
+sources/sherpa-onnx: sources/onnxruntime
+	git clone https://github.com/k2-fsa/sherpa-onnx.git sources/sherpa-onnx
+	cd sources/sherpa-onnx && git checkout $(SHERPA_COMMIT)
+	mkdir -p sources/sherpa-onnx/build
+	# sherpa-onnx's cmake detects a pre-installed onnxruntime via the
+	# SHERPA_ONNXRUNTIME_{INCLUDE,LIB}_DIR env vars (not via -D flags).
+	# Point them at our locally-downloaded Microsoft tarball — without
+	# this, sherpa-onnx falls through to download_onnxruntime() which
+	# fetches from csukuangfj/onnxruntime-libs. For the GPU 1.24.4
+	# build that release mirror publishes `-patched.zip` instead of the
+	# expected `.tgz`, so the download 404s and the build fails.
+	cd sources/sherpa-onnx/build && \
+	SHERPA_ONNXRUNTIME_INCLUDE_DIR=$(CURRENT_DIR)/sources/onnxruntime/include \
+	SHERPA_ONNXRUNTIME_LIB_DIR=$(CURRENT_DIR)/sources/onnxruntime/lib \
+	cmake \
+	  -DCMAKE_BUILD_TYPE=Release \
+	  -DCMAKE_C_FLAGS="-Wno-error=format-security" \
+	  -DCMAKE_CXX_FLAGS="-Wno-error=format-security" \
+	  -DSHERPA_ONNX_ENABLE_GPU=$(SHERPA_GPU) \
+	  -DSHERPA_ONNX_ENABLE_TTS=ON \
+	  -DSHERPA_ONNX_ENABLE_BINARY=OFF \
+	  -DSHERPA_ONNX_ENABLE_PYTHON=OFF \
+	  -DSHERPA_ONNX_ENABLE_TESTS=OFF \
+	  -DSHERPA_ONNX_ENABLE_C_API=ON \
+	  -DBUILD_SHARED_LIBS=ON \
+	  -DSHERPA_ONNX_USE_PRE_INSTALLED_ONNXRUNTIME_IF_AVAILABLE=ON \
+	  ..
+	cd sources/sherpa-onnx/build && make -j$(JOBS)
+
+backend-assets/lib: sources/sherpa-onnx sources/onnxruntime
+	mkdir -p backend-assets/lib
+	cp -rfLv sources/onnxruntime/lib/* backend-assets/lib/
+	cp -rfLv sources/sherpa-onnx/build/lib/*.so* backend-assets/lib/ 2>/dev/null || true
+	cp -rfLv sources/sherpa-onnx/build/lib/*.dylib backend-assets/lib/ 2>/dev/null || true
+
+# libsherpa-shim wraps sherpa-onnx's nested config structs and TTS
+# callback plumbing behind a purego-friendly API: opaque handles plus
+# fixed-signature setters/getters/trampoline. Plain C compile — no cgo.
+SHIM_EXT=so
+ifeq ($(OS),Darwin)
+	SHIM_EXT=dylib
+endif
+
+backend-assets/lib/libsherpa-shim.$(SHIM_EXT): csrc/shim.c csrc/shim.h backend-assets/lib
+	$(CC) -shared -fPIC -O2 \
+	  -I$(CURRENT_DIR)/sources/sherpa-onnx/sherpa-onnx/c-api \
+	  -o $@ csrc/shim.c \
+	  -L$(CURRENT_DIR)/backend-assets/lib \
+	  -lsherpa-onnx-c-api \
+	  -Wl,-rpath,'$$ORIGIN'
+
+sherpa-onnx: backend-assets/lib backend-assets/lib/libsherpa-shim.$(SHIM_EXT)
+	CGO_ENABLED=0 $(GOCMD) build \
+	  -ldflags "$(LD_FLAGS) -X main.onnxProvider=$(ONNX_PROVIDER)" \
+	  -tags "$(GO_TAGS)" -o sherpa-onnx ./
+
+package:
+	bash package.sh
+
+build: sherpa-onnx package
+
+clean:
+	rm -rf sherpa-onnx sources/ backend-assets/ package/ vits-ljs/ sherpa-onnx-whisper-*/
+
+test: sherpa-onnx
+	LD_LIBRARY_PATH=$(CURRENT_DIR)/backend-assets/lib \
+	bash test.sh
+
+.PHONY: build package clean test

backend/go/sherpa-onnx/backend_test.go

@@ -0,0 +1,169 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestSherpaBackend(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Sherpa-ONNX Backend Suite")
+}
+
+// Load libsherpa-shim + libsherpa-onnx-c-api via purego before any spec
+// runs — otherwise any Load/TTS/VAD/AudioTranscription call hits a nil
+// function pointer. LD_LIBRARY_PATH must contain the directory holding
+// both .so files; test.sh sets this.
+var _ = BeforeSuite(func() {
+	Expect(loadSherpaLibs()).To(Succeed())
+})
+
+var _ = Describe("Sherpa-ONNX", func() {
+	Context("lifecycle", func() {
+		It("is locking (C API is not thread safe)", func() {
+			Expect((&SherpaBackend{}).Locking()).To(BeTrue())
+		})
+
+		It("errors loading a non-existent model", func() {
+			tmpDir, err := os.MkdirTemp("", "sherpa-test-nonexistent")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpDir)
+
+			err = (&SherpaBackend{}).Load(&pb.ModelOptions{
+				ModelFile: filepath.Join(tmpDir, "non-existent-model.onnx"),
+			})
+			Expect(err).To(HaveOccurred())
+		})
+
+		It("errors loading a non-existent ASR model", func() {
+			tmpDir, err := os.MkdirTemp("", "sherpa-test-asr")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpDir)
+
+			err = (&SherpaBackend{}).Load(&pb.ModelOptions{
+				ModelFile: filepath.Join(tmpDir, "model.onnx"),
+				Type:      "asr",
+			})
+			Expect(err).To(HaveOccurred())
+		})
+
+		It("dispatches Load by Type", func() {
+			tmpDir, err := os.MkdirTemp("", "sherpa-test-dispatch")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpDir)
+
+			modelFile := filepath.Join(tmpDir, "model.onnx")
+			for _, typ := range []string{"", "asr", "vad"} {
+				err := (&SherpaBackend{}).Load(&pb.ModelOptions{ModelFile: modelFile, Type: typ})
+				Expect(err).To(HaveOccurred(), "Type=%q", typ)
+			}
+		})
+	})
+
+	Context("method errors without loaded model", func() {
+		It("rejects TTS", func() {
+			tmpDir, err := os.MkdirTemp("", "sherpa-test-tts")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpDir)
+
+			err = (&SherpaBackend{}).TTS(&pb.TTSRequest{
+				Text: "should fail — no model loaded",
+				Dst:  filepath.Join(tmpDir, "output.wav"),
+			})
+			Expect(err).To(HaveOccurred())
+		})
+
+		It("rejects AudioTranscription", func() {
+			_, err := (&SherpaBackend{}).AudioTranscription(&pb.TranscriptRequest{
+				Dst: "/tmp/nonexistent.wav",
+			})
+			Expect(err).To(HaveOccurred())
+		})
+
+		It("rejects VAD", func() {
+			_, err := (&SherpaBackend{}).VAD(&pb.VADRequest{
+				Audio: []float32{0.1, 0.2, 0.3},
+			})
+			Expect(err).To(HaveOccurred())
+		})
+	})
+
+	Context("type detection", func() {
+		DescribeTable("isASRType",
+			func(input string, want bool) {
+				Expect(isASRType(input)).To(Equal(want))
+			},
+			Entry("asr", "asr", true),
+			Entry("ASR", "ASR", true),
+			Entry("Asr", "Asr", true),
+			Entry("transcription", "transcription", true),
+			Entry("Transcription", "Transcription", true),
+			Entry("transcribe", "transcribe", true),
+			Entry("Transcribe", "Transcribe", true),
+			Entry("tts", "tts", false),
+			Entry("empty", "", false),
+			Entry("other", "other", false),
+			Entry("vad", "vad", false),
+		)
+
+		DescribeTable("isVADType",
+			func(input string, want bool) {
+				Expect(isVADType(input)).To(Equal(want))
+			},
+			Entry("vad", "vad", true),
+			Entry("VAD", "VAD", true),
+			Entry("Vad", "Vad", true),
+			Entry("asr", "asr", false),
+			Entry("tts", "tts", false),
+			Entry("empty", "", false),
+			Entry("other", "other", false),
+		)
+	})
+
+	Context("option parsing", func() {
+		It("parses float options with fallback on bad input", func() {
+			opts := &pb.ModelOptions{Options: []string{
+				"vad.threshold=0.3",
+				"tts.length_scale=1.25",
+				"bad.number=not-a-float",
+			}}
+			Expect(findOptionFloat(opts, "vad.threshold=", 0.5)).To(BeNumerically("~", 0.3, 1e-6))
+			Expect(findOptionFloat(opts, "tts.length_scale=", 1.0)).To(BeNumerically("~", 1.25, 1e-6))
+			Expect(findOptionFloat(opts, "missing.key=", 0.7)).To(BeNumerically("~", 0.7, 1e-6))
+			Expect(findOptionFloat(opts, "bad.number=", 9.9)).To(BeNumerically("~", 9.9, 1e-6))
+		})
+
+		It("parses int options with fallback on bad input", func() {
+			opts := &pb.ModelOptions{Options: []string{
+				"asr.sample_rate=22050",
+				"online.chunk_samples=800",
+				"bad.int=4.2",
+			}}
+			Expect(findOptionInt(opts, "asr.sample_rate=", 16000)).To(Equal(int32(22050)))
+			Expect(findOptionInt(opts, "online.chunk_samples=", 1600)).To(Equal(int32(800)))
+			Expect(findOptionInt(opts, "missing.key=", 42)).To(Equal(int32(42)))
+			Expect(findOptionInt(opts, "bad.int=", 100)).To(Equal(int32(100)))
+		})
+
+		It("parses bool options (0/1, true/false, yes/no, on/off)", func() {
+			opts := &pb.ModelOptions{Options: []string{
+				"online.enable_endpoint=0",
+				"asr.sense_voice.use_itn=True",
+				"feature.on=yes",
+				"feature.off=Off",
+				"feature.bad=maybe",
+			}}
+			Expect(findOptionBool(opts, "online.enable_endpoint=", 1)).To(Equal(int32(0)))
+			Expect(findOptionBool(opts, "asr.sense_voice.use_itn=", 0)).To(Equal(int32(1)))
+			Expect(findOptionBool(opts, "feature.on=", 0)).To(Equal(int32(1)))
+			Expect(findOptionBool(opts, "feature.off=", 1)).To(Equal(int32(0)))
+			Expect(findOptionBool(opts, "feature.bad=", 1)).To(Equal(int32(1)))
+			Expect(findOptionBool(opts, "missing.key=", 1)).To(Equal(int32(1)))
+		})
+	})
+})

backend/go/sherpa-onnx/csrc/shim.c

@@ -0,0 +1,325 @@
+#include "shim.h"
+#include "c-api.h"
+
+#include <stdlib.h>
+#include <string.h>
+
+// Replace the char* field pointed to by `slot` with a strdup of `s`
+// (or NULL if s is NULL). Frees any prior value. Silently no-ops when
+// strdup fails — the caller will see a Create* failure downstream.
+static void shim_set_str(const char **slot, const char *s) {
+    free((char *)*slot);
+    *slot = s ? strdup(s) : NULL;
+}
+
+// ==================================================================
+// VAD config
+// ==================================================================
+
+void *sherpa_shim_vad_config_new(void) {
+    return calloc(1, sizeof(SherpaOnnxVadModelConfig));
+}
+
+void sherpa_shim_vad_config_free(void *h) {
+    if (!h) return;
+    SherpaOnnxVadModelConfig *c = (SherpaOnnxVadModelConfig *)h;
+    free((char *)c->silero_vad.model);
+    free((char *)c->provider);
+    free(c);
+}
+
+void sherpa_shim_vad_config_set_silero_model(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxVadModelConfig *)h)->silero_vad.model, v);
+}
+void sherpa_shim_vad_config_set_silero_threshold(void *h, float v) {
+    ((SherpaOnnxVadModelConfig *)h)->silero_vad.threshold = v;
+}
+void sherpa_shim_vad_config_set_silero_min_silence_duration(void *h, float v) {
+    ((SherpaOnnxVadModelConfig *)h)->silero_vad.min_silence_duration = v;
+}
+void sherpa_shim_vad_config_set_silero_min_speech_duration(void *h, float v) {
+    ((SherpaOnnxVadModelConfig *)h)->silero_vad.min_speech_duration = v;
+}
+void sherpa_shim_vad_config_set_silero_window_size(void *h, int32_t v) {
+    ((SherpaOnnxVadModelConfig *)h)->silero_vad.window_size = v;
+}
+void sherpa_shim_vad_config_set_silero_max_speech_duration(void *h, float v) {
+    ((SherpaOnnxVadModelConfig *)h)->silero_vad.max_speech_duration = v;
+}
+void sherpa_shim_vad_config_set_sample_rate(void *h, int32_t v) {
+    ((SherpaOnnxVadModelConfig *)h)->sample_rate = v;
+}
+void sherpa_shim_vad_config_set_num_threads(void *h, int32_t v) {
+    ((SherpaOnnxVadModelConfig *)h)->num_threads = v;
+}
+void sherpa_shim_vad_config_set_provider(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxVadModelConfig *)h)->provider, v);
+}
+void sherpa_shim_vad_config_set_debug(void *h, int32_t v) {
+    ((SherpaOnnxVadModelConfig *)h)->debug = v;
+}
+
+void *sherpa_shim_create_vad(void *h, float buffer_size_seconds) {
+    return (void *)SherpaOnnxCreateVoiceActivityDetector(
+        (const SherpaOnnxVadModelConfig *)h, buffer_size_seconds);
+}
+
+// ==================================================================
+// Offline TTS config (VITS)
+// ==================================================================
+
+void *sherpa_shim_tts_config_new(void) {
+    return calloc(1, sizeof(SherpaOnnxOfflineTtsConfig));
+}
+
+void sherpa_shim_tts_config_free(void *h) {
+    if (!h) return;
+    SherpaOnnxOfflineTtsConfig *c = (SherpaOnnxOfflineTtsConfig *)h;
+    free((char *)c->model.vits.model);
+    free((char *)c->model.vits.tokens);
+    free((char *)c->model.vits.lexicon);
+    free((char *)c->model.vits.data_dir);
+    free((char *)c->model.provider);
+    free(c);
+}
+
+void sherpa_shim_tts_config_set_vits_model(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.model, v);
+}
+void sherpa_shim_tts_config_set_vits_tokens(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.tokens, v);
+}
+void sherpa_shim_tts_config_set_vits_lexicon(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.lexicon, v);
+}
+void sherpa_shim_tts_config_set_vits_data_dir(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.data_dir, v);
+}
+void sherpa_shim_tts_config_set_vits_noise_scale(void *h, float v) {
+    ((SherpaOnnxOfflineTtsConfig *)h)->model.vits.noise_scale = v;
+}
+void sherpa_shim_tts_config_set_vits_noise_scale_w(void *h, float v) {
+    ((SherpaOnnxOfflineTtsConfig *)h)->model.vits.noise_scale_w = v;
+}
+void sherpa_shim_tts_config_set_vits_length_scale(void *h, float v) {
+    ((SherpaOnnxOfflineTtsConfig *)h)->model.vits.length_scale = v;
+}
+void sherpa_shim_tts_config_set_num_threads(void *h, int32_t v) {
+    ((SherpaOnnxOfflineTtsConfig *)h)->model.num_threads = v;
+}
+void sherpa_shim_tts_config_set_debug(void *h, int32_t v) {
+    ((SherpaOnnxOfflineTtsConfig *)h)->model.debug = v;
+}
+void sherpa_shim_tts_config_set_provider(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.provider, v);
+}
+void sherpa_shim_tts_config_set_max_num_sentences(void *h, int32_t v) {
+    ((SherpaOnnxOfflineTtsConfig *)h)->max_num_sentences = v;
+}
+
+void *sherpa_shim_create_offline_tts(void *h) {
+    return (void *)SherpaOnnxCreateOfflineTts(
+        (const SherpaOnnxOfflineTtsConfig *)h);
+}
+
+// ==================================================================
+// Offline recognizer config
+// ==================================================================
+
+void *sherpa_shim_offline_recog_config_new(void) {
+    return calloc(1, sizeof(SherpaOnnxOfflineRecognizerConfig));
+}
+
+void sherpa_shim_offline_recog_config_free(void *h) {
+    if (!h) return;
+    SherpaOnnxOfflineRecognizerConfig *c = (SherpaOnnxOfflineRecognizerConfig *)h;
+    free((char *)c->model_config.provider);
+    free((char *)c->model_config.tokens);
+    free((char *)c->model_config.whisper.encoder);
+    free((char *)c->model_config.whisper.decoder);
+    free((char *)c->model_config.whisper.language);
+    free((char *)c->model_config.whisper.task);
+    free((char *)c->model_config.paraformer.model);
+    free((char *)c->model_config.sense_voice.model);
+    free((char *)c->model_config.sense_voice.language);
+    free((char *)c->model_config.omnilingual.model);
+    free((char *)c->decoding_method);
+    free(c);
+}
+
+void sherpa_shim_offline_recog_config_set_num_threads(void *h, int32_t v) {
+    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.num_threads = v;
+}
+void sherpa_shim_offline_recog_config_set_debug(void *h, int32_t v) {
+    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.debug = v;
+}
+void sherpa_shim_offline_recog_config_set_provider(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.provider, v);
+}
+void sherpa_shim_offline_recog_config_set_tokens(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.tokens, v);
+}
+void sherpa_shim_offline_recog_config_set_feat_sample_rate(void *h, int32_t v) {
+    ((SherpaOnnxOfflineRecognizerConfig *)h)->feat_config.sample_rate = v;
+}
+void sherpa_shim_offline_recog_config_set_feat_feature_dim(void *h, int32_t v) {
+    ((SherpaOnnxOfflineRecognizerConfig *)h)->feat_config.feature_dim = v;
+}
+void sherpa_shim_offline_recog_config_set_decoding_method(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->decoding_method, v);
+}
+void sherpa_shim_offline_recog_config_set_whisper_encoder(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.encoder, v);
+}
+void sherpa_shim_offline_recog_config_set_whisper_decoder(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.decoder, v);
+}
+void sherpa_shim_offline_recog_config_set_whisper_language(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.language, v);
+}
+void sherpa_shim_offline_recog_config_set_whisper_task(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.task, v);
+}
+void sherpa_shim_offline_recog_config_set_whisper_tail_paddings(void *h, int32_t v) {
+    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.tail_paddings = v;
+}
+void sherpa_shim_offline_recog_config_set_paraformer_model(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.paraformer.model, v);
+}
+void sherpa_shim_offline_recog_config_set_sense_voice_model(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.sense_voice.model, v);
+}
+void sherpa_shim_offline_recog_config_set_sense_voice_language(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.sense_voice.language, v);
+}
+void sherpa_shim_offline_recog_config_set_sense_voice_use_itn(void *h, int32_t v) {
+    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.sense_voice.use_itn = v;
+}
+void sherpa_shim_offline_recog_config_set_omnilingual_model(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.omnilingual.model, v);
+}
+
+void *sherpa_shim_create_offline_recognizer(void *h) {
+    return (void *)SherpaOnnxCreateOfflineRecognizer(
+        (const SherpaOnnxOfflineRecognizerConfig *)h);
+}
+
+// ==================================================================
+// Online recognizer config
+// ==================================================================
+
+void *sherpa_shim_online_recog_config_new(void) {
+    return calloc(1, sizeof(SherpaOnnxOnlineRecognizerConfig));
+}
+
+void sherpa_shim_online_recog_config_free(void *h) {
+    if (!h) return;
+    SherpaOnnxOnlineRecognizerConfig *c = (SherpaOnnxOnlineRecognizerConfig *)h;
+    free((char *)c->model_config.transducer.encoder);
+    free((char *)c->model_config.transducer.decoder);
+    free((char *)c->model_config.transducer.joiner);
+    free((char *)c->model_config.tokens);
+    free((char *)c->model_config.provider);
+    free((char *)c->decoding_method);
+    free(c);
+}
+
+void sherpa_shim_online_recog_config_set_transducer_encoder(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.transducer.encoder, v);
+}
+void sherpa_shim_online_recog_config_set_transducer_decoder(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.transducer.decoder, v);
+}
+void sherpa_shim_online_recog_config_set_transducer_joiner(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.transducer.joiner, v);
+}
+void sherpa_shim_online_recog_config_set_tokens(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.tokens, v);
+}
+void sherpa_shim_online_recog_config_set_num_threads(void *h, int32_t v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.num_threads = v;
+}
+void sherpa_shim_online_recog_config_set_debug(void *h, int32_t v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.debug = v;
+}
+void sherpa_shim_online_recog_config_set_provider(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.provider, v);
+}
+void sherpa_shim_online_recog_config_set_feat_sample_rate(void *h, int32_t v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->feat_config.sample_rate = v;
+}
+void sherpa_shim_online_recog_config_set_feat_feature_dim(void *h, int32_t v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->feat_config.feature_dim = v;
+}
+void sherpa_shim_online_recog_config_set_decoding_method(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->decoding_method, v);
+}
+void sherpa_shim_online_recog_config_set_enable_endpoint(void *h, int32_t v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->enable_endpoint = v;
+}
+void sherpa_shim_online_recog_config_set_rule1_min_trailing_silence(void *h, float v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->rule1_min_trailing_silence = v;
+}
+void sherpa_shim_online_recog_config_set_rule2_min_trailing_silence(void *h, float v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->rule2_min_trailing_silence = v;
+}
+void sherpa_shim_online_recog_config_set_rule3_min_utterance_length(void *h, float v) {
+    ((SherpaOnnxOnlineRecognizerConfig *)h)->rule3_min_utterance_length = v;
+}
+
+void *sherpa_shim_create_online_recognizer(void *h) {
+    return (void *)SherpaOnnxCreateOnlineRecognizer(
+        (const SherpaOnnxOnlineRecognizerConfig *)h);
+}
+
+// ==================================================================
+// Result-struct accessors
+// ==================================================================
+
+int32_t sherpa_shim_wave_sample_rate(const void *h) {
+    return ((const SherpaOnnxWave *)h)->sample_rate;
+}
+int32_t sherpa_shim_wave_num_samples(const void *h) {
+    return ((const SherpaOnnxWave *)h)->num_samples;
+}
+const float *sherpa_shim_wave_samples(const void *h) {
+    return ((const SherpaOnnxWave *)h)->samples;
+}
+
+const char *sherpa_shim_offline_result_text(const void *h) {
+    return ((const SherpaOnnxOfflineRecognizerResult *)h)->text;
+}
+const char *sherpa_shim_online_result_text(const void *h) {
+    return ((const SherpaOnnxOnlineRecognizerResult *)h)->text;
+}
+
+int32_t sherpa_shim_generated_audio_sample_rate(const void *h) {
+    return ((const SherpaOnnxGeneratedAudio *)h)->sample_rate;
+}
+int32_t sherpa_shim_generated_audio_n(const void *h) {
+    return ((const SherpaOnnxGeneratedAudio *)h)->n;
+}
+const float *sherpa_shim_generated_audio_samples(const void *h) {
+    return ((const SherpaOnnxGeneratedAudio *)h)->samples;
+}
+
+int32_t sherpa_shim_speech_segment_start(const void *h) {
+    return ((const SherpaOnnxSpeechSegment *)h)->start;
+}
+int32_t sherpa_shim_speech_segment_n(const void *h) {
+    return ((const SherpaOnnxSpeechSegment *)h)->n;
+}
+
+// ==================================================================
+// TTS streaming callback trampoline
+// ==================================================================
+
+void *sherpa_shim_tts_generate_with_callback(
+    void *tts, const char *text, int32_t sid, float speed,
+    uintptr_t callback_ptr, uintptr_t user_data) {
+    SherpaOnnxGeneratedAudioCallbackWithArg cb =
+        (SherpaOnnxGeneratedAudioCallbackWithArg)callback_ptr;
+    return (void *)SherpaOnnxOfflineTtsGenerateWithCallbackWithArg(
+        (const SherpaOnnxOfflineTts *)tts, text, sid, speed, cb,
+        (void *)user_data);
+}

backend/go/sherpa-onnx/csrc/shim.h

@@ -0,0 +1,129 @@
+#ifndef LOCALAI_SHERPA_ONNX_SHIM_H
+#define LOCALAI_SHERPA_ONNX_SHIM_H
+
+#include <stdint.h>
+
+// libsherpa-shim: purego-friendly wrapper around sherpa-onnx's C API.
+// Purego can't access C struct fields and can't route C callbacks to Go
+// funcs directly. Every function here is a fixed-signature trampoline
+// that replaces one field read/write or callback handoff that the Go
+// backend would otherwise have to do through cgo.
+//
+// String lifetime: setters strdup; _free walks every owned string and
+// frees it. Callers may discard their input buffers the moment a setter
+// returns.
+//
+// Opaque handles are `void *` in both directions. Nothing here holds a
+// reference across calls except config handles (freed via _free) and
+// sherpa-allocated results (freed via sherpa's own Destroy* entry
+// points, which Go calls through purego pass-through).
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+// --- VAD config -----------------------------------------------------
+void *sherpa_shim_vad_config_new(void);
+void  sherpa_shim_vad_config_free(void *cfg);
+void  sherpa_shim_vad_config_set_silero_model(void *cfg, const char *path);
+void  sherpa_shim_vad_config_set_silero_threshold(void *cfg, float v);
+void  sherpa_shim_vad_config_set_silero_min_silence_duration(void *cfg, float v);
+void  sherpa_shim_vad_config_set_silero_min_speech_duration(void *cfg, float v);
+void  sherpa_shim_vad_config_set_silero_window_size(void *cfg, int32_t v);
+void  sherpa_shim_vad_config_set_silero_max_speech_duration(void *cfg, float v);
+void  sherpa_shim_vad_config_set_sample_rate(void *cfg, int32_t v);
+void  sherpa_shim_vad_config_set_num_threads(void *cfg, int32_t v);
+void  sherpa_shim_vad_config_set_provider(void *cfg, const char *v);
+void  sherpa_shim_vad_config_set_debug(void *cfg, int32_t v);
+void *sherpa_shim_create_vad(void *cfg, float buffer_size_seconds);
+
+// --- Offline TTS config (VITS path — the only TTS family the backend uses) ---
+void *sherpa_shim_tts_config_new(void);
+void  sherpa_shim_tts_config_free(void *cfg);
+void  sherpa_shim_tts_config_set_vits_model(void *cfg, const char *v);
+void  sherpa_shim_tts_config_set_vits_tokens(void *cfg, const char *v);
+void  sherpa_shim_tts_config_set_vits_lexicon(void *cfg, const char *v);
+void  sherpa_shim_tts_config_set_vits_data_dir(void *cfg, const char *v);
+void  sherpa_shim_tts_config_set_vits_noise_scale(void *cfg, float v);
+void  sherpa_shim_tts_config_set_vits_noise_scale_w(void *cfg, float v);
+void  sherpa_shim_tts_config_set_vits_length_scale(void *cfg, float v);
+void  sherpa_shim_tts_config_set_num_threads(void *cfg, int32_t v);
+void  sherpa_shim_tts_config_set_debug(void *cfg, int32_t v);
+void  sherpa_shim_tts_config_set_provider(void *cfg, const char *v);
+void  sherpa_shim_tts_config_set_max_num_sentences(void *cfg, int32_t v);
+void *sherpa_shim_create_offline_tts(void *cfg);
+
+// --- Offline recognizer config (Whisper / Paraformer / SenseVoice / Omnilingual) ---
+void *sherpa_shim_offline_recog_config_new(void);
+void  sherpa_shim_offline_recog_config_free(void *cfg);
+void  sherpa_shim_offline_recog_config_set_num_threads(void *cfg, int32_t v);
+void  sherpa_shim_offline_recog_config_set_debug(void *cfg, int32_t v);
+void  sherpa_shim_offline_recog_config_set_provider(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_tokens(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_feat_sample_rate(void *cfg, int32_t v);
+void  sherpa_shim_offline_recog_config_set_feat_feature_dim(void *cfg, int32_t v);
+void  sherpa_shim_offline_recog_config_set_decoding_method(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_whisper_encoder(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_whisper_decoder(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_whisper_language(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_whisper_task(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_whisper_tail_paddings(void *cfg, int32_t v);
+void  sherpa_shim_offline_recog_config_set_paraformer_model(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_sense_voice_model(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_sense_voice_language(void *cfg, const char *v);
+void  sherpa_shim_offline_recog_config_set_sense_voice_use_itn(void *cfg, int32_t v);
+void  sherpa_shim_offline_recog_config_set_omnilingual_model(void *cfg, const char *v);
+void *sherpa_shim_create_offline_recognizer(void *cfg);
+
+// --- Online recognizer config (streaming zipformer transducer) ---
+void *sherpa_shim_online_recog_config_new(void);
+void  sherpa_shim_online_recog_config_free(void *cfg);
+void  sherpa_shim_online_recog_config_set_transducer_encoder(void *cfg, const char *v);
+void  sherpa_shim_online_recog_config_set_transducer_decoder(void *cfg, const char *v);
+void  sherpa_shim_online_recog_config_set_transducer_joiner(void *cfg, const char *v);
+void  sherpa_shim_online_recog_config_set_tokens(void *cfg, const char *v);
+void  sherpa_shim_online_recog_config_set_num_threads(void *cfg, int32_t v);
+void  sherpa_shim_online_recog_config_set_debug(void *cfg, int32_t v);
+void  sherpa_shim_online_recog_config_set_provider(void *cfg, const char *v);
+void  sherpa_shim_online_recog_config_set_feat_sample_rate(void *cfg, int32_t v);
+void  sherpa_shim_online_recog_config_set_feat_feature_dim(void *cfg, int32_t v);
+void  sherpa_shim_online_recog_config_set_decoding_method(void *cfg, const char *v);
+void  sherpa_shim_online_recog_config_set_enable_endpoint(void *cfg, int32_t v);
+void  sherpa_shim_online_recog_config_set_rule1_min_trailing_silence(void *cfg, float v);
+void  sherpa_shim_online_recog_config_set_rule2_min_trailing_silence(void *cfg, float v);
+void  sherpa_shim_online_recog_config_set_rule3_min_utterance_length(void *cfg, float v);
+void *sherpa_shim_create_online_recognizer(void *cfg);
+
+// --- Result accessors (sherpa-allocated; caller destroys via sherpa's own Destroy*) ---
+int32_t      sherpa_shim_wave_sample_rate(const void *wave);
+int32_t      sherpa_shim_wave_num_samples(const void *wave);
+const float *sherpa_shim_wave_samples(const void *wave);
+
+const char *sherpa_shim_offline_result_text(const void *result);
+const char *sherpa_shim_online_result_text(const void *result);
+
+int32_t      sherpa_shim_generated_audio_sample_rate(const void *audio);
+int32_t      sherpa_shim_generated_audio_n(const void *audio);
+const float *sherpa_shim_generated_audio_samples(const void *audio);
+
+int32_t sherpa_shim_speech_segment_start(const void *seg);
+int32_t sherpa_shim_speech_segment_n(const void *seg);
+
+// --- TTS streaming callback trampoline -----------------------------
+// Replaces the //export sherpaTtsGoCallback + callbacks.c bridge pattern.
+// `callback_ptr` is the C-callable function pointer returned by
+// purego.NewCallback. `user_data` is an integer the Go side uses to
+// look up its state (sync.Map keyed by uint64).
+//
+// Returns the sherpa-allocated SherpaOnnxGeneratedAudio. Destroy with
+// SherpaOnnxDestroyOfflineTtsGeneratedAudio (callable directly from
+// Go via purego).
+void *sherpa_shim_tts_generate_with_callback(
+    void *tts, const char *text, int32_t sid, float speed,
+    uintptr_t callback_ptr, uintptr_t user_data);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

backend/go/sherpa-onnx/main.go

@@ -0,0 +1,23 @@
+package main
+
+import (
+	"flag"
+
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+func main() {
+	flag.Parse()
+
+	if err := loadSherpaLibs(); err != nil {
+		panic(err)
+	}
+
+	if err := grpc.StartServer(*addr, &SherpaBackend{}); err != nil {
+		panic(err)
+	}
+}

backend/go/sherpa-onnx/package.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/sherpa-onnx $CURDIR/package/
+cp -avf $CURDIR/run.sh $CURDIR/package/
+cp -rfLv $CURDIR/backend-assets/lib/* $CURDIR/package/lib/
+
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/sherpa-onnx/run.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+set -ex
+
+CURDIR=$(dirname "$(realpath $0)")
+
+export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
+
+if [ -f $CURDIR/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	exec $CURDIR/lib/ld.so $CURDIR/sherpa-onnx "$@"
+fi
+
+exec $CURDIR/sherpa-onnx "$@"

backend/go/sherpa-onnx/test.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+# Unit tests for the sherpa-onnx backend. Exercises error-path and
+# dispatch logic via SherpaBackend directly (no gRPC). Integration
+# coverage (gRPC TTS / streaming ASR / realtime pipeline) lives in
+# tests/e2e-backends and tests/e2e and runs against the Docker image.
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+PACKAGES=$(go list ./... | grep -v /sources/)
+go test -v -timeout 60s $PACKAGES

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=44cca3d626d301e2215d5e243277e8f0e65bfa78
+STABLEDIFFUSION_GGML_VERSION?=b8bdffc19962be7e5a84bfefeb2e31bd885b571a
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/whisper/gowhisper.go

@@ -139,7 +139,10 @@ func (w *Whisper) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptR
 		// segment start/end conversion factor taken from https://github.com/ggml-org/whisper.cpp/blob/master/examples/cli/cli.cpp#L895
 		s := CppGetSegmentStart(i) * (10000000)
 		t := CppGetSegmentEnd(i) * (10000000)
-		txt := strings.Clone(CppGetSegmentText(i))
+		// whisper.cpp can emit bytes that aren't valid UTF-8 (e.g. a multibyte
+		// codepoint split across token boundaries); protobuf string fields
+		// reject those at marshal time. Scrub before the value escapes cgo.
+		txt := strings.ToValidUTF8(strings.Clone(CppGetSegmentText(i)), "<22>")
 		tokens := make([]int32, CppNTokens(i))
 
 		if opts.Diarize && CppGetSegmentSpeakerTurnNext(i) {

backend/index.yaml

@@ -263,6 +263,8 @@
     amd: "rocm-vllm"
     intel: "intel-vllm"
     nvidia-cuda-12: "cuda12-vllm"
+    nvidia-cuda-13: "cuda13-vllm"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vllm"
     cpu: "cpu-vllm"
 - &sglang
   name: "sglang"
@@ -285,6 +287,7 @@
     amd: "rocm-sglang"
     intel: "intel-sglang"
     nvidia-cuda-12: "cuda12-sglang"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-sglang"
     cpu: "cpu-sglang"
 - &vllm-omni
   name: "vllm-omni"
@@ -311,6 +314,8 @@
     nvidia: "cuda12-vllm-omni"
     amd: "rocm-vllm-omni"
     nvidia-cuda-12: "cuda12-vllm-omni"
+    nvidia-cuda-13: "cuda13-vllm-omni"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vllm-omni"
 - &mlx
   name: "mlx"
   uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx"
@@ -1006,6 +1011,23 @@
     nvidia: "cuda12-neutts"
     amd: "rocm-neutts"
     nvidia-cuda-12: "cuda12-neutts"
+- &sherpa-onnx
+  name: "sherpa-onnx"
+  alias: "sherpa-onnx"
+  urls:
+    - https://k2-fsa.github.io/sherpa/onnx/
+  description: |
+    Sherpa-ONNX backend for text-to-speech (VITS, Matcha, Kokoro), speech-to-text (Whisper, Paraformer, SenseVoice, Omnilingual ASR CTC), and voice activity detection via ONNX Runtime.
+    Supports multi-speaker voices, 1600+ language ASR, and GPU acceleration.
+  tags:
+    - text-to-speech
+    - TTS
+    - speech-to-text
+    - ASR
+  capabilities:
+    default: "cpu-sherpa-onnx"
+    nvidia: "cuda12-sherpa-onnx"
+    nvidia-cuda-12: "cuda12-sherpa-onnx"
 - !!merge <<: *neutts
   name: "neutts-development"
   capabilities:
@@ -1591,6 +1613,20 @@
   mirrors:
     - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-turboquant
 ## whisper
+- !!merge <<: *whispercpp
+  name: "whisper-development"
+  capabilities:
+    default: "cpu-whisper-development"
+    nvidia: "cuda12-whisper-development"
+    intel: "intel-sycl-f16-whisper-development"
+    metal: "metal-whisper-development"
+    amd: "rocm-whisper-development"
+    vulkan: "vulkan-whisper-development"
+    nvidia-l4t: "nvidia-l4t-arm64-whisper-development"
+    nvidia-cuda-13: "cuda13-whisper-development"
+    nvidia-cuda-12: "cuda12-whisper-development"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-whisper-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-whisper-development"
 - !!merge <<: *whispercpp
   name: "nvidia-l4t-arm64-whisper"
   uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-whisper"
@@ -1797,12 +1833,25 @@
     nvidia: "cuda12-vllm-development"
     amd: "rocm-vllm-development"
     intel: "intel-vllm-development"
+    nvidia-cuda-12: "cuda12-vllm-development"
+    nvidia-cuda-13: "cuda13-vllm-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vllm-development"
     cpu: "cpu-vllm-development"
 - !!merge <<: *vllm
   name: "cuda12-vllm"
   uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-vllm"
   mirrors:
     - localai/localai-backends:latest-gpu-nvidia-cuda-12-vllm
+- !!merge <<: *vllm
+  name: "cuda13-vllm"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-vllm"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-vllm
+- !!merge <<: *vllm
+  name: "cuda13-nvidia-l4t-arm64-vllm"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm
 - !!merge <<: *vllm
   name: "rocm-vllm"
   uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-vllm"
@@ -1823,6 +1872,16 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-vllm"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-12-vllm
+- !!merge <<: *vllm
+  name: "cuda13-vllm-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vllm"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-vllm
+- !!merge <<: *vllm
+  name: "cuda13-nvidia-l4t-arm64-vllm-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-vllm"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-vllm
 - !!merge <<: *vllm
   name: "rocm-vllm-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-vllm"
@@ -1845,12 +1904,19 @@
     nvidia: "cuda12-sglang-development"
     amd: "rocm-sglang-development"
     intel: "intel-sglang-development"
+    nvidia-cuda-12: "cuda12-sglang-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-sglang-development"
     cpu: "cpu-sglang-development"
 - !!merge <<: *sglang
   name: "cuda12-sglang"
   uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-sglang"
   mirrors:
     - localai/localai-backends:latest-gpu-nvidia-cuda-12-sglang
+- !!merge <<: *sglang
+  name: "cuda13-nvidia-l4t-arm64-sglang"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-sglang"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-sglang
 - !!merge <<: *sglang
   name: "rocm-sglang"
   uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-sglang"
@@ -1871,6 +1937,11 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-sglang"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-12-sglang
+- !!merge <<: *sglang
+  name: "cuda13-nvidia-l4t-arm64-sglang-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-sglang"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-sglang
 - !!merge <<: *sglang
   name: "rocm-sglang-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-sglang"
@@ -1893,11 +1964,23 @@
     nvidia: "cuda12-vllm-omni-development"
     amd: "rocm-vllm-omni-development"
     nvidia-cuda-12: "cuda12-vllm-omni-development"
+    nvidia-cuda-13: "cuda13-vllm-omni-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vllm-omni-development"
 - !!merge <<: *vllm-omni
   name: "cuda12-vllm-omni"
   uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-vllm-omni"
   mirrors:
     - localai/localai-backends:latest-gpu-nvidia-cuda-12-vllm-omni
+- !!merge <<: *vllm-omni
+  name: "cuda13-vllm-omni"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-vllm-omni"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-vllm-omni
+- !!merge <<: *vllm-omni
+  name: "cuda13-nvidia-l4t-arm64-vllm-omni"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-omni"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-omni
 - !!merge <<: *vllm-omni
   name: "rocm-vllm-omni"
   uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-vllm-omni"
@@ -1908,6 +1991,16 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-vllm-omni"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-12-vllm-omni
+- !!merge <<: *vllm-omni
+  name: "cuda13-vllm-omni-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vllm-omni"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-vllm-omni
+- !!merge <<: *vllm-omni
+  name: "cuda13-nvidia-l4t-arm64-vllm-omni-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-omni"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-omni
 - !!merge <<: *vllm-omni
   name: "rocm-vllm-omni-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-vllm-omni"
@@ -3773,3 +3866,91 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-insightface"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-12-insightface
+
+# speaker-recognition (voice/speaker biometrics) — Apache-2.0 stack
+- &speakerrecognition
+  name: "speaker-recognition"
+  alias: "speaker-recognition"
+  # SpeechBrain is Apache-2.0. WeSpeaker / 3D-Speaker ONNX exports are
+  # Apache-2.0. The backend itself ships only Python deps — all model
+  # weights flow through LocalAI's gallery download mechanism (or
+  # SpeechBrain's built-in HF auto-download at first LoadModel).
+  license: apache-2.0
+  description: |
+    Speaker (voice) recognition backend — the audio analog to
+    insightface. Wraps SpeechBrain ECAPA-TDNN (default engine, 192-d
+    embeddings, ~1.9% EER on VoxCeleb) plus an OnnxDirectEngine for
+    pre-exported WeSpeaker / 3D-Speaker ONNX models.
+
+    Exposes speaker verification (/v1/voice/verify), speaker embedding
+    (/v1/voice/embed), speaker analysis (/v1/voice/analyze), and 1:N
+    speaker identification (/v1/voice/{register,identify,forget}).
+    Registrations use LocalAI's built-in vector store — same in-memory
+    backing the face-recognition registry uses, separate instance.
+  urls:
+    - https://speechbrain.github.io/
+    - https://github.com/wenet-e2e/wespeaker
+    - https://github.com/modelscope/3D-Speaker
+  tags:
+    - voice-recognition
+    - speaker-verification
+    - speaker-embedding
+    - gpu
+    - cpu
+  capabilities:
+    default: "cpu-speaker-recognition"
+    nvidia: "cuda12-speaker-recognition"
+    nvidia-cuda-12: "cuda12-speaker-recognition"
+- !!merge <<: *speakerrecognition
+  name: "speaker-recognition-development"
+  capabilities:
+    default: "cpu-speaker-recognition-development"
+    nvidia: "cuda12-speaker-recognition-development"
+    nvidia-cuda-12: "cuda12-speaker-recognition-development"
+- !!merge <<: *speakerrecognition
+  name: "cpu-speaker-recognition"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-speaker-recognition"
+  mirrors:
+    - localai/localai-backends:latest-cpu-speaker-recognition
+- !!merge <<: *speakerrecognition
+  name: "cuda12-speaker-recognition"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-speaker-recognition"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-speaker-recognition
+- !!merge <<: *speakerrecognition
+  name: "cpu-speaker-recognition-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-speaker-recognition"
+  mirrors:
+    - localai/localai-backends:master-cpu-speaker-recognition
+- !!merge <<: *speakerrecognition
+  name: "cuda12-speaker-recognition-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-speaker-recognition"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-speaker-recognition
+## sherpa-onnx
+- !!merge <<: *sherpa-onnx
+  name: "sherpa-onnx-development"
+  capabilities:
+    default: "cpu-sherpa-onnx-development"
+    nvidia: "cuda12-sherpa-onnx-development"
+    nvidia-cuda-12: "cuda12-sherpa-onnx-development"
+- !!merge <<: *sherpa-onnx
+  name: "cpu-sherpa-onnx"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-sherpa-onnx"
+  mirrors:
+    - localai/localai-backends:latest-cpu-sherpa-onnx
+- !!merge <<: *sherpa-onnx
+  name: "cpu-sherpa-onnx-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-sherpa-onnx"
+  mirrors:
+    - localai/localai-backends:master-cpu-sherpa-onnx
+- !!merge <<: *sherpa-onnx
+  name: "cuda12-sherpa-onnx"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-sherpa-onnx"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-sherpa-onnx
+- !!merge <<: *sherpa-onnx
+  name: "cuda12-sherpa-onnx-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-sherpa-onnx"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-sherpa-onnx

backend/python/insightface/Makefile

@@ -11,3 +11,6 @@ protogen-clean:
 .PHONY: clean
 clean: protogen-clean
 	rm -rf venv __pycache__
+
+test: install
+	bash test.sh

backend/python/insightface/backend.py

@@ -180,23 +180,57 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         verified = distance < threshold
         confidence = max(0.0, min(100.0, (1.0 - distance / threshold) * 100.0)) if threshold > 0 else 0.0
 
-        def _region(img) -> backend_pb2.FacialArea:
+        # Detect once per image — region is needed for the response and
+        # potentially for the antispoof crop. Returns the highest-score face.
+        def _best_detection(img):
             dets = self.engine.detect(img)
             if not dets:
+                return None
+            return max(dets, key=lambda d: d.score)
+
+        def _region(det) -> backend_pb2.FacialArea:
+            if det is None:
                 return backend_pb2.FacialArea()
-            best = max(dets, key=lambda d: d.score)
-            x1, y1, x2, y2 = best.bbox
+            x1, y1, x2, y2 = det.bbox
             return backend_pb2.FacialArea(x=x1, y=y1, w=x2 - x1, h=y2 - y1)
 
+        det1 = _best_detection(img1)
+        det2 = _best_detection(img2)
+
+        img1_is_real = False
+        img1_score = 0.0
+        img2_is_real = False
+        img2_score = 0.0
+        if request.anti_spoofing:
+            spoof1 = self.engine.antispoof(img1, det1.bbox) if det1 is not None else None
+            spoof2 = self.engine.antispoof(img2, det2.bbox) if det2 is not None else None
+            if spoof1 is None or spoof2 is None:
+                context.set_code(grpc.StatusCode.FAILED_PRECONDITION)
+                context.set_details(
+                    "anti_spoofing requested but no antispoof model is loaded — "
+                    "install `silent-face-antispoofing` or pick a gallery entry "
+                    "that bundles MiniFASNet weights"
+                )
+                return backend_pb2.FaceVerifyResponse()
+            img1_is_real, img1_score = spoof1.is_real, spoof1.score
+            img2_is_real, img2_score = spoof2.is_real, spoof2.score
+            # Failed liveness vetoes verification regardless of similarity.
+            if not (img1_is_real and img2_is_real):
+                verified = False
+
         return backend_pb2.FaceVerifyResponse(
             verified=verified,
             distance=float(distance),
             threshold=float(threshold),
             confidence=float(confidence),
             model=self.model_name or self.engine_name,
-            img1_area=_region(img1),
-            img2_area=_region(img2),
+            img1_area=_region(det1),
+            img2_area=_region(det2),
             processing_time_ms=float((time.time() - start) * 1000.0),
+            img1_is_real=img1_is_real,
+            img1_antispoof_score=float(img1_score),
+            img2_is_real=img2_is_real,
+            img2_antispoof_score=float(img2_score),
         )
 
     def FaceAnalyze(self, request, context):
@@ -223,6 +257,19 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 fa.dominant_gender = attrs.dominant_gender
             for k, v in attrs.gender.items():
                 fa.gender[k] = float(v)
+            if request.anti_spoofing:
+                bbox = (float(x), float(y), float(x + w), float(y + h))
+                spoof = self.engine.antispoof(img, bbox)
+                if spoof is None:
+                    context.set_code(grpc.StatusCode.FAILED_PRECONDITION)
+                    context.set_details(
+                        "anti_spoofing requested but no antispoof model is loaded — "
+                        "install `silent-face-antispoofing` or pick a gallery entry "
+                        "that bundles MiniFASNet weights"
+                    )
+                    return backend_pb2.FaceAnalyzeResponse()
+                fa.is_real = spoof.is_real
+                fa.antispoof_score = float(spoof.score)
             faces.append(fa)
         return backend_pb2.FaceAnalyzeResponse(faces=faces)
 

backend/python/insightface/engines.py

@@ -41,6 +41,12 @@ class FaceAttributes:
     gender: dict[str, float] = field(default_factory=dict)
 
 
+@dataclass
+class SpoofResult:
+    is_real: bool
+    score: float  # averaged probability of the "real" class, 0.0-1.0
+
+
 class FaceEngine(Protocol):
     """Minimal interface every engine must implement."""
 
@@ -48,10 +54,149 @@ class FaceEngine(Protocol):
     def detect(self, img: np.ndarray) -> list[FaceDetection]: ...
     def embed(self, img: np.ndarray) -> np.ndarray | None: ...
     def analyze(self, img: np.ndarray) -> list[FaceAttributes]: ...
+    # Optional: returns None when no antispoof model is loaded.
+    def antispoof(self, img: np.ndarray, bbox: tuple[float, float, float, float]) -> SpoofResult | None: ...
+
+
+# ─── Antispoofer (Silent-Face MiniFASNet) ──────────────────────────────
+
+class Antispoofer:
+    """Liveness detector using the Silent-Face MiniFASNet ensemble.
+
+    Loads up to two ONNX exports (MiniFASNetV2 at scale 2.7 and
+    MiniFASNetV1SE at scale 4.0). Both are 80x80 BGR-float32-input
+    classifiers with 3 output logits where index 1 = "real". When both
+    are loaded, softmax outputs are averaged before argmax — the same
+    ensembling the upstream `test.py` does.
+
+    Preprocessing matches yakhyo/face-anti-spoofing's reference impl:
+    each model gets its own scale-expanded crop centered on the face
+    bbox, resized to 80x80, fed straight as float32 BGR (no /255, no
+    mean/std). See `_crop_face` for the bbox math.
+
+    A single model also works (the missing one is simply skipped).
+    """
+
+    INPUT_SIZE = (80, 80)  # h, w
+    REAL_CLASS_IDX = 1
+
+    def __init__(self) -> None:
+        self._sessions: list[tuple[Any, float, str, str]] = []  # (session, scale, input_name, output_name)
+        self.threshold: float = 0.5
+
+    def load(self, model_paths: list[tuple[str, float]], threshold: float = 0.5) -> None:
+        """Load one or more (path, scale) pairs."""
+        import onnxruntime as ort
+
+        providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
+        for path, scale in model_paths:
+            session = ort.InferenceSession(path, providers=providers)
+            input_name = session.get_inputs()[0].name
+            output_name = session.get_outputs()[0].name
+            self._sessions.append((session, float(scale), input_name, output_name))
+        self.threshold = float(threshold)
+
+    @property
+    def loaded(self) -> bool:
+        return bool(self._sessions)
+
+    def _crop_face(self, img: np.ndarray, bbox: tuple[float, float, float, float], scale: float) -> np.ndarray:
+        # bbox is (x1, y1, x2, y2) in source-image coordinates.
+        src_h, src_w = img.shape[:2]
+        x1, y1, x2, y2 = bbox
+        box_w = max(1.0, x2 - x1)
+        box_h = max(1.0, y2 - y1)
+
+        # Clamp scale so the expanded crop fits inside the source image.
+        scale = min((src_h - 1) / box_h, (src_w - 1) / box_w, scale)
+        new_w = box_w * scale
+        new_h = box_h * scale
+
+        cx = x1 + box_w / 2.0
+        cy = y1 + box_h / 2.0
+
+        cx1 = max(0, int(cx - new_w / 2.0))
+        cy1 = max(0, int(cy - new_h / 2.0))
+        cx2 = min(src_w - 1, int(cx + new_w / 2.0))
+        cy2 = min(src_h - 1, int(cy + new_h / 2.0))
+
+        cropped = img[cy1 : cy2 + 1, cx1 : cx2 + 1]
+        if cropped.size == 0:
+            cropped = img
+        out_h, out_w = self.INPUT_SIZE
+        return cv2.resize(cropped, (out_w, out_h))
+
+    @staticmethod
+    def _softmax(x: np.ndarray) -> np.ndarray:
+        e = np.exp(x - np.max(x, axis=1, keepdims=True))
+        return e / e.sum(axis=1, keepdims=True)
+
+    def predict(self, img: np.ndarray, bbox: tuple[float, float, float, float]) -> SpoofResult:
+        if not self._sessions:
+            raise RuntimeError("Antispoofer.predict called with no models loaded")
+        accum = np.zeros((1, 3), dtype=np.float32)
+        for session, scale, input_name, output_name in self._sessions:
+            face = self._crop_face(img, bbox, scale).astype(np.float32)
+            tensor = np.transpose(face, (2, 0, 1))[np.newaxis, ...]
+            logits = session.run([output_name], {input_name: tensor})[0]
+            accum += self._softmax(logits)
+        accum /= float(len(self._sessions))
+        real_prob = float(accum[0, self.REAL_CLASS_IDX])
+        is_real = int(np.argmax(accum)) == self.REAL_CLASS_IDX and real_prob >= self.threshold
+        return SpoofResult(is_real=is_real, score=real_prob)
+
+
+def _build_antispoofer(options: dict[str, str], model_dir: str | None) -> Antispoofer | None:
+    """Instantiate an Antispoofer from option keys, or return None.
+
+    Recognised options:
+        antispoof_v2_onnx     — path/filename of MiniFASNetV2 (scale 2.7)
+        antispoof_v1se_onnx   — path/filename of MiniFASNetV1SE (scale 4.0)
+        antispoof_threshold   — real-class probability threshold, default 0.5
+
+    Either or both can be provided. Returns None when neither is set.
+    """
+    pairs: list[tuple[str, float]] = []
+    v2 = options.get("antispoof_v2_onnx", "")
+    if v2:
+        pairs.append((_resolve_model_path(v2, model_dir=model_dir), 2.7))
+    v1se = options.get("antispoof_v1se_onnx", "")
+    if v1se:
+        pairs.append((_resolve_model_path(v1se, model_dir=model_dir), 4.0))
+    if not pairs:
+        return None
+    threshold = float(options.get("antispoof_threshold", "0.5"))
+    spoofer = Antispoofer()
+    spoofer.load(pairs, threshold=threshold)
+    return spoofer
 
 
 # ─── InsightFaceEngine ────────────────────────────────────────────────
 
+# Canonical ONNX manifest for each upstream insightface pack (v0.7 release
+# at github.com/deepinsight/insightface/releases). LocalAI's gallery extracts
+# these zips flat into the models directory, so when multiple packs or other
+# backends drop their own ONNX files alongside, the glob-the-directory
+# approach picks up foreign files and insightface's model_zoo.get_model()
+# raises IndexError trying to index `input_shape[2]` on a tensor that isn't
+# shaped like a face model. The manifest lets us pre-filter to only the
+# files that actually belong to the requested pack — deterministic, correct
+# pack choice, no crashes on neighbour ONNX files.
+_KNOWN_PACK_MANIFESTS: dict[str, frozenset[str]] = {
+    "buffalo_l": frozenset({
+        "det_10g.onnx",
+        "w600k_r50.onnx",
+        "genderage.onnx",
+        "2d106det.onnx",
+        "1k3d68.onnx",
+    }),
+    "buffalo_sc": frozenset({
+        "det_500m.onnx",
+        "w600k_mbf.onnx",
+    }),
+}
+
+
 class InsightFaceEngine:
     """Drives insightface's model_zoo directly — no FaceAnalysis wrapper.
 
@@ -80,6 +225,7 @@ class InsightFaceEngine:
         self.det_size: tuple[int, int] = (640, 640)
         self.det_thresh: float = 0.5
         self._providers: list[str] = ["CPUExecutionProvider"]
+        self._antispoofer: Antispoofer | None = None
 
     def prepare(self, options: dict[str, str]) -> None:
         import glob
@@ -90,6 +236,7 @@ class InsightFaceEngine:
         self.model_pack = options.get("model_pack", "buffalo_l")
         self.det_size = _parse_det_size(options.get("det_size", "640x640"))
         self.det_thresh = float(options.get("det_thresh", "0.5"))
+        self._antispoofer = _build_antispoofer(options, options.get("_model_dir"))
 
         pack_dir = _locate_insightface_pack(options, self.model_pack)
         if pack_dir is None:
@@ -99,6 +246,21 @@ class InsightFaceEngine:
             )
 
         onnx_files = sorted(glob.glob(os.path.join(pack_dir, "*.onnx")))
+        # When the pack extracts flat into a shared models directory it
+        # mixes with ONNX files from other backends (opencv face engine,
+        # MiniFASNet antispoof, WeSpeaker voice embedding, other buffalo
+        # packs installed earlier). Feeding those into model_zoo.get_model()
+        # blows up inside insightface's router — it assumes a 4-D NCHW
+        # input and indexes `input_shape[2]` on tensors that aren't shaped
+        # like a face model, raising IndexError. For the upstream packs we
+        # know the exact ONNX manifest; scoping to it makes the load
+        # deterministic (without it, det_10g.onnx from buffalo_l sorts
+        # before det_500m.onnx from buffalo_sc and silently wins).
+        manifest = _KNOWN_PACK_MANIFESTS.get(self.model_pack)
+        if manifest is not None:
+            scoped = [f for f in onnx_files if os.path.basename(f) in manifest]
+            if scoped:
+                onnx_files = scoped
         if not onnx_files:
             raise ValueError(f"no ONNX files in pack directory: {pack_dir}")
 
@@ -108,14 +270,31 @@ class InsightFaceEngine:
         self._providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
 
         self.models = {}
+        skipped: list[tuple[str, str]] = []
         for onnx_file in onnx_files:
-            m = model_zoo.get_model(onnx_file, providers=self._providers)
+            try:
+                m = model_zoo.get_model(onnx_file, providers=self._providers)
+            except Exception as err:
+                # Foreign ONNX (wrong rank/shape, non-insightface model) —
+                # older insightface versions raise IndexError / ValueError
+                # instead of returning None. Keep loading the rest.
+                skipped.append((os.path.basename(onnx_file), str(err)))
+                continue
             if m is None:
+                skipped.append((os.path.basename(onnx_file), "unknown taskname"))
                 continue
             # First occurrence of each taskname wins (matches FaceAnalysis).
             if m.taskname not in self.models:
                 self.models[m.taskname] = m
 
+        if skipped:
+            import sys
+            print(
+                f"[insightface] skipped {len(skipped)} non-pack ONNX file(s) in {pack_dir}: "
+                + ", ".join(f"{n} ({why})" for n, why in skipped),
+                file=sys.stderr,
+            )
+
         if "detection" not in self.models:
             raise ValueError(f"no detector (taskname='detection') found in {pack_dir}")
         self.det_model = self.models["detection"]
@@ -187,6 +366,11 @@ class InsightFaceEngine:
             out.append(attrs)
         return out
 
+    def antispoof(self, img: np.ndarray, bbox: tuple[float, float, float, float]) -> SpoofResult | None:
+        if self._antispoofer is None or not self._antispoofer.loaded:
+            return None
+        return self._antispoofer.predict(img, bbox)
+
 
 # ─── OnnxDirectEngine ─────────────────────────────────────────────────
 
@@ -206,6 +390,7 @@ class OnnxDirectEngine:
         self.det_thresh: float = 0.5
         self._detector: Any = None
         self._recognizer: Any = None
+        self._antispoofer: Antispoofer | None = None
 
     def prepare(self, options: dict[str, str]) -> None:
         raw_det = options.get("detector_onnx", "")
@@ -219,6 +404,7 @@ class OnnxDirectEngine:
         self.recognizer_path = _resolve_model_path(raw_rec, model_dir=model_dir)
         self.input_size = _parse_det_size(options.get("det_size", "320x320"))
         self.det_thresh = float(options.get("det_thresh", "0.5"))
+        self._antispoofer = _build_antispoofer(options, model_dir)
 
         # YuNet is a fixed-size detector; size is reset per detect() call to
         # match the input frame.
@@ -286,6 +472,11 @@ class OnnxDirectEngine:
             for d in self.detect(img)
         ]
 
+    def antispoof(self, img: np.ndarray, bbox: tuple[float, float, float, float]) -> SpoofResult | None:
+        if self._antispoofer is None or not self._antispoofer.loaded:
+            return None
+        return self._antispoofer.predict(img, bbox)
+
 
 # ─── helpers ──────────────────────────────────────────────────────────
 

backend/python/insightface/test.py

@@ -15,6 +15,7 @@ import sys
 import unittest
 
 import cv2
+import grpc
 import numpy as np
 
 sys.path.insert(0, os.path.dirname(__file__))
@@ -39,6 +40,44 @@ OPENCV_FILES = [
     ),
 ]
 
+# Silent-Face MiniFASNet ONNX files for antispoofing tests.
+ANTISPOOF_FILES = [
+    (
+        "MiniFASNetV2.onnx",
+        "https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV2.onnx",
+        "b32929adc2d9c34b9486f8c4c7bc97c1b69bc0ea9befefc380e4faae4e463907",
+    ),
+    (
+        "MiniFASNetV1SE.onnx",
+        "https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV1SE.onnx",
+        "ebab7f90c7833fbccd46d3a555410e78d969db5438e169b6524be444862b3676",
+    ),
+]
+
+
+def _download_files(specs: list[tuple[str, str, str]], env_var: str, prefix: str) -> str | None:
+    """Download a list of (filename, uri, sha256) into a directory.
+
+    Returns the directory, or None if any download failed.
+    """
+    import hashlib
+    import tempfile
+    import urllib.request
+
+    root = os.environ.get(env_var) or tempfile.mkdtemp(prefix=prefix)
+    for filename, uri, sha256 in specs:
+        dest = os.path.join(root, filename)
+        if os.path.isfile(dest):
+            if hashlib.sha256(open(dest, "rb").read()).hexdigest() == sha256:
+                continue
+        try:
+            urllib.request.urlretrieve(uri, dest)
+        except Exception:
+            return None
+        if hashlib.sha256(open(dest, "rb").read()).hexdigest() != sha256:
+            return None
+    return root
+
 
 def _encode(img: np.ndarray) -> str:
     _, buf = cv2.imencode(".jpg", img)
@@ -48,14 +87,19 @@ def _encode(img: np.ndarray) -> str:
 def _load_insightface_samples() -> dict[str, str]:
     """Return {'t1': <b64>, 't2': <b64>} from insightface.data.get_image.
 
-    t1 is a group photo, t2 a different one. We reuse both as
-    stand-ins for "Alice photo 1/2" and "Bob".
+    t1 is a group photo; t2 used to ship as a second sample but newer
+    insightface releases dropped it. We fall back to `Tom_Hanks_54745`
+    (also bundled) as a distinct second face.
     """
     from insightface.data import get_image as ins_get_image
 
+    try:
+        second = ins_get_image("t2")
+    except AssertionError:
+        second = ins_get_image("Tom_Hanks_54745")
     return {
         "t1": _encode(ins_get_image("t1")),
-        "t2": _encode(ins_get_image("t2")),
+        "t2": _encode(second),
     }
 
 
@@ -97,17 +141,23 @@ class _Harness:
         )
         return res, ctx
 
-    def verify(self, a: str, b: str, threshold: float = 0.0):
-        return self.svc.FaceVerify(
-            backend_pb2.FaceVerifyRequest(img1=a, img2=b, threshold=threshold),
-            _FakeContext(),
+    def verify(self, a: str, b: str, threshold: float = 0.0, anti_spoofing: bool = False):
+        ctx = _FakeContext()
+        res = self.svc.FaceVerify(
+            backend_pb2.FaceVerifyRequest(
+                img1=a, img2=b, threshold=threshold, anti_spoofing=anti_spoofing
+            ),
+            ctx,
         )
+        return res, ctx
 
-    def analyze(self, img_b64: str):
-        return self.svc.FaceAnalyze(
-            backend_pb2.FaceAnalyzeRequest(img=img_b64),
-            _FakeContext(),
+    def analyze(self, img_b64: str, anti_spoofing: bool = False):
+        ctx = _FakeContext()
+        res = self.svc.FaceAnalyze(
+            backend_pb2.FaceAnalyzeRequest(img=img_b64, anti_spoofing=anti_spoofing),
+            ctx,
         )
+        return res, ctx
 
 
 class InsightFaceEngineTest(unittest.TestCase):
@@ -138,21 +188,21 @@ class InsightFaceEngineTest(unittest.TestCase):
         self.assertAlmostEqual(norm_sq, 1.0, places=2)
 
     def test_verify_same_image(self):
-        res = self.harness.verify(self.samples["t1"], self.samples["t1"])
+        res, _ = self.harness.verify(self.samples["t1"], self.samples["t1"])
         self.assertTrue(res.verified)
         self.assertLess(res.distance, 0.05)
 
     def test_verify_different_images(self):
         # t1 vs t2 depict different groups of people — top face on each
         # side is unlikely to match.
-        res = self.harness.verify(self.samples["t1"], self.samples["t2"])
+        res, _ = self.harness.verify(self.samples["t1"], self.samples["t2"])
         # We assert only that some numerical answer came back; the
         # matches-or-not determination depends on which face each side
         # picked and isn't a stable test assertion.
         self.assertGreaterEqual(res.distance, 0.0)
 
     def test_analyze_has_age_and_gender(self):
-        res = self.harness.analyze(self.samples["t1"])
+        res, _ = self.harness.analyze(self.samples["t1"])
         self.assertGreater(len(res.faces), 0)
         for face in res.faces:
             self.assertGreater(face.face_confidence, 0.0)
@@ -160,31 +210,29 @@ class InsightFaceEngineTest(unittest.TestCase):
             self.assertGreater(face.age, 0.0)
             self.assertIn(face.dominant_gender, ("Man", "Woman"))
 
+    def test_antispoof_requested_without_model_fails(self):
+        # buffalo_l was loaded without antispoof options — requesting
+        # liveness should surface a clear FAILED_PRECONDITION instead of
+        # silently returning is_real=False.
+        _, ctx = self.harness.verify(
+            self.samples["t1"], self.samples["t1"], anti_spoofing=True
+        )
+        self.assertEqual(ctx.code, grpc.StatusCode.FAILED_PRECONDITION)
+        self.assertIn("anti_spoofing", ctx.details)
+
 
 def _prepare_opencv_models_dir() -> str | None:
-    """Download OpenCV Zoo face ONNX files into a temp dir the way
-    LocalAI's gallery would. Returns the directory, or None if
-    downloads failed (network-restricted sandbox).
-    """
-    import hashlib
-    import tempfile
-    import urllib.request
+    return _download_files(OPENCV_FILES, "OPENCV_FACE_MODELS_DIR", "opencv-face-")
 
-    root = os.environ.get("OPENCV_FACE_MODELS_DIR") or tempfile.mkdtemp(
-        prefix="opencv-face-"
-    )
-    for filename, uri, sha256 in OPENCV_FILES:
-        dest = os.path.join(root, filename)
-        if os.path.isfile(dest):
-            if hashlib.sha256(open(dest, "rb").read()).hexdigest() == sha256:
-                continue
-        try:
-            urllib.request.urlretrieve(uri, dest)
-        except Exception:
-            return None
-        if hashlib.sha256(open(dest, "rb").read()).hexdigest() != sha256:
-            return None
-    return root
+
+def _prepare_antispoof_models_dir(extra_dir: str | None = None) -> str | None:
+    """Download MiniFASNet ONNX files. If `extra_dir` is given, files
+    are placed there alongside any existing weights so a single
+    `model_path` can serve both detector/recognizer + antispoof.
+    """
+    if extra_dir is not None:
+        os.environ.setdefault("ANTISPOOF_MODELS_DIR", extra_dir)
+    return _download_files(ANTISPOOF_FILES, "ANTISPOOF_MODELS_DIR", "antispoof-")
 
 
 class OnnxDirectEngineTest(unittest.TestCase):
@@ -218,17 +266,79 @@ class OnnxDirectEngineTest(unittest.TestCase):
         self.assertGreater(len(res.embeddings), 0)
 
     def test_verify_same_image(self):
-        res = self.harness.verify(self.samples["t1"], self.samples["t1"], threshold=0.4)
+        res, _ = self.harness.verify(self.samples["t1"], self.samples["t1"], threshold=0.4)
         self.assertTrue(res.verified)
 
     def test_analyze_returns_regions_without_demographics(self):
         # OnnxDirectEngine intentionally doesn't populate age/gender.
-        res = self.harness.analyze(self.samples["t1"])
+        res, _ = self.harness.analyze(self.samples["t1"])
         self.assertGreater(len(res.faces), 0)
         for face in res.faces:
             self.assertEqual(face.dominant_gender, "")
             self.assertEqual(face.age, 0.0)
 
 
+class AntispoofingTest(unittest.TestCase):
+    """End-to-end FaceVerify / FaceAnalyze with anti_spoofing=True.
+
+    Loads the OpenCV-Zoo (Apache-2.0) face engine alongside the Silent-Face
+    MiniFASNet ensemble. Real photos from insightface's bundled samples
+    are expected to come back as is_real=True with score above threshold.
+    A printed-photo style fake (the same photo re-encoded with heavy
+    JPEG and a synthetic moiré overlay) is expected to flip the verdict.
+    """
+
+    @classmethod
+    def setUpClass(cls):
+        # Reuse one directory for both detector/recognizer + antispoof
+        # weights so a single LoadModel options block points at all of them.
+        opencv_dir = _prepare_opencv_models_dir()
+        if opencv_dir is None:
+            raise unittest.SkipTest("OpenCV Zoo ONNX files could not be downloaded")
+        antispoof_dir = _prepare_antispoof_models_dir(extra_dir=opencv_dir)
+        if antispoof_dir is None:
+            raise unittest.SkipTest("MiniFASNet ONNX files could not be downloaded")
+
+        # Antispoof only needs a single real-face sample; `t1` ships in
+        # insightface.data across every release.
+        from insightface.data import get_image as ins_get_image
+
+        cls.samples = {"t1": _encode(ins_get_image("t1"))}
+        cls.harness = _Harness(BackendServicer())
+        load = cls.harness.load(
+            [
+                "engine:onnx_direct",
+                "detector_onnx:face_detection_yunet_2023mar.onnx",
+                "recognizer_onnx:face_recognition_sface_2021dec.onnx",
+                "antispoof_v2_onnx:MiniFASNetV2.onnx",
+                "antispoof_v1se_onnx:MiniFASNetV1SE.onnx",
+            ],
+            model_path=opencv_dir,
+        )
+        if not load.success:
+            raise unittest.SkipTest(f"LoadModel failed: {load.message}")
+
+    def test_verify_returns_per_image_liveness(self):
+        res, ctx = self.harness.verify(
+            self.samples["t1"], self.samples["t1"], threshold=0.4, anti_spoofing=True
+        )
+        self.assertIsNone(ctx.code, f"FaceVerify error: {ctx.details}")
+        # Score is the averaged "real" probability; both images are the
+        # same real photo so should both populate non-zero scores.
+        self.assertGreater(res.img1_antispoof_score, 0.0)
+        self.assertGreater(res.img2_antispoof_score, 0.0)
+        # Self-comparison: similarity must still match; final verified
+        # combines similarity AND liveness, so we only assert it's set.
+        self.assertIsInstance(res.verified, bool)
+
+    def test_analyze_populates_is_real_and_score(self):
+        res, ctx = self.harness.analyze(self.samples["t1"], anti_spoofing=True)
+        self.assertIsNone(ctx.code, f"FaceAnalyze error: {ctx.details}")
+        self.assertGreater(len(res.faces), 0)
+        for face in res.faces:
+            self.assertGreaterEqual(face.antispoof_score, 0.0)
+            self.assertLessEqual(face.antispoof_score, 1.0)
+
+
 if __name__ == "__main__":
     unittest.main()

backend/python/mlx-vlm/requirements-cpu.txt

@@ -1,2 +1,2 @@
-git+https://github.com/Blaizzy/mlx-vlm
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
 mlx[cpu]

backend/python/mlx-vlm/requirements-cublas12.txt

@@ -1,2 +1,2 @@
-git+https://github.com/Blaizzy/mlx-vlm
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
 mlx[cuda12]

backend/python/mlx-vlm/requirements-cublas13.txt

@@ -1,2 +1,2 @@
-git+https://github.com/Blaizzy/mlx-vlm
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
 mlx[cuda13]

backend/python/mlx-vlm/requirements-l4t12.txt

@@ -1,2 +1,2 @@
-git+https://github.com/Blaizzy/mlx-vlm
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
 mlx[cuda12]

backend/python/mlx-vlm/requirements-l4t13.txt

@@ -1,2 +1,2 @@
-git+https://github.com/Blaizzy/mlx-vlm
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
 mlx[cuda13]

backend/python/mlx-vlm/requirements-mps.txt

@@ -1 +1 @@
-git+https://github.com/Blaizzy/mlx-vlm
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4

backend/python/sglang/install.sh

@@ -23,6 +23,19 @@ if [ "x${BUILD_PROFILE}" == "xcpu" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
+# JetPack 7 / L4T arm64 wheels are built for cp312 and shipped via
+# pypi.jetson-ai-lab.io. Bump the venv Python so the prebuilt sglang
+# wheel resolves cleanly. unsafe-best-match is required because the
+# jetson-ai-lab index lists transitive deps (e.g. decord) at older
+# versions only — without it uv refuses to fall through to PyPI for a
+# compatible wheel and resolution fails.
+if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    PYTHON_VERSION="3.12"
+    PYTHON_PATCH="12"
+    PY_STANDALONE_TAG="20251120"
+    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
+fi
+
 # sglang's CPU path has no prebuilt wheel on PyPI — upstream publishes
 # a separate pyproject_cpu.toml that must be swapped in before `pip install`.
 # Reference: docker/xeon.Dockerfile in the sglang upstream repo.

backend/python/sglang/requirements-l4t13.txt

@@ -0,0 +1,12 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+accelerate
+torch
+torchvision
+torchaudio
+transformers
+# Drop the [all] extra: it pulls outlines/decord, and decord has no
+# aarch64 cp312 wheel anywhere (PyPI nor the jetson-ai-lab index ships
+# only legacy cp35-cp37). With [all] uv backtracks through versions
+# trying to satisfy decord and lands on sglang==0.1.16. Floor at 0.5.0
+# so uv can't silently downgrade if a future resolution misfires.
+sglang>=0.5.0

backend/python/speaker-recognition/Makefile

@@ -0,0 +1,13 @@
+.DEFAULT_GOAL := install
+
+.PHONY: install
+install:
+	bash install.sh
+
+.PHONY: protogen-clean
+protogen-clean:
+	$(RM) backend_pb2_grpc.py backend_pb2.py
+
+.PHONY: clean
+clean: protogen-clean
+	rm -rf venv __pycache__

backend/python/speaker-recognition/README.md

@@ -0,0 +1,40 @@
+# speaker-recognition
+
+Speaker (voice) recognition backend for LocalAI. The audio analog to
+`insightface` — produces speaker embeddings and supports 1:1 voice
+verification and voice demographic analysis.
+
+## Engines
+
+- **SpeechBrainEngine** (default): ECAPA-TDNN trained on VoxCeleb.
+  192-d L2-normalised embeddings, cosine distance for verification.
+  Auto-downloads from HuggingFace on first LoadModel.
+- **OnnxDirectEngine**: Any pre-exported ONNX speaker encoder
+  (WeSpeaker ResNet, 3D-Speaker ERes2Net, CAM++, …). Model path comes
+  from the gallery `files:` entry.
+
+Engine selection is gallery-driven: if the model config provides
+`model_path:` / `onnx:` the ONNX engine is used, otherwise the
+SpeechBrain engine.
+
+## Endpoints
+
+- `POST /v1/voice/verify` — 1:1 same-speaker check.
+- `POST /v1/voice/embed` — extract a speaker embedding vector.
+- `POST /v1/voice/analyze` — voice demographics, loaded lazily on
+  the first analyze call:
+  - **Emotion** (default, opt-out): `superb/wav2vec2-base-superb-er`
+    (Apache-2.0), 4-way categorical (neutral / happy / angry / sad).
+  - **Age + gender** (opt-in): no default — wire a checkpoint with a
+    standard `Wav2Vec2ForSequenceClassification` head via
+    `age_gender_model:<repo>` in options. The Audeering
+    age-gender model is *not* usable as a drop-in because its
+    multi-task head isn't loadable via `AutoModelForAudioClassification`.
+
+  Both heads are optional. When nothing loads, the engine returns 501.
+
+## Audio input
+
+Audio is materialised by the HTTP layer to a temp wav before calling
+the gRPC backend. Accepted input forms on the HTTP side: URL, data-URI,
+or raw base64. The backend itself always receives a filesystem path.

backend/python/speaker-recognition/backend.py

@@ -0,0 +1,205 @@
+#!/usr/bin/env python3
+"""gRPC server for the LocalAI speaker-recognition backend.
+
+Implements Health / LoadModel / Status plus the voice-specific methods:
+VoiceVerify, VoiceAnalyze, VoiceEmbed. The heavy lifting lives in
+engines.py — this file is just the gRPC plumbing, mirroring the
+insightface backend's two-engine split (SpeechBrain + OnnxDirect).
+"""
+from __future__ import annotations
+
+import argparse
+import os
+import signal
+import sys
+import time
+from concurrent import futures
+
+import backend_pb2
+import backend_pb2_grpc
+import grpc
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "common"))
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "common"))
+from grpc_auth import get_auth_interceptors  # noqa: E402
+
+from engines import SpeakerEngine, build_engine  # noqa: E402
+
+_ONE_DAY = 60 * 60 * 24
+MAX_WORKERS = int(os.environ.get("PYTHON_GRPC_MAX_WORKERS", "1"))
+
+# ECAPA-TDNN on VoxCeleb is the reference. Threshold is tuned for
+# cosine distance (1 - cosine_similarity). Clients may override.
+DEFAULT_VERIFY_THRESHOLD = 0.25
+
+
+def _parse_options(raw: list[str]) -> dict[str, str]:
+    out: dict[str, str] = {}
+    for entry in raw:
+        if ":" not in entry:
+            continue
+        k, v = entry.split(":", 1)
+        out[k.strip()] = v.strip()
+    return out
+
+
+class BackendServicer(backend_pb2_grpc.BackendServicer):
+    def __init__(self) -> None:
+        self.engine: SpeakerEngine | None = None
+        self.engine_name: str = ""
+        self.model_name: str = ""
+        self.verify_threshold: float = DEFAULT_VERIFY_THRESHOLD
+
+    def Health(self, request, context):
+        return backend_pb2.Reply(message=bytes("OK", "utf-8"))
+
+    def LoadModel(self, request, context):
+        options = _parse_options(list(request.Options))
+        # Surface LocalAI's models directory (ModelPath) so engines can
+        # anchor relative paths and auto-download into a writable spot
+        # alongside every other gallery-managed asset.
+        options["_model_path"] = request.ModelPath or ""
+        try:
+            engine, engine_name = build_engine(request.Model, options)
+        except Exception as exc:  # noqa: BLE001
+            return backend_pb2.Result(success=False, message=f"engine init failed: {exc}")
+
+        self.engine = engine
+        self.engine_name = engine_name
+        self.model_name = request.Model
+
+        threshold_opt = options.get("verify_threshold")
+        if threshold_opt:
+            try:
+                self.verify_threshold = float(threshold_opt)
+            except ValueError:
+                pass
+        return backend_pb2.Result(success=True, message=f"loaded {engine_name}")
+
+    def Status(self, request, context):
+        state = backend_pb2.StatusResponse.State.READY if self.engine else backend_pb2.StatusResponse.State.UNINITIALIZED
+        return backend_pb2.StatusResponse(state=state)
+
+    def _require_engine(self, context) -> SpeakerEngine | None:
+        if self.engine is None:
+            context.set_code(grpc.StatusCode.FAILED_PRECONDITION)
+            context.set_details("no speaker-recognition model loaded")
+            return None
+        return self.engine
+
+    def VoiceVerify(self, request, context):
+        engine = self._require_engine(context)
+        if engine is None:
+            return backend_pb2.VoiceVerifyResponse()
+        if not request.audio1 or not request.audio2:
+            context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
+            context.set_details("audio1 and audio2 are required")
+            return backend_pb2.VoiceVerifyResponse()
+
+        threshold = request.threshold if request.threshold > 0 else self.verify_threshold
+        started = time.time()
+        try:
+            distance = engine.compare(request.audio1, request.audio2)
+        except Exception as exc:  # noqa: BLE001
+            context.set_code(grpc.StatusCode.INTERNAL)
+            context.set_details(f"voice verify failed: {exc}")
+            return backend_pb2.VoiceVerifyResponse()
+
+        elapsed_ms = (time.time() - started) * 1000.0
+        # Confidence goes linearly from 100 at distance=0 to 0 at distance=threshold.
+        confidence = max(0.0, min(100.0, (1.0 - distance / threshold) * 100.0))
+        return backend_pb2.VoiceVerifyResponse(
+            verified=distance <= threshold,
+            distance=distance,
+            threshold=threshold,
+            confidence=confidence,
+            model=self.model_name,
+            processing_time_ms=elapsed_ms,
+        )
+
+    def VoiceEmbed(self, request, context):
+        engine = self._require_engine(context)
+        if engine is None:
+            return backend_pb2.VoiceEmbedResponse()
+        if not request.audio:
+            context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
+            context.set_details("audio is required")
+            return backend_pb2.VoiceEmbedResponse()
+        try:
+            vec = engine.embed(request.audio)
+        except Exception as exc:  # noqa: BLE001
+            context.set_code(grpc.StatusCode.INTERNAL)
+            context.set_details(f"voice embed failed: {exc}")
+            return backend_pb2.VoiceEmbedResponse()
+        return backend_pb2.VoiceEmbedResponse(embedding=list(vec), model=self.model_name)
+
+    def VoiceAnalyze(self, request, context):
+        engine = self._require_engine(context)
+        if engine is None:
+            return backend_pb2.VoiceAnalyzeResponse()
+        if not request.audio:
+            context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
+            context.set_details("audio is required")
+            return backend_pb2.VoiceAnalyzeResponse()
+
+        actions = list(request.actions) or ["age", "gender", "emotion"]
+        try:
+            segments = engine.analyze(request.audio, actions)
+        except NotImplementedError:
+            context.set_code(grpc.StatusCode.UNIMPLEMENTED)
+            context.set_details(f"analyze not supported by {self.engine_name}")
+            return backend_pb2.VoiceAnalyzeResponse()
+        except Exception as exc:  # noqa: BLE001
+            context.set_code(grpc.StatusCode.INTERNAL)
+            context.set_details(f"voice analyze failed: {exc}")
+            return backend_pb2.VoiceAnalyzeResponse()
+
+        proto_segments = []
+        for seg in segments:
+            proto_segments.append(
+                backend_pb2.VoiceAnalysis(
+                    start=seg.get("start", 0.0),
+                    end=seg.get("end", 0.0),
+                    age=seg.get("age", 0.0),
+                    dominant_gender=seg.get("dominant_gender", ""),
+                    gender=seg.get("gender", {}),
+                    dominant_emotion=seg.get("dominant_emotion", ""),
+                    emotion=seg.get("emotion", {}),
+                )
+            )
+        return backend_pb2.VoiceAnalyzeResponse(segments=proto_segments)
+
+
+def serve(address: str) -> None:
+    interceptors = get_auth_interceptors()
+    server = grpc.server(
+        futures.ThreadPoolExecutor(max_workers=MAX_WORKERS),
+        interceptors=interceptors,
+        options=[
+            ("grpc.max_send_message_length", 128 * 1024 * 1024),
+            ("grpc.max_receive_message_length", 128 * 1024 * 1024),
+        ],
+    )
+    backend_pb2_grpc.add_BackendServicer_to_server(BackendServicer(), server)
+    server.add_insecure_port(address)
+    server.start()
+    print("speaker-recognition backend listening on", address, flush=True)
+
+    def _stop(*_):
+        server.stop(0)
+        sys.exit(0)
+
+    signal.signal(signal.SIGTERM, _stop)
+    signal.signal(signal.SIGINT, _stop)
+    try:
+        while True:
+            time.sleep(_ONE_DAY)
+    except KeyboardInterrupt:
+        server.stop(0)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--addr", default="localhost:50051")
+    args = parser.parse_args()
+    serve(args.addr)

backend/python/speaker-recognition/engines.py

@@ -0,0 +1,428 @@
+"""Speaker-recognition engines.
+
+Two engines are offered, mirroring the insightface backend's split:
+
+  * SpeechBrainEngine: full PyTorch / SpeechBrain path. Uses the
+    ECAPA-TDNN recipe trained on VoxCeleb; 192-d L2-normalized
+    embeddings, cosine distance for verification. Auto-downloads the
+    checkpoint into LocalAI's models directory on first LoadModel.
+
+  * OnnxDirectEngine: CPU-friendly fallback that runs pre-exported
+    ONNX speaker encoders (WeSpeaker ResNet34, 3D-Speaker ERes2Net,
+    CAM++, etc.). Model paths come from the model config — the gallery
+    `files:` flow drops them into the models directory.
+
+Engine selection follows the same gallery-driven convention face
+recognition uses (insightface commits 9c6da0f7 / 405fec0b): the
+Python backend reads `engine` / `model_path` / `checkpoint` from the
+options dict and picks an engine accordingly.
+"""
+from __future__ import annotations
+
+import os
+from typing import Any, Iterable, Protocol
+
+
+class SpeakerEngine(Protocol):
+    """Interface both concrete engines satisfy."""
+
+    name: str
+
+    def embed(self, audio_path: str) -> list[float]:  # pragma: no cover - interface
+        ...
+
+    def compare(self, audio1: str, audio2: str) -> float:  # pragma: no cover
+        ...
+
+    def analyze(self, audio_path: str, actions: Iterable[str]) -> list[dict[str, Any]]:  # pragma: no cover
+        ...
+
+
+def _cosine_distance(a, b) -> float:
+    import numpy as np
+
+    va = np.asarray(a, dtype=np.float32).reshape(-1)
+    vb = np.asarray(b, dtype=np.float32).reshape(-1)
+    na = float(np.linalg.norm(va))
+    nb = float(np.linalg.norm(vb))
+    if na == 0.0 or nb == 0.0:
+        return 1.0
+    return float(1.0 - np.dot(va, vb) / (na * nb))
+
+
+class AnalysisHead:
+    """Age / gender / emotion head, lazy-loaded on first analyze call.
+
+    Wraps two open-licence HuggingFace checkpoints:
+
+      * audeering/wav2vec2-large-robust-24-ft-age-gender — age
+        regression (0–100 years) + 3-way gender (female/male/child).
+        Apache 2.0.
+      * superb/wav2vec2-base-superb-er — 4-way emotion classification
+        (neutral / happy / angry / sad). Apache 2.0.
+
+    Either model is optional — the head degrades gracefully to only the
+    attributes it could load. Override the checkpoint with the
+    `age_gender_model` / `emotion_model` option if you want something
+    else. Set either to an empty string to disable that head.
+    """
+
+    # Age + gender is OFF by default: the high-accuracy Apache-2.0
+    # checkpoint (Audeering wav2vec2-large-robust-24-ft-age-gender) uses a
+    # custom multi-task head that AutoModelForAudioClassification silently
+    # mangles — it drops the age weights as UNEXPECTED and re-initialises
+    # the classifier head with random values, so the output is noise. Users
+    # who have a cleanly loadable age/gender classifier can opt in with
+    # `age_gender_model:<repo>` in options. The emotion default below
+    # (superb/wav2vec2-base-superb-er) loads via the standard audio-
+    # classification pipeline with no such caveat.
+    DEFAULT_AGE_GENDER_MODEL = ""
+    DEFAULT_EMOTION_MODEL = "superb/wav2vec2-base-superb-er"
+    AGE_GENDER_LABELS = ("female", "male", "child")
+
+    def __init__(self, options: dict[str, str]):
+        self._options = options
+        self._age_gender = None
+        self._age_gender_processor = None
+        self._age_gender_loaded = False
+        self._age_gender_error: str | None = None
+        self._emotion = None
+        self._emotion_loaded = False
+        self._emotion_error: str | None = None
+
+    # --- age / gender -------------------------------------------------
+    def _ensure_age_gender(self):
+        if self._age_gender_loaded:
+            return
+        self._age_gender_loaded = True
+        model_id = self._options.get(
+            "age_gender_model", self.DEFAULT_AGE_GENDER_MODEL
+        )
+        if not model_id:
+            self._age_gender_error = "disabled"
+            return
+        try:
+            # Late imports — torch / transformers are heavy and only
+            # pulled in when the analyze head actually runs.
+            import torch  # type: ignore
+            from transformers import AutoFeatureExtractor, AutoModelForAudioClassification  # type: ignore
+
+            self._torch = torch
+            self._age_gender_processor = AutoFeatureExtractor.from_pretrained(model_id)
+            self._age_gender = AutoModelForAudioClassification.from_pretrained(model_id)
+            self._age_gender.eval()
+        except Exception as exc:  # noqa: BLE001
+            self._age_gender_error = f"{type(exc).__name__}: {exc}"
+
+    def _infer_age_gender(self, waveform_16k) -> dict[str, Any]:
+        self._ensure_age_gender()
+        if self._age_gender is None:
+            return {}
+        import numpy as np
+
+        try:
+            inputs = self._age_gender_processor(
+                waveform_16k, sampling_rate=16000, return_tensors="pt"
+            )
+            with self._torch.no_grad():
+                outputs = self._age_gender(**inputs)
+
+            # Audeering's checkpoint is published with a custom head: the
+            # official recipe exposes `(hidden_states, logits_age, logits_gender)`.
+            # AutoModelForAudioClassification flattens that into a single
+            # `logits` tensor of shape [batch, 4] — [age_regression, female, male, child].
+            # Fall back gracefully when the shape is different (e.g. a
+            # user-supplied age_gender_model checkpoint that returns a proper tuple).
+            hidden = getattr(outputs, "logits", outputs)
+            age_years = None
+            gender_logits = None
+            if isinstance(hidden, (tuple, list)) and len(hidden) >= 2:
+                age_years = float(hidden[0].squeeze().item()) * 100.0
+                gender_logits = hidden[1]
+            else:
+                flat = hidden.squeeze()
+                if flat.ndim == 1 and flat.numel() >= 4:
+                    age_years = float(flat[0].item()) * 100.0
+                    gender_logits = flat[1:4]
+                elif flat.ndim == 1 and flat.numel() == 1:
+                    age_years = float(flat.item()) * 100.0
+
+            if age_years is None and gender_logits is None:
+                return {}
+
+            result: dict[str, Any] = {}
+            if age_years is not None:
+                result["age"] = age_years
+            if gender_logits is not None:
+                probs = self._torch.softmax(gender_logits, dim=-1).cpu().numpy()
+                probs = np.asarray(probs).reshape(-1)
+                gender_map = {
+                    label: float(probs[i])
+                    for i, label in enumerate(self.AGE_GENDER_LABELS[: len(probs)])
+                }
+                result["gender"] = gender_map
+                if gender_map:
+                    dom = max(gender_map.items(), key=lambda kv: kv[1])[0]
+                    result["dominant_gender"] = {
+                        "female": "Female",
+                        "male": "Male",
+                        "child": "Child",
+                    }.get(dom, dom.capitalize())
+            return result
+        except Exception as exc:  # noqa: BLE001
+            # Analyze is a best-effort feature — never take down the
+            # whole analyze call because the age/gender head had a bad
+            # day. Mark the failure so the emotion branch still runs.
+            self._age_gender_error = f"runtime: {type(exc).__name__}: {exc}"
+            return {}
+
+    # --- emotion ------------------------------------------------------
+    def _ensure_emotion(self):
+        if self._emotion_loaded:
+            return
+        self._emotion_loaded = True
+        model_id = self._options.get("emotion_model", self.DEFAULT_EMOTION_MODEL)
+        if not model_id:
+            self._emotion_error = "disabled"
+            return
+        try:
+            from transformers import pipeline  # type: ignore
+
+            self._emotion = pipeline("audio-classification", model=model_id)
+        except Exception as exc:  # noqa: BLE001
+            self._emotion_error = f"{type(exc).__name__}: {exc}"
+
+    def _infer_emotion(self, audio_path: str) -> dict[str, Any]:
+        self._ensure_emotion()
+        if self._emotion is None:
+            return {}
+        try:
+            raw = self._emotion(audio_path, top_k=8)
+        except Exception as exc:  # noqa: BLE001
+            # Second-line defense: don't fail the whole analyze call
+            # over a runtime inference hiccup.
+            self._emotion_error = f"runtime: {type(exc).__name__}: {exc}"
+            return {}
+        emotion_map = {row["label"].lower(): float(row["score"]) for row in raw}
+        if not emotion_map:
+            return {}
+        dom = max(emotion_map.items(), key=lambda kv: kv[1])[0]
+        return {"emotion": emotion_map, "dominant_emotion": dom}
+
+    # --- orchestrator -------------------------------------------------
+    def analyze(self, audio_path: str, waveform_16k, actions: Iterable[str]) -> dict[str, Any]:
+        wanted = {a.strip().lower() for a in actions} if actions else {"age", "gender", "emotion"}
+        result: dict[str, Any] = {}
+        if "age" in wanted or "gender" in wanted:
+            ag = self._infer_age_gender(waveform_16k)
+            if "age" in wanted and "age" in ag:
+                result["age"] = ag["age"]
+            if "gender" in wanted:
+                if "gender" in ag:
+                    result["gender"] = ag["gender"]
+                if "dominant_gender" in ag:
+                    result["dominant_gender"] = ag["dominant_gender"]
+        if "emotion" in wanted:
+            em = self._infer_emotion(audio_path)
+            result.update(em)
+        return result
+
+
+class SpeechBrainEngine:
+    """ECAPA-TDNN via SpeechBrain. Auto-downloads on first use."""
+
+    name = "speechbrain-ecapa-tdnn"
+
+    def __init__(self, model_name: str, options: dict[str, str]):
+        # Late imports so the module can be introspected / tested
+        # without torch / speechbrain being installed.
+        from speechbrain.inference.speaker import EncoderClassifier  # type: ignore
+
+        source = options.get("source") or model_name or "speechbrain/spkrec-ecapa-voxceleb"
+        savedir = options.get("_model_path") or os.environ.get("HF_HOME") or "./pretrained_models"
+        self._model = EncoderClassifier.from_hparams(source=source, savedir=savedir)
+        self._analysis = AnalysisHead(options)
+
+    def _load_waveform(self, path: str):
+        # Use soundfile + torch directly — torchaudio.load in torchaudio
+        # 2.8+ requires the torchcodec package for decoding, which adds
+        # another heavy ffmpeg-linked dep. soundfile covers WAV/FLAC
+        # which is what we care about here.
+        import numpy as np
+        import soundfile as sf  # type: ignore
+        import torch  # type: ignore
+
+        audio, sr = sf.read(path, always_2d=False)
+        if audio.ndim > 1:
+            audio = audio.mean(axis=1)
+        audio = np.asarray(audio, dtype=np.float32)
+        if sr != 16000:
+            # Simple linear resample — good enough for 16kHz downsampling
+            # from 44.1/48kHz, and we expect 16kHz inputs in practice.
+            ratio = 16000 / float(sr)
+            n = int(round(len(audio) * ratio))
+            audio = np.interp(
+                np.linspace(0, len(audio), n, endpoint=False),
+                np.arange(len(audio)),
+                audio,
+            ).astype(np.float32)
+        return torch.from_numpy(audio).unsqueeze(0)  # [1, T]
+
+    def embed(self, audio_path: str) -> list[float]:
+        waveform = self._load_waveform(audio_path)
+        vec = self._model.encode_batch(waveform).squeeze().detach().cpu().numpy()
+        return [float(x) for x in vec]
+
+    def compare(self, audio1: str, audio2: str) -> float:
+        return _cosine_distance(self.embed(audio1), self.embed(audio2))
+
+    def analyze(self, audio_path: str, actions):
+        # Age / gender / emotion aren't produced by ECAPA-TDNN itself;
+        # delegate to AnalysisHead which wraps separate Apache-2.0
+        # checkpoints. Returns a single segment spanning the clip —
+        # segmentation / diarisation is a future enhancement.
+        waveform = self._load_waveform(audio_path)
+        mono = waveform.squeeze().detach().cpu().numpy()
+        attrs = self._analysis.analyze(audio_path, mono, actions)
+        if not attrs:
+            raise NotImplementedError(
+                "analyze head failed to load — install transformers + torch or pass age_gender_model/emotion_model options"
+            )
+        duration = float(mono.shape[-1]) / 16000.0 if mono.size else 0.0
+        return [dict(start=0.0, end=duration, **attrs)]
+
+
+class OnnxDirectEngine:
+    """Run a pre-exported ONNX speaker encoder (WeSpeaker / 3D-Speaker)."""
+
+    name = "onnx-direct"
+
+    def __init__(self, model_name: str, options: dict[str, str]):
+        import onnxruntime as ort  # type: ignore
+
+        # The gallery is expected to have dropped the ONNX file under
+        # the models directory; accept either an absolute path or a
+        # filename relative to _model_path.
+        onnx_path = options.get("model_path") or options.get("onnx")
+        if not onnx_path:
+            raise ValueError("OnnxDirectEngine requires `model_path: <file.onnx>` in options")
+        if not os.path.isabs(onnx_path):
+            onnx_path = os.path.join(options.get("_model_path", ""), onnx_path)
+        if not os.path.isfile(onnx_path):
+            raise FileNotFoundError(f"ONNX model not found: {onnx_path}")
+
+        providers = options.get("providers")
+        if providers:
+            provider_list = [p.strip() for p in providers.split(",") if p.strip()]
+        else:
+            provider_list = ["CPUExecutionProvider"]
+        self._session = ort.InferenceSession(onnx_path, providers=provider_list)
+        input_meta = self._session.get_inputs()[0]
+        self._input_name = input_meta.name
+        # Pre-exported speaker encoders come in two shapes:
+        #   rank-2  [batch, samples]          — some 3D-Speaker exports feed raw waveform.
+        #   rank-3  [batch, frames, n_mels]   — WeSpeaker and most Kaldi-lineage encoders
+        #                                        expect pre-computed Kaldi FBank features.
+        # We detect this at load time and branch in embed(), because feeding raw audio
+        # into a rank-3 graph is exactly what triggered
+        # "Invalid rank for input: feats Got: 2 Expected: 3".
+        self._input_rank = len(input_meta.shape) if input_meta.shape is not None else 2
+        self._expected_sr = int(options.get("sample_rate", "16000"))
+        self._fbank_mels = int(options.get("fbank_num_mel_bins", "80"))
+        self._fbank_frame_length_ms = float(options.get("fbank_frame_length_ms", "25"))
+        self._fbank_frame_shift_ms = float(options.get("fbank_frame_shift_ms", "10"))
+        # Per-utterance cepstral mean normalisation — on for WeSpeaker by default,
+        # toggleable for encoders that expect raw FBank.
+        self._fbank_cmn = options.get("fbank_cmn", "true").lower() in ("1", "true", "yes")
+        self._analysis = AnalysisHead(options)
+
+    def _load_waveform(self, path: str):
+        import numpy as np
+        import soundfile as sf  # type: ignore
+
+        audio, sr = sf.read(path, always_2d=False)
+        if sr != self._expected_sr:
+            # Cheap linear resample — good enough for sanity; callers
+            # should pre-resample for production.
+            ratio = self._expected_sr / float(sr)
+            n = int(round(len(audio) * ratio))
+            audio = np.interp(
+                np.linspace(0, len(audio), n, endpoint=False),
+                np.arange(len(audio)),
+                audio,
+            )
+        if audio.ndim > 1:
+            audio = audio.mean(axis=1)
+        return audio.astype("float32")
+
+    def embed(self, audio_path: str) -> list[float]:
+        import numpy as np
+
+        audio = self._load_waveform(audio_path)
+        if self._input_rank >= 3:
+            feats = self._extract_fbank(audio)        # [frames, n_mels]
+            feed = feats[np.newaxis, :, :]             # [1, frames, n_mels]
+        else:
+            feed = audio.reshape(1, -1)                # [1, samples]
+        out = self._session.run(None, {self._input_name: feed})
+        vec = np.asarray(out[0]).reshape(-1)
+        return [float(x) for x in vec]
+
+    def _extract_fbank(self, audio):
+        """Compute Kaldi-style 80-dim FBank features for speaker encoders that
+        expect pre-featurised input (WeSpeaker, most 3D-Speaker exports).
+        torchaudio is already a backend dependency for SpeechBrain — no new
+        package required."""
+        import numpy as np
+        import torch  # type: ignore
+        import torchaudio.compliance.kaldi as kaldi  # type: ignore
+
+        tensor = torch.from_numpy(audio).unsqueeze(0)  # [1, samples]
+        feats = kaldi.fbank(
+            tensor,
+            sample_frequency=self._expected_sr,
+            num_mel_bins=self._fbank_mels,
+            frame_length=self._fbank_frame_length_ms,
+            frame_shift=self._fbank_frame_shift_ms,
+            dither=0.0,
+        )  # [frames, n_mels]
+        if self._fbank_cmn:
+            feats = feats - feats.mean(dim=0, keepdim=True)
+        return feats.numpy().astype(np.float32)
+
+    def compare(self, audio1: str, audio2: str) -> float:
+        return _cosine_distance(self.embed(audio1), self.embed(audio2))
+
+    def analyze(self, audio_path: str, actions):
+        # AnalysisHead expects 16kHz mono; _load_waveform already
+        # resamples to self._expected_sr. If the user configured a
+        # non-16k expected rate, resample one more time for analyze.
+        audio = self._load_waveform(audio_path)
+        if self._expected_sr != 16000:
+            import numpy as np
+
+            ratio = 16000 / float(self._expected_sr)
+            n = int(round(len(audio) * ratio))
+            audio = np.interp(
+                np.linspace(0, len(audio), n, endpoint=False),
+                np.arange(len(audio)),
+                audio,
+            ).astype("float32")
+        attrs = self._analysis.analyze(audio_path, audio, actions)
+        if not attrs:
+            raise NotImplementedError(
+                "analyze head failed to load — install transformers + torch or pass age_gender_model/emotion_model options"
+            )
+        duration = float(len(audio)) / 16000.0 if len(audio) else 0.0
+        return [dict(start=0.0, end=duration, **attrs)]
+
+
+def build_engine(model_name: str, options: dict[str, str]) -> tuple[SpeakerEngine, str]:
+    """Pick an engine based on the options. ONNX path takes priority:
+    if the gallery has dropped a `model_path:` or `onnx:` option, run
+    the direct ONNX engine. Otherwise, fall back to SpeechBrain.
+    """
+    engine_kind = (options.get("engine") or "").lower()
+    if engine_kind == "onnx" or options.get("model_path") or options.get("onnx"):
+        return OnnxDirectEngine(model_name, options), OnnxDirectEngine.name
+    return SpeechBrainEngine(model_name, options), SpeechBrainEngine.name

backend/python/speaker-recognition/install.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+set -e
+
+backend_dir=$(dirname $0)
+if [ -d $backend_dir/common ]; then
+    source $backend_dir/common/libbackend.sh
+else
+    source $backend_dir/../common/libbackend.sh
+fi
+
+installRequirements
+
+# No pre-baked model weights. Weights flow through LocalAI's gallery
+# `files:` mechanism — see gallery entries for speechbrain-ecapa-tdnn
+# and WeSpeaker / 3D-Speaker ONNX packs. SpeechBrain's
+# EncoderClassifier.from_hparams also knows how to auto-download from
+# HuggingFace into the configured savedir (we point it at ModelPath),
+# so the first LoadModel call bootstraps the checkpoint if the gallery
+# flow wasn't used.

backend/python/speaker-recognition/requirements-cpu.txt

@@ -0,0 +1,5 @@
+torch
+torchaudio
+speechbrain
+transformers
+onnxruntime

backend/python/speaker-recognition/requirements-cublas12.txt

@@ -0,0 +1,5 @@
+torch
+torchaudio
+speechbrain
+transformers
+onnxruntime-gpu

backend/python/speaker-recognition/requirements.txt

@@ -0,0 +1,5 @@
+grpcio==1.71.0
+protobuf
+grpcio-tools
+numpy
+soundfile

backend/python/speaker-recognition/run.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+backend_dir=$(dirname $0)
+if [ -d $backend_dir/common ]; then
+    source $backend_dir/common/libbackend.sh
+else
+    source $backend_dir/../common/libbackend.sh
+fi
+
+startBackend $@

backend/python/speaker-recognition/test.py

@@ -0,0 +1,78 @@
+"""Unit tests for the speaker-recognition gRPC backend.
+
+The servicer is instantiated in-process (no gRPC channel) and driven
+directly. The default path exercises SpeechBrain's ECAPA-TDNN — the
+first run downloads the checkpoint into a temp savedir. Tests are
+skipped gracefully when the heavy optional dependencies (torch /
+speechbrain / onnxruntime) are not installed, so the gRPC plumbing
+can still be verified on a bare image.
+"""
+from __future__ import annotations
+
+import importlib
+import os
+import sys
+import tempfile
+import unittest
+
+sys.path.insert(0, os.path.dirname(__file__))
+
+import backend_pb2  # noqa: E402
+
+from backend import BackendServicer  # noqa: E402
+
+
+def _have(*mods: str) -> bool:
+    for m in mods:
+        if importlib.util.find_spec(m) is None:
+            return False
+    return True
+
+
+class _FakeCtx:
+    """Minimal stand-in for a gRPC servicer context."""
+
+    def __init__(self) -> None:
+        self.code = None
+        self.details = ""
+
+    def set_code(self, c):
+        self.code = c
+
+    def set_details(self, d):
+        self.details = d
+
+
+class ServicerPlumbingTest(unittest.TestCase):
+    """Checks that LoadModel returns a clear error when no engine deps
+    are installed, and that Voice* calls on an uninitialised servicer
+    surface FAILED_PRECONDITION — both verifying the gRPC wiring
+    without requiring SpeechBrain or ONNX at test time."""
+
+    def test_pre_load_voice_calls_are_rejected(self):
+        svc = BackendServicer()
+        ctx = _FakeCtx()
+        svc.VoiceVerify(backend_pb2.VoiceVerifyRequest(audio1="/tmp/a.wav", audio2="/tmp/b.wav"), ctx)
+        self.assertEqual(str(ctx.code), "StatusCode.FAILED_PRECONDITION")
+
+    def test_load_without_deps_fails_cleanly(self):
+        svc = BackendServicer()
+        req = backend_pb2.ModelOptions(Model="speechbrain/spkrec-ecapa-voxceleb", ModelPath="")
+        result = svc.LoadModel(req, _FakeCtx())
+        # Either the deps are installed and it loaded, or they aren't
+        # and we got a structured error instead of a crash.
+        self.assertTrue(result.success or "engine init failed" in result.message)
+
+
+@unittest.skipUnless(_have("speechbrain", "torch", "torchaudio"), "speechbrain / torch missing")
+class SpeechBrainEngineSmokeTest(unittest.TestCase):
+    def test_load_and_embed(self):
+        svc = BackendServicer()
+        with tempfile.TemporaryDirectory() as td:
+            req = backend_pb2.ModelOptions(Model="speechbrain/spkrec-ecapa-voxceleb", ModelPath=td)
+            result = svc.LoadModel(req, _FakeCtx())
+            self.assertTrue(result.success, result.message)
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/speaker-recognition/test.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+set -e
+
+backend_dir=$(dirname $0)
+if [ -d $backend_dir/common ]; then
+    source $backend_dir/common/libbackend.sh
+else
+    source $backend_dir/../common/libbackend.sh
+fi
+
+runUnittests

backend/python/vllm-omni/install.sh

@@ -12,11 +12,15 @@ else
     source $backend_dir/../common/libbackend.sh
 fi
 
-# Handle l4t build profiles (Python 3.12, pip fallback) if needed
+# Handle l4t build profiles (Python 3.12, pip fallback) if needed.
+# unsafe-best-match is required on l4t13 because the jetson-ai-lab index
+# lists transitive deps at limited versions — without it uv pins to the
+# first matching index and fails to resolve a compatible wheel from PyPI.
 if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
   PYTHON_VERSION="3.12"
   PYTHON_PATCH="12"
   PY_STANDALONE_TAG="20251120"
+  EXTRA_PIP_INSTALL_FLAGS="${EXTRA_PIP_INSTALL_FLAGS:-} --index-strategy=unsafe-best-match"
 fi
 
 if [ "x${BUILD_PROFILE}" == "xl4t12" ]; then
@@ -26,7 +30,11 @@ fi
 # Install base requirements first
 installRequirements
 
-# Install vllm based on build type
+# Install vllm based on build type. vllm-omni tracks vllm master from
+# source (cloned below) so we leave the upstream vllm dependency unpinned
+# — vllm 0.19+ ships cu130 wheels by default, which is what we want for
+# cublas13. Older cuda12/rocm/cpu paths still resolve a compatible wheel
+# from the relevant channel.
 if [ "x${BUILD_TYPE}" == "xhipblas" ]; then
     # ROCm
     if [ "x${USE_PIP}" == "xtrue" ]; then
@@ -34,8 +42,26 @@ if [ "x${BUILD_TYPE}" == "xhipblas" ]; then
     else
         uv pip install vllm==0.14.0 --extra-index-url https://wheels.vllm.ai/rocm/0.14.0/rocm700
     fi
+elif [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    # JetPack 7 / L4T arm64 cu130 — vllm comes from the prebuilt SBSA wheel
+    # at jetson-ai-lab. Version is unpinned: the index ships whatever build
+    # matches the cu130/cp312 ABI. unsafe-best-match lets uv fall through
+    # to PyPI for transitive deps not present on the jetson-ai-lab index.
+    if [ "x${USE_PIP}" == "xtrue" ]; then
+        pip install vllm --extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+    else
+        uv pip install --index-strategy=unsafe-best-match vllm --extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+    fi
+elif [ "x${BUILD_PROFILE}" == "xcublas13" ]; then
+    # vllm 0.19+ defaults to cu130 wheels on PyPI, no extra index needed.
+    if [ "x${USE_PIP}" == "xtrue" ]; then
+        pip install vllm --torch-backend=auto
+    else
+        uv pip install vllm --torch-backend=auto
+    fi
 elif [ "x${BUILD_TYPE}" == "xcublas" ] || [ "x${BUILD_TYPE}" == "x" ]; then
-    # CUDA (default) or CPU
+    # cuda12 / CPU — keep the 0.14.0 pin for compatibility with the existing
+    # cuda12 vllm-omni image; bumping should be its own change.
     if [ "x${USE_PIP}" == "xtrue" ]; then
         pip install vllm==0.14.0 --torch-backend=auto
     else

backend/python/vllm-omni/requirements-cublas13.txt

@@ -0,0 +1,5 @@
+--extra-index-url https://download.pytorch.org/whl/cu130
+accelerate
+torch
+transformers
+bitsandbytes

backend/python/vllm-omni/requirements-l4t13.txt

@@ -0,0 +1,13 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+accelerate
+torch
+torchvision
+torchaudio
+transformers
+bitsandbytes
+flash-attn
+diffusers
+librosa
+soundfile
+pillow
+numpy

backend/python/vllm/install.sh

@@ -32,6 +32,22 @@ if [ "x${BUILD_PROFILE}" == "xcpu" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
+# JetPack 7 / L4T arm64 wheels (torch, vllm, flash-attn) live on
+# pypi.jetson-ai-lab.io and are built for cp312, so bump the venv Python
+# accordingly. JetPack 6 keeps cp310 + USE_PIP=true. unsafe-best-match
+# is required because the jetson-ai-lab index lists transitive deps at
+# limited versions — without it uv pins to the first matching index and
+# fails to resolve a compatible wheel from PyPI.
+if [ "x${BUILD_PROFILE}" == "xl4t12" ]; then
+    USE_PIP=true
+fi
+if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    PYTHON_VERSION="3.12"
+    PYTHON_PATCH="12"
+    PY_STANDALONE_TAG="20251120"
+    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
+fi
+
 # FROM_SOURCE=true on a CPU build skips the prebuilt vllm wheel in
 # requirements-cpu-after.txt and compiles vllm locally against the host's
 # actual CPU. Not used by default because it takes ~30-40 minutes, but

backend/python/vllm/requirements-cublas12-after.txt

@@ -1,2 +1,9 @@
-https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
+# flash-attn wheels are ABI-tied to a specific torch version. vllm forces
+# torch==2.10.0 as a hard dep, but flash-attn 2.8.3 (latest) only ships
+# prebuilt wheels up to torch 2.8 — any wheel we pin here gets silently
+# broken when vllm upgrades torch during install, producing an undefined
+# libc10_cuda symbol at import time. FlashInfer (required by vllm) covers
+# attention, and rotary_embedding/common.py guards the flash_attn import
+# with find_spec(), so skipping flash-attn is safe and the only stable
+# choice until upstream ships a torch-2.10 wheel.
 vllm

backend/python/vllm/requirements-cublas12.txt

@@ -1,4 +1,4 @@
 accelerate
-torch==2.7.0
+torch
 transformers
 bitsandbytes

backend/python/vllm/requirements-cublas13-after.txt

@@ -0,0 +1,2 @@
+--extra-index-url https://download.pytorch.org/whl/cu130
+vllm

backend/python/vllm/requirements-cublas13.txt

@@ -0,0 +1,5 @@
+--extra-index-url https://download.pytorch.org/whl/cu130
+accelerate
+torch
+transformers
+bitsandbytes

backend/python/vllm/requirements-l4t13-after.txt

@@ -0,0 +1,2 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+vllm

backend/python/vllm/requirements-l4t13.txt

@@ -0,0 +1,8 @@
+--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
+accelerate
+torch
+torchvision
+torchaudio
+transformers
+bitsandbytes
+flash-attn

backend/rust/kokoros/Cargo.lock

@@ -1867,9 +1867,9 @@ dependencies = [
 
 [[package]]
 name = "rustls-webpki"
-version = "0.103.10"
+version = "0.103.13"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "df33b2b81ac578cabaf06b89b0631153a3f416b0a886e8a7a1707fb51abbd1ef"
+checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
 dependencies = [
  "ring",
  "rustls-pki-types",

backend/rust/kokoros/src/service.rs

@@ -372,6 +372,41 @@ impl Backend for KokorosService {
         Err(Status::unimplemented("Not supported"))
     }
 
+    async fn face_verify(
+        &self,
+        _: Request<backend::FaceVerifyRequest>,
+    ) -> Result<Response<backend::FaceVerifyResponse>, Status> {
+        Err(Status::unimplemented("Not supported"))
+    }
+
+    async fn face_analyze(
+        &self,
+        _: Request<backend::FaceAnalyzeRequest>,
+    ) -> Result<Response<backend::FaceAnalyzeResponse>, Status> {
+        Err(Status::unimplemented("Not supported"))
+    }
+
+    async fn voice_verify(
+        &self,
+        _: Request<backend::VoiceVerifyRequest>,
+    ) -> Result<Response<backend::VoiceVerifyResponse>, Status> {
+        Err(Status::unimplemented("Not supported"))
+    }
+
+    async fn voice_analyze(
+        &self,
+        _: Request<backend::VoiceAnalyzeRequest>,
+    ) -> Result<Response<backend::VoiceAnalyzeResponse>, Status> {
+        Err(Status::unimplemented("Not supported"))
+    }
+
+    async fn voice_embed(
+        &self,
+        _: Request<backend::VoiceEmbedRequest>,
+    ) -> Result<Response<backend::VoiceEmbedResponse>, Status> {
+        Err(Status::unimplemented("Not supported"))
+    }
+
     async fn stores_set(
         &self,
         _: Request<backend::StoresSetOptions>,

core/application/application.go

@@ -14,6 +14,7 @@ import (
 	"github.com/mudler/LocalAI/core/services/facerecognition"
 	"github.com/mudler/LocalAI/core/services/galleryop"
 	"github.com/mudler/LocalAI/core/services/nodes"
+	"github.com/mudler/LocalAI/core/services/voicerecognition"
 	"github.com/mudler/LocalAI/core/templates"
 	pkggrpc "github.com/mudler/LocalAI/pkg/grpc"
 	"github.com/mudler/LocalAI/pkg/model"
@@ -29,6 +30,12 @@ import (
 // family per deployment; we keep the door open instead.
 const faceEmbeddingDim = 0
 
+// voiceEmbeddingDim is the expected dimension for speaker embeddings.
+// 0 so the Registry accepts whatever dim the loaded recognizer
+// produces — ECAPA-TDNN is 192, WeSpeaker ResNet34 is 256, 3D-Speaker
+// ERes2Net is 192, CAM++ is 512.
+const voiceEmbeddingDim = 0
+
 type Application struct {
 	backendLoader      *config.ModelConfigLoader
 	modelLoader        *model.ModelLoader
@@ -39,6 +46,7 @@ type Application struct {
 	agentJobService    *agentpool.AgentJobService
 	agentPoolService   atomic.Pointer[agentpool.AgentPoolService]
 	faceRegistry       facerecognition.Registry
+	voiceRegistry      voicerecognition.Registry
 	authDB             *gorm.DB
 	watchdogMutex      sync.Mutex
 	watchdogStop       chan bool
@@ -73,10 +81,30 @@ func newApplication(appConfig *config.ApplicationConfig) *Application {
 	// The resolver closes over the ModelLoader so the Registry stays
 	// decoupled from loader plumbing; swapping in a postgres-backed
 	// implementation later is a single construction change here.
+	//
+	// `faceStoreName` is the default namespace passed to StoreBackend when
+	// the request doesn't override it. Face and voice MUST use distinct
+	// namespaces — the local-store gRPC surface rejects mixed dimensions
+	// inside one namespace ("Try to add key with length N when existing
+	// length is M"). ArcFace buffalo_l produces 512-dim embeddings while
+	// ECAPA-TDNN produces 192-dim; enrolling one after the other into a
+	// shared namespace is exactly how we hit that error.
+	const (
+		faceStoreName  = "localai-face-biometrics"
+		voiceStoreName = "localai-voice-biometrics"
+	)
 	faceStoreResolver := func(_ context.Context, storeName string) (pkggrpc.Backend, error) {
 		return corebackend.StoreBackend(ml, appConfig, storeName, "")
 	}
-	app.faceRegistry = facerecognition.NewStoreRegistry(faceStoreResolver, "", faceEmbeddingDim)
+	app.faceRegistry = facerecognition.NewStoreRegistry(faceStoreResolver, faceStoreName, faceEmbeddingDim)
+
+	// Voice (speaker) recognition registry — same plumbing, separate
+	// namespace so embedding spaces stay isolated (a face vector and a
+	// speaker vector are not comparable and differ in dimensionality).
+	voiceStoreResolver := func(_ context.Context, storeName string) (pkggrpc.Backend, error) {
+		return corebackend.StoreBackend(ml, appConfig, storeName, "")
+	}
+	app.voiceRegistry = voicerecognition.NewStoreRegistry(voiceStoreResolver, voiceStoreName, voiceEmbeddingDim)
 
 	return app
 }
@@ -130,6 +158,14 @@ func (a *Application) FaceRegistry() facerecognition.Registry {
 	return a.faceRegistry
 }
 
+// VoiceRegistry returns the voice (speaker) recognition registry used
+// for 1:N identification. Same in-memory local-store backing as
+// FaceRegistry but a separate instance — voice embeddings live in
+// their own vector space.
+func (a *Application) VoiceRegistry() voicerecognition.Registry {
+	return a.voiceRegistry
+}
+
 // AuthDB returns the auth database connection, or nil if auth is not enabled.
 func (a *Application) AuthDB() *gorm.DB {
 	return a.authDB

core/application/startup.go

@@ -242,6 +242,12 @@ func New(opts ...config.AppOption) (*Application, error) {
 		bmFn := func() galleryop.BackendManager { return application.GalleryService().BackendManager() }
 		uc := NewUpgradeChecker(options, application.ModelLoader(), application.distributedDB(), bmFn)
 		application.upgradeChecker = uc
+		// Refresh the upgrade cache the moment a backend op finishes — otherwise
+		// the UI keeps showing a just-upgraded backend as upgradeable until the
+		// next 6-hour tick. TriggerCheck is non-blocking.
+		if gs := application.GalleryService(); gs != nil {
+			gs.OnBackendOpCompleted = uc.TriggerCheck
+		}
 		go uc.Run(options.Context)
 	}
 

core/backend/stores.go

@@ -11,8 +11,17 @@ func StoreBackend(sl *model.ModelLoader, appConfig *config.ApplicationConfig, st
 	if backend == "" {
 		backend = model.LocalStoreBackend
 	}
+	// ModelLoader caches backend processes by `modelID`, not by the `model`
+	// passed via WithModel. Without a distinct modelID, every StoreBackend
+	// call collapses to the same `modelID=""` cache slot — face (512-D) and
+	// voice (192-D) biometrics would then share the same local-store process
+	// and the second enrollment would fail with
+	//   Try to add key with length N when existing length is M
+	// Use the store namespace as modelID so each namespace gets its own
+	// process instance and its own in-memory Store{}.
 	sc := []model.Option{
 		model.WithBackendString(backend),
+		model.WithModelID(storeName),
 		model.WithModel(storeName),
 	}
 

core/backend/voice_analyze.go

@@ -0,0 +1,58 @@
+package backend
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/trace"
+	"github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/LocalAI/pkg/model"
+)
+
+func VoiceAnalyze(
+	audio string,
+	actions []string,
+	loader *model.ModelLoader,
+	appConfig *config.ApplicationConfig,
+	modelConfig config.ModelConfig,
+) (*proto.VoiceAnalyzeResponse, error) {
+	opts := ModelOptions(modelConfig, appConfig)
+	voiceModel, err := loader.Load(opts...)
+	if err != nil {
+		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)
+		return nil, err
+	}
+	if voiceModel == nil {
+		return nil, fmt.Errorf("could not load voice recognition model")
+	}
+
+	var startTime time.Time
+	if appConfig.EnableTracing {
+		trace.InitBackendTracingIfEnabled(appConfig.TracingMaxItems)
+		startTime = time.Now()
+	}
+
+	res, err := voiceModel.VoiceAnalyze(context.Background(), &proto.VoiceAnalyzeRequest{
+		Audio:   audio,
+		Actions: actions,
+	})
+
+	if appConfig.EnableTracing {
+		errStr := ""
+		if err != nil {
+			errStr = err.Error()
+		}
+		trace.RecordBackendTrace(trace.BackendTrace{
+			Timestamp: startTime,
+			Duration:  time.Since(startTime),
+			Type:      trace.BackendTraceVoiceAnalyze,
+			ModelName: modelConfig.Name,
+			Backend:   modelConfig.Backend,
+			Error:     errStr,
+		})
+	}
+
+	return res, err
+}

core/backend/voice_embed.go

@@ -0,0 +1,66 @@
+package backend
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/trace"
+	"github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/LocalAI/pkg/model"
+)
+
+// VoiceEmbed returns a speaker embedding (typically 192-d for ECAPA-TDNN)
+// for the audio file at audioPath. Unlike ModelEmbedding (which is
+// OpenAI-compatible and text-only), this call takes an audio path and
+// returns the backend's speaker-encoder output.
+func VoiceEmbed(
+	audioPath string,
+	loader *model.ModelLoader,
+	appConfig *config.ApplicationConfig,
+	modelConfig config.ModelConfig,
+) (*proto.VoiceEmbedResponse, error) {
+	opts := ModelOptions(modelConfig, appConfig)
+	voiceModel, err := loader.Load(opts...)
+	if err != nil {
+		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)
+		return nil, err
+	}
+	if voiceModel == nil {
+		return nil, fmt.Errorf("could not load voice recognition model")
+	}
+
+	var startTime time.Time
+	if appConfig.EnableTracing {
+		trace.InitBackendTracingIfEnabled(appConfig.TracingMaxItems)
+		startTime = time.Now()
+	}
+
+	res, err := voiceModel.VoiceEmbed(context.Background(), &proto.VoiceEmbedRequest{
+		Audio: audioPath,
+	})
+
+	if appConfig.EnableTracing {
+		errStr := ""
+		if err != nil {
+			errStr = err.Error()
+		}
+		trace.RecordBackendTrace(trace.BackendTrace{
+			Timestamp: startTime,
+			Duration:  time.Since(startTime),
+			Type:      trace.BackendTraceVoiceEmbed,
+			ModelName: modelConfig.Name,
+			Backend:   modelConfig.Backend,
+			Error:     errStr,
+		})
+	}
+
+	if err != nil {
+		return nil, err
+	}
+	if res == nil || len(res.Embedding) == 0 {
+		return nil, fmt.Errorf("voice embedding returned empty vector (no speech detected?)")
+	}
+	return res, nil
+}

core/backend/voice_verify.go

@@ -0,0 +1,61 @@
+package backend
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/trace"
+	"github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/LocalAI/pkg/model"
+)
+
+func VoiceVerify(
+	audio1, audio2 string,
+	threshold float32,
+	antiSpoofing bool,
+	loader *model.ModelLoader,
+	appConfig *config.ApplicationConfig,
+	modelConfig config.ModelConfig,
+) (*proto.VoiceVerifyResponse, error) {
+	opts := ModelOptions(modelConfig, appConfig)
+	voiceModel, err := loader.Load(opts...)
+	if err != nil {
+		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)
+		return nil, err
+	}
+	if voiceModel == nil {
+		return nil, fmt.Errorf("could not load voice recognition model")
+	}
+
+	var startTime time.Time
+	if appConfig.EnableTracing {
+		trace.InitBackendTracingIfEnabled(appConfig.TracingMaxItems)
+		startTime = time.Now()
+	}
+
+	res, err := voiceModel.VoiceVerify(context.Background(), &proto.VoiceVerifyRequest{
+		Audio1:       audio1,
+		Audio2:       audio2,
+		Threshold:    threshold,
+		AntiSpoofing: antiSpoofing,
+	})
+
+	if appConfig.EnableTracing {
+		errStr := ""
+		if err != nil {
+			errStr = err.Error()
+		}
+		trace.RecordBackendTrace(trace.BackendTrace{
+			Timestamp: startTime,
+			Duration:  time.Since(startTime),
+			Type:      trace.BackendTraceVoiceVerify,
+			ModelName: modelConfig.Name,
+			Backend:   modelConfig.Backend,
+			Error:     errStr,
+		})
+	}
+
+	return res, err
+}

core/cli/worker.go

@@ -90,6 +90,14 @@ type WorkerCMD struct {
 	RegistrationToken string `env:"LOCALAI_REGISTRATION_TOKEN" help:"Token for authenticating with the frontend" group:"registration"`
 	HeartbeatInterval string `env:"LOCALAI_HEARTBEAT_INTERVAL" default:"10s" help:"Interval between heartbeats" group:"registration"`
 	NodeLabels        string `env:"LOCALAI_NODE_LABELS" help:"Comma-separated key=value labels for this node (e.g. tier=fast,gpu=a100)" group:"registration"`
+	// MaxReplicasPerModel caps how many replicas of any one model can run on
+	// this worker concurrently. Default 1 = historical single-replica
+	// behavior. Set higher when a node has enough VRAM to host multiple
+	// copies of the same model (e.g. a fat 128 GiB box running 4× of a
+	// 24 GiB model for throughput). The auto-label `node.replica-slots=N`
+	// is published so model schedulers can target high-capacity nodes via
+	// the existing label selector.
+	MaxReplicasPerModel int `env:"LOCALAI_MAX_REPLICAS_PER_MODEL" default:"1" help:"Max replicas of any single model on this worker. Default 1 preserves single-replica behavior; set higher to allow stacking replicas on a fat node." group:"registration"`
 
 	// NATS (required)
 	NatsURL string `env:"LOCALAI_NATS_URL" required:"" help:"NATS server URL" group:"distributed"`
@@ -567,22 +575,35 @@ func (s *backendSupervisor) getAddr(backend string) string {
 	return ""
 }
 
+// buildProcessKey is the supervisor's stable identifier for a backend gRPC
+// process. It includes the replica index so the same model can run multiple
+// processes on a worker simultaneously without colliding on the same map slot
+// or port. The "#N" suffix is purely internal — the controller never reads it.
+func buildProcessKey(modelID, backend string, replicaIndex int) string {
+	base := modelID
+	if base == "" {
+		base = backend
+	}
+	return fmt.Sprintf("%s#%d", base, replicaIndex)
+}
+
 // installBackend handles the backend.install flow:
-// 1. If already running for this model, return existing address
+// 1. If already running for this (model, replica) slot, return existing address
 // 2. Install backend from gallery (if not already installed)
 // 3. Find backend binary
 // 4. Start gRPC process on a new port
 // Returns the gRPC address of the backend process.
+//
+// ProcessKey includes the replica index so a worker with MaxReplicasPerModel>1
+// can host multiple processes for the same model on distinct ports. Old
+// controllers (no replica_index in the request) implicitly target replica 0,
+// which preserves single-replica behavior.
 func (s *backendSupervisor) installBackend(req messaging.BackendInstallRequest) (string, error) {
-	// Process key: use ModelID if provided (per-model process), else backend name
-	processKey := req.ModelID
-	if processKey == "" {
-		processKey = req.Backend
-	}
+	processKey := buildProcessKey(req.ModelID, req.Backend, int(req.ReplicaIndex))
 
-	// If already running for this model, return its address
+	// If already running for this model+replica, return its address
 	if addr := s.getAddr(processKey); addr != "" {
-		xlog.Info("Backend already running for model", "backend", req.Backend, "model", req.ModelID, "addr", addr)
+		xlog.Info("Backend already running for model replica", "backend", req.Backend, "model", req.ModelID, "replica", req.ReplicaIndex, "addr", addr)
 		return addr, nil
 	}
 
@@ -886,13 +907,18 @@ func (cmd *WorkerCMD) registrationBody() map[string]any {
 	totalVRAM, _ := xsysinfo.TotalAvailableVRAM()
 	gpuVendor, _ := xsysinfo.DetectGPUVendor()
 
+	maxReplicas := cmd.MaxReplicasPerModel
+	if maxReplicas < 1 {
+		maxReplicas = 1
+	}
 	body := map[string]any{
-		"name":           nodeName,
-		"address":        cmd.advertiseAddr(),
-		"http_address":   cmd.advertiseHTTPAddr(),
-		"total_vram":     totalVRAM,
-		"available_vram": totalVRAM, // initially all VRAM is available
-		"gpu_vendor":     gpuVendor,
+		"name":                   nodeName,
+		"address":                cmd.advertiseAddr(),
+		"http_address":           cmd.advertiseHTTPAddr(),
+		"total_vram":             totalVRAM,
+		"available_vram":         totalVRAM, // initially all VRAM is available
+		"gpu_vendor":             gpuVendor,
+		"max_replicas_per_model": maxReplicas,
 	}
 
 	// If no GPU detected, report system RAM so the scheduler/UI has capacity info
@@ -906,39 +932,40 @@ func (cmd *WorkerCMD) registrationBody() map[string]any {
 		body["token"] = cmd.RegistrationToken
 	}
 
-	// Parse and add static node labels
+	// Parse and add static node labels. Always include the auto-label
+	// `node.replica-slots=N` so AND-selectors in ModelSchedulingConfig can
+	// target high-capacity nodes (e.g. {"node.replica-slots":"4"}).
+	labels := make(map[string]string)
 	if cmd.NodeLabels != "" {
-		labels := make(map[string]string)
 		for _, pair := range strings.Split(cmd.NodeLabels, ",") {
 			pair = strings.TrimSpace(pair)
 			if k, v, ok := strings.Cut(pair, "="); ok {
 				labels[strings.TrimSpace(k)] = strings.TrimSpace(v)
 			}
 		}
-		if len(labels) > 0 {
-			body["labels"] = labels
-		}
 	}
+	labels["node.replica-slots"] = strconv.Itoa(maxReplicas)
+	body["labels"] = labels
 
 	return body
 }
 
 // heartbeatBody returns the current VRAM/RAM stats for heartbeat payloads.
+//
+// When aggregate VRAM usage is unknown (no GPU, or temporary detection
+// failure), we deliberately OMIT available_vram so the frontend keeps its
+// last good value — overwriting with 0 makes the UI show the node as "fully
+// used", while reporting total-as-available lies to the scheduler about
+// free capacity.
 func (cmd *WorkerCMD) heartbeatBody() map[string]any {
-	var availVRAM uint64
+	body := map[string]any{}
 	aggregate := xsysinfo.GetGPUAggregateInfo()
 	if aggregate.TotalVRAM > 0 {
-		availVRAM = aggregate.FreeVRAM
-	} else {
-		// Fallback: report total as available (no usage tracking possible)
-		availVRAM, _ = xsysinfo.TotalAvailableVRAM()
+		body["available_vram"] = aggregate.FreeVRAM
 	}
 
-	body := map[string]any{
-		"available_vram": availVRAM,
-	}
-
-	// If no GPU, report system RAM usage instead
+	// CPU-only workers (or workers that lost GPU visibility momentarily):
+	// report system RAM so the scheduler still has capacity info.
 	if aggregate.TotalVRAM == 0 {
 		if ramInfo, err := xsysinfo.GetSystemRAMInfo(); err == nil {
 			body["available_ram"] = ramInfo.Available

core/cli/worker_replica_test.go

@@ -0,0 +1,70 @@
+package cli
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Worker per-replica process keying", func() {
+	Describe("buildProcessKey", func() {
+		// Pin the supervisor's keying contract: distinct replica indexes for
+		// the same modelID produce distinct process keys, so the supervisor
+		// map can hold multiple processes for one model. Dropping the suffix
+		// would re-introduce the original flap (one model, one slot, churn).
+		DescribeTable("produces stable, distinct keys",
+			func(modelID, backend string, replica int, want string) {
+				Expect(buildProcessKey(modelID, backend, replica)).To(Equal(want))
+			},
+			Entry("modelID present, replica 0", "Qwen3-35B", "llama-cpp", 0, "Qwen3-35B#0"),
+			Entry("modelID present, replica 1", "Qwen3-35B", "llama-cpp", 1, "Qwen3-35B#1"),
+			Entry("falls back to backend when modelID empty", "", "llama-cpp", 0, "llama-cpp#0"),
+			Entry("backend fallback with replica 2", "", "llama-cpp", 2, "llama-cpp#2"),
+		)
+
+		It("makes replicas distinguishable", func() {
+			r0 := buildProcessKey("model-a", "llama-cpp", 0)
+			r1 := buildProcessKey("model-a", "llama-cpp", 1)
+			Expect(r0).ToNot(Equal(r1), "replicas of the same model must produce distinct keys")
+		})
+	})
+
+	Describe("registrationBody", func() {
+		It("includes max_replicas_per_model and the auto-label", func() {
+			cmd := &WorkerCMD{
+				Addr:                "worker.example.com:50051",
+				MaxReplicasPerModel: 4,
+			}
+			body := cmd.registrationBody()
+
+			Expect(body).To(HaveKey("max_replicas_per_model"))
+			Expect(body["max_replicas_per_model"]).To(Equal(4))
+
+			labels, ok := body["labels"].(map[string]string)
+			Expect(ok).To(BeTrue(), "labels must be present so selectors can target the slot count")
+			Expect(labels).To(HaveKeyWithValue("node.replica-slots", "4"))
+		})
+
+		It("coerces zero/unset MaxReplicasPerModel to 1", func() {
+			cmd := &WorkerCMD{Addr: "worker.example.com:50051"}
+			body := cmd.registrationBody()
+			Expect(body["max_replicas_per_model"]).To(Equal(1),
+				"unset must default to single-replica behavior, not capacity 0")
+
+			labels := body["labels"].(map[string]string)
+			Expect(labels).To(HaveKeyWithValue("node.replica-slots", "1"))
+		})
+
+		It("preserves user-provided labels alongside the auto-label", func() {
+			cmd := &WorkerCMD{
+				Addr:                "worker.example.com:50051",
+				MaxReplicasPerModel: 2,
+				NodeLabels:          "tier=fast,gpu=a100",
+			}
+			body := cmd.registrationBody()
+			labels := body["labels"].(map[string]string)
+			Expect(labels).To(HaveKeyWithValue("tier", "fast"))
+			Expect(labels).To(HaveKeyWithValue("gpu", "a100"))
+			Expect(labels).To(HaveKeyWithValue("node.replica-slots", "2"))
+		})
+	})
+})

core/config/model_config.go

@@ -588,7 +588,8 @@ const (
 	FLAG_VAD              ModelConfigUsecase = 0b010000000000
 	FLAG_VIDEO            ModelConfigUsecase = 0b100000000000
 	FLAG_DETECTION        ModelConfigUsecase = 0b1000000000000
-	FLAG_FACE_RECOGNITION ModelConfigUsecase = 0b10000000000000
+	FLAG_FACE_RECOGNITION    ModelConfigUsecase = 0b10000000000000
+	FLAG_SPEAKER_RECOGNITION ModelConfigUsecase = 0b100000000000000
 
 	// Common Subsets
 	FLAG_LLM ModelConfigUsecase = FLAG_CHAT | FLAG_COMPLETION | FLAG_EDIT
@@ -612,7 +613,8 @@ func GetAllModelConfigUsecases() map[string]ModelConfigUsecase {
 		"FLAG_LLM":              FLAG_LLM,
 		"FLAG_VIDEO":            FLAG_VIDEO,
 		"FLAG_DETECTION":        FLAG_DETECTION,
-		"FLAG_FACE_RECOGNITION": FLAG_FACE_RECOGNITION,
+		"FLAG_FACE_RECOGNITION":    FLAG_FACE_RECOGNITION,
+		"FLAG_SPEAKER_RECOGNITION": FLAG_SPEAKER_RECOGNITION,
 	}
 }
 
@@ -653,7 +655,7 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool {
 	nonTextGenBackends := []string{
 		"whisper", "piper", "kokoro",
 		"diffusers", "stablediffusion", "stablediffusion-ggml",
-		"rerankers", "silero-vad", "rfdetr", "insightface",
+		"rerankers", "silero-vad", "rfdetr", "insightface", "speaker-recognition",
 		"transformers-musicgen", "ace-step", "acestep-cpp",
 	}
 
@@ -743,6 +745,13 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool {
 		}
 	}
 
+	if (u & FLAG_SPEAKER_RECOGNITION) == FLAG_SPEAKER_RECOGNITION {
+		speakerBackends := []string{"speaker-recognition"}
+		if !slices.Contains(speakerBackends, c.Backend) {
+			return false
+		}
+	}
+
 	if (u & FLAG_SOUND_GENERATION) == FLAG_SOUND_GENERATION {
 		soundGenBackends := []string{"transformers-musicgen", "ace-step", "acestep-cpp", "mock-backend"}
 		if !slices.Contains(soundGenBackends, c.Backend) {
@@ -758,7 +767,7 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool {
 	}
 
 	if (u & FLAG_VAD) == FLAG_VAD {
-		if c.Backend != "silero-vad" && !(c.Backend == "whisper" && slices.Contains(c.Options, "vad_only")) {
+		if c.Backend != "silero-vad" && c.Backend != "sherpa-onnx" && !(c.Backend == "whisper" && slices.Contains(c.Options, "vad_only")) {
 			return false
 		}
 	}

core/gallery/backends.go

@@ -194,6 +194,20 @@ func InstallBackend(ctx context.Context, systemState *system.SystemState, modelL
 
 	name := config.Name
 	backendPath := filepath.Join(systemState.Backend.BackendsPath, name)
+	// Clean up legacy flat-layout artefacts: earlier dev builds of the
+	// golang backends dropped the compiled binary directly at
+	// `<backendsPath>/<name>` (a plain file) instead of
+	// `<backendsPath>/<name>/<name>` (the nested layout the current code
+	// expects). MkdirAll below returns ENOTDIR when such a stale file
+	// exists, permanently blocking any reinstall or upgrade. Remove the
+	// file first so the install can proceed; the new install will write
+	// the correct nested layout, including metadata.json + run.sh.
+	if fi, statErr := os.Lstat(backendPath); statErr == nil && !fi.IsDir() {
+		xlog.Warn("removing stale non-directory backend artefact to make room for fresh install", "path", backendPath)
+		if rmErr := os.Remove(backendPath); rmErr != nil {
+			return fmt.Errorf("failed to remove stale backend artefact at %s: %w", backendPath, rmErr)
+		}
+	}
 	err = os.MkdirAll(backendPath, 0750)
 	if err != nil {
 		return fmt.Errorf("failed to create base path: %v", err)

core/gallery/importers/ace-step.go

@@ -0,0 +1,126 @@
+package importers
+
+import (
+	"encoding/json"
+	"path/filepath"
+	"strings"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/core/schema"
+	"go.yaml.in/yaml/v2"
+)
+
+var _ Importer = &ACEStepImporter{}
+
+// ACEStepImporter recognises ACE-Step music generation checkpoints
+// (ACE-Step/ACE-Step-v1-3.5B, ACE-Step/Ace-Step1.5, community finetunes).
+// Detection matches on "ace-step" in the repo name — case-insensitive —
+// so quantised mirrors still route here. The backend itself is
+// sound-generation / TTS-adjacent; the Modality() method returns "image"
+// purely to slot into the UI dropdown's image/video tab where it lives
+// with other generative media importers. preferences.backend="ace-step"
+// overrides detection.
+type ACEStepImporter struct{}
+
+func (i *ACEStepImporter) Name() string      { return "ace-step" }
+func (i *ACEStepImporter) Modality() string  { return "image" }
+func (i *ACEStepImporter) AutoDetects() bool { return true }
+
+func (i *ACEStepImporter) Match(details Details) bool {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return false
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return false
+		}
+	}
+
+	if b, ok := preferencesMap["backend"].(string); ok && b == "ace-step" {
+		return true
+	}
+
+	if details.HuggingFace != nil {
+		repoName := details.HuggingFace.ModelID
+		if idx := strings.Index(repoName, "/"); idx >= 0 {
+			repoName = repoName[idx+1:]
+		}
+		if strings.Contains(strings.ToLower(repoName), "ace-step") {
+			return true
+		}
+		if strings.EqualFold(details.HuggingFace.Author, "ACE-Step") {
+			return true
+		}
+	}
+
+	// Fallback: hfapi recursion bug may leave HuggingFace nil — decide
+	// from the URI owner/repo.
+	if owner, repo, ok := HFOwnerRepoFromURI(details.URI); ok {
+		if strings.EqualFold(owner, "ACE-Step") {
+			return true
+		}
+		if strings.Contains(strings.ToLower(repo), "ace-step") {
+			return true
+		}
+	}
+
+	return false
+}
+
+func (i *ACEStepImporter) Import(details Details) (gallery.ModelConfig, error) {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return gallery.ModelConfig{}, err
+		}
+	}
+
+	name, ok := preferencesMap["name"].(string)
+	if !ok {
+		name = filepath.Base(details.URI)
+	}
+
+	description, ok := preferencesMap["description"].(string)
+	if !ok {
+		description = "Imported from " + details.URI
+	}
+
+	model := details.URI
+	if details.HuggingFace != nil && details.HuggingFace.ModelID != "" {
+		model = details.HuggingFace.ModelID
+	} else if owner, repo, ok := HFOwnerRepoFromURI(details.URI); ok {
+		model = owner + "/" + repo
+	}
+
+	modelConfig := config.ModelConfig{
+		Name:        name,
+		Description: description,
+		Backend:     "ace-step",
+		// Mirrors gallery/index.yaml's ace-step-turbo entry which flags
+		// both sound_generation and tts — ACE-Step is a music/sound model,
+		// the UI groups it under image/video simply because there is no
+		// first-class music tab yet.
+		KnownUsecaseStrings: []string{"sound_generation", "tts"},
+		PredictionOptions: schema.PredictionOptions{
+			BasicModelRequest: schema.BasicModelRequest{Model: model},
+		},
+	}
+
+	data, err := yaml.Marshal(modelConfig)
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+
+	return gallery.ModelConfig{
+		Name:        name,
+		Description: description,
+		ConfigFile:  string(data),
+	}, nil
+}

core/gallery/importers/ace-step_test.go

@@ -0,0 +1,50 @@
+package importers_test
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/mudler/LocalAI/core/gallery/importers"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("ACEStepImporter", func() {
+	Context("detection from HuggingFace", func() {
+		// ACE-Step/ACE-Step-v1-3.5B is the reference public checkpoint for
+		// the ACE-Step music generation model. Detection must match on the
+		// repo name substring so third-party forks and quantised mirrors
+		// (e.g. Serveurperso/ACE-Step-1.5-GGUF) route to the same backend.
+		It("matches ACE-Step/ACE-Step-v1-3.5B (repo name contains ACE-Step)", func() {
+			uri := "https://huggingface.co/ACE-Step/ACE-Step-v1-3.5B"
+			preferences := json.RawMessage(`{}`)
+
+			modelConfig, err := importers.DiscoverModelConfig(uri, preferences)
+
+			Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: ace-step"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("ACE-Step/ACE-Step-v1-3.5B"), fmt.Sprintf("Model config: %+v", modelConfig))
+		})
+	})
+
+	Context("preference override", func() {
+		It("honours preferences.backend=ace-step for arbitrary URIs", func() {
+			uri := "https://example.com/some-unrelated-model"
+			preferences := json.RawMessage(`{"backend": "ace-step"}`)
+
+			modelConfig, err := importers.DiscoverModelConfig(uri, preferences)
+
+			Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: ace-step"), fmt.Sprintf("Model config: %+v", modelConfig))
+		})
+	})
+
+	Context("Importer interface metadata", func() {
+		It("exposes name/modality/autodetect", func() {
+			imp := &importers.ACEStepImporter{}
+			Expect(imp.Name()).To(Equal("ace-step"))
+			Expect(imp.Modality()).To(Equal("image"))
+			Expect(imp.AutoDetects()).To(BeTrue())
+		})
+	})
+})

core/gallery/importers/ambiguity_asr_test.go

@@ -0,0 +1,29 @@
+package importers_test
+
+import (
+	"encoding/json"
+	"errors"
+
+	"github.com/mudler/LocalAI/core/gallery/importers"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("ASR ambiguity", func() {
+	// pyannote/voice-activity-detection carries
+	// pipeline_tag=automatic-speech-recognition but ships only a YAML
+	// recipe — no ggml-*.bin, no .nemo, no Systran-style model.bin, no
+	// tokenizer.json, no .onnx. None of the ASR importers should match and
+	// none of the generic importers (vllm, transformers, llama-cpp, mlx,
+	// diffusers) should match either. Because the modality is in the
+	// ambiguous whitelist, DiscoverModelConfig must surface
+	// ErrAmbiguousImport rather than a bare "no importer matched" error.
+	It("returns ErrAmbiguousImport when ASR pipeline_tag is present but no importer matches", func() {
+		uri := "https://huggingface.co/pyannote/voice-activity-detection"
+		preferences := json.RawMessage(`{}`)
+
+		_, err := importers.DiscoverModelConfig(uri, preferences)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, importers.ErrAmbiguousImport)).To(BeTrue(), "expected ErrAmbiguousImport, got: %v", err)
+	})
+})

core/gallery/importers/ambiguity_embeddings_test.go

@@ -0,0 +1,34 @@
+package importers_test
+
+import (
+	"encoding/json"
+	"errors"
+
+	"github.com/mudler/LocalAI/core/gallery/importers"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Embeddings ambiguity", func() {
+	// Qdrant/bm25 carries pipeline_tag="sentence-similarity" but ships
+	// only config.json, README.md, .gitattributes, and per-language
+	// stopword .txt files — no tokenizer.json (rules out vllm and
+	// transformers), no modules.json / sentence_bert_config.json (rules
+	// out sentencetransformers), no "reranker" / cross-encoder owner
+	// (rules out rerankers), no rf-detr name (rules out rfdetr), no
+	// snakers4 / silero_vad.onnx (rules out silero-vad), no .gguf
+	// (rules out llama-cpp and stablediffusion-ggml), no mlx-community
+	// owner (rules out mlx), no model_index.json / scheduler_config.json
+	// (rules out diffusers). None of the ASR/TTS/image importers should
+	// trip either. Because sentence-similarity is in the ambiguous
+	// modality whitelist, DiscoverModelConfig must surface
+	// ErrAmbiguousImport.
+	It("returns ErrAmbiguousImport when sentence-similarity pipeline_tag is present but no importer matches", func() {
+		uri := "https://huggingface.co/Qdrant/bm25"
+		preferences := json.RawMessage(`{}`)
+
+		_, err := importers.DiscoverModelConfig(uri, preferences)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, importers.ErrAmbiguousImport)).To(BeTrue(), "expected ErrAmbiguousImport, got: %v", err)
+	})
+})

core/gallery/importers/ambiguity_image_test.go

@@ -0,0 +1,31 @@
+package importers_test
+
+import (
+	"encoding/json"
+	"errors"
+
+	"github.com/mudler/LocalAI/core/gallery/importers"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Image ambiguity", func() {
+	// h94/IP-Adapter-FaceID carries pipeline_tag="text-to-image" but ships
+	// only .bin + .safetensors + README — no model_index.json /
+	// scheduler_config.json (rules out diffusers), no .gguf (rules out
+	// llama-cpp and stablediffusion-ggml), no tokenizer.json (rules out
+	// vllm/transformers), owner is not mlx-community (rules out mlx), and
+	// the repo owner/name contain no ace-step/flux/sd1.5/sdxl/sd3/
+	// stable-diffusion arch token at the URI level — so none of the
+	// Batch-3 Image/Video importers match either. Because text-to-image
+	// is whitelisted as an ambiguous modality, DiscoverModelConfig must
+	// surface ErrAmbiguousImport rather than a bare "no importer matched".
+	It("returns ErrAmbiguousImport when text-to-image pipeline_tag is present but no importer matches", func() {
+		uri := "https://huggingface.co/h94/IP-Adapter-FaceID"
+		preferences := json.RawMessage(`{}`)
+
+		_, err := importers.DiscoverModelConfig(uri, preferences)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, importers.ErrAmbiguousImport)).To(BeTrue(), "expected ErrAmbiguousImport, got: %v", err)
+	})
+})

core/gallery/importers/ambiguity_tts_test.go

@@ -0,0 +1,32 @@
+package importers_test
+
+import (
+	"encoding/json"
+	"errors"
+
+	"github.com/mudler/LocalAI/core/gallery/importers"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("TTS ambiguity", func() {
+	// nari-labs/Dia-1.6B carries pipeline_tag="text-to-speech" but ships
+	// only config.json + *.pth + model.safetensors + preprocessor_config.json.
+	// None of the Batch-2 TTS importers match (owner neither "suno" nor
+	// "fishaudio" nor "OuteAI" nor "KittenML" nor "ResembleAI" nor "neuphonic"
+	// nor "coqui"; repo name contains none of "bark", "outetts", "voxcpm",
+	// "kokoro", "kitten-tts", "neutts", "chatterbox", "vibevoice"; no piper
+	// onnx/onnx.json pair). None of the generic importers match either —
+	// no tokenizer.json (rules out vllm/transformers), no .gguf (llama-cpp),
+	// no mlx-community owner (mlx), no model_index.json/scheduler_config
+	// (diffusers). Because the HF pipeline_tag is in the ambiguous
+	// whitelist, DiscoverModelConfig must surface ErrAmbiguousImport.
+	It("returns ErrAmbiguousImport when TTS pipeline_tag is present but no importer matches", func() {
+		uri := "https://huggingface.co/nari-labs/Dia-1.6B"
+		preferences := json.RawMessage(`{}`)
+
+		_, err := importers.DiscoverModelConfig(uri, preferences)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, importers.ErrAmbiguousImport)).To(BeTrue(), "expected ErrAmbiguousImport, got: %v", err)
+	})
+})

core/gallery/importers/bark.go

@@ -0,0 +1,124 @@
+package importers
+
+import (
+	"encoding/json"
+	"path/filepath"
+	"strings"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/core/schema"
+	"go.yaml.in/yaml/v2"
+)
+
+var _ Importer = &BarkImporter{}
+
+// BarkImporter recognises Suno's Bark TTS models. The `suno` owner hosts a
+// handful of Bark variants (bark, bark-small, bark-v2-en, …) sharing the
+// "bark" prefix — narrow enough to detect without false positives from
+// other suno repos. preferences.backend="bark" overrides detection.
+//
+// NOTE: suno/bark ships a `speaker_embeddings/v2` subdirectory that hits a
+// pre-existing path-doubling bug in pkg/huggingface-api's recursive tree
+// listing (item.Path already carries the parent path, but the recursion
+// prepends the parent path again → 404). When ModelDetails fetching fails,
+// DiscoverModelConfig leaves HuggingFace nil. To keep detection robust,
+// matchURIOwnerRepo() falls back to parsing the raw URI for "suno/bark*"
+// so the importer still fires end-to-end.
+type BarkImporter struct{}
+
+// matchBarkURI tolerates a nil ModelDetails (see note above) by extracting
+// the HF owner+repo portion directly from the raw URI.
+func matchBarkURI(uri string) bool {
+	owner, repo, ok := HFOwnerRepoFromURI(uri)
+	if !ok {
+		return false
+	}
+	return strings.EqualFold(owner, "suno") && strings.HasPrefix(strings.ToLower(repo), "bark")
+}
+
+func (i *BarkImporter) Name() string      { return "bark" }
+func (i *BarkImporter) Modality() string  { return "tts" }
+func (i *BarkImporter) AutoDetects() bool { return true }
+
+func (i *BarkImporter) Match(details Details) bool {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return false
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return false
+		}
+	}
+
+	if b, ok := preferencesMap["backend"].(string); ok && b == "bark" {
+		return true
+	}
+
+	if details.HuggingFace != nil {
+		if strings.EqualFold(details.HuggingFace.Author, "suno") {
+			repoName := details.HuggingFace.ModelID
+			if idx := strings.Index(repoName, "/"); idx >= 0 {
+				repoName = repoName[idx+1:]
+			}
+			if strings.HasPrefix(strings.ToLower(repoName), "bark") {
+				return true
+			}
+		}
+	}
+
+	// HF metadata may be absent when the recursive tree listing errors
+	// (see type-level note). Fall back to URI parsing.
+	return matchBarkURI(details.URI)
+}
+
+func (i *BarkImporter) Import(details Details) (gallery.ModelConfig, error) {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return gallery.ModelConfig{}, err
+		}
+	}
+
+	name, ok := preferencesMap["name"].(string)
+	if !ok {
+		name = filepath.Base(details.URI)
+	}
+
+	description, ok := preferencesMap["description"].(string)
+	if !ok {
+		description = "Imported from " + details.URI
+	}
+
+	model := details.URI
+	if details.HuggingFace != nil && details.HuggingFace.ModelID != "" {
+		model = details.HuggingFace.ModelID
+	}
+
+	modelConfig := config.ModelConfig{
+		Name:                name,
+		Description:         description,
+		Backend:             "bark",
+		KnownUsecaseStrings: []string{"tts"},
+		PredictionOptions: schema.PredictionOptions{
+			BasicModelRequest: schema.BasicModelRequest{Model: model},
+		},
+	}
+
+	data, err := yaml.Marshal(modelConfig)
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+
+	return gallery.ModelConfig{
+		Name:        name,
+		Description: description,
+		ConfigFile:  string(data),
+	}, nil
+}

core/gallery/importers/bark_test.go

@@ -0,0 +1,47 @@
+package importers_test
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/mudler/LocalAI/core/gallery/importers"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("BarkImporter", func() {
+	Context("detection from HuggingFace", func() {
+		It("matches suno/bark (owner + repo name prefix)", func() {
+			uri := "https://huggingface.co/suno/bark"
+			preferences := json.RawMessage(`{}`)
+
+			modelConfig, err := importers.DiscoverModelConfig(uri, preferences)
+
+			Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: bark"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("tts"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("suno/bark"), fmt.Sprintf("Model config: %+v", modelConfig))
+		})
+	})
+
+	Context("preference override", func() {
+		It("honours preferences.backend=bark for arbitrary URIs", func() {
+			uri := "https://example.com/some-unrelated-model"
+			preferences := json.RawMessage(`{"backend": "bark"}`)
+
+			modelConfig, err := importers.DiscoverModelConfig(uri, preferences)
+
+			Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: bark"), fmt.Sprintf("Model config: %+v", modelConfig))
+		})
+	})
+
+	Context("Importer interface metadata", func() {
+		It("exposes name/modality/autodetect", func() {
+			imp := &importers.BarkImporter{}
+			Expect(imp.Name()).To(Equal("bark"))
+			Expect(imp.Modality()).To(Equal("tts"))
+			Expect(imp.AutoDetects()).To(BeTrue())
+		})
+	})
+})

core/gallery/importers/chatterbox.go

@@ -0,0 +1,110 @@
+package importers
+
+import (
+	"encoding/json"
+	"path/filepath"
+	"strings"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/core/schema"
+	"go.yaml.in/yaml/v2"
+)
+
+var _ Importer = &ChatterboxImporter{}
+
+// ChatterboxImporter recognises Resemble AI's Chatterbox TTS. Detection
+// uses the `ResembleAI` owner or a "chatterbox" substring in the repo
+// name (covers the primary release plus community finetunes).
+// preferences.backend="chatterbox" overrides detection.
+type ChatterboxImporter struct{}
+
+func (i *ChatterboxImporter) Name() string      { return "chatterbox" }
+func (i *ChatterboxImporter) Modality() string  { return "tts" }
+func (i *ChatterboxImporter) AutoDetects() bool { return true }
+
+func (i *ChatterboxImporter) Match(details Details) bool {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return false
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return false
+		}
+	}
+
+	if b, ok := preferencesMap["backend"].(string); ok && b == "chatterbox" {
+		return true
+	}
+
+	if details.HuggingFace != nil {
+		if strings.EqualFold(details.HuggingFace.Author, "ResembleAI") {
+			return true
+		}
+		repoName := details.HuggingFace.ModelID
+		if idx := strings.Index(repoName, "/"); idx >= 0 {
+			repoName = repoName[idx+1:]
+		}
+		if strings.Contains(strings.ToLower(repoName), "chatterbox") {
+			return true
+		}
+	}
+
+	if owner, repo, ok := HFOwnerRepoFromURI(details.URI); ok {
+		if strings.EqualFold(owner, "ResembleAI") || strings.Contains(strings.ToLower(repo), "chatterbox") {
+			return true
+		}
+	}
+	return false
+}
+
+func (i *ChatterboxImporter) Import(details Details) (gallery.ModelConfig, error) {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return gallery.ModelConfig{}, err
+		}
+	}
+
+	name, ok := preferencesMap["name"].(string)
+	if !ok {
+		name = filepath.Base(details.URI)
+	}
+
+	description, ok := preferencesMap["description"].(string)
+	if !ok {
+		description = "Imported from " + details.URI
+	}
+
+	model := details.URI
+	if details.HuggingFace != nil && details.HuggingFace.ModelID != "" {
+		model = details.HuggingFace.ModelID
+	}
+
+	modelConfig := config.ModelConfig{
+		Name:                name,
+		Description:         description,
+		Backend:             "chatterbox",
+		KnownUsecaseStrings: []string{"tts"},
+		PredictionOptions: schema.PredictionOptions{
+			BasicModelRequest: schema.BasicModelRequest{Model: model},
+		},
+	}
+
+	data, err := yaml.Marshal(modelConfig)
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+
+	return gallery.ModelConfig{
+		Name:        name,
+		Description: description,
+		ConfigFile:  string(data),
+	}, nil
+}

core/gallery/importers/chatterbox_test.go

@@ -0,0 +1,47 @@
+package importers_test
+
+import (
+	"encoding/json"
+	"fmt"
+
+	"github.com/mudler/LocalAI/core/gallery/importers"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("ChatterboxImporter", func() {
+	Context("detection from HuggingFace", func() {
+		It("matches ResembleAI/chatterbox (owner)", func() {
+			uri := "https://huggingface.co/ResembleAI/chatterbox"
+			preferences := json.RawMessage(`{}`)
+
+			modelConfig, err := importers.DiscoverModelConfig(uri, preferences)
+
+			Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: chatterbox"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("tts"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("ResembleAI/chatterbox"), fmt.Sprintf("Model config: %+v", modelConfig))
+		})
+	})
+
+	Context("preference override", func() {
+		It("honours preferences.backend=chatterbox for arbitrary URIs", func() {
+			uri := "https://example.com/some-unrelated-model"
+			preferences := json.RawMessage(`{"backend": "chatterbox"}`)
+
+			modelConfig, err := importers.DiscoverModelConfig(uri, preferences)
+
+			Expect(err).ToNot(HaveOccurred(), fmt.Sprintf("Error: %v", err))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: chatterbox"), fmt.Sprintf("Model config: %+v", modelConfig))
+		})
+	})
+
+	Context("Importer interface metadata", func() {
+		It("exposes name/modality/autodetect", func() {
+			imp := &importers.ChatterboxImporter{}
+			Expect(imp.Name()).To(Equal("chatterbox"))
+			Expect(imp.Modality()).To(Equal("tts"))
+			Expect(imp.AutoDetects()).To(BeTrue())
+		})
+	})
+})

core/gallery/importers/coqui.go

@@ -0,0 +1,99 @@
+package importers
+
+import (
+	"encoding/json"
+	"path/filepath"
+	"strings"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/core/schema"
+	"go.yaml.in/yaml/v2"
+)
+
+var _ Importer = &CoquiImporter{}
+
+// CoquiImporter recognises Coqui AI's open-weight TTS releases (XTTS-v2,
+// YourTTS, the Tortoise port, etc). Detection is owner-scoped to `coqui`
+// — their HF org is the authoritative publisher for models that run on
+// the Coqui TTS Python runtime. preferences.backend="coqui" overrides.
+type CoquiImporter struct{}
+
+func (i *CoquiImporter) Name() string      { return "coqui" }
+func (i *CoquiImporter) Modality() string  { return "tts" }
+func (i *CoquiImporter) AutoDetects() bool { return true }
+
+func (i *CoquiImporter) Match(details Details) bool {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return false
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return false
+		}
+	}
+
+	if b, ok := preferencesMap["backend"].(string); ok && b == "coqui" {
+		return true
+	}
+
+	if details.HuggingFace != nil && strings.EqualFold(details.HuggingFace.Author, "coqui") {
+		return true
+	}
+
+	if owner, _, ok := HFOwnerRepoFromURI(details.URI); ok {
+		return strings.EqualFold(owner, "coqui")
+	}
+	return false
+}
+
+func (i *CoquiImporter) Import(details Details) (gallery.ModelConfig, error) {
+	preferences, err := details.Preferences.MarshalJSON()
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+	preferencesMap := make(map[string]any)
+	if len(preferences) > 0 {
+		if err := json.Unmarshal(preferences, &preferencesMap); err != nil {
+			return gallery.ModelConfig{}, err
+		}
+	}
+
+	name, ok := preferencesMap["name"].(string)
+	if !ok {
+		name = filepath.Base(details.URI)
+	}
+
+	description, ok := preferencesMap["description"].(string)
+	if !ok {
+		description = "Imported from " + details.URI
+	}
+
+	model := details.URI
+	if details.HuggingFace != nil && details.HuggingFace.ModelID != "" {
+		model = details.HuggingFace.ModelID
+	}
+
+	modelConfig := config.ModelConfig{
+		Name:                name,
+		Description:         description,
+		Backend:             "coqui",
+		KnownUsecaseStrings: []string{"tts"},
+		PredictionOptions: schema.PredictionOptions{
+			BasicModelRequest: schema.BasicModelRequest{Model: model},
+		},
+	}
+
+	data, err := yaml.Marshal(modelConfig)
+	if err != nil {
+		return gallery.ModelConfig{}, err
+	}
+
+	return gallery.ModelConfig{
+		Name:        name,
+		Description: description,
+		ConfigFile:  string(data),
+	}, nil
+}