chore: ⬆️ Update ggml-org/llama.cpp to 389ff61d77b5c71cec0cf92fe4e5d01ace80b797 (#9752)

⬆️ Update ggml-org/llama.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>

.agents/adding-backends.md

@@ -28,9 +28,9 @@ For Rust backends, you'll typically need (see `backend/rust/kokoros/` as a refer
 - `run.sh` - Sets `LD_LIBRARY_PATH`/`SSL_CERT_DIR` and execs the binary via the bundled `lib/ld.so`
 - `sources/<UpstreamProject>/` - Git submodule with the upstream Rust crate
 
-## 2. Add Build Configurations to `.github/workflows/backend.yml`
+## 2. Add Build Configurations to `.github/backend-matrix.yml`
 
-Add build matrix entries for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
+The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `backend.yml` itself). `backend.yml` (master push) and `backend_pr.yml` (PR) load it via `scripts/changed-backends.js`, which also handles per-file path filtering so only touched backends rebuild on PRs and master pushes alike. Add build matrix entries to `.github/backend-matrix.yml` for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
 
 **Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
 
@@ -46,6 +46,14 @@ If you add a new language bucket, `scripts/changed-backends.js` also needs a bra
 - Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"`
 - L4T (ARM): Use `build-type: 'l4t'` with `platforms: 'linux/arm64'` and `runs-on: 'ubuntu-24.04-arm'`
 
+**Per-arch native builds (`linux/amd64` + `linux/arm64`):**
+
+Multi-arch backends are NOT a single matrix entry with `platforms: 'linux/amd64,linux/arm64'`. Instead, add **two** entries — one with `platforms: 'linux/amd64'` + `platform-tag: 'amd64'` + `runs-on: 'ubuntu-latest'`, one with `platforms: 'linux/arm64'` + `platform-tag: 'arm64'` + `runs-on: 'ubuntu-24.04-arm'` — both sharing the same `tag-suffix`. The script detects the shared `tag-suffix` and emits a `merge-matrix` entry, so `backend-merge-jobs` (in `backend.yml`/`backend_pr.yml`) automatically assembles the manifest list from per-arch digest artifacts. See `-cpu-faster-whisper` in `.github/backend-matrix.yml` for a reference shape.
+
+**llama-cpp / ik-llama-cpp / turboquant variants only — `builder-base-image`:**
+
+Entries whose `dockerfile` is `./backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` must also set a `builder-base-image` field pointing at a prebuilt base from `quay.io/go-skynet/ci-cache:base-grpc-*` (CI builds these via `.github/workflows/base-images.yml`). The mapping is by `(build-type, platforms)` — see existing entries for the pattern. CI uses these prebuilt bases to skip the gRPC compile (~25–35 min cold). Local `make backends/<name>` ignores `builder-base-image` and uses the from-source path inside the Dockerfile, so you don't need quay access for local builds.
+
 ## 3. Add Backend Metadata to `backend/index.yaml`
 
 **Step 3a: Add Meta Definition**
@@ -145,7 +153,7 @@ docker-build-backends: ... docker-build-<backend-name>
 After adding a new backend, verify:
 
 - [ ] Backend directory structure is complete with all necessary files
-- [ ] Build configurations added to `.github/workflows/backend.yml` for all desired platforms
+- [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
 - [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
 - [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
 - [ ] Tag suffixes match between workflow file and index.yaml

.agents/building-and-testing.md

@@ -8,8 +8,9 @@ Let's say the user wants to build a particular backend for a given platform. For
 
 - The Makefile has targets like `docker-build-coqui` created with `generate-docker-build-target` at the time of writing. Recently added backends may require a new target.
 - At a minimum we need to set the BUILD_TYPE, BASE_IMAGE build-args
-  - Use .github/workflows/backend.yml as a reference it lists the needed args in the `include` job strategy matrix
-  - l4t and cublas also requires the CUDA major and minor version
+  - Use `.github/backend-matrix.yml` as a reference — it's the data-only YAML that lists every backend variant's `build-type`, `base-image`, `platforms`, etc. (`backend.yml` and `backend_pr.yml` consume it via `scripts/changed-backends.js`).
+  - l4t and cublas also require the CUDA major and minor version.
+  - For llama-cpp / ik-llama-cpp / turboquant the matrix also sets `builder-base-image` pointing at a prebuilt `quay.io/go-skynet/ci-cache:base-grpc-*` tag. Local `make backends/<name>` defaults to `BUILDER_TARGET=builder-fromsource` and doesn't need it — the Dockerfile's from-source stage installs everything itself.
 - You can pretty print a command like `DOCKER_MAKEFLAGS=-j$(nproc --ignore=1) BUILD_TYPE=hipblas BASE_IMAGE=rocm/dev-ubuntu-24.04:7.2.1 make docker-build-coqui`
 - Unless the user specifies that they want you to run the command, then just print it because not all agent frontends handle long running jobs well and the output may overflow your context
 - The user may say they want to build AMD or ROCM instead of hipblas, or Intel instead of SYCL or NVIDIA insted of l4t or cublas. Ask for confirmation if there is ambiguity.

.agents/ci-caching.md

@@ -1,33 +1,120 @@
 # CI Build Caching
 
-Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache. This file explains how that cache is laid out, what invalidates it, and how to bypass it.
+Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache plus a layered set of prebuilt base images. This file explains how the cache is laid out, what invalidates it, and how to bypass it.
+
+## Workflow surfaces
+
+| Workflow | Purpose | Triggers |
+|---|---|---|
+| `.github/workflows/backend.yml` | Backend container images on master | `push` to master + tags, weekly Sunday cron, `workflow_dispatch` |
+| `.github/workflows/backend_pr.yml` | Backend container images on PRs | `pull_request` |
+| `.github/workflows/backend_build.yml` | Reusable: builds one backend (one arch) by digest | `workflow_call` from above |
+| `.github/workflows/backend_merge.yml` | Reusable: assembles per-arch digests into a multi-arch manifest list | `workflow_call` |
+| `.github/workflows/backend_build_darwin.yml` | Reusable: macOS-native backend builds | `workflow_call` |
+| `.github/workflows/image.yml` / `image-pr.yml` | Root LocalAI image (push / PR) | push / PR |
+| `.github/workflows/image_build.yml` / `image_merge.yml` | Reusable: per-arch root-image build + merge | `workflow_call` |
+| `.github/workflows/base-images.yml` | Builds the prebuilt `base-grpc-*` builder bases | Saturdays 05:00 UTC cron, `workflow_dispatch`, master push touching `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/apt-mirror.sh`, or this workflow |
+
+The matrix that drives `backend.yml` / `backend_pr.yml` lives in **`.github/backend-matrix.yml`** (data-only YAML, not embedded in the workflow). `scripts/changed-backends.js` parses it, applies path-filter logic against the PR diff (PR events) or the GitHub Compare API (push events), and emits the filtered matrix plus a `merge-matrix` for backends with multiple per-arch entries.
 
 ## Cache layout
 
 - **Cache registry**: `quay.io/go-skynet/ci-cache`
-- **One tag per matrix entry**, derived from the existing `tag-suffix`:
-  - Backend builds (`backend_build.yml`): `cache<tag-suffix>`
-    - e.g. `cache-gpu-nvidia-cuda-12-llama-cpp`, `cache-cpu-vllm`, `cache-nvidia-l4t-cuda-13-arm64-vllm`
-  - Root image builds (`image_build.yml`): `cache-localai<tag-suffix>`
-    - e.g. `cache-localai-gpu-nvidia-cuda-12`, `cache-localai-gpu-vulkan`
+- **One tag per matrix entry per arch**, derived from `tag-suffix` and `platform-tag`:
+  - Backend builds (`backend_build.yml`): `cache<tag-suffix>-<platform-tag>`
+    - e.g. `cache-cpu-faster-whisper-amd64`, `cache-cpu-faster-whisper-arm64`, `cache-gpu-nvidia-cuda-13-llama-cpp-amd64`
+  - Root image builds (`image_build.yml`): `cache-localai<tag-suffix>-<platform-tag>` (with a `-core` placeholder when `tag-suffix` is empty, so `cache-localai-core-amd64` for the core image)
+  - Pre-built base images (`base-images.yml`): `cache-base-grpc-<variant>` (one per `(BUILD_TYPE, arch)` permutation)
 - Each tag stores a multi-arch BuildKit cache manifest (`mode=max`), so every intermediate stage is re-usable, not just the final image.
 
+The per-arch suffix exists because amd64 and arm64 builds produce different intermediate content; sharing one cache key would thrash on every cross-arch rebuild.
+
 ## Read/write semantics
 
 | Trigger | `cache-from` | `cache-to` |
 |---|---|---|
-| `push` to `master` / tag | yes | yes (`mode=max,ignore-error=true`) |
+| `push` to `master` / tag / cron / dispatch | yes | yes (`mode=max,ignore-error=true`) |
 | `pull_request` | yes | **no** |
 
 PR builds read master's warm cache but never write — this prevents PRs from polluting the shared cache with their experimental state. After merge, the master build for that matrix entry refreshes the cache.
 
 `ignore-error=true` on the write side means a transient quay push failure does not fail the build; the next master push retries.
 
-## Self-warming, no separate populator
+## Pre-built base images (`base-grpc-*`)
 
-There is no cron job that pre-warms the cache. The production builds *are* the populator. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`, Python wheel installs, etc.).
+The C++ backend Dockerfiles (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) compile gRPC from source. On a cold build that's ~25–35 min before any LocalAI source compiles. To skip that on CI, `.github/workflows/base-images.yml` builds and pushes a set of pre-prepped builder bases:
 
-Historically there was a `generate_grpc_cache.yaml` cron that targeted a `grpc` stage in the root Dockerfile. That stage was removed in July 2025 and the cron silently failed every night for 9 months without writing anything. It was deleted along with the registry-cache rollout.
+| Tag | Contents |
+|---|---|
+| `base-grpc-amd64` / `base-grpc-arm64` | Ubuntu 24.04 + apt build deps + protoc + cmake + gRPC at `/opt/grpc` |
+| `base-grpc-cuda-12-amd64` | the above + CUDA 12.8 toolkit |
+| `base-grpc-cuda-13-amd64` | the above + CUDA 13.0 toolkit (Ubuntu 22.04 base) |
+| `base-grpc-cuda-13-arm64` | the above + CUDA 13.0 sbsa toolkit (Ubuntu 24.04 base) |
+| `base-grpc-l4t-cuda-12-arm64` | JetPack r36.4.0 base (CUDA preinstalled, `SKIP_DRIVERS=true`) + gRPC |
+| `base-grpc-rocm-amd64` | rocm/dev-ubuntu-24.04:7.2.1 base + hipblas/hipblaslt/rocblas + gRPC |
+| `base-grpc-vulkan-amd64` / `base-grpc-vulkan-arm64` | Ubuntu 24.04 + Vulkan SDK 1.4.335 + gRPC |
+| `base-grpc-intel-amd64` | intel/oneapi-basekit:2025.3.2 base + gRPC |
+
+**Single source of truth**: the install logic for all 10 variants lives in `.docker/install-base-deps.sh`. Both `Dockerfile.base-grpc-builder` AND each variant Dockerfile's `builder-fromsource` stage bind-mount and execute the same script — so the prebuilt CI base and the local from-source path are bit-equivalent by construction.
+
+### How variant Dockerfiles consume the base
+
+`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` are multi-target. Three stages plus a final aliasing stage:
+
+- `builder-fromsource` — `FROM ${BASE_IMAGE}` then runs `install-base-deps.sh` and the per-backend compile script. Used when `BUILDER_TARGET=builder-fromsource` (the default; local `make backends/<name>`).
+- `builder-prebuilt` — `FROM ${BUILDER_BASE_IMAGE}` (one of the prebuilt `base-grpc-*` tags) and runs only the per-backend compile script. Used when `BUILDER_TARGET=builder-prebuilt` (CI when the matrix entry sets `builder-base-image`).
+- `FROM ${BUILDER_TARGET} AS builder` — alias resolves the ARG-selected stage to a fixed name (BuildKit doesn't allow ARG expansion in `COPY --from=`).
+- `FROM scratch` + `COPY --from=builder ...package/. ./` — emits the final scratch image with just the package contents.
+
+BuildKit prunes the unreferenced builder stage, so each build only runs the path it needs. `backend_build.yml` derives `BUILDER_TARGET=builder-prebuilt` automatically when the matrix entry has a non-empty `builder-base-image`; otherwise it defaults to `builder-fromsource`.
+
+The matrix `(build-type, platforms)` → `builder-base-image` mapping for llama-cpp / ik-llama-cpp / turboquant entries:
+
+| `build-type` | `platforms` | tag |
+|---|---|---|
+| `''` | `linux/amd64` | `base-grpc-amd64` |
+| `''` | `linux/arm64` | `base-grpc-arm64` |
+| `cublas` cuda 12 | `linux/amd64` | `base-grpc-cuda-12-amd64` |
+| `cublas` cuda 13 | `linux/amd64` | `base-grpc-cuda-13-amd64` |
+| `cublas` cuda 13 | `linux/arm64` | `base-grpc-cuda-13-arm64` |
+| `cublas` cuda 12 + JetPack base | `linux/arm64` | `base-grpc-l4t-cuda-12-arm64` |
+| `hipblas` | `linux/amd64` | `base-grpc-rocm-amd64` |
+| `vulkan` | `linux/amd64` | `base-grpc-vulkan-amd64` |
+| `vulkan` | `linux/arm64` | `base-grpc-vulkan-arm64` |
+| `sycl_*` | `linux/amd64` | `base-grpc-intel-amd64` |
+
+### Bootstrap order when adding a new variant
+
+If you add a new entry to `base-images.yml`'s matrix, the new tag does not exist on quay until the workflow runs. To consume it from a variant entry safely, dispatch the base-images workflow on the branch first:
+
+```bash
+gh workflow run base-images.yml --ref <feature-branch>
+```
+
+Wait for the new variant to push, then merge the consumer change. Otherwise the consumer's CI fails with "image not found."
+
+## Per-arch native builds + manifest merge
+
+Multi-arch backends (and the core LocalAI image) build natively per arch instead of running both arches under QEMU emulation on a single x86 runner. The pattern:
+
+- The matrix has TWO entries per multi-arch backend, sharing the same `tag-suffix` but distinct `platforms` + `platform-tag` + `runs-on`. Example: `-cpu-faster-whisper` has one amd64 entry on `ubuntu-latest` and one arm64 entry on `ubuntu-24.04-arm`.
+- Each per-arch build pushes by **canonical digest only** (no tags) via `outputs: type=image,push-by-digest=true,name-canonical=true,push=true`. The digest is uploaded as an artifact named `digests<tag-suffix>-<platform-tag>` (or `digests-localai<...>` for root-image builds).
+- `scripts/changed-backends.js` detects shared `tag-suffix` and emits a `merge-matrix` output. `backend.yml` / `backend_pr.yml` have a `backend-merge-jobs` job that consumes it and calls `backend_merge.yml`.
+- `backend_merge.yml` downloads all matching digest artifacts and runs `docker buildx imagetools create` to publish the final tagged manifest list pointing at both per-arch digests. Same `docker/metadata-action` config as the original monolithic build, so consumers see no tag-shape change.
+- `image_merge.yml` is the equivalent for the root LocalAI image (`-core` placeholder when `tag-suffix` is empty so the artifact-name glob doesn't over-match across `core` and `gpu-vulkan`).
+
+**`provenance: false` is required on multi-registry digest pushes**: with the default `mode=max` provenance attestation, BuildKit bundles a per-registry attestation manifest into each registry's manifest list, making the resulting list digest diverge across registries. `steps.build.outputs.digest` only matches one of them and the merge step's `imagetools create <reg>@sha256:<digest>` lookup fails on the other. Setting `provenance: false` keeps the digest content-only and identical across registries.
+
+## Path filter on master push
+
+Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynamically through `scripts/changed-backends.js`:
+
+- **PR events**: paginated `pulls/{n}/files` API → filter the matrix to entries whose `dockerfile` path prefix matches the PR diff.
+- **Push events**: GitHub Compare API (`/repos/{owner}/{repo}/compare/{before}...{after}`) → same path-filter logic. Falls back to "run everything" on first-branch push (`event.before` zero), API truncation (≥300 changed files), missing API token, or any thrown error.
+- **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
+- **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
+
+The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
 
 ## The `DEPS_REFRESH` cache-buster (Python backends)
 
@@ -42,18 +129,57 @@ Most Python backends ship `requirements*.txt` files that **do not pin every tran
 
 `DEPS_REFRESH` defends against that:
 
-- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W17`) before each build and passes it as a build-arg.
+- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W19`) before each build and passes it as a build-arg.
 - The `RUN ... make` layer's BuildKit hash now includes that string, so the layer invalidates **at most once per week**, automatically picking up newer wheels.
 - Within a week, builds stay warm.
 
 This applies only to `Dockerfile.python` because:
 - Go (`Dockerfile.golang`) pins versions in `go.mod` / `go.sum`.
 - Rust (`Dockerfile.rust`) pins via `Cargo.lock`.
-- C++ backends (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) clone gRPC at a pinned tag (`v1.65.0`) and llama.cpp at a pinned commit; their inputs don't drift between rebuilds.
+- C++ backends pin gRPC (`v1.65.0`) and llama.cpp at a specific commit; their inputs don't drift between rebuilds.
 
 ### Adjusting the cadence
 
-If you need a faster refresh (e.g. while debugging an upstream flake), bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`). If you need a one-shot rebuild for a specific backend without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
+Bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`) for faster refreshes. For one-shot rebuilds without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
+
+## ccache for C++ backend builds
+
+`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` declare a BuildKit cache mount on `/root/.ccache`:
+
+```dockerfile
+RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
+```
+
+The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
+
+On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
+
+## Composite actions
+
+Two composite actions handle runner-side prep:
+
+- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
+- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
+
+Both actions run before any docker buildx step.
+
+## Concurrency
+
+All `backend.yml` / `image.yml` / `test.yml` / etc. workflows use:
+
+```yaml
+concurrency:
+  group: ci-<workflow>-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+```
+
+- **PR events** group by PR number → newer pushes to the same PR cancel old runs (intended).
+- **Push events** group by `github.sha` → each master commit gets its own run; rapid-fire merges don't cancel each other (this was a real issue prior — two master pushes 11 seconds apart would cancel the first's CI).
+
+## Self-warming, no separate populator
+
+There is no cron job that pre-warms the BuildKit cache for individual backends. The production builds *are* the populators. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in the variant `builder-fromsource` stage or skipped entirely when consuming `base-grpc-*`, Python wheel installs, etc.). The base-images workflow's weekly cron is the closest thing to a populator and only refreshes the prebuilt builder bases.
 
 ## Manually evicting cache
 
@@ -63,19 +189,19 @@ To force a fully cold build for one backend or the whole image:
 # Delete a single tag (requires quay credentials with admin on the repo)
 curl -X DELETE \
   -H "Authorization: Bearer ${QUAY_TOKEN}" \
-  https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm
+  https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm-amd64
 
 # List all tags
 curl -s -H "Authorization: Bearer ${QUAY_TOKEN}" \
   "https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/?limit=100" | jq '.tags[].name'
 ```
 
-Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry so a stale tag never bleeds into a different build.
+Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry per arch so a stale tag never bleeds into a different build.
 
-## What the cache **does not** cover
+## What the cache does **not** cover
 
-- The "Free Disk Space" / "Release space from worker" steps run on every job — these reclaim ~6 GB on `ubuntu-latest` runners. They are runner-state cleanup, not Docker, and BuildKit caches don't apply.
-- Intermediate artifacts of `Build and push (PR)` are not pushed anywhere — PRs only build for verification.
+- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
+- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
 - Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
 
 ## Darwin native caches
@@ -95,17 +221,30 @@ The Python wheel cache uses the same ISO-week cache-buster as the Linux `DEPS_RE
 
 The brew Cellar cache requires `HOMEBREW_NO_AUTO_UPDATE=1` and `HOMEBREW_NO_INSTALL_CLEANUP=1` (set as job-level env). Without those, `brew install` would mutate the very directories that were just restored, defeating the cache.
 
+**Force-link after cache restore**: `actions/cache` restores `/opt/homebrew/Cellar/*` but NOT the `/opt/homebrew/bin/*` symlinks. After a cache hit, `brew install` sees the Cellar entries and decides "already installed" without re-running its link step, leaving the formulas off PATH. The Dependencies step explicitly runs `brew link --overwrite` for every cached formula afterwards to ensure the symlinks exist.
+
 For ccache, the workflow exports `CMAKE_ARGS=… -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache` via `$GITHUB_ENV` before running `make build-darwin-go-backend`. The Makefile in `backend/cpp/llama-cpp/` already forwards `CMAKE_ARGS` through to each variant build (`fallback`, `grpc`, `rpc-server`), so no script changes are needed. The three variants share most TUs, so ccache dedupes object files across them.
 
+`backend_build_darwin.yml` also has a llama-cpp-specific build-step branch that runs `make backends/llama-cpp-darwin` (the bespoke script that compiles three CMake variants and bundles dylibs via `otool`), distinct from the generic `make build-darwin-${lang}-backend` path. This was consolidated from a previously-bespoke top-level `llama-cpp-darwin` job in `backend.yml` so llama-cpp on Darwin honors the same path filter as the other 34 Darwin backends.
+
 ### Cache budget on Darwin
 
 GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~800 MB Go cache + ~2 GB brew Cellar + up to 2 GB ccache + ~1.5 GB × 5 python backends. If the cap is hit, prefer collapsing the per-backend Python keys into a shared `pyenv-darwin-shared-<week>` key (accepts more cross-backend churn for a smaller footprint) before reducing other caches.
 
+## Self-hosted runners
+
+`.github/backend-matrix.yml` has zero references to `arc-runner-set` or `bigger-runner` — all backends run on GHA free-tier hosted runners (`ubuntu-latest` for amd64, `ubuntu-24.04-arm` for arm64 native, `macos-14` for Darwin). The migration off self-hosted relied on the per-arch native split (no QEMU emulation) plus `setup-build-disk`'s `/mnt` relocation (~100 GB working space, enough for ROCm dev image + vLLM/torch installs).
+
+One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
+
 ## Touching the cache pipeline
 
-When changing `image_build.yml`, `backend_build.yml`, or any of the `backend/Dockerfile.*` files:
+When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
 
 1. **Don't drop `DEPS_REFRESH=...` from the build-args** without a replacement strategy (lockfiles, pinned requirements). Otherwise master will silently freeze on whichever versions were cached at the time.
-2. **Keep `tag-suffix` unique per matrix entry** — it's the cache namespace. Two matrix entries sharing a tag-suffix would clobber each other's cache.
+2. **Keep `(tag-suffix, platform-tag)` unique per matrix entry** — together they're the cache namespace. Two matrix entries sharing a key would clobber each other's cache.
 3. **Keep `cache-to` gated on `github.event_name != 'pull_request'`** — PRs must not write.
 4. **Keep `ignore-error=true` on `cache-to`** — quay registry hiccups must not fail builds.
+5. **Keep `provenance: false` on push-by-digest steps** — multi-registry digest divergence is the Bug We Already Fixed; reintroducing provenance attestation re-breaks the merge.
+6. **`install-base-deps.sh` is the single source of truth for base contents.** Both `Dockerfile.base-grpc-builder` (CI) and the variant Dockerfiles' `builder-fromsource` (local) bind-mount and execute it. If you add a package to one path, add it to the script — don't fork the logic into a Dockerfile RUN.
+7. **After adding a `base-images.yml` matrix variant, run the workflow on your branch before merging consumer changes** that depend on the new tag — otherwise the consumer's CI fails "image not found."

.agents/sglang-backend.md

@@ -0,0 +1,62 @@
+# Working on the SGLang Backend
+
+The SGLang backend lives at `backend/python/sglang/backend.py` (async gRPC). It wraps SGLang's `Engine` (`sglang.srt.entrypoints.engine.Engine`) and translates LocalAI's gRPC `PredictOptions` into SGLang sampling params + outputs into `Reply.chat_deltas`. Structurally it mirrors `backend/python/vllm/backend.py` — keep them shaped the same so changes in one have an obvious analog in the other.
+
+## `engine_args` is the universal escape hatch
+
+A small fixed set of fields on `ModelOptions` is mapped to typed SGLang kwargs in `LoadModel` (model, quantization, load_format, gpu_memory_utilization → mem_fraction_static, trust_remote_code, enforce_eager → disable_cuda_graph, tensor_parallel_size → tp_size, max_model_len → context_length, dtype). **Everything else** flows through the `engine_args:` YAML map.
+
+Validation happens in `_apply_engine_args`. Keys are checked against `dataclasses.fields(ServerArgs)` (`sglang.srt.server_args.ServerArgs` is a flat `@dataclass` with ~380 fields). Unknown keys raise `ValueError` at LoadModel time with a `difflib.get_close_matches` suggestion — same shape as the vLLM backend.
+
+**Precedence:** typed `ModelOptions` fields populate `engine_kwargs` first, then `engine_args` overrides them. So a YAML that sets both `gpu_memory_utilization: 0.9` and `engine_args.mem_fraction_static: 0.5` ends up at `0.5`. Document this when answering "why didn't my YAML field stick?".
+
+**ServerArgs is flat.** Unlike vLLM, where speculative decoding is nested under `engine_args.speculative_config: {...}`, SGLang exposes flat top-level fields: `speculative_algorithm`, `speculative_draft_model_path`, `speculative_num_steps`, `speculative_eagle_topk`, `speculative_num_draft_tokens`, `speculative_dflash_block_size`, etc. There is no `speculative_config:` dict. Same goes for compilation, kv-transfer, attention — all flat.
+
+The canonical reference is `python/sglang/srt/server_args.py:ServerArgs` (line ~304). When SGLang adds new flags, no LocalAI code change is needed — they're automatically available via `engine_args:`. The validator picks them up because it introspects the live dataclass.
+
+## Speculative decoding cheatsheet
+
+`--speculative-algorithm` accepts `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`, `DFLASH`. `NEXTN` is silently rewritten to `EAGLE` in `ServerArgs.__post_init__` (`server_args.py:3286-3287`). MTP (Multi-Token Prediction) is the same EAGLE path with `num_steps=1, eagle_topk=1, num_draft_tokens=2` against a target whose architecture has multi-token heads (e.g. MiMo-7B-RL, DeepSeek-V3-MTP).
+
+| Algorithm | Drafter requirement | Gallery demo target | Gallery demo drafter |
+|-----------|--------------------|---------------------|----------------------|
+| `NEXTN` / `EAGLE` (MTP) | Assistant drafter or built-in heads | google/gemma-4-E2B-it, google/gemma-4-E4B-it | google/gemma-4-E2B-it-assistant, google/gemma-4-E4B-it-assistant |
+| `EAGLE3` | EAGLE3 draft head | (no gallery entry yet) | e.g. jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B |
+| `DFLASH` | Block-diffusion drafter | (no gallery entry yet) | e.g. z-lab/Qwen3-4B-DFlash-b16 |
+| `STANDALONE` | Smaller LLM as drafter | (no gallery entry yet) | any smaller chat-tuned LLM in the same family |
+| `NGRAM` | None — uses prefix history | (no gallery entry yet) | n/a |
+
+The Gemma 4 demos use `mem_fraction_static: 0.85` (cookbook default) and the cookbook's `num_steps=5, num_draft_tokens=6, eagle_topk=1` parameters. Other algorithms are reachable from any user YAML via `engine_args:` but don't have shipped demos yet — that's a deliberate gallery scope choice, not a backend limitation.
+
+Gemma 4 support requires sglang built from a commit that includes [PR #21952](https://github.com/sgl-project/sglang/pull/21952). LocalAI's pinned release for cublas12 / cublas13 includes it. The `l4t13` (JetPack 7 / sbsa cu130) build floors at `sglang>=0.5.0` because the `pypi.jetson-ai-lab.io` mirror still ships only `0.5.1.post2` as of 2026-05-06 — Gemma 4 / MTP recipes are therefore not available on l4t13 until that mirror catches up. `backend.py` keeps backward compat with the 0.5.x → 0.5.11 `SamplingParams.seed` → `sampling_seed` rename via runtime detection.
+
+Compatibility caveats per the SGLang docs: DFLASH and NGRAM are incompatible with `enable_dp_attention`; DFLASH requires `pp_size == 1`; STANDALONE is incompatible with `enable_dp_attention`; NGRAM is CUDA-only and disables the overlap scheduler.
+
+### `mem_fraction_static` + quantization + MTP on consumer GPUs
+
+When combining online weight quantization (`engine_args.quantization: fp8` / `awq` / etc.) with built-in-head MTP (`speculative_algorithm: EAGLE`/`NEXTN`) on a tight VRAM budget, sglang's default `mem_fraction_static: 0.85` will OOM during draft-worker init. The reason: sglang quantizes the **target** model's transformer blocks but loads the **MTP draft worker's vocab embedding** at the source dtype (typically bf16). For a 7 B-class model with a 150k-token vocab × 4096 hidden, that's another ~1.2 GiB allocated *after* the static pool is reserved. At 0.85 fraction on a 16 GB card there's no room left.
+
+Workaround: drop `mem_fraction_static` to ~0.7 so the post-static heap can absorb the MTP embedding alloc + CUDA graph private pools. Verified end-to-end on MiMo-7B-RL + fp8 + MTP on a 16 GB RTX 5070 Ti (`gallery/sglang-mimo-7b-mtp.yaml`) at ~88 tok/s. Models with larger vocabs or more MTP layers (e.g. DeepSeek-V3-MTP) need an even smaller fraction.
+
+This isn't documented anywhere upstream as of 2026-05-06 — the SGLang Gemma 4 cookbook uses 0.85 because their MTP path doesn't go through `eagle_worker_v2.py` for an embedding-bearing draft module. Don't blanket-apply 0.7 across all sglang YAMLs; only when MTP-with-built-in-heads + quantization combine.
+
+## Tool-call and reasoning parsers stay on `Options[]`
+
+ServerArgs has `tool_call_parser` and `reasoning_parser` fields, and the backend does pass them through to `Engine` so SGLang's own HTTP/OAI surface keeps working. But for the **LocalAI** request path the backend constructs fresh per-request parser instances in `_make_parsers` (`backend.py:286`) because the parsers are stateful — the streaming and non-streaming paths each need their own.
+
+So the user-facing knob stays on `Options[]`:
+
+```yaml
+options:
+  - tool_parser:hermes
+  - reasoning_parser:deepseek_r1
+```
+
+Putting these in `engine_args:` will set them on `ServerArgs` but the LocalAI-level streaming `ChatDelta` will not pick them up. Don't recommend that path.
+
+## What's missing today (out of scope, but worth tracking)
+
+- `core/config/hooks_sglang.go` — there is no SGLang equivalent of `hooks_vllm.go`. The vLLM hook auto-selects parsers for known model families from `parser_defaults.json` and seeds production engine_args defaults. A symmetric hook for SGLang could reuse the same `parser_defaults.json` (the SGLang parser names are different but the family detection is shared) and seed defaults like `enable_metrics: true` or attention-backend choices.
+- `core/gallery/importers/sglang.go` — vLLM has an importer that resolves model architecture → parser defaults at gallery-import time. A matching importer for SGLang would let `local-ai install` populate sensible parsers automatically.
+
+These should be a follow-up PR, not a blocker for the engine_args feature.

.docker/ik-llama-cpp-compile.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# Shared compile logic for backend/Dockerfile.ik-llama-cpp.
+# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
+
+set -euxo pipefail
+
+export CCACHE_DIR=/root/.ccache
+ccache --max-size=5G || true
+ccache -z || true
+
+export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
+fi
+
+cd /LocalAI/backend/cpp/ik-llama-cpp
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  # ARM64 / ROCm: build without x86 SIMD
+  make ik-llama-cpp-fallback
+else
+  # ik_llama.cpp's IQK kernels require at least AVX2
+  make ik-llama-cpp-avx2
+fi
+
+ccache -s || true

.docker/install-base-deps.sh

@@ -0,0 +1,244 @@
+#!/usr/bin/env bash
+# Single source of truth for builder-base contents.
+#
+# Used by:
+#   - backend/Dockerfile.base-grpc-builder        (CI prebuilt-base source of truth)
+#   - backend/Dockerfile.llama-cpp                (builder-fromsource stage)
+#   - backend/Dockerfile.ik-llama-cpp             (builder-fromsource stage)
+#   - backend/Dockerfile.turboquant               (builder-fromsource stage)
+#
+# All four files invoke this script via
+#   RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
+#       --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+#       bash /usr/local/sbin/install-base-deps
+#
+# so the prebuilt CI base image and the from-source local-dev path are
+# bit-equivalent by construction.
+#
+# Inputs (env, populated from Dockerfile ARG/ENV):
+#   BUILD_TYPE                ("cublas"|"l4t"|"hipblas"|"vulkan"|"sycl"|"clblas"|"")
+#   CUDA_MAJOR_VERSION        ("12" | "13" | "")
+#   CUDA_MINOR_VERSION        ("8" | "0" | "")
+#   TARGETARCH                ("amd64" | "arm64")
+#   UBUNTU_VERSION            ("2204" | "2404")
+#   SKIP_DRIVERS              ("false" | "true")
+#   CMAKE_FROM_SOURCE         ("false" | "true")
+#   CMAKE_VERSION             ("3.31.10")
+#   GRPC_VERSION              ("v1.65.0")
+#   GRPC_MAKEFLAGS            ("-j4 -Otarget")
+#   APT_MIRROR / APT_PORTS_MIRROR  (optional; consumed by /usr/local/sbin/apt-mirror)
+#   AMDGPU_TARGETS            (optional; only relevant for hipblas downstream)
+#
+# IMPORTANT: install logic is copied verbatim from the prior in-Dockerfile
+# RUN blocks. Do not paraphrase apt invocations / version pins / sed line
+# numbers / deb URLs — the bit-equivalence guarantee depends on it.
+
+set -eux
+
+# --- 0. apt mirror rewrite (no-op when APT_MIRROR / APT_PORTS_MIRROR unset) ---
+if [ -x /usr/local/sbin/apt-mirror ]; then
+    APT_MIRROR="${APT_MIRROR:-}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR:-}" \
+        sh /usr/local/sbin/apt-mirror
+fi
+
+export DEBIAN_FRONTEND=noninteractive
+export MAKEFLAGS="${GRPC_MAKEFLAGS:-}"
+
+# --- 1. Base apt build deps ---
+apt-get update
+apt-get install -y --no-install-recommends \
+    build-essential \
+    ccache git \
+    ca-certificates \
+    make \
+    pkg-config libcurl4-openssl-dev \
+    curl unzip \
+    libssl-dev wget
+apt-get clean
+rm -rf /var/lib/apt/lists/*
+
+# --- 2. Vulkan SDK (BUILD_TYPE=vulkan) ---
+# NB: this block intentionally installs `cmake` via apt as part of the
+# Vulkan tooling — must run before the dedicated CMake step below.
+if [ "${BUILD_TYPE:-}" = "vulkan" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
+    apt-get update
+    apt-get install -y  --no-install-recommends \
+        software-properties-common pciutils wget gpg-agent
+    apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+        libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+        libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+        git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+        ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+        clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
+    if [ "amd64" = "${TARGETARCH:-}" ]; then
+        wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz"
+        tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz
+        rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz
+        mkdir -p /opt/vulkan-sdk
+        mv 1.4.335.0 /opt/vulkan-sdk/
+        ( cd /opt/vulkan-sdk/1.4.335.0 && \
+          ./vulkansdk --no-deps --maxjobs \
+              vulkan-loader \
+              vulkan-validationlayers \
+              vulkan-extensionlayer \
+              vulkan-tools \
+              shaderc )
+        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/
+        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/
+        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/
+        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/
+        rm -rf /opt/vulkan-sdk
+    fi
+    if [ "arm64" = "${TARGETARCH:-}" ]; then
+        mkdir vulkan
+        ( cd vulkan && \
+          curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
+          tar -xvf vulkan-sdk.tar.xz && \
+          rm vulkan-sdk.tar.xz && \
+          cd 1.4.335.0 && \
+          cp -rfv aarch64/bin/* /usr/bin/ && \
+          cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
+          cp -rfv aarch64/include/* /usr/include/ && \
+          cp -rfv aarch64/share/* /usr/share/ )
+        rm -rf vulkan
+    fi
+    ldconfig
+    apt-get clean
+    rm -rf /var/lib/apt/lists/*
+fi
+
+# --- 3. CUDA toolkit (BUILD_TYPE=cublas|l4t) ---
+if { [ "${BUILD_TYPE:-}" = "cublas" ] || [ "${BUILD_TYPE:-}" = "l4t" ]; } && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
+    apt-get update
+    apt-get install -y  --no-install-recommends \
+        software-properties-common pciutils
+    if [ "amd64" = "${TARGETARCH:-}" ]; then
+        curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb"
+    fi
+    if [ "arm64" = "${TARGETARCH:-}" ]; then
+        if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+            curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb"
+        else
+            curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb"
+        fi
+    fi
+    dpkg -i cuda-keyring_1.1-1_all.deb
+    rm -f cuda-keyring_1.1-1_all.deb
+    apt-get update
+    apt-get install -y --no-install-recommends \
+        "cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
+        "libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
+        "libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
+        "libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
+        "libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
+        "libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
+    if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "${TARGETARCH:-}" ]; then
+        apt-get install -y --no-install-recommends \
+            "libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
+            "libcudnn9-cuda-${CUDA_MAJOR_VERSION}" \
+            "cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
+            "libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
+    fi
+    apt-get clean
+    rm -rf /var/lib/apt/lists/*
+fi
+
+# --- 4. cuDSS / NVPL on arm64 + cublas (legacy JetPack / Tegra) ---
+# https://github.com/NVIDIA/Isaac-GR00T/issues/343
+if [ "${BUILD_TYPE:-}" = "cublas" ] && [ "${TARGETARCH:-}" = "arm64" ]; then
+    wget "https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
+    dpkg -i "cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
+    cp /var/cudss-local-tegra-repo-ubuntu"${UBUNTU_VERSION}"-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/
+    apt-get update
+    apt-get -y install cudss "cudss-cuda-${CUDA_MAJOR_VERSION}"
+    wget "https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
+    dpkg -i "nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
+    cp /var/nvpl-local-repo-ubuntu"${UBUNTU_VERSION}"-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/
+    apt-get update
+    apt-get install -y nvpl
+fi
+
+# --- 5. clBLAS (BUILD_TYPE=clblas) ---
+# Present in variant Dockerfiles' from-source path but not in master's
+# Dockerfile.base-grpc-builder. No CI matrix entry currently uses this,
+# but keep parity so a future BUILD_TYPE=clblas build doesn't drift.
+if [ "${BUILD_TYPE:-}" = "clblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
+    apt-get update
+    apt-get install -y --no-install-recommends \
+        libclblast-dev
+    apt-get clean
+    rm -rf /var/lib/apt/lists/*
+fi
+
+# --- 6. ROCm / HIP build deps (BUILD_TYPE=hipblas) ---
+if [ "${BUILD_TYPE:-}" = "hipblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
+    apt-get update
+    apt-get install -y --no-install-recommends \
+        hipblas-dev \
+        hipblaslt-dev \
+        rocblas-dev
+    apt-get clean
+    rm -rf /var/lib/apt/lists/*
+    # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install,
+    # which results in local-ai and others not being able to locate the libraries.
+    # We run ldconfig ourselves to work around this packaging deficiency.
+    ldconfig
+    # Log which GPU architectures have rocBLAS kernel support
+    echo "rocBLAS library data architectures:"
+    (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
+        echo "WARNING: No rocBLAS kernel data found"
+fi
+
+echo "TARGETARCH: ${TARGETARCH:-}"
+
+# --- 7. protoc (always) ---
+# The version in 22.04 is too old. We will create one as part of installing
+# the GRPC build below but that will also bring in a newer version of absl
+# which stablediffusion cannot compile with. This version of protoc is only
+# here so that we can generate the grpc code for the stablediffusion build.
+if [ "amd64" = "${TARGETARCH:-}" ]; then
+    curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip
+    unzip -j -d /usr/local/bin protoc.zip bin/protoc
+    rm protoc.zip
+fi
+if [ "arm64" = "${TARGETARCH:-}" ]; then
+    curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip
+    unzip -j -d /usr/local/bin protoc.zip bin/protoc
+    rm protoc.zip
+fi
+
+# --- 8. CMake (apt or compiled from source) ---
+# The version in 22.04 is too old. Vulkan path above already pulled cmake
+# via apt; the from-source branch here will install over it which is fine.
+if [ "${CMAKE_FROM_SOURCE:-false}" = "true" ]; then
+    curl -L -s "https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz" -o cmake.tar.gz
+    tar xvf cmake.tar.gz
+    ( cd "cmake-${CMAKE_VERSION}" && ./configure && make && make install )
+else
+    apt-get update
+    apt-get install -y \
+        cmake
+    apt-get clean
+    rm -rf /var/lib/apt/lists/*
+fi
+
+# --- 9. gRPC compile + install at /opt/grpc ---
+# We install GRPC to a different prefix here so that we can copy in only
+# the build artifacts later — saves several hundred MB on the final docker
+# image size vs copying in the entire GRPC source tree and running
+# `make install` in the target container.
+#
+# The TESTONLY abseil sed patch and /opt/grpc prefix are load-bearing —
+# downstream Dockerfiles `COPY` /opt/grpc to /usr/local (or rely on the
+# prebuilt base having it at /opt/grpc).
+mkdir -p /build
+cd /build
+git clone --recurse-submodules --jobs 4 -b "${GRPC_VERSION}" --depth 1 --shallow-submodules https://github.com/grpc/grpc
+mkdir -p /build/grpc/cmake/build
+cd /build/grpc/cmake/build
+sed -i "216i\\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt"
+cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../..
+make
+make install
+cd /
+rm -rf /build

.docker/llama-cpp-compile.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Shared compile logic for backend/Dockerfile.llama-cpp.
+# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
+
+set -euxo pipefail
+
+export CCACHE_DIR=/root/.ccache
+ccache --max-size=5G || true
+ccache -z || true
+
+export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
+fi
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  cd /LocalAI/backend/cpp/llama-cpp
+  make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
+else
+  cd /LocalAI/backend/cpp/llama-cpp
+  make llama-cpp-avx
+  make llama-cpp-avx2
+  make llama-cpp-avx512
+  make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
+fi
+
+ccache -s || true

.docker/turboquant-compile.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Shared compile logic for backend/Dockerfile.turboquant.
+# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
+
+set -euxo pipefail
+
+export CCACHE_DIR=/root/.ccache
+ccache --max-size=5G || true
+ccache -z || true
+
+export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/turboquant-*-build
+fi
+
+cd /LocalAI/backend/cpp/turboquant
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  make turboquant-fallback
+  make turboquant-grpc
+  make turboquant-rpc-server
+else
+  make turboquant-avx
+  make turboquant-avx2
+  make turboquant-avx512
+  make turboquant-fallback
+  make turboquant-grpc
+  make turboquant-rpc-server
+fi
+
+ccache -s || true

.github/actions/configure-apt-mirror/action.yml

@@ -28,11 +28,20 @@ inputs:
   self-hosted-mirror:
     description: 'archive/security mirror URL for self-hosted runners (empty = upstream)'
     required: false
-    default: 'https://mirrors.edge.kernel.org'
+    # HTTP, not HTTPS: the bare ubuntu:24.04 builder image doesn't ship
+    # ca-certificates, so the very first apt-get update over TLS would
+    # fail with "No system certificates available" before it can install
+    # anything. apt validates package integrity via GPG signatures, so
+    # plain HTTP is safe for the archive itself.
+    default: 'http://mirrors.edge.kernel.org'
   self-hosted-ports-mirror:
     description: 'ports.ubuntu.com mirror URL for self-hosted runners (empty = upstream)'
     required: false
-    default: 'https://mirrors.edge.kernel.org'
+    # mirrors.edge.kernel.org does NOT carry /ubuntu-ports/ — only the
+    # main /ubuntu/ archive — so arm64 builds 404 there. Leave ports
+    # upstream by default. The original DDoS was on archive.ubuntu.com
+    # so ports.ubuntu.com remains the path of least surprise.
+    default: ''
 
 outputs:
   effective-mirror:

.github/actions/free-disk-space/action.yml

@@ -0,0 +1,65 @@
+name: 'Free disk space on hosted runners'
+description: |
+  Aggressively clean GitHub-hosted ubuntu-latest runners to reclaim ~6-10 GB
+  of working space before docker buildx steps. Combines jlumbroso/free-disk-space
+  with explicit apt purges of large packages we never use (dotnet, ghc, mono,
+  android, jdk, ...).
+
+  No-op on self-hosted runners; pass mode=skip to force-disable.
+
+inputs:
+  mode:
+    description: 'hosted (default — clean) or skip (no-op)'
+    required: false
+    default: 'hosted'
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Free Disk Space (Ubuntu)
+      if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
+      uses: jlumbroso/free-disk-space@main
+      with:
+        tool-cache: true
+        android: true
+        dotnet: true
+        haskell: true
+        large-packages: true
+        docker-images: true
+        swap-storage: true
+
+    - name: Release space from worker
+      if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
+      shell: bash
+      run: |
+        echo "Listing top largest packages"
+        pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+        head -n 30 <<< "${pkgs}"
+        df -h
+        sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+        sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
+        sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
+        sudo rm -rf /usr/local/lib/android
+        sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+        sudo rm -rf /usr/share/dotnet
+        sudo apt-get remove -y '^mono-.*' || true
+        sudo apt-get remove -y '^ghc-.*' || true
+        sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+        sudo apt-get remove -y 'php.*' || true
+        sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+        sudo apt-get remove -y '^google-.*' || true
+        sudo apt-get remove -y azure-cli || true
+        sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+        sudo apt-get remove -y '^gfortran-.*' || true
+        sudo apt-get remove -y microsoft-edge-stable || true
+        sudo apt-get remove -y firefox || true
+        sudo apt-get remove -y powershell || true
+        sudo apt-get remove -y r-base-core || true
+        sudo apt-get autoremove -y
+        sudo apt-get clean
+        sudo rm -rfv build || true
+        sudo rm -rf /usr/share/dotnet || true
+        sudo rm -rf /opt/ghc || true
+        sudo rm -rf "/usr/local/share/boost" || true
+        sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
+        df -h

.github/actions/setup-build-disk/action.yml

@@ -0,0 +1,59 @@
+name: 'Set up build disk on hosted runners'
+description: |
+  Relocate Docker's data-root to /mnt (which has ~75 GB free, vs ~20 GB
+  on / after free-disk-space). Combined with the apt cleanup, gives
+  ~100 GB working space for buildx — enough for ROCm dev image + vLLM
+  torch install + flash-attn build.
+
+  No-op on:
+    - self-hosted runners (no /mnt expectation)
+    - non-X64 runners (verify /mnt shape on ubuntu-24.04-arm separately
+      before enabling there — see Task 3.2 in the migration plan)
+    - mode=skip (force-disable from caller)
+
+  Must run after free-disk-space (which removes large packages — would
+  fail mid-uninstall if Docker were stopped) and before any Docker
+  operation (setup-qemu, setup-buildx, login, build) so the relocated
+  data-root catches all subsequent docker activity.
+
+inputs:
+  mode:
+    description: 'auto (default — relocate on hosted X64 only) or skip'
+    required: false
+    default: 'auto'
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Relocate Docker data-root to /mnt
+      if: inputs.mode == 'auto' && runner.environment == 'github-hosted' && runner.arch == 'X64'
+      shell: bash
+      run: |
+        set -euo pipefail
+        echo "Before relocation:"
+        df -h / /mnt || true
+        sudo systemctl stop docker docker.socket
+        sudo mkdir -p /mnt/docker-data /mnt/docker-tmp
+        # buildx CLI runs as the unprivileged runner user and creates
+        # config dirs under TMPDIR before binding them into the buildkit
+        # container. /mnt is owned by root by default; mirror /tmp's
+        # 1777 (world-writable + sticky) so non-root processes can write.
+        sudo chmod 1777 /mnt/docker-tmp
+        if [ -d /var/lib/docker ] && [ ! -L /var/lib/docker ]; then
+          sudo rsync -a /var/lib/docker/ /mnt/docker-data/
+          sudo rm -rf /var/lib/docker
+          sudo ln -s /mnt/docker-data /var/lib/docker
+        fi
+        # daemon.json may not exist; merge data-root in or create minimal.
+        if [ -f /etc/docker/daemon.json ]; then
+          sudo jq '."data-root" = "/mnt/docker-data"' /etc/docker/daemon.json | sudo tee /etc/docker/daemon.json.new >/dev/null
+          sudo mv /etc/docker/daemon.json.new /etc/docker/daemon.json
+        else
+          echo '{"data-root":"/mnt/docker-data"}' | sudo tee /etc/docker/daemon.json
+        fi
+        sudo systemctl start docker
+        # Make TMPDIR persist for subsequent steps in the same job.
+        echo "TMPDIR=/mnt/docker-tmp" >> "$GITHUB_ENV"
+        echo "After relocation:"
+        df -h / /mnt
+        docker info | grep -i 'docker root dir' || true

.github/workflows/backend_build.yml

@@ -24,6 +24,17 @@ on:
         description: 'Platforms'
         default: ''
         type: string
+      platform-tag:
+        description: |
+          Short tag identifying the platform leg, e.g. "amd64" or "arm64".
+          Used to scope the per-arch registry cache and the digest artifact name.
+          Required for split-and-merge multi-arch builds; pass "amd64" for
+          single-arch amd64 builds too. Optional (default '') during the
+          migration to per-arch matrix expansion; will be flipped to
+          required: true in Phase 6 once all callers pass an explicit value.
+        required: false
+        default: ''
+        type: string
       tag-latest:
         description: 'Tag latest'
         default: ''
@@ -63,6 +74,15 @@ on:
         required: false
         default: ''
         type: string
+      builder-base-image:
+        description: |
+          Pre-built builder base image (e.g. quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64).
+          When set, the variant Dockerfile uses its `builder-prebuilt` stage which FROMs this
+          image directly instead of running its own gRPC stage + apt installs. Empty for
+          backends whose Dockerfile doesn't support a prebuilt base.
+        required: false
+        default: ''
+        type: string
     secrets:
       dockerUsername:
         required: false
@@ -89,63 +109,13 @@ jobs:
         id: apt_mirror
         uses: ./.github/actions/configure-apt-mirror
 
-      - name: Free Disk Space (Ubuntu)
-        if: inputs.runs-on == 'ubuntu-latest'
-        uses: jlumbroso/free-disk-space@main
+      - name: Free disk space
+        uses: ./.github/actions/free-disk-space
         with:
-          # this might remove tools that are actually needed,
-          # if set to "true" but frees about 6 GB
-          tool-cache: true
-          # all of these default to true, but feel free to set to
-          # "false" if necessary for your workflow
-          android: true
-          dotnet: true
-          haskell: true
-          large-packages: true
-          docker-images: true
-          swap-storage: true
+          mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
 
-      - name: Release space from worker
-        if: inputs.runs-on == 'ubuntu-latest'
-        run: |
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          df -h
-          echo
-          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-          sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
-          sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
-          sudo rm -rf /usr/local/lib/android
-          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-          sudo rm -rf /usr/share/dotnet
-          sudo apt-get remove -y '^mono-.*' || true
-          sudo apt-get remove -y '^ghc-.*' || true
-          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-          sudo apt-get remove -y 'php.*' || true
-          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-          sudo apt-get remove -y '^google-.*' || true
-          sudo apt-get remove -y azure-cli || true
-          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-          sudo apt-get remove -y '^gfortran-.*' || true
-          sudo apt-get remove -y microsoft-edge-stable || true
-          sudo apt-get remove -y firefox || true
-          sudo apt-get remove -y powershell || true
-          sudo apt-get remove -y r-base-core || true
-          sudo apt-get autoremove -y
-          sudo apt-get clean
-          echo
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          sudo rm -rfv build || true
-          sudo rm -rf /usr/share/dotnet || true
-          sudo rm -rf /opt/ghc || true
-          sudo rm -rf "/usr/local/share/boost" || true
-          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
-          df -h
+      - name: Set up build disk
+        uses: ./.github/actions/setup-build-disk
 
       - name: Docker meta
         id: meta
@@ -211,7 +181,8 @@ jobs:
         id: deps_refresh
         run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
 
-      - name: Build and push
+      - name: Build and push by digest
+        id: build
         uses: docker/build-push-action@v7
         if: github.event_name != 'pull_request'
         with:
@@ -228,16 +199,45 @@ jobs:
             APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
             APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
+            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
+            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }},mode=max,ignore-error=true
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
           platforms: ${{ inputs.platforms }}
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: ${{ steps.meta.outputs.tags }}
+          outputs: |
+            type=image,name=quay.io/go-skynet/local-ai-backends,push-by-digest=true,name-canonical=true,push=true
+            type=image,name=localai/localai-backends,push-by-digest=true,name-canonical=true,push=true
+          # Disable provenance: with mode=max (the default for push:true)
+          # buildx bundles a per-registry attestation manifest into each
+          # registry's manifest list, which makes the resulting list digest
+          # diverge across registries. steps.build.outputs.digest then
+          # only matches one of them, and the merge job's
+          # `imagetools create <reg>@sha256:<digest>` lookup fails on the
+          # other. Disabling provenance keeps the digest content-only and
+          # identical across both registries — required for digest-based
+          # cross-registry merge.
+          provenance: false
           labels: ${{ steps.meta.outputs.labels }}
 
-      - name: Build and push (PR)
+      - name: Export digest
+        if: github.event_name != 'pull_request'
+        run: |
+          mkdir -p /tmp/digests
+          digest="${{ steps.build.outputs.digest }}"
+          touch "/tmp/digests/${digest#sha256:}"
+
+      - name: Upload digest artifact
+        if: github.event_name != 'pull_request'
+        uses: actions/upload-artifact@v4
+        with:
+          name: digests${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          path: /tmp/digests/*
+          if-no-files-found: error
+          retention-days: 1
+
+      - name: Build (PR)
         uses: docker/build-push-action@v7
         if: github.event_name == 'pull_request'
         with:
@@ -254,9 +254,11 @@ jobs:
             APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
             APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
+            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
+            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
           platforms: ${{ inputs.platforms }}
           push: ${{ env.quay_username != '' }}
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/backend_build_darwin.yml

@@ -93,6 +93,11 @@ jobs:
             /opt/homebrew/Cellar/libomp
             /opt/homebrew/Cellar/llvm
             /opt/homebrew/Cellar/ccache
+            /opt/homebrew/Cellar/blake3
+            /opt/homebrew/Cellar/fmt
+            /opt/homebrew/Cellar/hiredis
+            /opt/homebrew/Cellar/xxhash
+            /opt/homebrew/Cellar/zstd
           key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
 
       - name: Dependencies
@@ -100,7 +105,23 @@ jobs:
           # ccache is always installed (used by the llama-cpp variant build) so
           # the brew cache content stays stable across every backend in the
           # matrix — they all share one cache key.
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache
+          # blake3, fmt, hiredis, xxhash, zstd are ccache's runtime dylib deps.
+          # Without explicitly installing them, a brew cache-hit run restores
+          # ccache's Cellar dir but skips installing those transitive deps,
+          # and ccache fails at runtime with `dyld: Library not loaded`.
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd
+          # Force-reinstall ccache so brew re-validates its full runtime-dep
+          # closure on every run. This is the durable fix: when the upstream
+          # ccache formula gains a new transitive dep (as it has multiple times
+          # already), we don't have to chase missing dylibs one at a time.
+          # The downloads cache makes the reinstall fast (~5s on a hit).
+          brew reinstall ccache
+          # The brew cache restores the Cellar dirs but NOT the bin symlinks
+          # at /opt/homebrew/bin/*. brew install above sees the Cellar present
+          # and decides "already installed" without re-linking, so on a cache-
+          # hit run the formulas aren't on PATH. Force-link them; --overwrite
+          # tolerates pre-existing symlinks from earlier installs.
+          brew link --overwrite protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd 2>/dev/null || true
 
       - name: Save Homebrew cache
         if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
@@ -115,6 +136,11 @@ jobs:
             /opt/homebrew/Cellar/libomp
             /opt/homebrew/Cellar/llvm
             /opt/homebrew/Cellar/ccache
+            /opt/homebrew/Cellar/blake3
+            /opt/homebrew/Cellar/fmt
+            /opt/homebrew/Cellar/hiredis
+            /opt/homebrew/Cellar/xxhash
+            /opt/homebrew/Cellar/zstd
           key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
 
       # ---- ccache for llama.cpp CMake builds ----
@@ -175,7 +201,18 @@ jobs:
           restore-keys: |
             pyenv-darwin-${{ inputs.backend }}-
 
+      # llama-cpp on Darwin uses a bespoke build script (scripts/build/llama-cpp-darwin.sh)
+      # that compiles three CMake variants from backend/cpp/llama-cpp and bundles dylibs
+      # via otool — it doesn't fit the build-darwin-go-backend / build-darwin-python-backend
+      # mold. Drive it via its dedicated `backends/llama-cpp-darwin` make target instead.
+      - name: Build ${{ inputs.backend }}-darwin (llama-cpp)
+        if: inputs.backend == 'llama-cpp'
+        run: |
+          make protogen-go
+          make backends/llama-cpp-darwin
+
       - name: Build ${{ inputs.backend }}-darwin
+        if: inputs.backend != 'llama-cpp'
         run: |
           make protogen-go
           BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend

.github/workflows/backend_merge.yml

@@ -0,0 +1,133 @@
+---
+name: 'merge backend manifest list (reusable)'
+
+# Reusable workflow that joins per-arch digest artifacts (uploaded by
+# backend_build.yml when called with platform-tag) into a single tagged
+# multi-arch manifest list. Called once per backend by backend.yml after
+# both per-arch build jobs succeed.
+
+on:
+  workflow_call:
+    inputs:
+      tag-latest:
+        description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
+        required: false
+        type: string
+        default: ''
+      tag-suffix:
+        description: 'Backend tag suffix (e.g. -cpu-faster-whisper). Used to compute the artifact pattern and the final tag suffix.'
+        required: true
+        type: string
+    secrets:
+      dockerUsername:
+        required: false
+      dockerPassword:
+        required: false
+      quayUsername:
+        required: true
+      quayPassword:
+        required: true
+
+jobs:
+  merge:
+    runs-on: ubuntu-latest
+    env:
+      quay_username: ${{ secrets.quayUsername }}
+    steps:
+      - name: Download digests
+        uses: actions/download-artifact@v4
+        with:
+          pattern: digests${{ inputs.tag-suffix }}-*
+          merge-multiple: true
+          path: /tmp/digests
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Login to DockerHub
+        if: github.event_name != 'pull_request'
+        uses: docker/login-action@v4
+        with:
+          username: ${{ secrets.dockerUsername }}
+          password: ${{ secrets.dockerPassword }}
+
+      - name: Login to Quay.io
+        if: ${{ env.quay_username != '' }}
+        uses: docker/login-action@v4
+        with:
+          registry: quay.io
+          username: ${{ secrets.quayUsername }}
+          password: ${{ secrets.quayPassword }}
+
+      - name: Docker meta
+        id: meta
+        if: github.event_name != 'pull_request'
+        uses: docker/metadata-action@v6
+        with:
+          images: |
+            quay.io/go-skynet/local-ai-backends
+            localai/localai-backends
+          tags: |
+            type=ref,event=branch
+            type=semver,pattern={{raw}}
+            type=sha
+          flavor: |
+            latest=${{ inputs.tag-latest }}
+            suffix=${{ inputs.tag-suffix }},onlatest=true
+
+      - name: Create manifest list and push (quay)
+        if: github.event_name != 'pull_request'
+        working-directory: /tmp/digests
+        run: |
+          set -euo pipefail
+          tags=$(jq -cr '
+            .tags
+            | map(select(startswith("quay.io/")))
+            | map("-t " + .)
+            | join(" ")
+          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
+          if [ -z "$tags" ]; then
+            echo "No quay.io tags from docker/metadata-action; skipping quay merge"
+          else
+            # shellcheck disable=SC2086
+            docker buildx imagetools create $tags \
+              $(printf 'quay.io/go-skynet/local-ai-backends@sha256:%s ' *)
+          fi
+
+      - name: Create manifest list and push (dockerhub)
+        if: github.event_name != 'pull_request'
+        working-directory: /tmp/digests
+        run: |
+          set -euo pipefail
+          tags=$(jq -cr '
+            .tags
+            | map(select(startswith("localai/")))
+            | map("-t " + .)
+            | join(" ")
+          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
+          if [ -z "$tags" ]; then
+            echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
+          else
+            # shellcheck disable=SC2086
+            docker buildx imagetools create $tags \
+              $(printf 'localai/localai-backends@sha256:%s ' *)
+          fi
+
+      - name: Inspect manifest
+        if: github.event_name != 'pull_request'
+        run: |
+          set -euo pipefail
+          first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
+          if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
+            docker buildx imagetools inspect "$first_tag"
+          fi
+
+      - name: Job summary
+        if: github.event_name != 'pull_request'
+        run: |
+          set -euo pipefail
+          echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
+          jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
+          echo >> "$GITHUB_STEP_SUMMARY"
+          echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
+          ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"

.github/workflows/backend_pr.yml

@@ -4,17 +4,21 @@ on:
   pull_request:
 
 concurrency:
-  group: ci-backends-pr-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: ci-backends-pr-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   generate-matrix:
     runs-on: ubuntu-latest
     outputs:
-      matrix: ${{ steps.set-matrix.outputs.matrix }}
-      matrix-darwin: ${{ steps.set-matrix.outputs.matrix-darwin }}
-      has-backends: ${{ steps.set-matrix.outputs.has-backends }}
-      has-backends-darwin: ${{ steps.set-matrix.outputs.has-backends-darwin }}
+      matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
+      matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
+      matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
+      merge-matrix: ${{ steps.set-matrix.outputs['merge-matrix'] }}
+      has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
+      has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
+      has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
+      has-merges: ${{ steps.set-matrix.outputs['has-merges'] }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -27,7 +31,9 @@ jobs:
           bun add js-yaml
           bun add @octokit/core
 
-      # filters the matrix in backend.yml
+      # filters the matrix in backend.yml; splits into single-arch and
+      # multi-arch groups so backend-merge-jobs can `needs:` only the latter
+      # (matches backend.yml's structure).
       - name: Filter matrix for changed backends
         id: set-matrix
         env:
@@ -35,10 +41,10 @@ jobs:
           GITHUB_EVENT_PATH: ${{ github.event_path }}
         run: bun run scripts/changed-backends.js
 
-  backend-jobs:
+  backend-jobs-multiarch:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build.yml
-    if: needs.generate-matrix.outputs.has-backends == 'true'
+    if: needs.generate-matrix.outputs['has-backends-multiarch'] == 'true'
     with:
       tag-latest: ${{ matrix.tag-latest }}
       tag-suffix: ${{ matrix.tag-suffix }}
@@ -46,7 +52,9 @@ jobs:
       cuda-major-version: ${{ matrix.cuda-major-version }}
       cuda-minor-version: ${{ matrix.cuda-minor-version }}
       platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
       runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
       base-image: ${{ matrix.base-image }}
       backend: ${{ matrix.backend }}
       dockerfile: ${{ matrix.dockerfile }}
@@ -59,7 +67,52 @@ jobs:
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
     strategy:
       fail-fast: true
-      matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }}
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
+  backend-jobs-singlearch:
+    needs: generate-matrix
+    uses: ./.github/workflows/backend_build.yml
+    if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
+      runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
+      base-image: ${{ matrix.base-image }}
+      backend: ${{ matrix.backend }}
+      dockerfile: ${{ matrix.dockerfile }}
+      skip-drivers: ${{ matrix.skip-drivers }}
+      context: ${{ matrix.context }}
+      ubuntu-version: ${{ matrix.ubuntu-version }}
+      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: true
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
+  backend-merge-jobs:
+    needs: [generate-matrix, backend-jobs-multiarch]
+    # backend_merge.yml's push-side steps are all gated on
+    # github.event_name != 'pull_request', so on a PR the merge job would
+    # do nothing. Skip it entirely to avoid spinning up an empty runner.
+    if: github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges'] == 'true'
+    uses: ./.github/workflows/backend_merge.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix']) }}
   backend-jobs-darwin:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build_darwin.yml
@@ -67,7 +120,7 @@ jobs:
     with:
       backend: ${{ matrix.backend }}
       build-type: ${{ matrix.build-type }}
-      go-version: "1.24.x"
+      go-version: "1.25.x"
       tag-suffix: ${{ matrix.tag-suffix }}
       lang: ${{ matrix.lang || 'python' }}
       use-pip: ${{ matrix.backend == 'diffusers' }}

.github/workflows/base-images.yml

@@ -0,0 +1,161 @@
+---
+name: 'build base-grpc images'
+
+# Builds + pushes pre-compiled builder base images that downstream
+# llama-cpp / ik-llama-cpp / turboquant variant Dockerfiles will FROM
+# (PR 2). Each base contains apt deps + protoc + cmake + gRPC at
+# /opt/grpc + (conditionally) CUDA / ROCm / Vulkan toolchains.
+#
+# Triggers:
+#   - schedule (Saturdays 05:00 UTC) - picks up Ubuntu/CUDA/ROCm
+#     security updates and re-runs ahead of the backend.yml weekly
+#     cron (Sundays 06:00 UTC).
+#   - workflow_dispatch - manual one-off rebuild.
+#   - push to master that touches Dockerfile.base-grpc-builder or
+#     this workflow itself - keeps bases in sync with their inputs.
+#
+# Bootstrap (one-time after this PR merges):
+#   gh workflow run base-images.yml --ref master
+# Wait ~30 min for all 9 matrix variants to push to
+# quay.io/go-skynet/ci-cache:base-grpc-* before merging PR 2.
+
+on:
+  schedule:
+    - cron: '0 5 * * 6'
+  workflow_dispatch:
+  push:
+    branches: [master]
+    paths:
+      - 'backend/Dockerfile.base-grpc-builder'
+      - '.github/workflows/base-images.yml'
+      # The install logic and apt-mirror helper are bind-mounted into
+      # Dockerfile.base-grpc-builder at build time — changes to either
+      # affect the produced base images and must trigger a rebuild.
+      - '.docker/install-base-deps.sh'
+      - '.docker/apt-mirror.sh'
+
+concurrency:
+  group: ci-base-images-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  build:
+    if: github.repository == 'mudler/LocalAI'
+    runs-on: ${{ matrix.runs-on }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - tag: 'base-grpc-amd64'
+            runs-on: 'ubuntu-latest'
+            base-image: 'ubuntu:24.04'
+            build-type: ''
+            cuda-major-version: ''
+            cuda-minor-version: ''
+            ubuntu-version: '2404'
+          - tag: 'base-grpc-arm64'
+            runs-on: 'ubuntu-24.04-arm'
+            base-image: 'ubuntu:24.04'
+            build-type: ''
+            cuda-major-version: ''
+            cuda-minor-version: ''
+            ubuntu-version: '2404'
+          - tag: 'base-grpc-cuda-12-amd64'
+            runs-on: 'ubuntu-latest'
+            base-image: 'ubuntu:24.04'
+            build-type: 'cublas'
+            cuda-major-version: '12'
+            cuda-minor-version: '8'
+            ubuntu-version: '2404'
+          - tag: 'base-grpc-cuda-13-amd64'
+            runs-on: 'ubuntu-latest'
+            base-image: 'ubuntu:22.04'
+            build-type: 'cublas'
+            cuda-major-version: '13'
+            cuda-minor-version: '0'
+            ubuntu-version: '2204'
+          - tag: 'base-grpc-cuda-13-arm64'
+            runs-on: 'ubuntu-24.04-arm'
+            base-image: 'ubuntu:24.04'
+            build-type: 'cublas'
+            cuda-major-version: '13'
+            cuda-minor-version: '0'
+            ubuntu-version: '2404'
+          - tag: 'base-grpc-rocm-amd64'
+            runs-on: 'ubuntu-latest'
+            base-image: 'rocm/dev-ubuntu-24.04:7.2.1'
+            build-type: 'hipblas'
+            cuda-major-version: ''
+            cuda-minor-version: ''
+            ubuntu-version: '2404'
+          - tag: 'base-grpc-vulkan-amd64'
+            runs-on: 'ubuntu-latest'
+            base-image: 'ubuntu:24.04'
+            build-type: 'vulkan'
+            cuda-major-version: ''
+            cuda-minor-version: ''
+            ubuntu-version: '2404'
+          - tag: 'base-grpc-vulkan-arm64'
+            runs-on: 'ubuntu-24.04-arm'
+            base-image: 'ubuntu:24.04'
+            build-type: 'vulkan'
+            cuda-major-version: ''
+            cuda-minor-version: ''
+            ubuntu-version: '2404'
+          - tag: 'base-grpc-intel-amd64'
+            runs-on: 'ubuntu-latest'
+            base-image: 'intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04'
+            build-type: 'sycl'
+            cuda-major-version: ''
+            cuda-minor-version: ''
+            ubuntu-version: '2404'
+          # Legacy JetPack r36.4.0 base for older Jetson devices (CUDA 12).
+          # Distinct from base-grpc-cuda-13-arm64 (Ubuntu 24.04 + CUDA 13 sbsa)
+          # which targets newer Jetsons. Some matrix entries
+          # (-nvidia-l4t-arm64-llama-cpp / -turboquant) still build against
+          # the JetPack image, so we need a matching base.
+          - tag: 'base-grpc-l4t-cuda-12-arm64'
+            runs-on: 'ubuntu-24.04-arm'
+            base-image: 'nvcr.io/nvidia/l4t-jetpack:r36.4.0'
+            build-type: 'l4t'
+            cuda-major-version: '12'
+            cuda-minor-version: '0'
+            ubuntu-version: '2204'
+            # JetPack r36.4.0 already ships CUDA preinstalled at /usr/local/cuda;
+            # apt-installing cuda-nvcc-12-0 from the public repos fails because
+            # those packages aren't published for the JetPack apt feed. Match
+            # the original l4t matrix entry which set skip-drivers: 'true'.
+            skip-drivers: 'true'
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          submodules: false
+      - name: Free disk space
+        uses: ./.github/actions/free-disk-space
+      - name: Set up build disk
+        uses: ./.github/actions/setup-build-disk
+      - uses: docker/setup-qemu-action@master
+        with:
+          platforms: all
+      - uses: docker/setup-buildx-action@master
+      - uses: docker/login-action@v4
+        with:
+          registry: quay.io
+          username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+          password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+      - uses: docker/build-push-action@v7
+        with:
+          context: .
+          file: ./backend/Dockerfile.base-grpc-builder
+          build-args: |
+            BASE_IMAGE=${{ matrix.base-image }}
+            BUILD_TYPE=${{ matrix.build-type }}
+            CUDA_MAJOR_VERSION=${{ matrix.cuda-major-version }}
+            CUDA_MINOR_VERSION=${{ matrix.cuda-minor-version }}
+            UBUNTU_VERSION=${{ matrix.ubuntu-version }}
+            SKIP_DRIVERS=${{ matrix.skip-drivers || 'false' }}
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }}
+          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }},mode=max,ignore-error=true
+          provenance: false
+          tags: quay.io/go-skynet/ci-cache:${{ matrix.tag }}
+          push: true

.github/workflows/bump_deps.yaml

@@ -50,6 +50,10 @@ jobs:
             variable: "QWEN3TTS_CPP_VERSION"
             branch: "main"
             file: "backend/go/qwen3-tts-cpp/Makefile"
+          - repository: "localai-org/vibevoice.cpp"
+            variable: "VIBEVOICE_CPP_VERSION"
+            branch: "master"
+            file: "backend/go/vibevoice-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6

.github/workflows/generate_intel_image.yaml

@@ -7,8 +7,8 @@ on:
       - master
 
 concurrency:
-  group: intel-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: intel-cache-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   generate_caches:

.github/workflows/image-pr.yml

@@ -5,8 +5,8 @@
     pull_request:
   
   concurrency:
-    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
-    cancel-in-progress: true
+    group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+    cancel-in-progress: ${{ github.event_name == 'pull_request' }}
   
   jobs:
     image-build:
@@ -18,6 +18,7 @@
         cuda-major-version: ${{ matrix.cuda-major-version }}
         cuda-minor-version: ${{ matrix.cuda-minor-version }}
         platforms: ${{ matrix.platforms }}
+        platform-tag: ${{ matrix.platform-tag || '' }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
         makeflags: ${{ matrix.makeflags }}
@@ -71,13 +72,23 @@
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'vulkan'
-              platforms: 'linux/amd64,linux/arm64'
+              platforms: 'linux/amd64'
+              platform-tag: 'amd64'
               tag-latest: 'false'
               tag-suffix: '-vulkan-core'
               runs-on: 'ubuntu-latest'
               base-image: "ubuntu:24.04"
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'
+            - build-type: 'vulkan'
+              platforms: 'linux/arm64'
+              platform-tag: 'arm64'
+              tag-latest: 'false'
+              tag-suffix: '-vulkan-core'
+              runs-on: 'ubuntu-24.04-arm'
+              base-image: "ubuntu:24.04"
+              makeflags: "--jobs=4 --output-sync=target"
+              ubuntu-version: '2404'
             - build-type: 'cublas'
               cuda-major-version: "13"
               cuda-minor-version: "0"

.github/workflows/image.yml

@@ -9,8 +9,8 @@
         - '*'
   
   concurrency:
-    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
-    cancel-in-progress: true
+    group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+    cancel-in-progress: ${{ github.event_name == 'pull_request' }}
   
   jobs:
     hipblas-jobs:
@@ -56,6 +56,7 @@
         cuda-major-version: ${{ matrix.cuda-major-version }}
         cuda-minor-version: ${{ matrix.cuda-minor-version }}
         platforms: ${{ matrix.platforms }}
+        platform-tag: ${{ matrix.platform-tag || '' }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
         makeflags: ${{ matrix.makeflags }}
@@ -72,7 +73,8 @@
         matrix:
           include:
             - build-type: ''
-              platforms: 'linux/amd64,linux/arm64'
+              platforms: 'linux/amd64'
+              platform-tag: 'amd64'
               tag-latest: 'auto'
               tag-suffix: ''
               base-image: "ubuntu:24.04"
@@ -81,6 +83,17 @@
               skip-drivers: 'false'
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
+            - build-type: ''
+              platforms: 'linux/arm64'
+              platform-tag: 'arm64'
+              tag-latest: 'auto'
+              tag-suffix: ''
+              base-image: "ubuntu:24.04"
+              runs-on: 'ubuntu-24.04-arm'
+              makeflags: "--jobs=4 --output-sync=target"
+              skip-drivers: 'false'
+              ubuntu-version: '2404'
+              ubuntu-codename: 'noble'
             - build-type: 'cublas'
               cuda-major-version: "12"
               cuda-minor-version: "8"
@@ -106,7 +119,8 @@
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
             - build-type: 'vulkan'
-              platforms: 'linux/amd64,linux/arm64'
+              platforms: 'linux/amd64'
+              platform-tag: 'amd64'
               tag-latest: 'auto'
               tag-suffix: '-gpu-vulkan'
               runs-on: 'ubuntu-latest'
@@ -115,6 +129,17 @@
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
+            - build-type: 'vulkan'
+              platforms: 'linux/arm64'
+              platform-tag: 'arm64'
+              tag-latest: 'auto'
+              tag-suffix: '-gpu-vulkan'
+              runs-on: 'ubuntu-24.04-arm'
+              base-image: "ubuntu:24.04"
+              skip-drivers: 'false'
+              makeflags: "--jobs=4 --output-sync=target"
+              ubuntu-version: '2404'
+              ubuntu-codename: 'noble'
             - build-type: 'intel'
               platforms: 'linux/amd64'
               tag-latest: 'auto'
@@ -124,6 +149,32 @@
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
+
+    core-image-merge:
+      if: github.repository == 'mudler/LocalAI'
+      needs: core-image-build
+      uses: ./.github/workflows/image_merge.yml
+      with:
+        tag-latest: 'auto'
+        tag-suffix: ''
+      secrets:
+        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+
+    gpu-vulkan-image-merge:
+      if: github.repository == 'mudler/LocalAI'
+      needs: core-image-build
+      uses: ./.github/workflows/image_merge.yml
+      with:
+        tag-latest: 'auto'
+        tag-suffix: '-gpu-vulkan'
+      secrets:
+        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
   
     gh-runner:
       if: github.repository == 'mudler/LocalAI'

.github/workflows/image_build.yml

@@ -24,6 +24,15 @@ on:
         description: 'Platforms'
         default: ''
         type: string
+      platform-tag:
+        description: |
+          Short tag identifying the platform leg, e.g. "amd64" or "arm64".
+          Used to scope the per-arch registry cache and the digest artifact name.
+          Optional during the migration; will be flipped to required: true once
+          every caller passes an explicit value.
+        required: false
+        default: ''
+        type: string
       tag-latest:
         description: 'Tag latest'
         default: ''
@@ -77,63 +86,13 @@ jobs:
         id: apt_mirror
         uses: ./.github/actions/configure-apt-mirror
 
-      - name: Free Disk Space (Ubuntu)
-        if: inputs.runs-on == 'ubuntu-latest'
-        uses: jlumbroso/free-disk-space@main
+      - name: Free disk space
+        uses: ./.github/actions/free-disk-space
         with:
-          # this might remove tools that are actually needed,
-          # if set to "true" but frees about 6 GB
-          tool-cache: true
-          # all of these default to true, but feel free to set to
-          # "false" if necessary for your workflow
-          android: true
-          dotnet: true
-          haskell: true
-          large-packages: true
-          docker-images: true
-          swap-storage: true
+          mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
 
-      - name: Release space from worker
-        if: inputs.runs-on == 'ubuntu-latest'
-        run: |
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          df -h
-          echo
-          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-          sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
-          sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
-          sudo rm -rf /usr/local/lib/android
-          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-          sudo rm -rf /usr/share/dotnet
-          sudo apt-get remove -y '^mono-.*' || true
-          sudo apt-get remove -y '^ghc-.*' || true
-          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-          sudo apt-get remove -y 'php.*' || true
-          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-          sudo apt-get remove -y '^google-.*' || true
-          sudo apt-get remove -y azure-cli || true
-          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-          sudo apt-get remove -y '^gfortran-.*' || true
-          sudo apt-get remove -y microsoft-edge-stable || true
-          sudo apt-get remove -y firefox || true
-          sudo apt-get remove -y powershell || true
-          sudo apt-get remove -y r-base-core || true
-          sudo apt-get autoremove -y
-          sudo apt-get clean
-          echo
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          sudo rm -rfv build || true
-          sudo rm -rf /usr/share/dotnet || true
-          sudo rm -rf /opt/ghc || true
-          sudo rm -rf "/usr/local/share/boost" || true
-          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
-          df -h
+      - name: Set up build disk
+        uses: ./.github/actions/setup-build-disk
 
       - name: Docker meta
         id: meta
@@ -188,7 +147,8 @@ jobs:
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
-      - name: Build and push
+      - name: Build and push by digest
+        id: build
         uses: docker/build-push-action@v7
         if: github.event_name != 'pull_request'
         with:
@@ -206,12 +166,33 @@ jobs:
             APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
           context: .
           file: ./Dockerfile
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }},mode=max,ignore-error=true
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
           platforms: ${{ inputs.platforms }}
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: ${{ steps.meta.outputs.tags }}
+          outputs: |
+            type=image,name=quay.io/go-skynet/local-ai,push-by-digest=true,name-canonical=true,push=true
+            type=image,name=localai/localai,push-by-digest=true,name-canonical=true,push=true
+          # See backend_build.yml for the rationale — provenance=mode=max
+          # diverges the manifest-list digest per registry, breaking the
+          # downstream imagetools create lookup.
+          provenance: false
           labels: ${{ steps.meta.outputs.labels }}
+
+      - name: Export digest
+        if: github.event_name != 'pull_request'
+        run: |
+          mkdir -p /tmp/digests
+          digest="${{ steps.build.outputs.digest }}"
+          touch "/tmp/digests/${digest#sha256:}"
+
+      - name: Upload digest artifact
+        if: github.event_name != 'pull_request'
+        uses: actions/upload-artifact@v4
+        with:
+          name: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          path: /tmp/digests/*
+          if-no-files-found: error
+          retention-days: 1
 ### Start testing image
       - name: Build and push
         uses: docker/build-push-action@v7
@@ -231,7 +212,7 @@ jobs:
             APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
           context: .
           file: ./Dockerfile
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
+          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
           platforms: ${{ inputs.platforms }}
           #push: true
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/image_merge.yml

@@ -0,0 +1,117 @@
+---
+name: 'merge LocalAI image manifest list (reusable)'
+
+# Reusable workflow that joins per-arch digest artifacts (uploaded by
+# image_build.yml when called with platform-tag) into a single tagged
+# multi-arch manifest list.
+
+on:
+  workflow_call:
+    inputs:
+      tag-latest:
+        description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
+        required: false
+        type: string
+        default: ''
+      tag-suffix:
+        description: 'Image tag suffix (empty for core image). Used in artifact pattern with a -core placeholder for empty.'
+        required: true
+        type: string
+    secrets:
+      dockerUsername:
+        required: false
+      dockerPassword:
+        required: false
+      quayUsername:
+        required: true
+      quayPassword:
+        required: true
+
+jobs:
+  merge:
+    runs-on: ubuntu-latest
+    env:
+      quay_username: ${{ secrets.quayUsername }}
+    steps:
+      - name: Download digests
+        uses: actions/download-artifact@v4
+        with:
+          pattern: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}-*
+          merge-multiple: true
+          path: /tmp/digests
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Login to DockerHub
+        if: github.event_name != 'pull_request'
+        uses: docker/login-action@v4
+        with:
+          username: ${{ secrets.dockerUsername }}
+          password: ${{ secrets.dockerPassword }}
+
+      - name: Login to Quay.io
+        uses: docker/login-action@v4
+        with:
+          registry: quay.io
+          username: ${{ secrets.quayUsername }}
+          password: ${{ secrets.quayPassword }}
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v6
+        with:
+          images: |
+            quay.io/go-skynet/local-ai
+            localai/localai
+          tags: |
+            type=ref,event=branch
+            type=semver,pattern={{raw}}
+            type=sha
+          flavor: |
+            latest=${{ inputs.tag-latest }}
+            suffix=${{ inputs.tag-suffix }},onlatest=true
+
+      - name: Create manifest list and push (quay)
+        working-directory: /tmp/digests
+        run: |
+          set -euo pipefail
+          tags=$(jq -cr '.tags | map(select(startswith("quay.io/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
+          if [ -z "$tags" ]; then
+            echo "No quay.io tags from docker/metadata-action; skipping quay merge"
+          else
+            # shellcheck disable=SC2086
+            docker buildx imagetools create $tags \
+              $(printf 'quay.io/go-skynet/local-ai@sha256:%s ' *)
+          fi
+
+      - name: Create manifest list and push (dockerhub)
+        if: github.event_name != 'pull_request'
+        working-directory: /tmp/digests
+        run: |
+          set -euo pipefail
+          tags=$(jq -cr '.tags | map(select(startswith("localai/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
+          if [ -z "$tags" ]; then
+            echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
+          else
+            # shellcheck disable=SC2086
+            docker buildx imagetools create $tags \
+              $(printf 'localai/localai@sha256:%s ' *)
+          fi
+
+      - name: Inspect manifest
+        run: |
+          set -euo pipefail
+          first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
+          if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
+            docker buildx imagetools inspect "$first_tag"
+          fi
+
+      - name: Job summary
+        run: |
+          set -euo pipefail
+          echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
+          jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
+          echo >> "$GITHUB_STEP_SUMMARY"
+          echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
+          ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"

.github/workflows/lint.yml

@@ -13,14 +13,14 @@ on:
       - master
 
 concurrency:
-  group: ci-lint-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: ci-lint-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   golangci-lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v6
         with:
           # Full history so golangci-lint's new-from-merge-base can reach
           # origin/master and compute the diff against it.

.github/workflows/test-extra.yml

@@ -10,8 +10,8 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-extra-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: ci-tests-extra-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   detect-changes:
@@ -37,11 +37,13 @@ jobs:
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
       qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
       vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
+      localvqe: ${{ steps.detect.outputs.localvqe }}
       voxtral: ${{ steps.detect.outputs.voxtral }}
       kokoros: ${{ steps.detect.outputs.kokoros }}
       insightface: ${{ steps.detect.outputs.insightface }}
       speaker-recognition: ${{ steps.detect.outputs.speaker-recognition }}
       sherpa-onnx: ${{ steps.detect.outputs.sherpa-onnx }}
+      whisper: ${{ steps.detect.outputs.whisper }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -582,6 +584,27 @@ jobs:
       - name: Build sherpa-onnx backend image and run streaming ASR gRPC e2e tests
         run: |
           make test-extra-backend-sherpa-onnx-transcription
+  # End-to-end transcription via the e2e-backends gRPC harness against
+  # the whisper.cpp backend. Drives AudioTranscription (offline) and
+  # AudioTranscriptionStream (real, segment-callback-driven deltas) on
+  # ggml-base.en + the JFK 11s clip.
+  tests-whisper-grpc-transcription:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.whisper == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25.4'
+      - name: Build whisper backend image and run transcription gRPC e2e tests
+        run: |
+          make test-extra-backend-whisper-transcription
   # VITS TTS via the sherpa-onnx backend. Drives both TTS (file write) and
   # TTSStream (PCM chunks) on the e2e-backends harness.
   tests-sherpa-onnx-grpc-tts:
@@ -884,6 +907,26 @@ jobs:
       - name: Build vibevoice-cpp backend image and run ASR gRPC e2e tests
         run: |
           make test-extra-backend-vibevoice-cpp-transcription
+  # End-to-end audio transform via the e2e-backends gRPC harness. The
+  # LocalVQE GGUF is small (~5 MB) and the model is real-time on CPU, so
+  # the default ubuntu-latest pool is plenty.
+  tests-localvqe-grpc-transform:
+    needs: detect-changes
+    if: needs.detect-changes.outputs.localvqe == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25.4'
+      - name: Build localvqe backend image and run audio_transform gRPC e2e tests
+        run: |
+          make test-extra-backend-localvqe-transform
   tests-voxtral:
     needs: detect-changes
     if: needs.detect-changes.outputs.voxtral == 'true' || needs.detect-changes.outputs.run-all == 'true'

.github/workflows/test.yml

@@ -3,12 +3,6 @@ name: 'tests'
 
 on:
   pull_request:
-    paths-ignore:
-      - 'docs/**'
-      - 'examples/**'
-      - 'README.md'
-      - '**/*.md'
-      - 'backend/**'
   push:
     branches:
       - master
@@ -16,8 +10,8 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: ci-tests-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   tests-linux:
@@ -26,56 +20,12 @@ jobs:
       matrix:
         go-version: ['1.26.x']
     steps:
-      - name: Free Disk Space (Ubuntu)
-        uses: jlumbroso/free-disk-space@main
-        with:
-          # this might remove tools that are actually needed,
-          # if set to "true" but frees about 6 GB
-          tool-cache: true
-          # all of these default to true, but feel free to set to
-          # "false" if necessary for your workflow
-          android: true
-          dotnet: true
-          haskell: true
-          large-packages: true
-          docker-images: true
-          swap-storage: true
-      - name: Release space from worker
-        run: |
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          df -h
-          echo
-          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
-          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
-          sudo rm -rf /usr/local/lib/android
-          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-          sudo rm -rf /usr/share/dotnet
-          sudo apt-get remove -y '^mono-.*' || true
-          sudo apt-get remove -y '^ghc-.*' || true
-          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-          sudo apt-get remove -y 'php.*' || true
-          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-          sudo apt-get remove -y '^google-.*' || true
-          sudo apt-get remove -y azure-cli || true
-          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-          sudo apt-get remove -y '^gfortran-.*' || true
-          sudo apt-get autoremove -y
-          sudo apt-get clean
-          echo
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          sudo rm -rfv build || true
-          df -h
       - name: Clone
         uses: actions/checkout@v6
         with:
           submodules: true
+      - name: Free disk space
+        uses: ./.github/actions/free-disk-space
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/tests-aio.yml

@@ -22,8 +22,8 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-aio-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: ci-tests-aio-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   tests-aio:

.github/workflows/tests-e2e.yml

@@ -3,12 +3,6 @@ name: 'E2E Backend Tests'
 
 on:
   pull_request:
-    paths-ignore:
-      - 'docs/**'
-      - 'examples/**'
-      - 'README.md'
-      - '**/*.md'
-      - 'backend/**'
   push:
     branches:
       - master
@@ -16,8 +10,8 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-e2e-backend-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: ci-tests-e2e-backend-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   tests-e2e-backend:

.github/workflows/tests-ui-e2e.yml

@@ -12,8 +12,8 @@ on:
       - master
 
 concurrency:
-  group: ci-tests-ui-e2e-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
+  group: ci-tests-ui-e2e-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
   tests-ui-e2e:

AGENTS.md

@@ -19,11 +19,12 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 |------|-------------|
 | [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md) | Policy for AI-assisted contributions — licensing, DCO, attribution |
 | [.agents/building-and-testing.md](.agents/building-and-testing.md) | Building the project, running tests, Docker builds for specific platforms |
-| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, manual eviction |
+| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache, per-arch keys), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, prebuilt `base-grpc-*` images for llama.cpp variants, per-arch native + manifest-merge pattern, `setup-build-disk` `/mnt` relocation, path filter on master push, manual eviction |
 | [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist, including importer integration (the `/import-model` dropdown is server-driven from `GET /backends/known`) |
 | [.agents/coding-style.md](.agents/coding-style.md) | Code style, editorconfig, logging, documentation conventions |
 | [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
 | [.agents/vllm-backend.md](.agents/vllm-backend.md) | Working on the vLLM / vLLM-omni backends — native parsers, ChatDelta, CPU build, libnuma packaging, backend hooks |
+| [.agents/sglang-backend.md](.agents/sglang-backend.md) | Working on the SGLang backend — `engine_args` validation against ServerArgs, speculative-decoding (EAGLE/EAGLE3/DFLASH/MTP) recipes, parser handling |
 | [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
 | [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
 | [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/vibevoice-cpp backends/tinygrad backends/sherpa-onnx
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -232,6 +232,20 @@ run-e2e-aio: protogen-go
 	@echo 'Running e2e AIO tests'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e-aio
 
+# vLLM multi-node DP smoke (CPU). Builds local-ai:tests and the
+# cpu-vllm backend from the current working tree, then drives a
+# head + headless follower via testcontainers-go and asserts a chat
+# completion. BuildKit caches both images, so re-runs only rebuild
+# what changed. The test lives under tests/e2e/distributed and is
+# selected by the VLLMMultinode label so it doesn't run alongside
+# the other distributed-suite tests by default.
+test-e2e-vllm-multinode: docker-build-e2e extract-backend-vllm protogen-go
+	@echo 'Running e2e vLLM multi-node DP test'
+	LOCALAI_IMAGE=local-ai \
+	LOCALAI_IMAGE_TAG=tests \
+	LOCALAI_VLLM_BACKEND_DIR=$(abspath ./local-backends/vllm) \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='VLLMMultinode' -v -r ./tests/e2e/distributed
+
 ########################################################
 ## E2E tests
 ########################################################
@@ -319,7 +333,7 @@ local-backends:
 
 extract-backend-%: docker-build-% local-backends
 	@echo "Extracting backend $*..."
-	@CID=$$(docker create local-ai-backend:$*) && \
+	@CID=$$(docker create --entrypoint=/run.sh local-ai-backend:$*) && \
 	  rm -rf local-backends/$* && mkdir -p local-backends/$* && \
 	  docker cp $$CID:/ - | tar -xf - -C local-backends/$* && \
 	  docker rm $$CID > /dev/null
@@ -580,6 +594,7 @@ test-extra-backend-llama-cpp-transcription: docker-build-llama-cpp
 	BACKEND_TEST_MMPROJ_URL=https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf \
 	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
 	BACKEND_TEST_CAPS=health,load,transcription \
+	BACKEND_TEST_CTX_SIZE=2048 \
 	$(MAKE) test-extra-backend
 
 ## vllm is resolved from a HuggingFace model id (no file download) and
@@ -594,6 +609,14 @@ test-extra-backend-vllm: docker-build-vllm
 	BACKEND_TEST_OPTIONS=tool_parser:hermes \
 	$(MAKE) test-extra-backend
 
+## vllm multi-node data-parallel smoke test. Runs LocalAI head + a
+## `local-ai p2p-worker vllm` follower in docker compose against
+## Qwen2.5-0.5B with data_parallel_size=2. Requires 2 NVIDIA GPUs and
+## nvidia-container-runtime on the host — vLLM v1's DP coordinator is
+## not viable on CPU so this cannot run in CI without GPU.
+test-extra-backend-vllm-multinode:
+	./tests/e2e/vllm-multinode/smoke.sh
+
 ## tinygrad mirrors the vllm target (same model, same caps, same parser) so
 ## the two backends are directly comparable. The LLM path covers Predict,
 ## streaming and native tool-call extraction. Companion targets below cover
@@ -874,6 +897,28 @@ test-extra-backend-vibevoice-cpp-transcription: docker-build-vibevoice-cpp
 	BACKEND_TEST_CAPS=health,load,transcription \
 	$(MAKE) test-extra-backend
 
+## Audio transcription wrapper for the whisper.cpp backend.
+## Drives the AudioTranscription / AudioTranscriptionStream RPCs against
+## ggml-base.en (~145 MB) using the JFK 11s clip. The streaming spec
+## asserts len(deltas) >= 1 and concat(deltas) == final.Text - whisper-
+## specific multi-segment assertions live in backend/go/whisper/gowhisper_test.go.
+test-extra-backend-whisper-transcription: docker-build-whisper
+	BACKEND_IMAGE=local-ai-backend:whisper \
+	BACKEND_TEST_MODEL_URL=https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin \
+	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
+	BACKEND_TEST_CAPS=health,load,transcription \
+	$(MAKE) test-extra-backend
+
+## LocalVQE audio transform (joint AEC + noise suppression + dereverb).
+## Exercises the audio_transform capability end-to-end: batch transform
+## of a real WAV fixture and bidi streaming of synthetic silent frames.
+test-extra-backend-localvqe-transform: docker-build-localvqe
+	BACKEND_IMAGE=local-ai-backend:localvqe \
+	BACKEND_TEST_MODEL_URL='https://huggingface.co/LocalAI-io/LocalVQE/resolve/main/localvqe-v1-1.3M-f32.gguf#localvqe-v1-1.3M-f32.gguf' \
+	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
+	BACKEND_TEST_CAPS=health,load,audio_transform \
+	$(MAKE) test-extra-backend
+
 ## sglang mirrors the vllm setup: HuggingFace model id, same tiny Qwen,
 ## tool-call extraction via sglang's native qwen parser. CPU builds use
 ## sglang's upstream pyproject_cpu.toml recipe (see backend/python/sglang/install.sh).
@@ -1017,6 +1062,7 @@ BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
+BACKEND_LOCALVQE = localvqe|golang|.|false|true
 BACKEND_OPUS = opus|golang|.|false|true
 BACKEND_SHERPA_ONNX = sherpa-onnx|golang|.|false|true
 
@@ -1127,6 +1173,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX_VLM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX_DISTRIBUTED)))
@@ -1141,7 +1188,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-vibevoice-cpp docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx
 
 ########################################################
 ### Mock Backend for E2E Tests

backend/Dockerfile.base-grpc-builder

@@ -0,0 +1,98 @@
+# syntax=docker/dockerfile:1.7
+#
+# Pre-built builder base image for LocalAI's C++ backends.
+#
+# This Dockerfile is the source of truth for the
+# `quay.io/go-skynet/ci-cache:base-grpc-*` images that
+# `.github/workflows/base-images.yml` builds and pushes. The output of a
+# build is a fully-prepped builder layer containing:
+#
+#   - apt build deps (build-essential, ccache, git, make, pkg-config,
+#     libcurl4-openssl-dev, libssl-dev, curl, unzip, wget, ca-certificates)
+#   - cmake (apt or, when CMAKE_FROM_SOURCE=true, compiled from
+#     ${CMAKE_VERSION})
+#   - protoc v27.1 at /usr/local/bin/protoc
+#   - gRPC ${GRPC_VERSION} compiled and installed at /opt/grpc
+#   - Conditional CUDA toolkit (BUILD_TYPE=cublas|l4t, SKIP_DRIVERS=false)
+#     including the cuda-13 + arm64 cudss/nvpl special case
+#   - Conditional ROCm/HIP build deps (BUILD_TYPE=hipblas)
+#   - Conditional Vulkan SDK 1.4.335.0 (BUILD_TYPE=vulkan)
+#
+# Variants built by the workflow (matrix in base-images.yml):
+#
+#   base-grpc-amd64                 ubuntu:24.04, CPU-only
+#   base-grpc-arm64                 ubuntu:24.04, CPU-only
+#   base-grpc-cuda-12-amd64         ubuntu:24.04 + CUDA 12.8
+#   base-grpc-cuda-13-amd64         ubuntu:22.04 + CUDA 13.0
+#   base-grpc-cuda-13-arm64         ubuntu:24.04 + CUDA 13.0 (sbsa)
+#   base-grpc-l4t-cuda-12-arm64     ubuntu:22.04 + CUDA 12.x (legacy JetPack)
+#   base-grpc-rocm-amd64            rocm/dev-ubuntu-24.04:7.2.1 + hipblas
+#   base-grpc-vulkan-amd64          ubuntu:24.04 + Vulkan SDK 1.4.335
+#   base-grpc-vulkan-arm64          ubuntu:24.04 + Vulkan SDK ARM 1.4.335
+#   base-grpc-intel-amd64           intel/oneapi-basekit:2025.3.2 (sycl)
+#
+# This is a SINGLE-stage Dockerfile by design: the final image IS the
+# builder base. The intermediate gRPC compile happens inside this same
+# stage so consumer Dockerfiles in PR 2 can simply
+# `FROM quay.io/go-skynet/ci-cache:base-grpc-<variant>` without needing a
+# COPY --from=grpc step. /opt/grpc is the canonical install prefix and
+# downstream builds will add it to CMAKE_PREFIX_PATH (or copy to
+# /usr/local) the same way Dockerfile.llama-cpp does today.
+#
+# Install logic lives in .docker/install-base-deps.sh, which is also
+# bind-mounted by the variant Dockerfiles' builder-fromsource stage.
+# This guarantees bit-equivalence between the prebuilt CI base and the
+# from-source local-dev path — both invoke the same script with the
+# same env inputs.
+
+ARG BASE_IMAGE=ubuntu:24.04
+
+FROM ${BASE_IMAGE}
+
+ARG BASE_IMAGE=ubuntu:24.04
+ARG BUILD_TYPE=""
+ARG CUDA_MAJOR_VERSION=""
+ARG CUDA_MINOR_VERSION=""
+ARG CMAKE_FROM_SOURCE=false
+# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain
+# detection / arch table issues.
+ARG CMAKE_VERSION=3.31.10
+ARG GRPC_VERSION=v1.65.0
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+ARG SKIP_DRIVERS=false
+ARG TARGETARCH
+ARG UBUNTU_VERSION=2404
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
+ARG AMDGPU_TARGETS=""
+
+ENV BUILD_TYPE=${BUILD_TYPE} \
+    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
+    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
+    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
+    CMAKE_VERSION=${CMAKE_VERSION} \
+    GRPC_VERSION=${GRPC_VERSION} \
+    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
+    SKIP_DRIVERS=${SKIP_DRIVERS} \
+    TARGETARCH=${TARGETARCH} \
+    UBUNTU_VERSION=${UBUNTU_VERSION} \
+    APT_MIRROR=${APT_MIRROR} \
+    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
+    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
+    MAKEFLAGS=${GRPC_MAKEFLAGS} \
+    DEBIAN_FRONTEND=noninteractive
+
+# CUDA on PATH (no-op when CUDA isn't installed)
+ENV PATH=/usr/local/cuda/bin:${PATH}
+# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
+ENV PATH=/opt/rocm/bin:${PATH}
+
+WORKDIR /build
+
+# Single RUN that delegates to .docker/install-base-deps.sh — the same
+# script the variant Dockerfiles' builder-fromsource stage runs.
+RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
+    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    bash /usr/local/sbin/install-base-deps
+
+WORKDIR /

backend/Dockerfile.golang

@@ -21,6 +21,12 @@ ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
 ARG APT_MIRROR
 ARG APT_PORTS_MIRROR
 
+# gcc-14 is the default on noble (ubuntu:24.04) but absent from jammy
+# (the L4T jetpack r36.4.0 base). LocalVQE specifically needs it; the
+# other Go backends compile fine with the default gcc shipped via
+# build-essential. So: try gcc-14 from the configured repos, fall back
+# gracefully when it's not available so jammy-based builds don't fail
+# at the apt step.
 RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
     APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
     apt-get update && \
@@ -31,6 +37,12 @@ RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mi
         make cmake wget libopenblas-dev \
         curl unzip \
         libssl-dev && \
+    if apt-cache show gcc-14 >/dev/null 2>&1 && apt-cache show g++-14 >/dev/null 2>&1; then \
+        apt-get install -y --no-install-recommends gcc-14 g++-14 && \
+        update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100 \
+            --slave /usr/bin/g++ g++ /usr/bin/g++-14 \
+            --slave /usr/bin/gcov gcov /usr/bin/gcov-14; \
+    fi && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 

backend/Dockerfile.ik-llama-cpp

@@ -1,290 +1,149 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
+# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
+# when no prebuilt base is supplied. The builder-prebuilt stage is only
+# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
+# content here is harmless — BuildKit prunes the unreferenced builder.
+ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
+# BUILDER_TARGET selects which builder stage the final scratch image copies
+# package output from. Declared at global scope (before any FROM) so it's
+# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
+# `make backends/ik-llama-cpp` on the from-source path.
+ARG BUILDER_TARGET=builder-fromsource
 ARG APT_MIRROR=""
 ARG APT_PORTS_MIRROR=""
 
 
-# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
-# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
-FROM ${GRPC_BASE_IMAGE} AS grpc
-
-# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG GRPC_VERSION=v1.65.0
+# ============================================================================
+# Stage: builder-fromsource — self-contained build path.
+# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
+# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
+# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
+# default; local `make backends/ik-llama-cpp`).
+#
+# The install script is the same one that backend/Dockerfile.base-grpc-builder
+# runs, so the result is bit-equivalent to the prebuilt-base path
+# (builder-prebuilt below).
+# ============================================================================
+FROM ${BASE_IMAGE} AS builder-fromsource
+ARG BUILD_TYPE
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-
-ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
-
-WORKDIR /build
-
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends \
-        ca-certificates \
-        build-essential curl libssl-dev \
-        git wget && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
-
-# Install CMake (the version in 22.04 is too old)
-RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
-        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
-    else
-        apt-get update && \
-        apt-get install -y \
-            cmake && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
-# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
-# and running make install in the target container
-RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
-    mkdir -p /build/grpc/cmake/build && \
-    cd /build/grpc/cmake/build && \
-    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
-    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
-    make && \
-    make install && \
-    rm -rf /build
-
-FROM ${BASE_IMAGE} AS builder
-ARG CMAKE_FROM_SOURCE=false
-ARG CMAKE_VERSION=3.31.10
-# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-ARG BACKEND=rerankers
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
+ARG GRPC_VERSION=v1.65.0
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
 ARG SKIP_DRIVERS=false
-ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
-ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
-ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
 ARG APT_MIRROR
 ARG APT_PORTS_MIRROR
+ARG AMDGPU_TARGETS=""
+ARG BACKEND=rerankers
+# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ARG CMAKE_ARGS
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends \
-        build-essential \
-        ccache git \
-        ca-certificates \
-        make \
-        pkg-config libcurl4-openssl-dev \
-        curl unzip \
-        libssl-dev wget && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
+ENV BUILD_TYPE=${BUILD_TYPE} \
+    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
+    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
+    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
+    CMAKE_VERSION=${CMAKE_VERSION} \
+    GRPC_VERSION=${GRPC_VERSION} \
+    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
+    SKIP_DRIVERS=${SKIP_DRIVERS} \
+    TARGETARCH=${TARGETARCH} \
+    UBUNTU_VERSION=${UBUNTU_VERSION} \
+    APT_MIRROR=${APT_MIRROR} \
+    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
+    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
+    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
+    CMAKE_ARGS=${CMAKE_ARGS} \
+    DEBIAN_FRONTEND=noninteractive
 
-# Cuda
+# CUDA on PATH (no-op when CUDA isn't installed)
 ENV PATH=/usr/local/cuda/bin:${PATH}
-
-# HipBLAS requirements
+# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
 ENV PATH=/opt/rocm/bin:${PATH}
 
+WORKDIR /build
 
-# Vulkan requirements
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
-        apt-get update && \
-        apt-get install -y  --no-install-recommends \
-            software-properties-common pciutils wget gpg-agent && \
-        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
-            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
-            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
-            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
-            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
-            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
-            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            mkdir -p /opt/vulkan-sdk && \
-            mv 1.4.335.0 /opt/vulkan-sdk/ && \
-            cd /opt/vulkan-sdk/1.4.335.0 && \
-            ./vulkansdk --no-deps --maxjobs \
-                vulkan-loader \
-                vulkan-validationlayers \
-                vulkan-extensionlayer \
-                vulkan-tools \
-                shaderc && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
-            rm -rf /opt/vulkan-sdk
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            mkdir vulkan && cd vulkan && \
-            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-            tar -xvf vulkan-sdk.tar.xz && \
-            rm vulkan-sdk.tar.xz && \
-            cd 1.4.335.0 && \
-            cp -rfv aarch64/bin/* /usr/bin/ && \
-            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-            cp -rfv aarch64/include/* /usr/include/ && \
-            cp -rfv aarch64/share/* /usr/share/ && \
-            cd ../.. && \
-            rm -rf vulkan
-        fi
-        ldconfig && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-# CuBLAS requirements
-RUN <<EOT bash
-    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
-        apt-get update && \
-        apt-get install -y  --no-install-recommends \
-            software-properties-common pciutils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
-            else
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
-            fi
-        fi
-        dpkg -i cuda-keyring_1.1-1_all.deb && \
-        rm -f cuda-keyring_1.1-1_all.deb && \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
-            apt-get install -y --no-install-recommends \
-            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        fi
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
-        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get install -y nvpl
-    fi
-EOT
-
-# If we are building with clblas support, we need the libraries for the builds
-RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            libclblast-dev && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/* \
-    ; fi
-
-RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            hipblas-dev \
-            hipblaslt-dev \
-            rocblas-dev && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/* && \
-        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
-        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
-        ldconfig \
-    ; fi
-
-RUN echo "TARGETARCH: $TARGETARCH"
-
-# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
-# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
-# here so that we can generate the grpc code for the stablediffusion build
-RUN <<EOT bash
-    if [ "amd64" = "$TARGETARCH" ]; then
-        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
-        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-        rm protoc.zip
-    fi
-    if [ "arm64" = "$TARGETARCH" ]; then
-        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
-        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-        rm protoc.zip
-    fi
-EOT
-
-# Install CMake (the version in 22.04 is too old)
-RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
-        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
-    else
-        apt-get update && \
-        apt-get install -y \
-            cmake && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-COPY --from=grpc /opt/grpc /usr/local
+# Install everything via the shared script — the same one that
+# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
+# this from-source path are bit-equivalent.
+RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
+    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    bash /usr/local/sbin/install-base-deps
 
+# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
+# CMake's find_package finds it at the canonical prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
 
 COPY . /LocalAI
 
-RUN <<'EOT' bash
-set -euxo pipefail
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
-fi
-
-cd /LocalAI/backend/cpp/ik-llama-cpp
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  # ARM64 / ROCm: build without x86 SIMD
-  make ik-llama-cpp-fallback
-else
-  # ik_llama.cpp's IQK kernels require at least AVX2
-  make ik-llama-cpp-avx2
-fi
-EOT
+# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
+# for the rationale. Distinct mount id so ik-llama-cpp's cache doesn't
+# overlap with llama-cpp's — ik_llama.cpp is a different fork with
+# different source.
+#
+# The compile body is shared with builder-prebuilt via .docker/ik-llama-cpp-compile.sh.
+RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
 
 
 # Copy libraries using a script to handle architecture differences
 RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
 
 
+# ============================================================================
+# Stage: builder-prebuilt — uses the pre-built base from
+# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
+# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
+# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
+# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
+# builder-base-image).
+# ============================================================================
+FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
+
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+ARG TARGETARCH
+ARG TARGETVARIANT
+
+# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
+# /usr/local. Mirror what the from-source path does so the compile step
+# can find gRPC at the canonical prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
+
+COPY . /LocalAI
+
+RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
+
+RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
+
+
+# ============================================================================
+# Final stage — copies package output from one of the two builders.
+# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
+#
+# BuildKit doesn't support variable expansion in `COPY --from=` directly,
+# so we resolve the ARG by aliasing the chosen builder to a fixed stage
+# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
+# BUILDER_TARGET itself is declared as a global ARG at the top of this
+# file (required for use in FROM), so we just re-import it into this
+# stage's scope before the FROM directive.
+# ============================================================================
+FROM ${BUILDER_TARGET} AS builder
+
 FROM scratch
 
 

backend/Dockerfile.llama-cpp

@@ -1,301 +1,155 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
+# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
+# when no prebuilt base is supplied. The builder-prebuilt stage is only
+# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
+# content here is harmless — BuildKit prunes the unreferenced builder.
+ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
+# BUILDER_TARGET selects which builder stage the final scratch image copies
+# package output from. Declared at global scope (before any FROM) so it's
+# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
+# `make backends/llama-cpp` on the from-source path.
+ARG BUILDER_TARGET=builder-fromsource
 ARG APT_MIRROR=""
 ARG APT_PORTS_MIRROR=""
 
 
-# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
-# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
-FROM ${GRPC_BASE_IMAGE} AS grpc
-
-# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG GRPC_VERSION=v1.65.0
+# ============================================================================
+# Stage: builder-fromsource — self-contained build path.
+# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
+# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
+# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
+# default; local `make backends/llama-cpp`).
+#
+# The install script is the same one that backend/Dockerfile.base-grpc-builder
+# runs, so the result is bit-equivalent to the prebuilt-base path
+# (builder-prebuilt below).
+# ============================================================================
+FROM ${BASE_IMAGE} AS builder-fromsource
+ARG BUILD_TYPE
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-
-ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
-
-WORKDIR /build
-
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends \
-        ca-certificates \
-        build-essential curl libssl-dev \
-        git wget && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
-
-# Install CMake (the version in 22.04 is too old)
-RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
-        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
-    else
-        apt-get update && \
-        apt-get install -y \
-            cmake && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
-# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
-# and running make install in the target container
-RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
-    mkdir -p /build/grpc/cmake/build && \
-    cd /build/grpc/cmake/build && \
-    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
-    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
-    make && \
-    make install && \
-    rm -rf /build
-
-FROM ${BASE_IMAGE} AS builder
-ARG CMAKE_FROM_SOURCE=false
-ARG CMAKE_VERSION=3.31.10
-# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-ARG AMDGPU_TARGETS
-ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
-ARG BACKEND=rerankers
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
+ARG GRPC_VERSION=v1.65.0
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
 ARG SKIP_DRIVERS=false
-ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
-ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
-ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
 ARG APT_MIRROR
 ARG APT_PORTS_MIRROR
+ARG AMDGPU_TARGETS
+# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ARG CMAKE_ARGS
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends \
-        build-essential \
-        ccache git \
-        ca-certificates \
-        make \
-        pkg-config libcurl4-openssl-dev \
-        curl unzip \
-        libssl-dev wget && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
+ENV BUILD_TYPE=${BUILD_TYPE} \
+    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
+    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
+    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
+    CMAKE_VERSION=${CMAKE_VERSION} \
+    GRPC_VERSION=${GRPC_VERSION} \
+    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
+    SKIP_DRIVERS=${SKIP_DRIVERS} \
+    TARGETARCH=${TARGETARCH} \
+    UBUNTU_VERSION=${UBUNTU_VERSION} \
+    APT_MIRROR=${APT_MIRROR} \
+    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
+    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
+    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
+    CMAKE_ARGS=${CMAKE_ARGS} \
+    DEBIAN_FRONTEND=noninteractive
 
-# Cuda
+# CUDA on PATH (no-op when CUDA isn't installed)
 ENV PATH=/usr/local/cuda/bin:${PATH}
-
-# HipBLAS requirements
+# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
 ENV PATH=/opt/rocm/bin:${PATH}
 
+WORKDIR /build
 
-# Vulkan requirements
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
-        apt-get update && \
-        apt-get install -y  --no-install-recommends \
-            software-properties-common pciutils wget gpg-agent && \
-        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
-            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
-            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
-            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
-            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
-            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
-            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            mkdir -p /opt/vulkan-sdk && \
-            mv 1.4.335.0 /opt/vulkan-sdk/ && \
-            cd /opt/vulkan-sdk/1.4.335.0 && \
-            ./vulkansdk --no-deps --maxjobs \
-                vulkan-loader \
-                vulkan-validationlayers \
-                vulkan-extensionlayer \
-                vulkan-tools \
-                shaderc && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
-            rm -rf /opt/vulkan-sdk
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            mkdir vulkan && cd vulkan && \
-            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-            tar -xvf vulkan-sdk.tar.xz && \
-            rm vulkan-sdk.tar.xz && \
-            cd 1.4.335.0 && \
-            cp -rfv aarch64/bin/* /usr/bin/ && \
-            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-            cp -rfv aarch64/include/* /usr/include/ && \
-            cp -rfv aarch64/share/* /usr/share/ && \
-            cd ../.. && \
-            rm -rf vulkan
-        fi
-        ldconfig && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-# CuBLAS requirements
-RUN <<EOT bash
-    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
-        apt-get update && \
-        apt-get install -y  --no-install-recommends \
-            software-properties-common pciutils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
-            else
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
-            fi
-        fi
-        dpkg -i cuda-keyring_1.1-1_all.deb && \
-        rm -f cuda-keyring_1.1-1_all.deb && \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
-            apt-get install -y --no-install-recommends \
-            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        fi
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
-        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get install -y nvpl
-    fi
-EOT
-
-# If we are building with clblas support, we need the libraries for the builds
-RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            libclblast-dev && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/* \
-    ; fi
-
-RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            hipblas-dev \
-            hipblaslt-dev \
-            rocblas-dev && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/* && \
-        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
-        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
-        ldconfig && \
-        # Log which GPU architectures have rocBLAS kernel support
-        echo "rocBLAS library data architectures:" && \
-        (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
-        echo "WARNING: No rocBLAS kernel data found" \
-    ; fi
-
-RUN echo "TARGETARCH: $TARGETARCH"
-
-# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
-# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
-# here so that we can generate the grpc code for the stablediffusion build
-RUN <<EOT bash
-    if [ "amd64" = "$TARGETARCH" ]; then
-        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
-        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-        rm protoc.zip
-    fi
-    if [ "arm64" = "$TARGETARCH" ]; then
-        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
-        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-        rm protoc.zip
-    fi
-EOT
-
-# Install CMake (the version in 22.04 is too old)
-RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
-        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
-    else
-        apt-get update && \
-        apt-get install -y \
-            cmake && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-COPY --from=grpc /opt/grpc /usr/local
+# Install everything via the shared script — the same one that
+# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
+# this from-source path are bit-equivalent.
+RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
+    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    bash /usr/local/sbin/install-base-deps
 
+# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
+# CMake's find_package finds it at the canonical prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
 
 COPY . /LocalAI
 
-RUN <<'EOT' bash
-set -euxo pipefail
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
-fi
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  cd /LocalAI/backend/cpp/llama-cpp
-  make llama-cpp-fallback
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-else
-  cd /LocalAI/backend/cpp/llama-cpp
-  make llama-cpp-avx
-  make llama-cpp-avx2
-  make llama-cpp-avx512
-  make llama-cpp-fallback
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-fi
-EOT
+# BuildKit cache mount for ccache. Persists compiler outputs across builds
+# via the registry cache (cache-to: type=registry,mode=max in CI). On a
+# LLAMA_VERSION bump most TUs are byte-identical to the previous version's
+# preprocessed source — ccache returns the previous .o file and skips the
+# real compile. Same for LocalAI source changes that don't touch llama.cpp.
+# CMAKE_*_COMPILER_LAUNCHER threads ccache through CMake to wrap gcc/g++/nvcc.
+# sharing=locked serializes concurrent writes if multiple matrix variants
+# share the same cache mount id.
+#
+# The compile body is shared with builder-prebuilt via .docker/llama-cpp-compile.sh.
+RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
 
 
 # Copy libraries using a script to handle architecture differences
 RUN make -BC /LocalAI/backend/cpp/llama-cpp package
 
 
+# ============================================================================
+# Stage: builder-prebuilt — uses the pre-built base from
+# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
+# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
+# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
+# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
+# builder-base-image).
+# ============================================================================
+FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
+
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+ARG AMDGPU_TARGETS
+ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
+ARG TARGETARCH
+ARG TARGETVARIANT
+
+# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
+# /usr/local. The variant Dockerfile's from-source path does that too;
+# mirror it here so the compile step can find gRPC at the canonical
+# prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
+
+COPY . /LocalAI
+
+RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
+
+RUN make -BC /LocalAI/backend/cpp/llama-cpp package
+
+
+# ============================================================================
+# Final stage — copies package output from one of the two builders.
+# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
+#
+# BuildKit doesn't support variable expansion in `COPY --from=` directly,
+# so we resolve the ARG by aliasing the chosen builder to a fixed stage
+# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
+# BUILDER_TARGET itself is declared as a global ARG at the top of this
+# file (required for use in FROM), so we just re-import it into this
+# stage's scope before the FROM directive.
+# ============================================================================
+FROM ${BUILDER_TARGET} AS builder
+
 FROM scratch
 
 

backend/Dockerfile.turboquant

@@ -1,299 +1,152 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
+# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
+# when no prebuilt base is supplied. The builder-prebuilt stage is only
+# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
+# content here is harmless — BuildKit prunes the unreferenced builder.
+ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
+# BUILDER_TARGET selects which builder stage the final scratch image copies
+# package output from. Declared at global scope (before any FROM) so it's
+# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
+# `make backends/turboquant` on the from-source path.
+ARG BUILDER_TARGET=builder-fromsource
 ARG APT_MIRROR=""
 ARG APT_PORTS_MIRROR=""
 
 
-# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
-# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
-FROM ${GRPC_BASE_IMAGE} AS grpc
-
-# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG GRPC_VERSION=v1.65.0
+# ============================================================================
+# Stage: builder-fromsource — self-contained build path.
+# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
+# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
+# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
+# default; local `make backends/turboquant`).
+#
+# The install script is the same one that backend/Dockerfile.base-grpc-builder
+# runs, so the result is bit-equivalent to the prebuilt-base path
+# (builder-prebuilt below).
+# ============================================================================
+FROM ${BASE_IMAGE} AS builder-fromsource
+ARG BUILD_TYPE
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-
-ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
-
-WORKDIR /build
-
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends \
-        ca-certificates \
-        build-essential curl libssl-dev \
-        git wget && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
-
-# Install CMake (the version in 22.04 is too old)
-RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
-        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
-    else
-        apt-get update && \
-        apt-get install -y \
-            cmake && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
-# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
-# and running make install in the target container
-RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
-    mkdir -p /build/grpc/cmake/build && \
-    cd /build/grpc/cmake/build && \
-    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
-    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
-    make && \
-    make install && \
-    rm -rf /build
-
-FROM ${BASE_IMAGE} AS builder
-ARG CMAKE_FROM_SOURCE=false
-ARG CMAKE_VERSION=3.31.10
-# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-ARG BACKEND=rerankers
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
+ARG GRPC_VERSION=v1.65.0
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
 ARG SKIP_DRIVERS=false
-ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
-ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
-ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
 ARG APT_MIRROR
 ARG APT_PORTS_MIRROR
+ARG AMDGPU_TARGETS=""
+ARG BACKEND=rerankers
+# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ARG CMAKE_ARGS
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends \
-        build-essential \
-        ccache git \
-        ca-certificates \
-        make \
-        pkg-config libcurl4-openssl-dev \
-        curl unzip \
-        libssl-dev wget && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
+ENV BUILD_TYPE=${BUILD_TYPE} \
+    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
+    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
+    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
+    CMAKE_VERSION=${CMAKE_VERSION} \
+    GRPC_VERSION=${GRPC_VERSION} \
+    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
+    SKIP_DRIVERS=${SKIP_DRIVERS} \
+    TARGETARCH=${TARGETARCH} \
+    UBUNTU_VERSION=${UBUNTU_VERSION} \
+    APT_MIRROR=${APT_MIRROR} \
+    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
+    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
+    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
+    CMAKE_ARGS=${CMAKE_ARGS} \
+    DEBIAN_FRONTEND=noninteractive
 
-# Cuda
+# CUDA on PATH (no-op when CUDA isn't installed)
 ENV PATH=/usr/local/cuda/bin:${PATH}
-
-# HipBLAS requirements
+# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
 ENV PATH=/opt/rocm/bin:${PATH}
 
+WORKDIR /build
 
-# Vulkan requirements
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
-        apt-get update && \
-        apt-get install -y  --no-install-recommends \
-            software-properties-common pciutils wget gpg-agent && \
-        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
-            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
-            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
-            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
-            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
-            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
-            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            mkdir -p /opt/vulkan-sdk && \
-            mv 1.4.335.0 /opt/vulkan-sdk/ && \
-            cd /opt/vulkan-sdk/1.4.335.0 && \
-            ./vulkansdk --no-deps --maxjobs \
-                vulkan-loader \
-                vulkan-validationlayers \
-                vulkan-extensionlayer \
-                vulkan-tools \
-                shaderc && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
-            rm -rf /opt/vulkan-sdk
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            mkdir vulkan && cd vulkan && \
-            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-            tar -xvf vulkan-sdk.tar.xz && \
-            rm vulkan-sdk.tar.xz && \
-            cd 1.4.335.0 && \
-            cp -rfv aarch64/bin/* /usr/bin/ && \
-            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-            cp -rfv aarch64/include/* /usr/include/ && \
-            cp -rfv aarch64/share/* /usr/share/ && \
-            cd ../.. && \
-            rm -rf vulkan
-        fi
-        ldconfig && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-# CuBLAS requirements
-RUN <<EOT bash
-    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
-        apt-get update && \
-        apt-get install -y  --no-install-recommends \
-            software-properties-common pciutils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
-            else
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
-            fi
-        fi
-        dpkg -i cuda-keyring_1.1-1_all.deb && \
-        rm -f cuda-keyring_1.1-1_all.deb && \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
-            apt-get install -y --no-install-recommends \
-            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        fi
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
-        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get install -y nvpl
-    fi
-EOT
-
-# If we are building with clblas support, we need the libraries for the builds
-RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            libclblast-dev && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/* \
-    ; fi
-
-RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
-        apt-get update && \
-        apt-get install -y --no-install-recommends \
-            hipblas-dev \
-            hipblaslt-dev \
-            rocblas-dev && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/* && \
-        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
-        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
-        ldconfig && \
-        # Log which GPU architectures have rocBLAS kernel support
-        echo "rocBLAS library data architectures:" && \
-        (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
-        echo "WARNING: No rocBLAS kernel data found" \
-    ; fi
-
-RUN echo "TARGETARCH: $TARGETARCH"
-
-# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
-# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
-# here so that we can generate the grpc code for the stablediffusion build
-RUN <<EOT bash
-    if [ "amd64" = "$TARGETARCH" ]; then
-        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
-        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-        rm protoc.zip
-    fi
-    if [ "arm64" = "$TARGETARCH" ]; then
-        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
-        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-        rm protoc.zip
-    fi
-EOT
-
-# Install CMake (the version in 22.04 is too old)
-RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
-        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
-    else
-        apt-get update && \
-        apt-get install -y \
-            cmake && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists/*
-    fi
-EOT
-
-COPY --from=grpc /opt/grpc /usr/local
+# Install everything via the shared script — the same one that
+# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
+# this from-source path are bit-equivalent.
+RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
+    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    bash /usr/local/sbin/install-base-deps
 
+# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
+# CMake's find_package finds it at the canonical prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
 
 COPY . /LocalAI
 
-RUN <<'EOT' bash
-set -euxo pipefail
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/turboquant-*-build
-fi
-
-cd /LocalAI/backend/cpp/turboquant
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  make turboquant-fallback
-  make turboquant-grpc
-  make turboquant-rpc-server
-else
-  make turboquant-avx
-  make turboquant-avx2
-  make turboquant-avx512
-  make turboquant-fallback
-  make turboquant-grpc
-  make turboquant-rpc-server
-fi
-EOT
+# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
+# for rationale. turboquant is a llama.cpp fork that reuses
+# backend/cpp/llama-cpp source via a thin wrapper Makefile, so MOST TUs
+# are content-identical to the upstream llama-cpp build. Sharing a cache
+# id with llama-cpp could give cross-fork hits — but for now keep them
+# separate so a regression in one doesn't poison the other. Revisit
+# sharing after measuring the actual hit rate.
+#
+# The compile body is shared with builder-prebuilt via .docker/turboquant-compile.sh.
+RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
 
 
 # Copy libraries using a script to handle architecture differences
 RUN make -BC /LocalAI/backend/cpp/turboquant package
 
 
+# ============================================================================
+# Stage: builder-prebuilt — uses the pre-built base from
+# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
+# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
+# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
+# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
+# builder-base-image).
+# ============================================================================
+FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
+
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+ARG TARGETARCH
+ARG TARGETVARIANT
+
+# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
+# /usr/local. Mirror what the from-source path does so the compile step
+# can find gRPC at the canonical prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
+
+COPY . /LocalAI
+
+RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
+
+RUN make -BC /LocalAI/backend/cpp/turboquant package
+
+
+# ============================================================================
+# Final stage — copies package output from one of the two builders.
+# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
+#
+# BuildKit doesn't support variable expansion in `COPY --from=` directly,
+# so we resolve the ARG by aliasing the chosen builder to a fixed stage
+# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
+# BUILDER_TARGET itself is declared as a global ARG at the top of this
+# file (required for use in FROM), so we just re-import it into this
+# stage's scope before the FROM directive.
+# ============================================================================
+FROM ${BUILDER_TARGET} AS builder
+
 FROM scratch
 
 

backend/backend.proto

@@ -41,9 +41,14 @@ service Backend {
 
   rpc VAD(VADRequest) returns (VADResponse) {}
 
+  rpc Diarize(DiarizeRequest) returns (DiarizeResponse) {}
+
   rpc AudioEncode(AudioEncodeRequest) returns (AudioEncodeResult) {}
   rpc AudioDecode(AudioDecodeRequest) returns (AudioDecodeResult) {}
 
+  rpc AudioTransform(AudioTransformRequest) returns (AudioTransformResult) {}
+  rpc AudioTransformStream(stream AudioTransformFrameRequest) returns (stream AudioTransformFrameResponse) {}
+
   rpc ModelMetadata(ModelOptions) returns (ModelMetadataResponse) {}
 
   // Fine-tuning RPCs
@@ -350,6 +355,12 @@ message TranscriptStreamResponse {
   TranscriptResult final_result = 2;
 }
 
+message TranscriptWord {
+  int64 start = 1;
+  int64 end = 2;
+  string text = 3;
+}
+
 message TranscriptSegment {
   int32 id = 1;
   int64 start = 2;
@@ -357,6 +368,7 @@ message TranscriptSegment {
   string text = 4;
   repeated int32 tokens = 5;
   string speaker = 6;
+  repeated TranscriptWord words = 7;
 }
 
 message GenerateImageRequest {
@@ -413,6 +425,43 @@ message VADResponse {
   repeated VADSegment segments = 1;
 }
 
+// --- Speaker diarization messages ---
+//
+// Pure speaker diarization: "who spoke when". Returns time-stamped segments
+// labelled with cluster IDs (the same string for the same speaker across
+// segments). Some backends (e.g. vibevoice.cpp) produce diarization as a
+// by-product of ASR and may also fill in `text` per segment; backends with a
+// dedicated diarization pipeline (e.g. sherpa-onnx pyannote) leave `text`
+// empty and emit only the segmentation.
+
+message DiarizeRequest {
+  string dst = 1;                      // path to audio file (HTTP layer materialises uploads to a temp file)
+  uint32 threads = 2;
+  string language = 3;                 // optional; only meaningful for transcription-bundling backends
+  int32  num_speakers = 4;             // exact speaker count if known (>0 forces); 0 = auto
+  int32  min_speakers = 5;             // hint when auto-detecting; 0 = unset
+  int32  max_speakers = 6;             // hint when auto-detecting; 0 = unset
+  float  clustering_threshold = 7;     // distance threshold when num_speakers unknown; 0 = backend default
+  float  min_duration_on = 8;          // discard segments shorter than this (seconds); 0 = backend default
+  float  min_duration_off = 9;         // merge gaps shorter than this (seconds); 0 = backend default
+  bool   include_text = 10;            // when the backend can emit per-segment transcript for free, ask it to populate `text`
+}
+
+message DiarizeSegment {
+  int32  id = 1;
+  float  start = 2;                    // seconds
+  float  end = 3;                      // seconds
+  string speaker = 4;                  // backend-emitted speaker label (e.g. "0", "SPEAKER_00")
+  string text = 5;                     // optional per-segment transcript (empty unless include_text and supported)
+}
+
+message DiarizeResponse {
+  repeated DiarizeSegment segments = 1;
+  int32  num_speakers = 2;             // count of distinct speaker labels in `segments`
+  float  duration = 3;                 // total audio duration in seconds (0 if unknown)
+  string language = 4;                 // optional, when the backend bundles transcription
+}
+
 message SoundGenerationRequest {
   string text = 1;
   string model = 2;
@@ -669,6 +718,56 @@ message AudioDecodeResult {
   int32 samples_per_frame = 3;
 }
 
+// Generic audio transform: an audio-in, audio-out operation, optionally
+// conditioned on a second reference signal. Concrete transforms include
+// AEC + noise suppression + dereverberation (LocalVQE), voice conversion
+// (reference = target speaker), pitch shifting, etc.
+message AudioTransformRequest {
+  string audio_path = 1;             // required, primary input file path
+  string reference_path = 2;         // optional auxiliary; empty => zero-fill
+  string dst = 3;                    // required, output file path
+  map<string, string> params = 4;    // backend-specific tuning
+}
+
+message AudioTransformResult {
+  string dst = 1;
+  int32  sample_rate = 2;
+  int32  samples = 3;
+  bool   reference_provided = 4;
+}
+
+// Bidirectional streaming audio transform. The first message MUST carry a
+// Config; subsequent messages carry Frames. A second Config mid-stream
+// resets streaming state before the next frame.
+message AudioTransformFrameRequest {
+  oneof payload {
+    AudioTransformStreamConfig config = 1;
+    AudioTransformFrame        frame  = 2;
+  }
+}
+
+message AudioTransformStreamConfig {
+  enum SampleFormat {
+    F32_LE = 0;
+    S16_LE = 1;
+  }
+  SampleFormat sample_format = 1;
+  int32 sample_rate = 2;             // 0 => backend default
+  int32 frame_samples = 3;           // 0 => backend default
+  map<string, string> params = 4;
+  bool reset = 5;                    // reset streaming state before next frame
+}
+
+message AudioTransformFrame {
+  bytes audio_pcm = 1;               // frame_samples samples in stream's format
+  bytes reference_pcm = 2;           // empty => zero-fill (silent reference)
+}
+
+message AudioTransformFrameResponse {
+  bytes pcm = 1;
+  int64 frame_index = 2;
+}
+
 message ModelMetadataResponse {
   bool supports_thinking = 1;
   string rendered_template = 2;  // The rendered chat template with enable_thinking=true (empty if not applicable)

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=a8aecbf15933295af96504f9a693998322185b5c
+IK_LLAMA_VERSION?=23127139cb6fa314899c3b5f4935b88b3374c56c
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=beb42fffa45eded44804a1fd4916146222371581
+LLAMA_VERSION?=389ff61d77b5c71cec0cf92fe4e5d01ace80b797
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/turboquant/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-TURBOQUANT_VERSION?=11a241d0db78a68e0a5b99fe6f36de6683100f6a
+TURBOQUANT_VERSION?=69d8e4be47243e83b3d0d71e932bc7aa61c644dc
 LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
 
 CMAKE_ARGS?=

backend/go/localvqe/.gitignore

@@ -0,0 +1,7 @@
+sources/
+build/
+package/
+liblocalvqe.so*
+libggml*.so*
+localvqe
+.localvqe-build.stamp

backend/go/localvqe/Makefile

@@ -0,0 +1,98 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+JOBS?=$(shell nproc --ignore=1)
+
+# LocalVQE upstream version pin. Bump to a specific commit when picking up
+# a new release; `main` works for development but is not reproducible.
+LOCALVQE_REPO?=https://github.com/localai-org/LocalVQE
+LOCALVQE_VERSION?=72bfb4c6
+
+# LocalVQE handles CPU feature selection internally (it ships the multiple
+# libggml-cpu-*.so variants and its loader picks the best one at runtime
+# via GGML_BACKEND_DL), so we build a single liblocalvqe.so + the per-CPU
+# ggml shared libs and let it sort itself out. No need for a wrapper
+# MODULE library or per-AVX backend variants here.
+
+CMAKE_ARGS+=-DLOCALVQE_BUILD_SHARED=ON
+CMAKE_ARGS+=-DGGML_BUILD_TESTS=OFF
+CMAKE_ARGS+=-DGGML_BUILD_EXAMPLES=OFF
+
+ifeq ($(NATIVE),false)
+	CMAKE_ARGS+=-DGGML_NATIVE=OFF
+endif
+
+# LocalVQE upstream supports CPU + Vulkan only. Other BUILD_TYPE values
+# fall through to the default CPU build — Vulkan is already as fast as the
+# specialised GPU paths would be on this 1.3 M-parameter model.
+ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DGGML_VULKAN=ON -DLOCALVQE_VULKAN=ON
+else ifeq ($(OS),Darwin)
+	CMAKE_ARGS+=-DGGML_METAL=OFF
+endif
+
+# --- Sources ---
+
+sources/LocalVQE:
+	mkdir -p sources/LocalVQE
+	cd sources/LocalVQE && \
+	git init && \
+	git remote add origin $(LOCALVQE_REPO) && \
+	git fetch origin && \
+	git checkout $(LOCALVQE_VERSION) && \
+	git submodule update --init --recursive --depth 1 --single-branch
+
+# --- Native build ---
+#
+# Drives cmake directly against the upstream LocalVQE/ggml CMakeLists.
+# Produces liblocalvqe.so plus the per-CPU libggml-cpu-*.so variants in
+# build/bin/, all of which we copy into the backend directory so package.sh
+# can pick them up. The `liblocalvqe.so` rule deliberately uses a sentinel
+# stamp file because Make's wildcard tracking would otherwise mis-decide
+# about freshness when SOVERSION symlinks are involved.
+
+LIB_SENTINEL=.localvqe-build.stamp
+
+$(LIB_SENTINEL): sources/LocalVQE
+	mkdir -p build && \
+	cd build && \
+	cmake ../sources/LocalVQE/ggml $(CMAKE_ARGS) -DCMAKE_BUILD_TYPE=Release && \
+	cmake --build . --config Release -j$(JOBS)
+	# Upstream's CPU build sets GGML_BACKEND_DL=ON + GGML_CPU_ALL_VARIANTS=ON,
+	# which produces multiple libggml-cpu-*.so files (SSE4.2 / AVX2 / AVX-512)
+	# that the loader picks at runtime. We must build every target — the
+	# default `--target localvqe_shared` drops these. CMAKE_LIBRARY_OUTPUT_DIRECTORY
+	# routes all of them into build/bin; copy them out next to the binary.
+	cp -P build/bin/liblocalvqe.so* . 2>/dev/null || cp -P build/liblocalvqe.so* .
+	cp -P build/bin/libggml*.so* . 2>/dev/null || true
+	touch $(LIB_SENTINEL)
+
+liblocalvqe.so: $(LIB_SENTINEL)
+
+# --- Go binary + packaging ---
+
+localvqe: main.go golocalvqe.go $(LIB_SENTINEL)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o localvqe ./
+
+package: localvqe
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf liblocalvqe.so* libggml*.so* package sources/LocalVQE localvqe $(LIB_SENTINEL)
+
+purge:
+	rm -rf build
+
+test: localvqe
+	@echo "Running localvqe tests..."
+	bash test.sh
+	@echo "localvqe tests completed."
+
+all: localvqe package
+
+.PHONY: build package clean purge test all

backend/go/localvqe/golocalvqe.go

@@ -0,0 +1,610 @@
+package main
+
+import (
+	"encoding/binary"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strconv"
+	"strings"
+	"unsafe"
+
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// localvqeSampleRate is the only sample rate currently supported by the
+// upstream LocalVQE model. We assert against it after Load() and reject
+// anything else with a clear error rather than letting the C side return
+// garbage.
+const localvqeSampleRate = 16000
+
+// Param map keys understood by LocalVQE. Keep these strings in sync with
+// schema.AudioTransformParam* (separate package — this is a standalone
+// backend module).
+const (
+	paramNoiseGate          = "noise_gate"
+	paramNoiseGateThreshold = "noise_gate_threshold_dbfs"
+)
+
+// Option keys read from ModelOptions.Options[] at Load() time. The backend
+// + device pair is forwarded to the upstream options builder; everything
+// else is consumed locally (noise gate state, etc.).
+const (
+	optionBackend = "backend"
+	optionDevice  = "device"
+)
+
+// purego-bound entry points from liblocalvqe.
+//
+// uintptr opaque handles model the C `uintptr_t ctx` / `uintptr_t opts`
+// tokens; we never dereference them on the Go side, just hand them
+// straight back to the library on every call. Construction always goes
+// through the options builder (CppOptionsNew + setters + CppNewWithOptions)
+// — the bare localvqe_new path doesn't expose backend / device selection.
+var (
+	CppOptionsNew           func() uintptr
+	CppOptionsFree          func(opts uintptr)
+	CppOptionsSetModelPath  func(opts uintptr, modelPath string) int32
+	CppOptionsSetBackend    func(opts uintptr, backend string) int32
+	CppOptionsSetDevice     func(opts uintptr, device int32) int32
+	CppNewWithOptions       func(opts uintptr) uintptr
+	CppFree                 func(ctx uintptr)
+	CppProcessF32           func(ctx uintptr, mic, ref uintptr, nSamples int32, out uintptr) int32
+	CppProcessS16           func(ctx uintptr, mic, ref uintptr, nSamples int32, out uintptr) int32
+	CppProcessFrameF32      func(ctx uintptr, mic, ref uintptr, hopSamples int32, out uintptr) int32
+	CppProcessFrameS16      func(ctx uintptr, mic, ref uintptr, hopSamples int32, out uintptr) int32
+	CppReset                func(ctx uintptr)
+	CppLastError            func(ctx uintptr) string
+	CppSampleRate           func(ctx uintptr) int32
+	CppHopLength            func(ctx uintptr) int32
+	CppFFTSize              func(ctx uintptr) int32
+	CppSetNoiseGate         func(ctx uintptr, enabled int32, thresholdDBFS float32) int32
+	CppGetNoiseGate         func(ctx uintptr, enabledOut, thresholdDBFSOut uintptr) int32
+)
+
+// LocalVQE speaks gRPC against LocalVQE's flat C ABI. The streaming
+// state is per-context, so we serialize calls through SingleThread —
+// concurrent streams would corrupt the overlap-add buffers.
+type LocalVQE struct {
+	base.SingleThread
+	ctx        uintptr // 0 when unloaded
+	sampleRate int
+	hopLength  int
+	fftSize    int
+
+	// modelRoot resolves relative paths from Options[].
+	modelRoot string
+
+	// Cached gate config so we can re-apply on each AudioTransform call
+	// without paying for a CGo round-trip every time. Sourced from
+	// Options[] at Load() time and overridable per-request via the
+	// gRPC params map.
+	gateEnabled bool
+	gateDbfs    float32
+
+	// Backend / device picked via Options[]. Empty backend leaves the
+	// default (CPU) selection to the upstream options builder.
+	backend string
+	device  int32
+}
+
+// parseOptions reads opts.Options[] for backend-specific tuning. Documented
+// keys: noise_gate=true|false and noise_gate_threshold_dbfs=<float> (also
+// settable per-request via AudioTransformRequest.params), plus backend=<name>
+// and device=<index> which route through the upstream options builder so
+// the user can force a non-default GGML backend (e.g. "Vulkan").
+func (v *LocalVQE) parseOptions(opts []string) {
+	for _, raw := range opts {
+		k, val, ok := strings.Cut(raw, "=")
+		if !ok {
+			k, val, ok = strings.Cut(raw, ":")
+			if !ok {
+				continue
+			}
+		}
+		key := strings.TrimSpace(strings.ToLower(k))
+		val = strings.TrimSpace(val)
+		switch key {
+		case paramNoiseGate:
+			if b, err := strconv.ParseBool(val); err == nil {
+				v.gateEnabled = b
+			}
+		case paramNoiseGateThreshold:
+			if f, err := strconv.ParseFloat(val, 32); err == nil {
+				v.gateDbfs = float32(f)
+			}
+		case optionBackend:
+			v.backend = val
+		case optionDevice:
+			if d, err := strconv.Atoi(val); err == nil && d >= 0 {
+				v.device = int32(d)
+			}
+		}
+	}
+}
+
+// newCtxWithOptions builds a context via the upstream options-builder so we
+// can pass backend / device in addition to the model path. Returns 0 on
+// failure; the caller logs/wraps the error since the C side has no
+// last-error channel for construction failures.
+func newCtxWithOptions(modelPath, backend string, device int32) uintptr {
+	o := CppOptionsNew()
+	if o == 0 {
+		return 0
+	}
+	defer CppOptionsFree(o)
+	if rc := CppOptionsSetModelPath(o, modelPath); rc != 0 {
+		return 0
+	}
+	if backend != "" {
+		if rc := CppOptionsSetBackend(o, backend); rc != 0 {
+			return 0
+		}
+	}
+	if device > 0 {
+		if rc := CppOptionsSetDevice(o, device); rc != 0 {
+			return 0
+		}
+	}
+	return CppNewWithOptions(o)
+}
+
+func (v *LocalVQE) Load(opts *pb.ModelOptions) error {
+	if opts.ModelFile == "" {
+		return fmt.Errorf("localvqe: ModelFile is required")
+	}
+
+	modelFile := opts.ModelFile
+	if !filepath.IsAbs(modelFile) && opts.ModelPath != "" {
+		modelFile = filepath.Join(opts.ModelPath, modelFile)
+	}
+	v.modelRoot = opts.ModelPath
+	if v.modelRoot == "" {
+		v.modelRoot = filepath.Dir(modelFile)
+	}
+
+	// Defaults — gate off, threshold at -45 dBFS as a reasonable starting
+	// point per the upstream localvqe_api.h documentation.
+	v.gateEnabled = false
+	v.gateDbfs = -45.0
+	v.parseOptions(opts.Options)
+
+	// localvqe_new reads GGML_NTHREADS at construction time; without it
+	// the C side falls back to single-threaded compute (~1× realtime
+	// instead of the documented ~9× on a multi-core CPU). Pass the
+	// model config's Threads through, defaulting to min(NumCPU, 4).
+	//
+	// LocalVQE is 1.3M parameters; per the upstream bench sweep 1–4
+	// threads is the sweet spot — beyond ~4 the per-frame budget gets
+	// dominated by sync overhead and p99 latency degrades. We cap at 4
+	// even when the user passes more so a globally-configured
+	// LOCALAI_THREADS=N tuned for a 70B LLM doesn't accidentally
+	// pessimise audio processing.
+	const localvqeMaxThreads = 4
+	threads := int(opts.Threads)
+	if threads <= 0 {
+		threads = runtime.NumCPU()
+	}
+	if threads > localvqeMaxThreads {
+		threads = localvqeMaxThreads
+	}
+	if threads < 1 {
+		threads = 1
+	}
+	if err := os.Setenv("GGML_NTHREADS", fmt.Sprintf("%d", threads)); err != nil {
+		return fmt.Errorf("localvqe: setenv GGML_NTHREADS: %w", err)
+	}
+
+	xlog.Info("[localvqe] loading model", "path", modelFile, "threads", threads, "backend", v.backend, "device", v.device, "noise_gate", v.gateEnabled, "threshold_dbfs", v.gateDbfs)
+
+	ctx := newCtxWithOptions(modelFile, v.backend, v.device)
+	if ctx == 0 {
+		return fmt.Errorf("localvqe: localvqe_new_with_options failed for %q (backend=%q device=%d)", modelFile, v.backend, v.device)
+	}
+	v.ctx = ctx
+
+	v.sampleRate = int(CppSampleRate(ctx))
+	v.hopLength = int(CppHopLength(ctx))
+	v.fftSize = int(CppFFTSize(ctx))
+
+	if v.sampleRate != localvqeSampleRate {
+		CppFree(ctx)
+		v.ctx = 0
+		return fmt.Errorf("localvqe: unsupported sample rate %d (only %d Hz is supported)", v.sampleRate, localvqeSampleRate)
+	}
+	if v.hopLength <= 0 || v.fftSize <= 0 {
+		CppFree(ctx)
+		v.ctx = 0
+		return fmt.Errorf("localvqe: model reports invalid hop=%d fft=%d", v.hopLength, v.fftSize)
+	}
+
+	if v.gateEnabled {
+		if rc := CppSetNoiseGate(ctx, 1, v.gateDbfs); rc != 0 {
+			err := fmt.Errorf("localvqe: localvqe_set_noise_gate failed (rc=%d): %s", rc, CppLastError(ctx))
+			CppFree(ctx)
+			v.ctx = 0
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (v *LocalVQE) Free() error {
+	if v.ctx != 0 {
+		CppFree(v.ctx)
+		v.ctx = 0
+	}
+	return nil
+}
+
+// applyParams forwards backend-specific tuning to the C side per call.
+func (v *LocalVQE) applyParams(params map[string]string) error {
+	if len(params) == 0 {
+		return nil
+	}
+	enabled := v.gateEnabled
+	threshold := v.gateDbfs
+	updated := false
+
+	if val, ok := params[paramNoiseGate]; ok {
+		if b, err := strconv.ParseBool(val); err == nil {
+			enabled = b
+			updated = true
+		}
+	}
+	if val, ok := params[paramNoiseGateThreshold]; ok {
+		if f, err := strconv.ParseFloat(val, 32); err == nil {
+			threshold = float32(f)
+			updated = true
+		}
+	}
+	if !updated {
+		return nil
+	}
+
+	gateOn := int32(0)
+	if enabled {
+		gateOn = 1
+	}
+	if rc := CppSetNoiseGate(v.ctx, gateOn, threshold); rc != 0 {
+		return fmt.Errorf("localvqe_set_noise_gate failed (rc=%d): %s", rc, CppLastError(v.ctx))
+	}
+	v.gateEnabled = enabled
+	v.gateDbfs = threshold
+	return nil
+}
+
+func (v *LocalVQE) AudioTransform(req *pb.AudioTransformRequest) (*pb.AudioTransformResult, error) {
+	if v.ctx == 0 {
+		return nil, fmt.Errorf("localvqe: no model loaded")
+	}
+	if req.AudioPath == "" || req.Dst == "" {
+		return nil, fmt.Errorf("localvqe: audio_path and dst are required")
+	}
+
+	if err := v.applyParams(req.Params); err != nil {
+		return nil, err
+	}
+
+	mic, micRate, err := readMonoWAVf32(req.AudioPath)
+	if err != nil {
+		return nil, fmt.Errorf("read audio: %w", err)
+	}
+	if micRate != v.sampleRate {
+		return nil, fmt.Errorf("localvqe: audio sample rate %d != model %d (resample upstream)", micRate, v.sampleRate)
+	}
+
+	refProvided := req.ReferencePath != ""
+	var ref []float32
+	if refProvided {
+		var refRate int
+		ref, refRate, err = readMonoWAVf32(req.ReferencePath)
+		if err != nil {
+			return nil, fmt.Errorf("read reference: %w", err)
+		}
+		if refRate != v.sampleRate {
+			return nil, fmt.Errorf("localvqe: reference sample rate %d != model %d", refRate, v.sampleRate)
+		}
+		// Length-mismatch policy: zero-pad a short reference (silence past
+		// the mic's tail), truncate a long one (the trailing reference
+		// can't have leaked into a mic that wasn't recording yet).
+		switch {
+		case len(ref) < len(mic):
+			padded := make([]float32, len(mic))
+			copy(padded, ref)
+			ref = padded
+		case len(ref) > len(mic):
+			ref = ref[:len(mic)]
+		}
+	} else {
+		ref = make([]float32, len(mic))
+	}
+
+	if len(mic) < v.fftSize {
+		return nil, fmt.Errorf("localvqe: audio too short (%d samples, need ≥ %d)", len(mic), v.fftSize)
+	}
+
+	out := make([]float32, len(mic))
+	rc := CppProcessF32(v.ctx,
+		uintptr(unsafe.Pointer(&mic[0])),
+		uintptr(unsafe.Pointer(&ref[0])),
+		int32(len(mic)),
+		uintptr(unsafe.Pointer(&out[0])))
+	if rc != 0 {
+		return nil, fmt.Errorf("localvqe_process_f32 failed (rc=%d): %s", rc, CppLastError(v.ctx))
+	}
+
+	if err := writeMonoWAVf32(req.Dst, out, v.sampleRate); err != nil {
+		return nil, fmt.Errorf("write output: %w", err)
+	}
+
+	return &pb.AudioTransformResult{
+		Dst:               req.Dst,
+		SampleRate:        int32(v.sampleRate),
+		Samples:           int32(len(out)),
+		ReferenceProvided: refProvided,
+	}, nil
+}
+
+// AudioTransformStream runs the bidirectional streaming path. The first
+// inbound message MUST be a Config; subsequent messages MUST be Frames.
+// A second Config mid-stream resets the streaming state.
+func (v *LocalVQE) AudioTransformStream(in <-chan *pb.AudioTransformFrameRequest, out chan<- *pb.AudioTransformFrameResponse) error {
+	defer close(out)
+
+	if v.ctx == 0 {
+		return fmt.Errorf("localvqe: no model loaded")
+	}
+
+	first, ok := <-in
+	if !ok {
+		return nil
+	}
+	cfg := first.GetConfig()
+	if cfg == nil {
+		return fmt.Errorf("localvqe: first stream message must be a Config")
+	}
+	if err := v.applyStreamConfig(cfg); err != nil {
+		return err
+	}
+
+	hop := v.hopLength
+	if cfg.FrameSamples != 0 && int(cfg.FrameSamples) != hop {
+		return fmt.Errorf("localvqe: frame_samples=%d != hop_length=%d", cfg.FrameSamples, hop)
+	}
+
+	// Pre-allocated scratch buffers for the C-side process call. The
+	// per-frame output []byte stays a fresh allocation: the response
+	// channel is buffered, so reusing one backing array would race with
+	// the gRPC send goroutine flushing prior queued frames.
+	micF32 := make([]float32, hop)
+	refF32 := make([]float32, hop)
+	outF32 := make([]float32, hop)
+	micS16 := make([]int16, hop)
+	refS16 := make([]int16, hop)
+	outS16 := make([]int16, hop)
+
+	useS16 := cfg.SampleFormat == pb.AudioTransformStreamConfig_S16_LE
+	frameSize := hop * 4
+	if useS16 {
+		frameSize = hop * 2
+	}
+
+	frameIndex := int64(0)
+	for req := range in {
+		switch payload := req.Payload.(type) {
+		case *pb.AudioTransformFrameRequest_Config:
+			if err := v.applyStreamConfig(payload.Config); err != nil {
+				return err
+			}
+			if payload.Config.Reset_ {
+				CppReset(v.ctx)
+				frameIndex = 0
+			}
+			continue
+		case *pb.AudioTransformFrameRequest_Frame:
+			if len(payload.Frame.AudioPcm) != frameSize {
+				return fmt.Errorf("localvqe: frame audio bytes=%d expected=%d", len(payload.Frame.AudioPcm), frameSize)
+			}
+			refBuf := payload.Frame.ReferencePcm
+			if len(refBuf) != 0 && len(refBuf) != frameSize {
+				return fmt.Errorf("localvqe: frame reference bytes=%d expected=%d (or 0)", len(refBuf), frameSize)
+			}
+
+			var outBytes []byte
+			if useS16 {
+				if err := decodeS16LE(payload.Frame.AudioPcm, micS16); err != nil {
+					return err
+				}
+				if len(refBuf) > 0 {
+					if err := decodeS16LE(refBuf, refS16); err != nil {
+						return err
+					}
+				} else {
+					zeroS16(refS16)
+				}
+				rc := CppProcessFrameS16(v.ctx,
+					uintptr(unsafe.Pointer(&micS16[0])),
+					uintptr(unsafe.Pointer(&refS16[0])),
+					int32(hop),
+					uintptr(unsafe.Pointer(&outS16[0])))
+				if rc != 0 {
+					return fmt.Errorf("localvqe_process_frame_s16 (rc=%d): %s", rc, CppLastError(v.ctx))
+				}
+				outBytes = make([]byte, hop*2)
+				encodeS16LE(outS16, outBytes)
+			} else {
+				if err := decodeF32LE(payload.Frame.AudioPcm, micF32); err != nil {
+					return err
+				}
+				if len(refBuf) > 0 {
+					if err := decodeF32LE(refBuf, refF32); err != nil {
+						return err
+					}
+				} else {
+					zeroF32(refF32)
+				}
+				rc := CppProcessFrameF32(v.ctx,
+					uintptr(unsafe.Pointer(&micF32[0])),
+					uintptr(unsafe.Pointer(&refF32[0])),
+					int32(hop),
+					uintptr(unsafe.Pointer(&outF32[0])))
+				if rc != 0 {
+					return fmt.Errorf("localvqe_process_frame_f32 (rc=%d): %s", rc, CppLastError(v.ctx))
+				}
+				outBytes = make([]byte, hop*4)
+				encodeF32LE(outF32, outBytes)
+			}
+			out <- &pb.AudioTransformFrameResponse{Pcm: outBytes, FrameIndex: frameIndex}
+			frameIndex++
+		default:
+			return fmt.Errorf("localvqe: unexpected stream payload %T", payload)
+		}
+	}
+	return nil
+}
+
+func zeroS16(s []int16) {
+	for i := range s {
+		s[i] = 0
+	}
+}
+
+func zeroF32(s []float32) {
+	for i := range s {
+		s[i] = 0
+	}
+}
+
+func (v *LocalVQE) applyStreamConfig(cfg *pb.AudioTransformStreamConfig) error {
+	if cfg.SampleRate != 0 && int(cfg.SampleRate) != v.sampleRate {
+		return fmt.Errorf("localvqe: sample_rate=%d != model %d", cfg.SampleRate, v.sampleRate)
+	}
+	return v.applyParams(cfg.Params)
+}
+
+// ---- WAV I/O ----------------------------------------------------------
+//
+// Minimal mono PCM WAV reader/writer. Only handles the subset LocalVQE
+// cares about (mono, 16-bit signed, no extensible chunks). For broader
+// audio support the HTTP layer's `audio.NormalizeAudioFile` already
+// converts arbitrary input to a canonical WAV before we see it; this
+// reader just decodes the canonical shape.
+
+func readMonoWAVf32(path string) ([]float32, int, error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return nil, 0, err
+	}
+	defer func() { _ = f.Close() }()
+	header := make([]byte, 44)
+	if _, err := io.ReadFull(f, header); err != nil {
+		return nil, 0, err
+	}
+	if string(header[0:4]) != "RIFF" || string(header[8:12]) != "WAVE" {
+		return nil, 0, fmt.Errorf("not a WAV file")
+	}
+	channels := binary.LittleEndian.Uint16(header[22:24])
+	sampleRate := binary.LittleEndian.Uint32(header[24:28])
+	bitsPerSample := binary.LittleEndian.Uint16(header[34:36])
+
+	if channels != 1 {
+		return nil, 0, fmt.Errorf("only mono WAV supported (got %d channels)", channels)
+	}
+	if bitsPerSample != 16 {
+		return nil, 0, fmt.Errorf("only 16-bit PCM supported (got %d bits)", bitsPerSample)
+	}
+
+	rest, err := io.ReadAll(f)
+	if err != nil {
+		return nil, 0, err
+	}
+	n := len(rest) / 2
+	out := make([]float32, n)
+	for i := 0; i < n; i++ {
+		s := int16(binary.LittleEndian.Uint16(rest[i*2 : i*2+2]))
+		out[i] = float32(s) / 32768.0
+	}
+	return out, int(sampleRate), nil
+}
+
+func writeMonoWAVf32(path string, samples []float32, sampleRate int) error {
+	f, err := os.Create(path)
+	if err != nil {
+		return err
+	}
+	defer func() { _ = f.Close() }()
+
+	dataLen := uint32(len(samples) * 2)
+	header := make([]byte, 44)
+	copy(header[0:4], []byte("RIFF"))
+	binary.LittleEndian.PutUint32(header[4:8], 36+dataLen)
+	copy(header[8:12], []byte("WAVE"))
+	copy(header[12:16], []byte("fmt "))
+	binary.LittleEndian.PutUint32(header[16:20], 16)        // fmt chunk size
+	binary.LittleEndian.PutUint16(header[20:22], 1)         // PCM
+	binary.LittleEndian.PutUint16(header[22:24], 1)         // mono
+	binary.LittleEndian.PutUint32(header[24:28], uint32(sampleRate))
+	binary.LittleEndian.PutUint32(header[28:32], uint32(sampleRate*2)) // byte rate
+	binary.LittleEndian.PutUint16(header[32:34], 2)         // block align
+	binary.LittleEndian.PutUint16(header[34:36], 16)        // bits per sample
+	copy(header[36:40], []byte("data"))
+	binary.LittleEndian.PutUint32(header[40:44], dataLen)
+	if _, err := f.Write(header); err != nil {
+		return err
+	}
+
+	body := make([]byte, len(samples)*2)
+	for i, s := range samples {
+		clamped := s * 32768.0
+		if clamped > 32767 {
+			clamped = 32767
+		} else if clamped < -32768 {
+			clamped = -32768
+		}
+		binary.LittleEndian.PutUint16(body[i*2:i*2+2], uint16(int16(clamped)))
+	}
+	_, err = f.Write(body)
+	return err
+}
+
+// ---- PCM endec helpers ------------------------------------------------
+
+func decodeS16LE(buf []byte, out []int16) error {
+	if len(buf) != len(out)*2 {
+		return fmt.Errorf("decodeS16LE: buf=%d out=%d", len(buf), len(out))
+	}
+	for i := range out {
+		out[i] = int16(binary.LittleEndian.Uint16(buf[i*2 : i*2+2]))
+	}
+	return nil
+}
+
+func encodeS16LE(in []int16, out []byte) {
+	for i, s := range in {
+		binary.LittleEndian.PutUint16(out[i*2:i*2+2], uint16(s))
+	}
+}
+
+func decodeF32LE(buf []byte, out []float32) error {
+	if len(buf) != len(out)*4 {
+		return fmt.Errorf("decodeF32LE: buf=%d out=%d", len(buf), len(out))
+	}
+	for i := range out {
+		bits := binary.LittleEndian.Uint32(buf[i*4 : i*4+4])
+		out[i] = *(*float32)(unsafe.Pointer(&bits))
+	}
+	return nil
+}
+
+func encodeF32LE(in []float32, out []byte) {
+	for i, s := range in {
+		bits := *(*uint32)(unsafe.Pointer(&s))
+		binary.LittleEndian.PutUint32(out[i*4:i*4+4], bits)
+	}
+}

backend/go/localvqe/localvqe_test.go

@@ -0,0 +1,120 @@
+package main
+
+import (
+	"os"
+	"testing"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestLocalVQE(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "LocalVQE-cpp Backend Suite")
+}
+
+// modelPathOrSkip returns the LocalVQE GGUF path or Skip()s the current
+// spec when LOCALVQE_MODEL_PATH is unset / unreadable.
+func modelPathOrSkip() string {
+	path := os.Getenv("LOCALVQE_MODEL_PATH")
+	if path == "" {
+		Skip("LOCALVQE_MODEL_PATH not set, skipping model-dependent specs")
+	}
+	if _, err := os.Stat(path); err != nil {
+		Skip("LOCALVQE_MODEL_PATH unreadable: " + err.Error())
+	}
+	return path
+}
+
+var _ = Describe("LocalVQE-cpp", func() {
+	Context("backend semantics (no purego load needed)", func() {
+		It("is locking - the engine has per-context streaming state", func() {
+			Expect((&LocalVQE{}).Locking()).To(BeTrue())
+		})
+
+		It("rejects Load with empty ModelFile", func() {
+			err := (&LocalVQE{}).Load(&pb.ModelOptions{})
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("ModelFile"))
+		})
+
+		It("rejects AudioTransform without a loaded model", func() {
+			_, err := (&LocalVQE{}).AudioTransform(&pb.AudioTransformRequest{
+				AudioPath: "/tmp/audio.wav",
+				Dst:       "/tmp/out.wav",
+			})
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("no model loaded"))
+		})
+
+		It("closes the output channel and errors on AudioTransformStream without a loaded model", func() {
+			in := make(chan *pb.AudioTransformFrameRequest, 1)
+			out := make(chan *pb.AudioTransformFrameResponse, 1)
+			close(in)
+			err := (&LocalVQE{}).AudioTransformStream(in, out)
+			Expect(err).To(HaveOccurred())
+			_, ok := <-out
+			Expect(ok).To(BeFalse(), "AudioTransformStream must close results channel even on error")
+		})
+
+		It("rejects AudioTransform with empty audio_path", func() {
+			v := &LocalVQE{ctx: 1, sampleRate: localvqeSampleRate, hopLength: 256, fftSize: 512}
+			_, err := v.AudioTransform(&pb.AudioTransformRequest{Dst: "/tmp/out.wav"})
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("audio_path"))
+		})
+	})
+
+	Context("parseOptions", func() {
+		It("reads noise_gate=true (=)", func() {
+			v := &LocalVQE{}
+			v.parseOptions([]string{"noise_gate=true"})
+			Expect(v.gateEnabled).To(BeTrue())
+		})
+
+		It("reads noise_gate_threshold_dbfs=-50 (:)", func() {
+			v := &LocalVQE{}
+			v.parseOptions([]string{"noise_gate_threshold_dbfs:-50"})
+			Expect(v.gateDbfs).To(BeNumerically("==", -50.0))
+		})
+
+		It("ignores unknown keys without error", func() {
+			v := &LocalVQE{}
+			v.parseOptions([]string{"unknown=value", "another:thing"})
+			Expect(v.gateEnabled).To(BeFalse())
+		})
+
+		It("is case-insensitive on keys", func() {
+			v := &LocalVQE{}
+			v.parseOptions([]string{"NOISE_GATE=true"})
+			Expect(v.gateEnabled).To(BeTrue())
+		})
+	})
+
+	Context("model-gated integration (LOCALVQE_MODEL_PATH)", func() {
+		It("load + sample rate + hop + fft", func() {
+			path := modelPathOrSkip()
+			v := &LocalVQE{}
+			Expect(v.Load(&pb.ModelOptions{ModelFile: path})).To(Succeed())
+			defer func() { _ = v.Free() }()
+			Expect(v.sampleRate).To(Equal(localvqeSampleRate))
+			Expect(v.hopLength).To(Equal(256))
+			Expect(v.fftSize).To(Equal(512))
+		})
+
+		It("sets reference_provided correctly", func() {
+			// This spec is best exercised against a real model + WAV
+			// fixture, which the e2e harness drives separately. Here
+			// we just assert the expectation when ref is empty.
+			path := modelPathOrSkip()
+			v := &LocalVQE{}
+			Expect(v.Load(&pb.ModelOptions{ModelFile: path})).To(Succeed())
+			defer func() { _ = v.Free() }()
+			// Synthetic input; the C side handles a constant-zero ref
+			// just fine. Skip writing the WAV: this spec is a smoke
+			// check — the SNR-improvement assertion lives in the e2e
+			// harness where we have a real fixture.
+		})
+	})
+})

backend/go/localvqe/main.go

@@ -0,0 +1,62 @@
+package main
+
+// Started internally by LocalAI - one gRPC server per loaded model.
+import (
+	"flag"
+	"os"
+
+	"github.com/ebitengine/purego"
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+type LibFuncs struct {
+	FuncPtr any
+	Name    string
+}
+
+func main() {
+	libName := os.Getenv("LOCALVQE_LIBRARY")
+	if libName == "" {
+		libName = "./liblocalvqe.so"
+	}
+
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		panic(err)
+	}
+
+	libFuncs := []LibFuncs{
+		{&CppOptionsNew, "localvqe_options_new"},
+		{&CppOptionsFree, "localvqe_options_free"},
+		{&CppOptionsSetModelPath, "localvqe_options_set_model_path"},
+		{&CppOptionsSetBackend, "localvqe_options_set_backend"},
+		{&CppOptionsSetDevice, "localvqe_options_set_device"},
+		{&CppNewWithOptions, "localvqe_new_with_options"},
+		{&CppFree, "localvqe_free"},
+		{&CppProcessF32, "localvqe_process_f32"},
+		{&CppProcessS16, "localvqe_process_s16"},
+		{&CppProcessFrameF32, "localvqe_process_frame_f32"},
+		{&CppProcessFrameS16, "localvqe_process_frame_s16"},
+		{&CppReset, "localvqe_reset"},
+		{&CppLastError, "localvqe_last_error"},
+		{&CppSampleRate, "localvqe_sample_rate"},
+		{&CppHopLength, "localvqe_hop_length"},
+		{&CppFFTSize, "localvqe_fft_size"},
+		{&CppSetNoiseGate, "localvqe_set_noise_gate"},
+		{&CppGetNoiseGate, "localvqe_get_noise_gate"},
+	}
+
+	for _, lf := range libFuncs {
+		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &LocalVQE{}); err != nil {
+		panic(err)
+	}
+}

backend/go/localvqe/package.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+
+# Bundle the localvqe binary, the upstream liblocalvqe.so + the per-CPU
+# libggml-*.so runtime variants, the run wrapper, and the runtime libs the
+# binary depends on so the package is self-contained.
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/localvqe $CURDIR/package/
+# liblocalvqe.so* (with SOVERSION symlinks) and the libggml-*.so runtime
+# variants — LocalVQE picks the matching CPU variant at load time.
+cp -P $CURDIR/liblocalvqe.so* $CURDIR/package/ 2>/dev/null || true
+cp -P $CURDIR/libggml*.so* $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/localvqe/run.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+set -ex
+
+CURDIR=$(dirname "$(realpath $0)")
+
+# LocalVQE's runtime CPU-variant loader (ggml_backend_load_all) searches
+# get_executable_path() and current_path() — the second one is what saves us
+# when /proc/self/exe resolves to lib/ld.so under the bundled-loader path.
+# So we cd into $CURDIR (where all the libggml-cpu-*.so files live) before
+# exec'ing the binary.
+cd "$CURDIR"
+
+export LD_LIBRARY_PATH=$CURDIR:$CURDIR/lib:$LD_LIBRARY_PATH
+export LOCALVQE_LIBRARY=$CURDIR/liblocalvqe.so
+
+if [ -f $CURDIR/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LOCALVQE_LIBRARY"
+	exec $CURDIR/lib/ld.so $CURDIR/localvqe "$@"
+fi
+
+echo "Using library: $LOCALVQE_LIBRARY"
+exec $CURDIR/localvqe "$@"

backend/go/localvqe/test.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+# The Go test suite uses a built localvqe binary for end-to-end
+# specs. It also opportunistically runs the integration tests when
+# LOCALVQE_MODEL_PATH points at a real GGUF; otherwise those specs Skip().
+
+export LOCALVQE_BINARY="${LOCALVQE_BINARY:-$CURDIR/localvqe}"
+export LD_LIBRARY_PATH="$CURDIR:$LD_LIBRARY_PATH"
+
+go test -v ./...

backend/go/sherpa-onnx/backend.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"bytes"
+	"context"
 	"encoding/binary"
 	"fmt"
 	"os"
@@ -29,6 +30,12 @@ type SherpaBackend struct {
 	vadWindowSize      int
 	ttsSpeed           float32
 	onlineChunkSamples int
+
+	// Speaker diarization (offline pyannote + embedding extractor + clustering).
+	// diarSampleRate is reported by sherpa at create time; we cache it so
+	// runDiarization can resample only when the input doesn't already match.
+	diarizer       uintptr
+	diarSampleRate int
 }
 
 var onnxProvider = "cpu"
@@ -128,6 +135,25 @@ var (
 
 	// TTS streaming callback trampoline
 	shimTtsGenerateWithCallback func(tts uintptr, text string, sid int32, speed float32, cb uintptr, ud uintptr) uintptr
+
+	// Diarization config + result accessors (see csrc/shim.h).
+	shimDiarizeConfigNew                       func() uintptr
+	shimDiarizeConfigFree                      func(uintptr)
+	shimDiarizeConfigSetSegmentationModel      func(uintptr, string)
+	shimDiarizeConfigSetSegmentationNumThreads func(uintptr, int32)
+	shimDiarizeConfigSetSegmentationProvider   func(uintptr, string)
+	shimDiarizeConfigSetSegmentationDebug      func(uintptr, int32)
+	shimDiarizeConfigSetEmbeddingModel         func(uintptr, string)
+	shimDiarizeConfigSetEmbeddingNumThreads    func(uintptr, int32)
+	shimDiarizeConfigSetEmbeddingProvider      func(uintptr, string)
+	shimDiarizeConfigSetEmbeddingDebug         func(uintptr, int32)
+	shimDiarizeConfigSetClusteringNumClusters  func(uintptr, int32)
+	shimDiarizeConfigSetClusteringThreshold    func(uintptr, float32)
+	shimDiarizeConfigSetMinDurationOn          func(uintptr, float32)
+	shimDiarizeConfigSetMinDurationOff         func(uintptr, float32)
+	shimCreateOfflineSpeakerDiarization        func(uintptr) uintptr
+	shimDiarizeSetClustering                   func(uintptr, int32, float32)
+	shimDiarizeSegmentAt                       func(segs uintptr, i int32, outStart unsafe.Pointer, outEnd unsafe.Pointer, outSpeaker unsafe.Pointer)
 )
 
 // libsherpa-onnx-c-api pass-throughs — called directly from Go via purego.
@@ -172,6 +198,18 @@ var (
 	sherpaOfflineTtsGenerate             func(tts uintptr, text string, sid int32, speed float32) uintptr
 	sherpaDestroyOfflineTtsGeneratedAudio func(audio uintptr)
 	sherpaOfflineTtsSampleRate           func(tts uintptr) int32
+
+	// Offline speaker diarization. Result handle owns the segment-array
+	// pointer returned by ResultSortByStartTime; destroy the segment
+	// array first, then the result, then (at backend Free()) the diarizer.
+	sherpaDestroyOfflineSpeakerDiarization                 func(sd uintptr)
+	sherpaOfflineSpeakerDiarizationGetSampleRate           func(sd uintptr) int32
+	sherpaOfflineSpeakerDiarizationProcess                 func(sd uintptr, samples unsafe.Pointer, n int32) uintptr
+	sherpaOfflineSpeakerDiarizationResultGetNumSegments    func(result uintptr) int32
+	sherpaOfflineSpeakerDiarizationResultGetNumSpeakers    func(result uintptr) int32
+	sherpaOfflineSpeakerDiarizationResultSortByStartTime   func(result uintptr) uintptr
+	sherpaOfflineSpeakerDiarizationDestroySegment          func(segs uintptr)
+	sherpaDestroyOfflineSpeakerDiarizationResult           func(result uintptr)
 )
 
 var (
@@ -292,6 +330,24 @@ func loadSherpaLibsOnce() error {
 		{&shimSpeechSegmentStart, "sherpa_shim_speech_segment_start"},
 		{&shimSpeechSegmentN, "sherpa_shim_speech_segment_n"},
 		{&shimTtsGenerateWithCallback, "sherpa_shim_tts_generate_with_callback"},
+
+		{&shimDiarizeConfigNew, "sherpa_shim_diarize_config_new"},
+		{&shimDiarizeConfigFree, "sherpa_shim_diarize_config_free"},
+		{&shimDiarizeConfigSetSegmentationModel, "sherpa_shim_diarize_config_set_segmentation_model"},
+		{&shimDiarizeConfigSetSegmentationNumThreads, "sherpa_shim_diarize_config_set_segmentation_num_threads"},
+		{&shimDiarizeConfigSetSegmentationProvider, "sherpa_shim_diarize_config_set_segmentation_provider"},
+		{&shimDiarizeConfigSetSegmentationDebug, "sherpa_shim_diarize_config_set_segmentation_debug"},
+		{&shimDiarizeConfigSetEmbeddingModel, "sherpa_shim_diarize_config_set_embedding_model"},
+		{&shimDiarizeConfigSetEmbeddingNumThreads, "sherpa_shim_diarize_config_set_embedding_num_threads"},
+		{&shimDiarizeConfigSetEmbeddingProvider, "sherpa_shim_diarize_config_set_embedding_provider"},
+		{&shimDiarizeConfigSetEmbeddingDebug, "sherpa_shim_diarize_config_set_embedding_debug"},
+		{&shimDiarizeConfigSetClusteringNumClusters, "sherpa_shim_diarize_config_set_clustering_num_clusters"},
+		{&shimDiarizeConfigSetClusteringThreshold, "sherpa_shim_diarize_config_set_clustering_threshold"},
+		{&shimDiarizeConfigSetMinDurationOn, "sherpa_shim_diarize_config_set_min_duration_on"},
+		{&shimDiarizeConfigSetMinDurationOff, "sherpa_shim_diarize_config_set_min_duration_off"},
+		{&shimCreateOfflineSpeakerDiarization, "sherpa_shim_create_offline_speaker_diarization"},
+		{&shimDiarizeSetClustering, "sherpa_shim_diarize_set_clustering"},
+		{&shimDiarizeSegmentAt, "sherpa_shim_diarize_segment_at"},
 	} {
 		purego.RegisterLibFunc(r.ptr, shim, r.name)
 	}
@@ -334,6 +390,15 @@ func loadSherpaLibsOnce() error {
 		{&sherpaOfflineTtsGenerate, "SherpaOnnxOfflineTtsGenerate"},
 		{&sherpaDestroyOfflineTtsGeneratedAudio, "SherpaOnnxDestroyOfflineTtsGeneratedAudio"},
 		{&sherpaOfflineTtsSampleRate, "SherpaOnnxOfflineTtsSampleRate"},
+
+		{&sherpaDestroyOfflineSpeakerDiarization, "SherpaOnnxDestroyOfflineSpeakerDiarization"},
+		{&sherpaOfflineSpeakerDiarizationGetSampleRate, "SherpaOnnxOfflineSpeakerDiarizationGetSampleRate"},
+		{&sherpaOfflineSpeakerDiarizationProcess, "SherpaOnnxOfflineSpeakerDiarizationProcess"},
+		{&sherpaOfflineSpeakerDiarizationResultGetNumSegments, "SherpaOnnxOfflineSpeakerDiarizationResultGetNumSegments"},
+		{&sherpaOfflineSpeakerDiarizationResultGetNumSpeakers, "SherpaOnnxOfflineSpeakerDiarizationResultGetNumSpeakers"},
+		{&sherpaOfflineSpeakerDiarizationResultSortByStartTime, "SherpaOnnxOfflineSpeakerDiarizationResultSortByStartTime"},
+		{&sherpaOfflineSpeakerDiarizationDestroySegment, "SherpaOnnxOfflineSpeakerDiarizationDestroySegment"},
+		{&sherpaDestroyOfflineSpeakerDiarizationResult, "SherpaOnnxOfflineSpeakerDiarizationDestroyResult"},
 	} {
 		purego.RegisterLibFunc(r.ptr, capi, r.name)
 	}
@@ -383,6 +448,11 @@ func isVADType(t string) bool {
 	return t == "vad"
 }
 
+func isDiarizationType(t string) bool {
+	t = strings.ToLower(t)
+	return t == "diarization" || t == "diarize" || t == "speaker-diarization"
+}
+
 // Model-options prefixes recognised by this backend. Kept as typed
 // constants so the asrFamily / loadWhisperASR / loadGenericASR paths
 // can all speak the same vocabulary.
@@ -423,6 +493,19 @@ const (
 	optionOnlineRule2          = "online.rule2_min_trailing_silence="
 	optionOnlineRule3          = "online.rule3_min_utterance_length="
 	optionOnlineChunkSamples   = "online.chunk_samples="
+
+	// Speaker diarization (offline pyannote + speaker-embedding extractor).
+	// `diarize.segmentation_model` overrides the auto-detected pyannote
+	// segmentation .onnx in modelDir; `diarize.embedding_model` does the
+	// same for the speaker-embedding extractor. `diarize.num_clusters`
+	// pins a known speaker count at load time; per-call DiarizeRequest
+	// fields take precedence at process time.
+	optionDiarizeSegmentationModel = "diarize.segmentation_model="
+	optionDiarizeEmbeddingModel    = "diarize.embedding_model="
+	optionDiarizeNumClusters       = "diarize.num_clusters="
+	optionDiarizeThreshold         = "diarize.threshold="
+	optionDiarizeMinDurationOn     = "diarize.min_duration_on="
+	optionDiarizeMinDurationOff    = "diarize.min_duration_off="
 )
 
 func hasOption(opts *pb.ModelOptions, prefix string) bool {
@@ -493,6 +576,9 @@ func (s *SherpaBackend) Load(opts *pb.ModelOptions) error {
 	if isVADType(opts.Type) {
 		return s.loadVAD(opts)
 	}
+	if isDiarizationType(opts.Type) {
+		return s.loadDiarization(opts)
+	}
 	// An explicit `subtype=...` option routes to ASR even when Type is
 	// unset — handy for the e2e-backends harness, which doesn't know
 	// about ModelOptions.Type.
@@ -913,7 +999,7 @@ func (s *SherpaBackend) loadOnlineASR(opts *pb.ModelOptions) error {
 // Transcription
 // =============================================================
 
-func (s *SherpaBackend) AudioTranscription(req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
+func (s *SherpaBackend) AudioTranscription(_ context.Context, req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
 	if s.onlineRecognizer != 0 {
 		return s.runOnlineASR(req, nil)
 	}
@@ -971,6 +1057,7 @@ func (s *SherpaBackend) AudioTranscription(req *pb.TranscriptRequest) (pb.Transc
 // Closes `results` before returning so the server wrapper's reader
 // goroutine can exit.
 func (s *SherpaBackend) AudioTranscriptionStream(
+	_ context.Context,
 	req *pb.TranscriptRequest,
 	results chan *pb.TranscriptStreamResponse,
 ) error {
@@ -1247,3 +1334,176 @@ func (s *SherpaBackend) TTSStream(req *pb.TTSRequest, results chan []byte) error
 	}
 	return nil
 }
+
+// =============================================================
+// Speaker diarization (offline)
+// =============================================================
+//
+// Conventions:
+//   - opts.ModelFile is the pyannote segmentation .onnx (e.g. model.onnx
+//     under sherpa-onnx-pyannote-segmentation-3-0/). Override with
+//     `diarize.segmentation_model=` if the gallery layout differs.
+//   - The speaker-embedding extractor must be provided via
+//     `diarize.embedding_model=`. There's no reliable filename heuristic
+//     we can rely on (3dspeaker, NeMo, WeSpeaker all ship with
+//     model-specific names), so we require it to be explicit.
+//   - Both paths are resolved relative to opts.ModelPath if not absolute.
+
+func (s *SherpaBackend) loadDiarization(opts *pb.ModelOptions) error {
+	if s.diarizer != 0 {
+		return nil
+	}
+
+	modelDir := filepath.Dir(opts.ModelFile)
+	segModel := findOptionValue(opts, optionDiarizeSegmentationModel, opts.ModelFile)
+	if segModel != "" && !filepath.IsAbs(segModel) && opts.ModelPath != "" {
+		segModel = filepath.Join(opts.ModelPath, segModel)
+	}
+	if !fileExists(segModel) {
+		return fmt.Errorf("sherpa-onnx diarization: pyannote segmentation model not found at %q (set diarize.segmentation_model=...)", segModel)
+	}
+
+	embModel := findOptionValue(opts, optionDiarizeEmbeddingModel, "")
+	if embModel == "" {
+		return fmt.Errorf("sherpa-onnx diarization: speaker-embedding model is required — pass options: [diarize.embedding_model=<path>] (e.g. 3dspeaker_speech_campplus_sv_zh-cn_16k-common.onnx)")
+	}
+	if !filepath.IsAbs(embModel) {
+		base := opts.ModelPath
+		if base == "" {
+			base = modelDir
+		}
+		embModel = filepath.Join(base, embModel)
+	}
+	if !fileExists(embModel) {
+		return fmt.Errorf("sherpa-onnx diarization: speaker-embedding model not found at %q", embModel)
+	}
+
+	threads := int32(1)
+	if opts.Threads != 0 {
+		threads = opts.Threads
+	}
+
+	cfg := shimDiarizeConfigNew()
+	defer shimDiarizeConfigFree(cfg)
+
+	shimDiarizeConfigSetSegmentationModel(cfg, segModel)
+	shimDiarizeConfigSetSegmentationNumThreads(cfg, threads)
+	shimDiarizeConfigSetSegmentationProvider(cfg, onnxProvider)
+	shimDiarizeConfigSetSegmentationDebug(cfg, 0)
+
+	shimDiarizeConfigSetEmbeddingModel(cfg, embModel)
+	shimDiarizeConfigSetEmbeddingNumThreads(cfg, threads)
+	shimDiarizeConfigSetEmbeddingProvider(cfg, onnxProvider)
+	shimDiarizeConfigSetEmbeddingDebug(cfg, 0)
+
+	shimDiarizeConfigSetClusteringNumClusters(cfg, findOptionInt(opts, optionDiarizeNumClusters, -1))
+	shimDiarizeConfigSetClusteringThreshold(cfg, findOptionFloat(opts, optionDiarizeThreshold, 0.5))
+	shimDiarizeConfigSetMinDurationOn(cfg, findOptionFloat(opts, optionDiarizeMinDurationOn, 0.3))
+	shimDiarizeConfigSetMinDurationOff(cfg, findOptionFloat(opts, optionDiarizeMinDurationOff, 0.5))
+
+	sd := shimCreateOfflineSpeakerDiarization(cfg)
+	if sd == 0 {
+		return fmt.Errorf("sherpa-onnx diarization: failed to create diarizer (segmentation=%s embedding=%s)", segModel, embModel)
+	}
+	s.diarizer = sd
+	s.diarSampleRate = int(sherpaOfflineSpeakerDiarizationGetSampleRate(sd))
+	return nil
+}
+
+// applyDiarizeOverrides re-applies clustering knobs onto an existing
+// diarizer when per-call DiarizeRequest fields are set. Both -1/0 sentinels
+// follow sherpa's convention: num_clusters<=0 → use threshold-based
+// clustering, threshold<=0 → keep load-time default.
+func (s *SherpaBackend) applyDiarizeOverrides(req *pb.DiarizeRequest) {
+	num := int32(-1)
+	if req.NumSpeakers > 0 {
+		num = req.NumSpeakers
+	}
+	threshold := float32(0)
+	if req.ClusteringThreshold > 0 {
+		threshold = req.ClusteringThreshold
+	}
+	if num > 0 || threshold > 0 {
+		shimDiarizeSetClustering(s.diarizer, num, threshold)
+	}
+}
+
+func (s *SherpaBackend) Diarize(req *pb.DiarizeRequest) (pb.DiarizeResponse, error) {
+	if s.diarizer == 0 {
+		return pb.DiarizeResponse{}, fmt.Errorf("sherpa-onnx diarization not loaded (model must be loaded with type=diarization)")
+	}
+	if req.Dst == "" {
+		return pb.DiarizeResponse{}, fmt.Errorf("sherpa-onnx diarization: DiarizeRequest.dst (audio path) is required")
+	}
+
+	dir, err := os.MkdirTemp("", "sherpa-diarize")
+	if err != nil {
+		return pb.DiarizeResponse{}, fmt.Errorf("failed to create temp dir: %w", err)
+	}
+	defer func() { _ = os.RemoveAll(dir) }()
+
+	wavPath := filepath.Join(dir, "input.wav")
+	if err := utils.AudioToWav(req.Dst, wavPath); err != nil {
+		return pb.DiarizeResponse{}, fmt.Errorf("failed to convert audio to wav: %w", err)
+	}
+
+	wave := sherpaReadWave(wavPath)
+	if wave == 0 {
+		return pb.DiarizeResponse{}, fmt.Errorf("failed to read wav %s", wavPath)
+	}
+	defer sherpaFreeWave(wave)
+
+	sr := int(shimWaveSampleRate(wave))
+	nSamples := shimWaveNumSamples(wave)
+	samples := shimWaveSamples(wave)
+	duration := float32(nSamples) / float32(sr)
+	if sr != s.diarSampleRate {
+		// AudioToWav already targets 16 kHz; pyannote-3.0 also wants 16 kHz, so
+		// this branch should be unreachable. Fail loudly instead of silently
+		// passing mismatched audio to the model.
+		return pb.DiarizeResponse{}, fmt.Errorf("sherpa-onnx diarization: input sample rate %d Hz does not match model %d Hz", sr, s.diarSampleRate)
+	}
+
+	s.applyDiarizeOverrides(req)
+
+	result := sherpaOfflineSpeakerDiarizationProcess(s.diarizer, samples, nSamples)
+	if result == 0 {
+		return pb.DiarizeResponse{}, fmt.Errorf("sherpa-onnx diarization: process failed")
+	}
+	defer sherpaDestroyOfflineSpeakerDiarizationResult(result)
+
+	numSegments := sherpaOfflineSpeakerDiarizationResultGetNumSegments(result)
+	numSpeakers := sherpaOfflineSpeakerDiarizationResultGetNumSpeakers(result)
+	if numSegments <= 0 {
+		return pb.DiarizeResponse{
+			Segments:    []*pb.DiarizeSegment{},
+			NumSpeakers: numSpeakers,
+			Duration:    duration,
+		}, nil
+	}
+
+	segs := sherpaOfflineSpeakerDiarizationResultSortByStartTime(result)
+	if segs == 0 {
+		return pb.DiarizeResponse{}, fmt.Errorf("sherpa-onnx diarization: failed to retrieve segments")
+	}
+	defer sherpaOfflineSpeakerDiarizationDestroySegment(segs)
+
+	out := make([]*pb.DiarizeSegment, 0, numSegments)
+	for i := range int(numSegments) {
+		var start, end float32
+		var spk int32
+		shimDiarizeSegmentAt(segs, int32(i),
+			unsafe.Pointer(&start), unsafe.Pointer(&end), unsafe.Pointer(&spk))
+		out = append(out, &pb.DiarizeSegment{
+			Id:      int32(i),
+			Start:   start,
+			End:     end,
+			Speaker: strconv.FormatInt(int64(spk), 10),
+		})
+	}
+	return pb.DiarizeResponse{
+		Segments:    out,
+		NumSpeakers: numSpeakers,
+		Duration:    duration,
+	}, nil
+}

backend/go/sherpa-onnx/backend_test.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"os"
 	"path/filepath"
 	"testing"
@@ -79,7 +80,7 @@ var _ = Describe("Sherpa-ONNX", func() {
 		})
 
 		It("rejects AudioTranscription", func() {
-			_, err := (&SherpaBackend{}).AudioTranscription(&pb.TranscriptRequest{
+			_, err := (&SherpaBackend{}).AudioTranscription(context.Background(), &pb.TranscriptRequest{
 				Dst: "/tmp/nonexistent.wav",
 			})
 			Expect(err).To(HaveOccurred())

backend/go/sherpa-onnx/csrc/shim.c

@@ -310,6 +310,87 @@ int32_t sherpa_shim_speech_segment_n(const void *h) {
     return ((const SherpaOnnxSpeechSegment *)h)->n;
 }
 
+// ==================================================================
+// Offline speaker diarization config
+// ==================================================================
+
+void *sherpa_shim_diarize_config_new(void) {
+    return calloc(1, sizeof(SherpaOnnxOfflineSpeakerDiarizationConfig));
+}
+
+void sherpa_shim_diarize_config_free(void *h) {
+    if (!h) return;
+    SherpaOnnxOfflineSpeakerDiarizationConfig *c =
+        (SherpaOnnxOfflineSpeakerDiarizationConfig *)h;
+    free((char *)c->segmentation.pyannote.model);
+    free((char *)c->segmentation.provider);
+    free((char *)c->embedding.model);
+    free((char *)c->embedding.provider);
+    free(c);
+}
+
+void sherpa_shim_diarize_config_set_segmentation_model(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.pyannote.model, v);
+}
+void sherpa_shim_diarize_config_set_segmentation_num_threads(void *h, int32_t v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.num_threads = v;
+}
+void sherpa_shim_diarize_config_set_segmentation_provider(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.provider, v);
+}
+void sherpa_shim_diarize_config_set_segmentation_debug(void *h, int32_t v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.debug = v;
+}
+void sherpa_shim_diarize_config_set_embedding_model(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.model, v);
+}
+void sherpa_shim_diarize_config_set_embedding_num_threads(void *h, int32_t v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.num_threads = v;
+}
+void sherpa_shim_diarize_config_set_embedding_provider(void *h, const char *v) {
+    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.provider, v);
+}
+void sherpa_shim_diarize_config_set_embedding_debug(void *h, int32_t v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.debug = v;
+}
+void sherpa_shim_diarize_config_set_clustering_num_clusters(void *h, int32_t v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->clustering.num_clusters = v;
+}
+void sherpa_shim_diarize_config_set_clustering_threshold(void *h, float v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->clustering.threshold = v;
+}
+void sherpa_shim_diarize_config_set_min_duration_on(void *h, float v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->min_duration_on = v;
+}
+void sherpa_shim_diarize_config_set_min_duration_off(void *h, float v) {
+    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->min_duration_off = v;
+}
+
+void *sherpa_shim_create_offline_speaker_diarization(void *h) {
+    return (void *)SherpaOnnxCreateOfflineSpeakerDiarization(
+        (const SherpaOnnxOfflineSpeakerDiarizationConfig *)h);
+}
+
+void sherpa_shim_diarize_set_clustering(void *sd, int32_t num_clusters, float threshold) {
+    if (!sd) return;
+    SherpaOnnxOfflineSpeakerDiarizationConfig cfg;
+    memset(&cfg, 0, sizeof(cfg));
+    cfg.clustering.num_clusters = num_clusters;
+    cfg.clustering.threshold    = threshold;
+    SherpaOnnxOfflineSpeakerDiarizationSetConfig(
+        (const SherpaOnnxOfflineSpeakerDiarization *)sd, &cfg);
+}
+
+void sherpa_shim_diarize_segment_at(const void *segs, int32_t i,
+                                    float *out_start, float *out_end,
+                                    int32_t *out_speaker) {
+    const SherpaOnnxOfflineSpeakerDiarizationSegment *arr =
+        (const SherpaOnnxOfflineSpeakerDiarizationSegment *)segs;
+    if (out_start)   *out_start   = arr[i].start;
+    if (out_end)     *out_end     = arr[i].end;
+    if (out_speaker) *out_speaker = arr[i].speaker;
+}
+
 // ==================================================================
 // TTS streaming callback trampoline
 // ==================================================================

backend/go/sherpa-onnx/csrc/shim.h

@@ -109,6 +109,41 @@ const float *sherpa_shim_generated_audio_samples(const void *audio);
 int32_t sherpa_shim_speech_segment_start(const void *seg);
 int32_t sherpa_shim_speech_segment_n(const void *seg);
 
+// --- Offline speaker diarization config -----------------------------
+// Pyannote segmentation + speaker-embedding extractor + fast clustering.
+// The upstream config is a struct of nested structs; purego can't read or
+// build those across dlopen, so we expose a calloc'd opaque holder plus
+// flat setters, then hand it to sherpa via the create wrapper.
+void *sherpa_shim_diarize_config_new(void);
+void  sherpa_shim_diarize_config_free(void *cfg);
+void  sherpa_shim_diarize_config_set_segmentation_model(void *cfg, const char *path);
+void  sherpa_shim_diarize_config_set_segmentation_num_threads(void *cfg, int32_t v);
+void  sherpa_shim_diarize_config_set_segmentation_provider(void *cfg, const char *v);
+void  sherpa_shim_diarize_config_set_segmentation_debug(void *cfg, int32_t v);
+void  sherpa_shim_diarize_config_set_embedding_model(void *cfg, const char *path);
+void  sherpa_shim_diarize_config_set_embedding_num_threads(void *cfg, int32_t v);
+void  sherpa_shim_diarize_config_set_embedding_provider(void *cfg, const char *v);
+void  sherpa_shim_diarize_config_set_embedding_debug(void *cfg, int32_t v);
+void  sherpa_shim_diarize_config_set_clustering_num_clusters(void *cfg, int32_t v);
+void  sherpa_shim_diarize_config_set_clustering_threshold(void *cfg, float v);
+void  sherpa_shim_diarize_config_set_min_duration_on(void *cfg, float v);
+void  sherpa_shim_diarize_config_set_min_duration_off(void *cfg, float v);
+void *sherpa_shim_create_offline_speaker_diarization(void *cfg);
+
+// Apply just the clustering knobs onto a loaded diarizer (sherpa
+// supports re-clustering after Create), so per-call overrides like
+// num_speakers don't require re-loading the heavy ONNX models.
+void  sherpa_shim_diarize_set_clustering(void *sd, int32_t num_clusters, float threshold);
+
+// Sherpa's ResultSortByStartTime returns a sherpa-allocated array of
+// SherpaOnnxOfflineSpeakerDiarizationSegment structs (free with
+// SherpaOnnxOfflineSpeakerDiarizationDestroySegment). Purego can't read
+// fields out of an array of C structs, so this getter copies one
+// segment's fields into the caller-supplied float/int32 cells.
+void  sherpa_shim_diarize_segment_at(const void *segs, int32_t i,
+                                     float *out_start, float *out_end,
+                                     int32_t *out_speaker);
+
 // --- TTS streaming callback trampoline -----------------------------
 // Replaces the //export sherpaTtsGoCallback + callbacks.c bridge pattern.
 // `callback_ptr` is the C-callable function pointer returned by

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=3d6064b37ef4607917f8acf2ca8c8906d5087413
+STABLEDIFFUSION_GGML_VERSION?=90e87bc846f17059771efb8aaa31e9ef0cab6f78
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/vibevoice-cpp/Makefile

@@ -6,9 +6,12 @@ GOCMD?=go
 GO_TAGS?=
 JOBS?=$(shell nproc --ignore=1)
 
-# vibevoice.cpp version
+# vibevoice.cpp version. Pinned to a commit hash and auto-bumped by
+# .github/workflows/bump_deps.yaml (the matrix entry mirrors what we
+# already do for ik_llama.cpp / llama.cpp / whisper.cpp). Floating on
+# `master` led to silent ABI breaks reaching CI — pin it.
 VIBEVOICE_REPO?=https://github.com/mudler/vibevoice.cpp
-VIBEVOICE_CPP_VERSION?=master
+VIBEVOICE_CPP_VERSION?=ad856bda6b1311b7f3d7c4a667be43eeb8a8249a
 SO_TARGET?=libgovibevoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/vibevoice-cpp/govibevoicecpp.go

@@ -1,10 +1,14 @@
 package main
 
 import (
+	"context"
 	"encoding/json"
 	"fmt"
+	"io"
 	"os"
+	"os/exec"
 	"path/filepath"
+	"runtime"
 	"strings"
 
 	laudio "github.com/mudler/LocalAI/pkg/audio"
@@ -12,15 +16,102 @@ import (
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 )
 
+// vv_capi_asr loads audio with load_wav_24k_mono — a 24 kHz mono s16le
+// WAV is the format the model was trained on. Inputs already in that
+// format pass through; everything else is converted via ffmpeg, which
+// is therefore a runtime requirement only when callers upload non-WAV
+// (or non-24 kHz mono s16le WAV) audio. Skipping ffmpeg on the happy
+// path matters for the e2e-backends test container, which does not
+// ship ffmpeg but feeds the backend pre-cooked 24 kHz mono WAVs.
+const vibevoiceASRSampleRate = 24000
+
+// prepareWavInput resolves `src` to a 24 kHz mono s16le WAV path that
+// vv_capi_asr's load_wav_24k_mono accepts. Returns the resolved path
+// plus a cleanup func; both must be honoured by the caller.
+//
+// Pass-through happens when `src` already has the right WAV format —
+// no ffmpeg required. Otherwise we shell out to ffmpeg into a temp
+// dir; if ffmpeg isn't on PATH we surface a clear error mentioning the
+// underlying format mismatch.
+func prepareWavInput(src string) (string, func(), error) {
+	if src == "" {
+		return "", func() {}, fmt.Errorf("empty audio path")
+	}
+	if isVibevoiceCompatibleWav(src) {
+		return src, func() {}, nil
+	}
+
+	dir, err := os.MkdirTemp("", "vibevoice-asr")
+	if err != nil {
+		return "", func() {}, fmt.Errorf("mkdtemp: %w", err)
+	}
+	cleanup := func() { _ = os.RemoveAll(dir) }
+	wavPath := filepath.Join(dir, "input.wav")
+
+	// -y: overwrite, -ar 24000: target sample rate, -ac 1: mono,
+	// -acodec pcm_s16le: signed 16-bit little-endian PCM (load_wav_24k_mono
+	// only accepts s16le).
+	cmd := exec.Command("ffmpeg",
+		"-y", "-i", src,
+		"-ar", fmt.Sprintf("%d", vibevoiceASRSampleRate),
+		"-ac", "1",
+		"-acodec", "pcm_s16le",
+		wavPath,
+	)
+	cmd.Env = []string{}
+	if out, err := cmd.CombinedOutput(); err != nil {
+		cleanup()
+		return "", func() {}, fmt.Errorf("ffmpeg convert to 24k mono wav: %w (output: %s)", err, string(out))
+	}
+	return wavPath, cleanup, nil
+}
+
+// isVibevoiceCompatibleWav returns true when `src` carries the RIFF/WAVE
+// magic bytes. vibevoice's load_wav_24k_mono uses drwav under the hood,
+// which accepts any PCM/IEEE-float WAV at any sample rate and downmixes
+// multi-channel input to mono on its own — so any valid WAV passes
+// through to the C side without conversion. Anything else (MP3, OGG,
+// FLAC, ...) needs ffmpeg.
+func isVibevoiceCompatibleWav(src string) bool {
+	f, err := os.Open(src)
+	if err != nil {
+		return false
+	}
+	defer func() { _ = f.Close() }()
+
+	// 0..3 = "RIFF", 8..11 = "WAVE".
+	var hdr [12]byte
+	if _, err := io.ReadFull(f, hdr[:]); err != nil {
+		return false
+	}
+	return string(hdr[0:4]) == "RIFF" && string(hdr[8:12]) == "WAVE"
+}
+
+// asrMaxNewTokens caps the ASR generation budget. The C ABI defaults to
+// 256 when 0 is passed — far too small for anything past ~10s of speech.
+// Vibevoice generates ~30 tokens per second of audio, so 16 384 covers
+// roughly 9 minutes of dialogue, well past any normal /v1/audio/diarization
+// upload. Going higher costs little since generation stops at EOS.
+const asrMaxNewTokens = 16384
+
 // vibevoice.cpp synthesizes 24 kHz mono 16-bit PCM. Hardcoded - the
 // model itself is fixed-rate; if the upstream ever changes this we'll
 // pick it up via vv_capi_version().
 const vibevoiceSampleRate = uint32(24000)
 
 // purego-bound entry points from libgovibevoicecpp.
+//
+// vv_capi_tts takes a `const char* const* ref_audio_paths` array (used
+// by the 1.5B variant for runtime voice cloning; the realtime-0.5B
+// path leaves it NULL and uses voice_path instead). purego marshals a
+// Go []*byte slice as **char by passing the underlying array's address.
+// A nil/empty slice marshals to NULL, which matches the C contract for
+// "no reference audio".
 var (
 	CppLoad func(ttsModel, asrModel, tokenizer, voice string, threads int32) int32
-	CppTTS  func(text, voicePath, dstWav string,
+	CppTTS  func(text, voicePath string,
+		refAudioPaths []*byte, nRefAudioPaths int32,
+		dstWav string,
 		nSteps int32, cfgScale float32, maxSpeechFrames int32, seed uint32) int32
 	CppASR func(srcWav string, outJSON []byte, capacity uint64,
 		maxNewTokens int32) int32
@@ -44,6 +135,14 @@ type VibevoiceCpp struct {
 	asrModel  string
 	tokenizer string
 	voice     string
+
+	// refAudio is the load-time default list of reference WAVs used by
+	// the 1.5B model (one per speaker). Sourced from
+	// ModelOptions.AudioPath (config_file's `audio_path:`) — comma-
+	// separated for multi-speaker. Per-call TTSRequest.Voice can
+	// override it. Empty for the realtime-0.5B path, which conditions
+	// on a pre-baked voice gguf via `voice` instead.
+	refAudio []string
 }
 
 // resolvePath joins a relative path onto `relTo`. The gallery
@@ -89,6 +188,25 @@ func (v *VibevoiceCpp) parseOptions(opts []string, relTo string) string {
 	return role
 }
 
+// parseRefAudio splits a comma-separated audio_path value into a
+// resolved list of WAVs. The 1.5B model uses one WAV per speaker;
+// callers that only need a single reference set audio_path to a single
+// path. Empty / whitespace-only entries are skipped.
+func parseRefAudio(audioPath, relTo string) []string {
+	if audioPath == "" {
+		return nil
+	}
+	var out []string
+	for _, p := range strings.Split(audioPath, ",") {
+		p = strings.TrimSpace(p)
+		if p == "" {
+			continue
+		}
+		out = append(out, resolvePath(p, relTo))
+	}
+	return out
+}
+
 func (v *VibevoiceCpp) Load(opts *pb.ModelOptions) error {
 	if opts.ModelFile == "" {
 		return fmt.Errorf("vibevoice-cpp: ModelFile is required")
@@ -109,6 +227,12 @@ func (v *VibevoiceCpp) Load(opts *pb.ModelOptions) error {
 	}
 	role := v.parseOptions(opts.Options, v.modelRoot)
 
+	// 1.5B reference WAVs ride on ModelOptions.AudioPath (config_file's
+	// `audio_path:` key) — same convention other audio backends already
+	// follow. Single-speaker = single path; multi-speaker = comma list,
+	// one WAV per Speaker N: tag in TTSRequest.text.
+	v.refAudio = parseRefAudio(opts.AudioPath, v.modelRoot)
+
 	// ModelFile fills the "primary" role-slot determined by `type=`
 	// in Options (defaults to tts). The other slot stays exactly as
 	// Options set it - so a closed-loop config with ModelFile=tts.gguf
@@ -142,8 +266,8 @@ func (v *VibevoiceCpp) Load(opts *pb.ModelOptions) error {
 	v.threads = threads
 
 	fmt.Fprintf(os.Stderr,
-		"[vibevoice-cpp] Loading: tts=%q asr=%q tokenizer=%q voice=%q threads=%d\n",
-		v.ttsModel, v.asrModel, v.tokenizer, v.voice, threads)
+		"[vibevoice-cpp] Loading: tts=%q asr=%q tokenizer=%q voice=%q ref_audio=%v threads=%d\n",
+		v.ttsModel, v.asrModel, v.tokenizer, v.voice, v.refAudio, threads)
 
 	if rc := CppLoad(v.ttsModel, v.asrModel, v.tokenizer, v.voice, int32(threads)); rc != 0 {
 		return fmt.Errorf("vibevoice-cpp: vv_capi_load failed (rc=%d)", rc)
@@ -161,10 +285,35 @@ func (v *VibevoiceCpp) TTS(req *pb.TTSRequest) error {
 		return fmt.Errorf("vibevoice-cpp: TTS requires both text and dst")
 	}
 
-	// req.Voice may be a bare filename (e.g. "voice-en-Emma.gguf") or an
-	// absolute path. Resolve via the same modelRoot Load() used for
-	// Options[] so a swap-voice request mirrors the gallery's layout.
-	voice := resolvePath(req.Voice, v.modelRoot)
+	// TTSRequest.Voice carries the per-call override. Routing depends
+	// on the loaded model variant:
+	//   * realtime-0.5B → expects a baked voice .gguf (single path).
+	//   * 1.5B          → expects one or more raw 24 kHz mono .wav
+	//                     reference clips for runtime voice cloning;
+	//                     comma-separated to address multi-speaker
+	//                     dialogs (Speaker 0..n-1 follow the order).
+	// We pick the branch by extension / shape of the override; if no
+	// override is given, fall back to the load-time defaults.
+	voice := ""
+	var refAudio []string
+	if reqVoice := strings.TrimSpace(req.Voice); reqVoice != "" {
+		if isRefAudioOverride(reqVoice) {
+			for _, p := range strings.Split(reqVoice, ",") {
+				p = strings.TrimSpace(p)
+				if p == "" {
+					continue
+				}
+				refAudio = append(refAudio, resolvePath(p, v.modelRoot))
+			}
+		} else {
+			voice = resolvePath(reqVoice, v.modelRoot)
+		}
+	} else {
+		// No per-call override. v.voice already went to vv_capi_load
+		// for realtime-0.5B; ref_audio is per-call only on the C ABI,
+		// so the gallery's `ref_audio:` defaults are re-passed here.
+		refAudio = append(refAudio, v.refAudio...)
+	}
 
 	if req.Language != nil && *req.Language != "" {
 		fmt.Fprintf(os.Stderr,
@@ -177,13 +326,51 @@ func (v *VibevoiceCpp) TTS(req *pb.TTSRequest) error {
 		defaultMaxFrames = 200
 	)
 	defaultCfg := float32(1.3)
-	if rc := CppTTS(text, voice, dst,
-		int32(defaultSteps), defaultCfg, int32(defaultMaxFrames), 0); rc != 0 {
+
+	refPtrs, refKeep := newCStringArray(refAudio)
+	rc := CppTTS(text, voice, refPtrs, int32(len(refPtrs)), dst,
+		int32(defaultSteps), defaultCfg, int32(defaultMaxFrames), 0)
+	// Hold the backing buffers past the cgo call. purego marshals
+	// []*byte by handing the C side the underlying array address; the
+	// pointed-to NUL-terminated bytes must outlive the call.
+	runtime.KeepAlive(refKeep)
+	runtime.KeepAlive(refPtrs)
+	if rc != 0 {
 		return fmt.Errorf("vibevoice-cpp: vv_capi_tts failed (rc=%d)", rc)
 	}
 	return nil
 }
 
+// isRefAudioOverride decides whether a TTSRequest.Voice override should
+// be routed to ref_audio_paths (1.5B path) instead of voice_path
+// (realtime-0.5B). Either a comma-separated list (multi-speaker) or a
+// single .wav clip qualifies; a bare voice .gguf falls through.
+func isRefAudioOverride(s string) bool {
+	if strings.Contains(s, ",") {
+		return true
+	}
+	return strings.HasSuffix(strings.ToLower(s), ".wav")
+}
+
+// newCStringArray builds the **char array vv_capi_tts expects, plus the
+// keep-alive slice the caller must runtime.KeepAlive across the cgo
+// call. A nil/empty input returns (nil, nil) which purego marshals to
+// the C NULL pointer.
+func newCStringArray(in []string) ([]*byte, [][]byte) {
+	if len(in) == 0 {
+		return nil, nil
+	}
+	keep := make([][]byte, len(in))
+	ptrs := make([]*byte, len(in))
+	for i, s := range in {
+		b := make([]byte, len(s)+1)
+		copy(b, s)
+		keep[i] = b
+		ptrs[i] = &b[0]
+	}
+	return ptrs, keep
+}
+
 // asrSegment matches vibevoice's JSON output:
 //
 //	[{"Start":0.0,"End":2.8,"Speaker":0,"Content":"…"}, ...]
@@ -294,7 +481,7 @@ func (w *byteWriter) Write(p []byte) (int, error) {
 	return len(p), nil
 }
 
-func (v *VibevoiceCpp) AudioTranscription(req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
+func (v *VibevoiceCpp) AudioTranscription(_ context.Context, req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
 	if v.asrModel == "" {
 		return pb.TranscriptResult{}, fmt.Errorf("vibevoice-cpp: AudioTranscription requested but no ASR model was loaded")
 	}
@@ -302,7 +489,13 @@ func (v *VibevoiceCpp) AudioTranscription(req *pb.TranscriptRequest) (pb.Transcr
 		return pb.TranscriptResult{}, fmt.Errorf("vibevoice-cpp: TranscriptRequest.dst (audio path) is required")
 	}
 
-	out, err := v.callASR(req.Dst, 0)
+	wavPath, cleanup, err := prepareWavInput(req.Dst)
+	if err != nil {
+		return pb.TranscriptResult{}, fmt.Errorf("vibevoice-cpp: %w", err)
+	}
+	defer cleanup()
+
+	out, err := v.callASR(wavPath, asrMaxNewTokens)
 	if err != nil {
 		return pb.TranscriptResult{}, err
 	}
@@ -346,6 +539,83 @@ func (v *VibevoiceCpp) AudioTranscription(req *pb.TranscriptRequest) (pb.Transcr
 	}, nil
 }
 
+// Diarize runs vibevoice's ASR and projects the speaker-labelled segment
+// list it returns natively. vibevoice.cpp's ASR prompt asks the model to
+// emit `[{"Start":..,"End":..,"Speaker":..,"Content":..}]`, so diarization
+// is a by-product of the same pass — we reuse callASR and re-shape.
+//
+// Speaker hints (num_speakers/min/max/threshold) and min_duration_on/off are
+// not actionable here: vibevoice's model picks the speaker count itself and
+// has no clustering knob. The HTTP layer documents this; we accept the
+// fields for API symmetry and ignore them.
+func (v *VibevoiceCpp) Diarize(req *pb.DiarizeRequest) (pb.DiarizeResponse, error) {
+	if v.asrModel == "" {
+		return pb.DiarizeResponse{}, fmt.Errorf("vibevoice-cpp: Diarize requires an ASR model (load options: type=asr)")
+	}
+	if req.Dst == "" {
+		return pb.DiarizeResponse{}, fmt.Errorf("vibevoice-cpp: DiarizeRequest.dst (audio path) is required")
+	}
+
+	wavPath, cleanup, err := prepareWavInput(req.Dst)
+	if err != nil {
+		return pb.DiarizeResponse{}, fmt.Errorf("vibevoice-cpp: %w", err)
+	}
+	defer cleanup()
+
+	out, err := v.callASR(wavPath, asrMaxNewTokens)
+	if err != nil {
+		return pb.DiarizeResponse{}, err
+	}
+	if out == "" {
+		return pb.DiarizeResponse{}, nil
+	}
+
+	var segs []asrSegment
+	if err := json.Unmarshal([]byte(out), &segs); err != nil {
+		// Mirror AudioTranscription's fallback: vibevoice's ASR sometimes
+		// emits free-form text instead of JSON for short or unusual audio.
+		// Surface a single unknown-speaker segment carrying the full text
+		// (when include_text is set) so the caller still gets coverage of
+		// the whole clip rather than a hard failure.
+		fmt.Fprintf(os.Stderr,
+			"[vibevoice-cpp] WARNING: vv_capi_asr returned non-JSON for diarization, falling back to single segment: %v\n", err)
+		text := strings.TrimSpace(out)
+		seg := &pb.DiarizeSegment{Id: 0, Speaker: "0"}
+		if req.IncludeText {
+			seg.Text = text
+		}
+		return pb.DiarizeResponse{
+			Segments:    []*pb.DiarizeSegment{seg},
+			NumSpeakers: 1,
+		}, nil
+	}
+
+	speakers := make(map[int]struct{})
+	segments := make([]*pb.DiarizeSegment, 0, len(segs))
+	var duration float32
+	for i, s := range segs {
+		ds := &pb.DiarizeSegment{
+			Id:      int32(i),
+			Start:   float32(s.Start),
+			End:     float32(s.End),
+			Speaker: fmt.Sprintf("%d", s.Speaker),
+		}
+		if req.IncludeText {
+			ds.Text = strings.TrimSpace(s.Content)
+		}
+		segments = append(segments, ds)
+		speakers[s.Speaker] = struct{}{}
+		if float32(s.End) > duration {
+			duration = float32(s.End)
+		}
+	}
+	return pb.DiarizeResponse{
+		Segments:    segments,
+		NumSpeakers: int32(len(speakers)),
+		Duration:    duration,
+	}, nil
+}
+
 // AudioTranscriptionStream wraps AudioTranscription so the streaming
 // gRPC endpoint (server.go:AudioTranscriptionStream) sees its channel
 // close and the client doesn't sit waiting until deadline. vibevoice's
@@ -354,9 +624,9 @@ func (v *VibevoiceCpp) AudioTranscription(req *pb.TranscriptRequest) (pb.Transcr
 // transcription, emit each segment's content as a delta, then close
 // with a final_result whose Text equals the concatenated deltas (the
 // e2e harness asserts those match).
-func (v *VibevoiceCpp) AudioTranscriptionStream(req *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
+func (v *VibevoiceCpp) AudioTranscriptionStream(ctx context.Context, req *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
 	defer close(results)
-	res, err := v.AudioTranscription(req)
+	res, err := v.AudioTranscription(ctx, req)
 	if err != nil {
 		return err
 	}

backend/go/vibevoice-cpp/vibevoicecpp_test.go

@@ -107,7 +107,7 @@ var _ = Describe("VibeVoice-cpp", func() {
 		})
 
 		It("rejects AudioTranscription without a loaded ASR model", func() {
-			_, err := (&VibevoiceCpp{}).AudioTranscription(&pb.TranscriptRequest{
+			_, err := (&VibevoiceCpp{}).AudioTranscription(context.Background(), &pb.TranscriptRequest{
 				Dst: "/tmp/some.wav",
 			})
 			Expect(err).To(HaveOccurred())
@@ -255,7 +255,7 @@ var _ = Describe("VibeVoice-cpp", func() {
 
 		It("closes the channel and errors on AudioTranscriptionStream without a loaded model", func() {
 			ch := make(chan *pb.TranscriptStreamResponse, 4)
-			err := (&VibevoiceCpp{}).AudioTranscriptionStream(&pb.TranscriptRequest{
+			err := (&VibevoiceCpp{}).AudioTranscriptionStream(context.Background(), &pb.TranscriptRequest{
 				Dst: "/tmp/some.wav",
 			}, ch)
 			Expect(err).To(HaveOccurred())

backend/go/voxtral/govoxtral.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"os"
 	"strings"
@@ -27,7 +28,7 @@ func (v *Voxtral) Load(opts *pb.ModelOptions) error {
 	return nil
 }
 
-func (v *Voxtral) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
+func (v *Voxtral) AudioTranscription(_ context.Context, opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
 	dir, err := os.MkdirTemp("", "voxtral")
 	if err != nil {
 		return pb.TranscriptResult{}, err

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=fc674574ca27cac59a15e5b22a09b9d9ad62aafe
+WHISPER_CPP_VERSION?=c33c5618b72bb345df029b730b36bc0e369845a3
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/whisper/cpp/gowhisper.cpp

@@ -1,12 +1,47 @@
 #include "gowhisper.h"
 #include "ggml-backend.h"
 #include "whisper.h"
+#include <atomic>
 #include <vector>
 
 static struct whisper_vad_context *vctx;
 static struct whisper_context *ctx;
 static std::vector<float> flat_segs;
 
+static std::atomic<int> g_abort{0};
+
+static std::atomic<uintptr_t> g_go_new_segment_cb{0};
+static std::atomic<uintptr_t> g_go_new_segment_user_data{0};
+
+static bool abort_cb(void * /*user_data*/) {
+    return g_abort.load(std::memory_order_relaxed) != 0;
+}
+
+static void new_segment_cb(struct whisper_context *cb_ctx,
+                           struct whisper_state * /*state*/, int n_new,
+                           void * /*user_data*/) {
+    uintptr_t go_cb = g_go_new_segment_cb.load(std::memory_order_relaxed);
+    if (go_cb == 0) {
+        return;
+    }
+    int total = whisper_full_n_segments(cb_ctx);
+    int idx_first = total - n_new;
+    if (idx_first < 0) {
+        idx_first = 0;
+    }
+    uintptr_t ud = g_go_new_segment_user_data.load(std::memory_order_relaxed);
+    reinterpret_cast<go_new_segment_cb>(go_cb)(idx_first, n_new, ud);
+}
+
+extern "C" void set_abort(int v) {
+    g_abort.store(v, std::memory_order_relaxed);
+}
+
+extern "C" void set_new_segment_callback(uintptr_t cb_ptr, uintptr_t user_data) {
+    g_go_new_segment_cb.store(cb_ptr, std::memory_order_relaxed);
+    g_go_new_segment_user_data.store(user_data, std::memory_order_relaxed);
+}
+
 static void ggml_log_cb(enum ggml_log_level level, const char *log,
                         void *data) {
   const char *level_str;
@@ -124,10 +159,28 @@ int transcribe(uint32_t threads, char *lang, bool translate, bool tdrz,
   wparams.tdrz_enable = tdrz;
   wparams.initial_prompt = prompt;
 
+  // Reset stale abort flag from any prior cancelled call, then install the
+  // ggml abort hook so a subsequent set_abort(1) from Go aborts the next
+  // compute graph step.
+  g_abort.store(0, std::memory_order_relaxed);
+  // Only install the new-segment callback when streaming is requested
+  // (Go side calls set_new_segment_callback before transcribe()). Leaving
+  // it always-on is harmless but adds a function-pointer dispatch per
+  // segment for the offline path.
+  if (g_go_new_segment_cb.load(std::memory_order_relaxed) != 0) {
+      wparams.new_segment_callback = new_segment_cb;
+      wparams.new_segment_callback_user_data = nullptr;
+  }
+  wparams.abort_callback = abort_cb;
+  wparams.abort_callback_user_data = nullptr;
+
   fprintf(stderr, "info: Enable tdrz: %d\n", tdrz);
   fprintf(stderr, "info: Initial prompt: \"%s\"\n", prompt);
 
   if (whisper_full(ctx, wparams, pcmf32, pcmf32_len)) {
+    if (g_abort.load(std::memory_order_relaxed)) {
+      return 2;   // aborted by client
+    }
     fprintf(stderr, "error: transcription failed\n");
     return 1;
   }

backend/go/whisper/cpp/gowhisper.h

@@ -15,4 +15,16 @@ int64_t get_segment_t1(int i);
 int n_tokens(int i);
 int32_t get_token_id(int i, int j);
 bool get_segment_speaker_turn_next(int i);
+void set_abort(int v);
+
+// Function pointer from Go (returned by purego.NewCallback). Invoked once
+// per new-segment event during whisper_full(). The callback runs on the
+// decode thread - if Go blocks (slow gRPC consumer), the decode blocks
+// too. That is the intended backpressure path.
+typedef void (*go_new_segment_cb)(int idx_first, int n_new, uintptr_t user_data);
+
+// Install the callback used by the next transcribe() call. Pass cb=0 to
+// clear. user_data is opaque to C; the Go side uses it to look up
+// per-call state.
+void set_new_segment_callback(uintptr_t cb_ptr, uintptr_t user_data);
 }

backend/go/whisper/gowhisper.go

@@ -1,16 +1,21 @@
 package main
 
 import (
+	"context"
 	"fmt"
 	"os"
 	"path/filepath"
 	"strings"
+	"sync"
+	"sync/atomic"
 	"unsafe"
 
 	"github.com/go-audio/wav"
 	"github.com/mudler/LocalAI/pkg/grpc/base"
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 	"github.com/mudler/LocalAI/pkg/utils"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
 )
 
 var (
@@ -24,8 +29,84 @@ var (
 	CppNTokens                   func(i int) int
 	CppGetTokenID                func(i int, j int) int
 	CppGetSegmentSpeakerTurnNext func(i int) bool
+	CppSetAbort                  func(v int)
+	// Set by main.go via purego.RegisterLibFunc. Installs (or clears with cb=0)
+	// the C-side trampoline that whisper.cpp invokes per new segment.
+	CppSetNewSegmentCallback func(cbPtr uintptr, userData uintptr)
 )
 
+// streamCallStates maps per-AudioTranscriptionStream call IDs to the
+// state the Go callback needs to emit deltas. Only one entry is ever
+// live today (base.SingleThread), but the map shape mirrors
+// sherpa-onnx's TTS callback registry and survives a future SingleThread
+// removal without a contract change.
+var (
+	streamCallStates sync.Map // uint64 -> *streamCallState
+	streamCallSeq    atomic.Uint64
+	goNewSegmentCb   uintptr // purego.NewCallback(onNewSegment) result; set in main.go at boot
+)
+
+type streamCallState struct {
+	results chan *pb.TranscriptStreamResponse
+	diarize bool
+	// nextIdx tracks how many segments we've already emitted. The C
+	// trampoline passes idx_first = total - n_new, but we walk from
+	// nextIdx to (idx_first + n_new) defensively in case whisper.cpp ever
+	// coalesces multiple commits into a single callback invocation.
+	nextIdx int
+	// assembled mirrors the literal concat of every Delta sent on results.
+	// We reuse it as the final TranscriptResult.Text so the e2e
+	// invariant `final.Text == concat(deltas)` holds exactly. Written from
+	// the cgo decode thread inside onNewSegment and read by the streaming
+	// method after CppTranscribe returns; the cgo boundary provides the
+	// happens-before edge.
+	assembled strings.Builder
+}
+
+// onNewSegment is the Go side of the C trampoline declared in
+// gowhisper.cpp:new_segment_cb. Whisper.cpp invokes it once per
+// new-segment event during whisper_full(). Reads segment text via the
+// existing CppGetSegment* getters (safe to call against the singleton
+// ctx; whisper.cpp is the only writer and it has already published the
+// segments by the time this fires).
+//
+// Sends deltas synchronously: if the channel is full, this blocks the
+// whisper decode thread. That's the intended backpressure path -
+// dropping deltas would break the concat(deltas) == final.Text invariant
+// the e2e suite asserts.
+func onNewSegment(idxFirst int32, nNew int32, userData uintptr) {
+	v, ok := streamCallStates.Load(uint64(userData))
+	if !ok {
+		return // call already torn down (race with cancel + cb fire)
+	}
+	state := v.(*streamCallState)
+	end := int(idxFirst) + int(nNew)
+	for i := state.nextIdx; i < end; i++ {
+		txt := strings.ToValidUTF8(strings.Clone(CppGetSegmentText(i)), "<22>")
+		txt = strings.TrimSpace(txt)
+		if state.diarize && CppGetSegmentSpeakerTurnNext(i) {
+			txt += " [SPEAKER_TURN]"
+		}
+		if txt == "" {
+			state.nextIdx = i + 1
+			continue
+		}
+		// Prefix subsequent deltas with a single space so the assembled
+		// stream reads as one space-joined transcript. The first delta has
+		// no leading space, otherwise concat(deltas) would not match
+		// final.Text and the e2e invariant would break.
+		var delta string
+		if state.assembled.Len() == 0 {
+			delta = txt
+		} else {
+			delta = " " + txt
+		}
+		state.results <- &pb.TranscriptStreamResponse{Delta: delta}
+		state.assembled.WriteString(delta)
+		state.nextIdx = i + 1
+	}
+}
+
 type Whisper struct {
 	base.SingleThread
 }
@@ -92,7 +173,11 @@ func (w *Whisper) VAD(req *pb.VADRequest) (pb.VADResponse, error) {
 	}, nil
 }
 
-func (w *Whisper) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
+func (w *Whisper) AudioTranscription(ctx context.Context, opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
+	if err := ctx.Err(); err != nil {
+		return pb.TranscriptResult{}, status.Error(codes.Canceled, "transcription cancelled")
+	}
+
 	dir, err := os.MkdirTemp("", "whisper")
 	if err != nil {
 		return pb.TranscriptResult{}, err
@@ -105,14 +190,12 @@ func (w *Whisper) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptR
 		return pb.TranscriptResult{}, err
 	}
 
-	// Open samples
 	fh, err := os.Open(convertedPath)
 	if err != nil {
 		return pb.TranscriptResult{}, err
 	}
 	defer fh.Close()
 
-	// Read samples
 	d := wav.NewDecoder(fh)
 	buf, err := d.FullPCMBuffer()
 	if err != nil {
@@ -120,8 +203,6 @@ func (w *Whisper) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptR
 	}
 
 	data := buf.AsFloat32Buffer().Data
-	// whisper.cpp resamples to 16 kHz internally; this matches buf.Format.SampleRate
-	// for the converted file produced by AudioToWav above.
 	var duration float32
 	if buf.Format != nil && buf.Format.SampleRate > 0 {
 		duration = float32(len(data)) / float32(buf.Format.SampleRate)
@@ -129,7 +210,31 @@ func (w *Whisper) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptR
 	segsLen := uintptr(0xdeadbeef)
 	segsLenPtr := unsafe.Pointer(&segsLen)
 
-	if ret := CppTranscribe(opts.Threads, opts.Language, opts.Translate, opts.Diarize, data, uintptr(len(data)), segsLenPtr, opts.Prompt); ret != 0 {
+	// Watcher: flips the C-side abort flag when ctx is cancelled. The
+	// goroutine is joined synchronously (close(done) signals it to exit,
+	// wg.Wait() blocks until it has) so a late CppSetAbort(1) cannot fire
+	// after the function returns and corrupt the next transcription call.
+	done := make(chan struct{})
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		select {
+		case <-ctx.Done():
+			CppSetAbort(1)
+		case <-done:
+		}
+	}()
+	defer func() {
+		close(done)
+		wg.Wait()
+	}()
+
+	ret := CppTranscribe(opts.Threads, opts.Language, opts.Translate, opts.Diarize, data, uintptr(len(data)), segsLenPtr, opts.Prompt)
+	if ret == 2 {
+		return pb.TranscriptResult{}, status.Error(codes.Canceled, "transcription cancelled")
+	}
+	if ret != 0 {
 		return pb.TranscriptResult{}, fmt.Errorf("Failed Transcribe")
 	}
 
@@ -171,3 +276,120 @@ func (w *Whisper) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptR
 		Duration: duration,
 	}, nil
 }
+
+// AudioTranscriptionStream runs whisper_full() and emits deltas via
+// whisper.cpp's new_segment_callback as segments are decoded, then a
+// final TranscriptResult. The offline AudioTranscription is unchanged;
+// both paths share whisper's single-instance ctx and the SingleThread
+// concurrency model.
+func (w *Whisper) AudioTranscriptionStream(ctx context.Context, opts *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
+	defer close(results)
+
+	if err := ctx.Err(); err != nil {
+		return status.Error(codes.Canceled, "transcription cancelled")
+	}
+
+	dir, err := os.MkdirTemp("", "whisper")
+	if err != nil {
+		return err
+	}
+	defer func() { _ = os.RemoveAll(dir) }()
+
+	convertedPath := filepath.Join(dir, "converted.wav")
+	if err := utils.AudioToWav(opts.Dst, convertedPath); err != nil {
+		return err
+	}
+
+	fh, err := os.Open(convertedPath)
+	if err != nil {
+		return err
+	}
+	defer func() { _ = fh.Close() }()
+
+	d := wav.NewDecoder(fh)
+	buf, err := d.FullPCMBuffer()
+	if err != nil {
+		return err
+	}
+	data := buf.AsFloat32Buffer().Data
+	var duration float32
+	if buf.Format != nil && buf.Format.SampleRate > 0 {
+		duration = float32(len(data)) / float32(buf.Format.SampleRate)
+	}
+
+	// Register per-call state and install the C-side callback. defer
+	// teardown so even a panic clears the C pointer (otherwise a stale
+	// callback fires on the next AudioTranscription call).
+	callID := streamCallSeq.Add(1)
+	state := &streamCallState{
+		results: results,
+		diarize: opts.Diarize,
+	}
+	streamCallStates.Store(callID, state)
+	CppSetNewSegmentCallback(goNewSegmentCb, uintptr(callID))
+	defer func() {
+		CppSetNewSegmentCallback(0, 0)
+		streamCallStates.Delete(callID)
+	}()
+
+	// Same abort-watcher pattern as AudioTranscription. Joined synchronously
+	// so a late CppSetAbort(1) cannot fire after this function returns.
+	done := make(chan struct{})
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		select {
+		case <-ctx.Done():
+			CppSetAbort(1)
+		case <-done:
+		}
+	}()
+	defer func() {
+		close(done)
+		wg.Wait()
+	}()
+
+	segsLen := uintptr(0xdeadbeef)
+	segsLenPtr := unsafe.Pointer(&segsLen)
+	ret := CppTranscribe(opts.Threads, opts.Language, opts.Translate, opts.Diarize, data, uintptr(len(data)), segsLenPtr, opts.Prompt)
+	if ret == 2 {
+		return status.Error(codes.Canceled, "transcription cancelled")
+	}
+	if ret != 0 {
+		return fmt.Errorf("Failed Transcribe")
+	}
+
+	// Build the final TranscriptResult. Segments[] mirrors the offline
+	// path so the SSE done event carries the same per-segment shape.
+	// final.Text reuses the assembled stream so concat(deltas) == final.Text
+	// holds exactly, matching the e2e contract.
+	segments := []*pb.TranscriptSegment{}
+	for i := range int(segsLen) {
+		s := CppGetSegmentStart(i) * 10000000
+		t := CppGetSegmentEnd(i) * 10000000
+		txt := strings.ToValidUTF8(strings.Clone(CppGetSegmentText(i)), "<22>")
+		tokens := make([]int32, CppNTokens(i))
+		if opts.Diarize && CppGetSegmentSpeakerTurnNext(i) {
+			txt += " [SPEAKER_TURN]"
+		}
+		for j := range tokens {
+			tokens[j] = int32(CppGetTokenID(i, j))
+		}
+		segments = append(segments, &pb.TranscriptSegment{
+			Id:    int32(i),
+			Text:  txt,
+			Start: s, End: t,
+			Tokens: tokens,
+		})
+	}
+
+	final := &pb.TranscriptResult{
+		Segments: segments,
+		Text:     state.assembled.String(),
+		Language: opts.Language,
+		Duration: duration,
+	}
+	results <- &pb.TranscriptStreamResponse{FinalResult: final}
+	return nil
+}

backend/go/whisper/gowhisper_test.go

@@ -0,0 +1,174 @@
+package main
+
+import (
+	"context"
+	"os"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+func TestWhisper(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Whisper Backend Suite")
+}
+
+var (
+	libLoadOnce sync.Once
+	libLoadErr  error
+)
+
+// ensureLibLoaded mirrors main.go's bootstrap so a Go test can drive the
+// bridge without spinning up the gRPC server. Skips the current spec when the
+// shared library isn't present (e.g. running before `make backends/whisper`).
+func ensureLibLoaded() {
+	libLoadOnce.Do(func() {
+		libName := os.Getenv("WHISPER_LIBRARY")
+		if libName == "" {
+			libName = "./libgowhisper-fallback.so"
+		}
+		if _, err := os.Stat(libName); err != nil {
+			libLoadErr = err
+			return
+		}
+		gosd, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+		if err != nil {
+			libLoadErr = err
+			return
+		}
+		purego.RegisterLibFunc(&CppLoadModel, gosd, "load_model")
+		purego.RegisterLibFunc(&CppTranscribe, gosd, "transcribe")
+		purego.RegisterLibFunc(&CppGetSegmentText, gosd, "get_segment_text")
+		purego.RegisterLibFunc(&CppGetSegmentStart, gosd, "get_segment_t0")
+		purego.RegisterLibFunc(&CppGetSegmentEnd, gosd, "get_segment_t1")
+		purego.RegisterLibFunc(&CppNTokens, gosd, "n_tokens")
+		purego.RegisterLibFunc(&CppGetTokenID, gosd, "get_token_id")
+		purego.RegisterLibFunc(&CppGetSegmentSpeakerTurnNext, gosd, "get_segment_speaker_turn_next")
+		purego.RegisterLibFunc(&CppSetAbort, gosd, "set_abort")
+		purego.RegisterLibFunc(&CppSetNewSegmentCallback, gosd, "set_new_segment_callback")
+	})
+	if libLoadErr != nil {
+		Skip("whisper library not loadable: " + libLoadErr.Error())
+	}
+}
+
+// fixturesOrSkip returns the model + audio paths or skips the spec if either
+// env var is unset. The test never runs in default CI — it requires a real
+// whisper model and a long audio file (~3 minutes) on disk.
+func fixturesOrSkip() (string, string) {
+	modelPath := os.Getenv("WHISPER_MODEL_PATH")
+	audioPath := os.Getenv("WHISPER_AUDIO_PATH")
+	if modelPath == "" || audioPath == "" {
+		Skip("set WHISPER_MODEL_PATH and WHISPER_AUDIO_PATH to run this spec")
+	}
+	return modelPath, audioPath
+}
+
+var _ = Describe("Whisper", func() {
+	Context("AudioTranscription cancellation", func() {
+		It("returns codes.Canceled and resets the abort flag for the next call", func() {
+			modelPath, audioPath := fixturesOrSkip()
+			ensureLibLoaded()
+
+			w := &Whisper{}
+			Expect(w.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
+
+			ctx, cancel := context.WithCancel(context.Background())
+			go func() {
+				time.Sleep(100 * time.Millisecond)
+				cancel()
+			}()
+
+			start := time.Now()
+			_, err := w.AudioTranscription(ctx, &pb.TranscriptRequest{
+				Dst:      audioPath,
+				Threads:  4,
+				Language: "en",
+			})
+			elapsed := time.Since(start)
+
+			Expect(err).To(HaveOccurred(), "transcription completed in %s without cancel — try a longer audio file", elapsed)
+			st, ok := status.FromError(err)
+			Expect(ok).To(BeTrue(), "expected gRPC status error, got %v", err)
+			Expect(st.Code()).To(Equal(codes.Canceled), "expected codes.Canceled, got %v", err)
+			Expect(elapsed).To(BeNumerically("<", 5*time.Second), "cancellation took %s, expected <5s", elapsed)
+
+			// Subsequent transcription must succeed — proves g_abort reset.
+			res, err := w.AudioTranscription(context.Background(), &pb.TranscriptRequest{
+				Dst:      audioPath,
+				Threads:  4,
+				Language: "en",
+			})
+			Expect(err).ToNot(HaveOccurred(), "post-cancel transcription failed")
+			Expect(res.Text).ToNot(BeEmpty(), "post-cancel transcription returned empty text")
+		})
+	})
+
+	Context("AudioTranscriptionStream", func() {
+		It("emits multiple deltas progressively for a multi-segment clip", func() {
+			modelPath, audioPath := fixturesOrSkip()
+			ensureLibLoaded()
+
+			// The streaming method dispatches through the package-level
+			// goNewSegmentCb. main.go normally builds it; in this test
+			// process main() is never called, so build it here lazily.
+			// purego.NewCallback returns a stable pointer; calling it once
+			// per process is correct.
+			if goNewSegmentCb == 0 {
+				goNewSegmentCb = purego.NewCallback(onNewSegment)
+			}
+
+			w := &Whisper{}
+			Expect(w.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
+
+			results := make(chan *pb.TranscriptStreamResponse, 64)
+			done := make(chan error, 1)
+			go func() {
+				done <- w.AudioTranscriptionStream(context.Background(), &pb.TranscriptRequest{
+					Dst:      audioPath,
+					Threads:  4,
+					Language: "en",
+					Stream:   true,
+				}, results)
+			}()
+
+			var deltas []string
+			var assembled strings.Builder
+			var finalText string
+			var finalSegmentCount int
+			for chunk := range results {
+				if d := chunk.GetDelta(); d != "" {
+					deltas = append(deltas, d)
+					assembled.WriteString(d)
+				}
+				if final := chunk.GetFinalResult(); final != nil {
+					finalText = final.GetText()
+					finalSegmentCount = len(final.GetSegments())
+				}
+			}
+			Expect(<-done).ToNot(HaveOccurred())
+
+			// The whisper-specific bar: real streaming via new_segment_callback
+			// fires once per decoded segment, so a multi-segment clip MUST
+			// produce >=2 delta events. A faked-streaming impl (run
+			// whisper_full to completion, then walk the segment list) would
+			// also pass len(deltas) >= 1, which is why the generic e2e spec
+			// is not strict enough.
+			Expect(len(deltas)).To(BeNumerically(">=", 2),
+				"expected multiple deltas from a multi-segment clip, got %d (assembled=%q)",
+				len(deltas), assembled.String())
+			Expect(finalSegmentCount).To(BeNumerically(">=", 2),
+				"expected final to carry multiple segments")
+			Expect(assembled.String()).To(Equal(finalText),
+				"concat(deltas) must equal final.Text")
+		})
+	})
+})

backend/go/whisper/main.go

@@ -41,12 +41,19 @@ func main() {
 		{&CppNTokens, "n_tokens"},
 		{&CppGetTokenID, "get_token_id"},
 		{&CppGetSegmentSpeakerTurnNext, "get_segment_speaker_turn_next"},
+		{&CppSetAbort, "set_abort"},
+		{&CppSetNewSegmentCallback, "set_new_segment_callback"},
 	}
 
 	for _, lf := range libFuncs {
 		purego.RegisterLibFunc(lf.FuncPtr, gosd, lf.Name)
 	}
 
+	// Build a stable C-callable function pointer from the Go callback. The
+	// pointer lives for the lifetime of the process; per-call dispatch is
+	// keyed by user_data through streamCallStates.
+	goNewSegmentCb = purego.NewCallback(onNewSegment)
+
 	flag.Parse()
 
 	if err := grpc.StartServer(*addr, &Whisper{}); err != nil {

backend/index.yaml

@@ -287,6 +287,7 @@
     amd: "rocm-sglang"
     intel: "intel-sglang"
     nvidia-cuda-12: "cuda12-sglang"
+    nvidia-cuda-13: "cuda13-sglang"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-sglang"
     cpu: "cpu-sglang"
 - &vllm-omni
@@ -600,6 +601,38 @@
     nvidia-l4t: "nvidia-l4t-arm64-vibevoice-cpp"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-vibevoice-cpp"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vibevoice-cpp"
+- &localvqecpp
+  name: "localvqe"
+  description: |
+    LocalVQE C++ backend using GGML — joint acoustic echo cancellation, noise
+    suppression, and dereverberation (DeepVQE-style architecture). 16 kHz mono
+    in / out, supports both batch and low-latency streaming. Implements the
+    audio-transform capability.
+  urls:
+    - https://github.com/localai-org/LocalVQE
+  tags:
+    - audio-transform
+    - aec
+    - acoustic-echo-cancellation
+    - noise-suppression
+    - dereverberation
+  license: apache2
+  alias: "localvqe"
+  # Upstream LocalVQE only supports CPU and Vulkan; no CUDA/ROCm/SYCL/Metal
+  # builds. GPU-class hardware that exposes a Vulkan ICD (NVIDIA, AMD, Intel
+  # discrete + iGPU, Tegra) routes to the Vulkan image; everything else
+  # falls back to the CPU build, which is already ~9× realtime on a desktop.
+  capabilities:
+    default: "cpu-localvqe"
+    nvidia: "vulkan-localvqe"
+    nvidia-cuda-12: "vulkan-localvqe"
+    nvidia-cuda-13: "vulkan-localvqe"
+    intel: "vulkan-localvqe"
+    amd: "vulkan-localvqe"
+    vulkan: "vulkan-localvqe"
+    nvidia-l4t: "vulkan-localvqe"
+    nvidia-l4t-cuda-12: "vulkan-localvqe"
+    nvidia-l4t-cuda-13: "vulkan-localvqe"
 - &faster-whisper
   icon: https://avatars.githubusercontent.com/u/1520500?s=200&v=4
   description: |
@@ -1933,6 +1966,7 @@
     amd: "rocm-sglang-development"
     intel: "intel-sglang-development"
     nvidia-cuda-12: "cuda12-sglang-development"
+    nvidia-cuda-13: "cuda13-sglang-development"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-sglang-development"
     cpu: "cpu-sglang-development"
 - !!merge <<: *sglang
@@ -1940,6 +1974,11 @@
   uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-sglang"
   mirrors:
     - localai/localai-backends:latest-gpu-nvidia-cuda-12-sglang
+- !!merge <<: *sglang
+  name: "cuda13-sglang"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-sglang"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-sglang
 - !!merge <<: *sglang
   name: "cuda13-nvidia-l4t-arm64-sglang"
   uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-sglang"
@@ -1965,6 +2004,11 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-sglang"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-12-sglang
+- !!merge <<: *sglang
+  name: "cuda13-sglang-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-sglang"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-sglang
 - !!merge <<: *sglang
   name: "cuda13-nvidia-l4t-arm64-sglang-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-sglang"
@@ -2785,6 +2829,27 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vibevoice-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-vibevoice-cpp
+## localvqe
+- !!merge <<: *localvqecpp
+  name: "cpu-localvqe"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-localvqe"
+  mirrors:
+    - localai/localai-backends:latest-cpu-localvqe
+- !!merge <<: *localvqecpp
+  name: "cpu-localvqe-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-localvqe"
+  mirrors:
+    - localai/localai-backends:master-cpu-localvqe
+- !!merge <<: *localvqecpp
+  name: "vulkan-localvqe"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-localvqe"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-localvqe
+- !!merge <<: *localvqecpp
+  name: "vulkan-localvqe-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-localvqe"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-localvqe
 ## kokoro
 - !!merge <<: *kokoro
   name: "kokoro-development"

backend/python/chatterbox/install.sh

@@ -23,3 +23,15 @@ fi
 
 
 installRequirements
+
+# chatterbox-tts upstream pulls `russian-text-stresser` (unpinned git URL) which
+# transitively pins spacy==3.6.* and other ancient packages. That cascade forces
+# pip to backtrack through Jinja2/MarkupSafe/omegaconf/ruamel.yaml into Python-2-era
+# sdists that no longer build. We install chatterbox-tts itself with --no-deps and
+# list its real runtime deps in requirements-*.txt instead.
+echo "Installing chatterbox-tts with --no-deps"
+if [ "x${USE_PIP}" == "xtrue" ]; then
+    pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --no-deps "chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster"
+else
+    uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --no-deps "chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster"
+fi

backend/python/chatterbox/requirements-cpu.txt

@@ -4,6 +4,16 @@ torch
 torchaudio
 numpy>=1.24.0,<1.26.0
 transformers
-# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
-#chatterbox-tts==0.1.4
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0

backend/python/chatterbox/requirements-cublas12.txt

@@ -2,6 +2,17 @@ torch
 torchaudio
 transformers
 numpy>=1.24.0,<1.26.0
-# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0
 accelerate

backend/python/chatterbox/requirements-cublas13.txt

@@ -3,6 +3,17 @@ torch
 torchaudio
 transformers
 numpy>=1.24.0,<1.26.0
-# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0
 accelerate

backend/python/chatterbox/requirements-hipblas.txt

@@ -3,6 +3,17 @@ torch==2.10.0+rocm7.0
 torchaudio==2.10.0+rocm7.0
 transformers
 numpy>=1.24.0,<1.26.0
-# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0
 accelerate

backend/python/chatterbox/requirements-intel.txt

@@ -3,8 +3,19 @@ torch
 torchaudio
 transformers
 numpy>=1.24.0,<1.26.0
-# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0
 accelerate
 oneccl_bind_pt==2.3.100+xpu
 optimum[openvino]

backend/python/chatterbox/requirements-l4t12.txt

@@ -3,5 +3,17 @@ torch
 torchaudio
 transformers
 numpy>=1.24.0,<1.26.0
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0
 accelerate

backend/python/chatterbox/requirements-l4t13.txt

@@ -3,5 +3,17 @@ torch
 torchaudio
 transformers
 numpy>=1.24.0,<1.26.0
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0
 accelerate

backend/python/chatterbox/requirements-mps.txt

@@ -3,5 +3,16 @@ torchaudio
 accelerate
 numpy>=1.24.0,<1.26.0
 transformers
-# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
-chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
+# chatterbox-tts itself is installed with --no-deps in install.sh.
+# These are its real runtime deps, mirroring upstream's pyproject.toml
+# minus russian-text-stresser (whose ancient pins break the resolver).
+omegaconf==2.3.0
+resampy==0.4.3
+librosa
+s3tokenizer
+diffusers
+resemble-perth==1.0.1
+conformer
+safetensors
+spacy-pkuseg
+pykakasi==2.3.0

backend/python/common/libbackend.sh

@@ -318,6 +318,21 @@ _makeVenvPortable() {
 }
 
 
+# Apply the venv to the current process: VIRTUAL_ENV, PATH, PYTHONHOME hygiene.
+# Equivalent to the runtime portion of `source bin/activate`, but computed from
+# $EDIR (resolved at runtime via realpath) instead of the path baked into
+# bin/activate at venv-create time. `uv venv` (and `python -m venv`) both bake
+# the create-time absolute path in, so sourcing activate on a relocated venv —
+# e.g. one built at /vllm/venv inside a Docker stage and unpacked under
+# /backends/cuda13-vllm-development/venv at runtime — silently prepends a
+# stale, non-existent path to $PATH. Doing the setup ourselves sidesteps that;
+# this is the same approach `uv run` takes internally.
+_activateVenv() {
+    export VIRTUAL_ENV="${EDIR}/venv"
+    export PATH="${EDIR}/venv/bin:${PATH}"
+    unset PYTHONHOME
+}
+
 # ensureVenv makes sure that the venv for the backend both exists, and is activated.
 #
 # This function is idempotent, so you can call it as many times as you want and it will
@@ -354,7 +369,7 @@ function ensureVenv() {
                 venv_args="--copies"
             fi
             "${interpreter}" -m venv ${venv_args} "${EDIR}/venv"
-            source "${EDIR}/venv/bin/activate"
+            _activateVenv
             "${interpreter}" -m pip install --upgrade pip
         else
             if [ "x${PORTABLE_PYTHON}" == "xtrue" ]; then
@@ -375,7 +390,7 @@ function ensureVenv() {
     fi
 
     if [ "x${VIRTUAL_ENV:-}" != "x${EDIR}/venv" ]; then
-        source "${EDIR}/venv/bin/activate"
+        _activateVenv
     fi
 }
 

backend/python/faster-whisper/Makefile

@@ -10,4 +10,5 @@ protogen-clean:
 
 .PHONY: clean
 clean: protogen-clean
-	rm -rf venv __pycache__
+	rm -rf venv __pycache__
+# trigger per-arch+merge rebuild for faster-whisper pilot

backend/python/faster-whisper/backend.py

@@ -55,11 +55,27 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         resultSegments = []
         text = ""
         try:
-            segments, info = self.model.transcribe(request.dst, beam_size=5, condition_on_previous_text=False)
+            word_timestamps = "word" in request.timestamp_granularities
+            segments, info = self.model.transcribe(request.dst, beam_size=5, condition_on_previous_text=False, word_timestamps=word_timestamps)
             id = 0
             for segment in segments:
                 print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
-                resultSegments.append(backend_pb2.TranscriptSegment(id=id, start=int(segment.start)*1e9, end=int(segment.end)*1e9, text=segment.text))
+                words = []
+                if word_timestamps and hasattr(segment, 'words'):
+                    for word in segment.words:
+                        words.append(backend_pb2.TranscriptWord(
+                            start=int(word.start * 1e9),
+                            end=int(word.end * 1e9),
+                            text=word.word
+                        ))
+
+                resultSegments.append(backend_pb2.TranscriptSegment(
+                    id=id,
+                    start=int(segment.start * 1e9),
+                    end=int(segment.end * 1e9),
+                    text=segment.text,
+                    words=words
+                ))
                 text += segment.text
                 id += 1
         except Exception as err:

backend/python/rerankers/requirements-cpu.txt

@@ -1,4 +1,4 @@
 transformers
 accelerate
-torch==2.4.1
+torch==2.7.1
 rerankers[transformers]

backend/python/rerankers/requirements-cublas12.txt

@@ -1,4 +1,4 @@
 transformers
 accelerate
-torch==2.4.1
+torch==2.7.1
 rerankers[transformers]

backend/python/sglang/Makefile

@@ -8,6 +8,12 @@ run: sglang
 	bash run.sh
 	@echo "sglang run."
 
+.PHONY: test
+test: sglang
+	@echo "Testing sglang..."
+	bash test.sh
+	@echo "sglang tested."
+
 .PHONY: protogen-clean
 protogen-clean:
 	$(RM) backend_pb2_grpc.py backend_pb2.py

backend/python/sglang/backend.py

@@ -9,10 +9,18 @@ The streaming path applies sglang's per-request FunctionCallParser and
 ReasoningParser so tool_calls and reasoning_content are emitted
 incrementally inside ChatDelta, which is a capability sglang exposes
 natively and vLLM does not.
+
+Like the vLLM backend, this one accepts an arbitrary ``engine_args:``
+map in the model YAML; keys are validated against ``ServerArgs`` fields
+and forwarded to ``Engine(**kwargs)``. That covers speculative decoding
+(EAGLE/EAGLE3/DFLASH/NGRAM/STANDALONE plus MTP via NEXTN), attention
+backend selection, MoE knobs, hierarchical cache, and so on.
 """
 import asyncio
 from concurrent import futures
 import argparse
+import dataclasses
+import difflib
 import signal
 import sys
 import os
@@ -38,6 +46,7 @@ from grpc_auth import get_auth_interceptors
 # are wrapped in try/except so older / leaner installs that omit them
 # still load the backend for plain text generation.
 from sglang.srt.entrypoints.engine import Engine
+from sglang.srt.server_args import ServerArgs
 
 try:
     from sglang.srt.function_call.function_call_parser import FunctionCallParser
@@ -66,6 +75,19 @@ except Exception:
     HAS_TRANSFORMERS = False
 
 
+# sglang 0.5.11 renamed SamplingParams.seed -> sampling_seed (PR #21952).
+# Earlier 0.5.x releases (e.g. 0.5.1.post2 — the wheel still pinned by the
+# pypi.jetson-ai-lab.io sbsa/cu130 mirror used by the l4t13 build profile)
+# accept only `seed`. Detect the supported keyword once at import time so
+# both versions work without a hard pin floor.
+try:
+    import inspect as _inspect
+    from sglang.srt.sampling.sampling_params import SamplingParams as _SamplingParams
+    _SEED_KEY = "sampling_seed" if "sampling_seed" in _inspect.signature(_SamplingParams).parameters else "seed"
+except Exception:
+    _SEED_KEY = "sampling_seed"
+
+
 _ONE_DAY_IN_SECONDS = 60 * 60 * 24
 MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1'))
 
@@ -82,6 +104,37 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             opts[key.strip()] = value.strip()
         return opts
 
+    def _apply_engine_args(self, engine_kwargs: dict, engine_args_json: str) -> dict:
+        """Merge user-supplied engine_args (JSON object) into the kwargs dict
+        that will be forwarded to ``sglang.Engine`` (which constructs a
+        ``ServerArgs`` from them).
+
+        Mirrors ``backend/python/vllm/backend.py::_apply_engine_args`` but
+        operates on the kwargs dict because sglang's ``Engine.__init__``
+        accepts ``**kwargs`` directly rather than a pre-built dataclass.
+        Validation happens against ``ServerArgs`` fields so a typo fails
+        early with a close-match suggestion instead of producing a confusing
+        ``TypeError`` deep inside engine startup.
+        """
+        if not engine_args_json:
+            return engine_kwargs
+        try:
+            extra = json.loads(engine_args_json)
+        except json.JSONDecodeError as e:
+            raise ValueError(f"engine_args is not valid JSON: {e}") from e
+        if not isinstance(extra, dict):
+            raise ValueError(
+                f"engine_args must be a JSON object, got {type(extra).__name__}"
+            )
+        valid = {f.name for f in dataclasses.fields(ServerArgs)}
+        for key in extra:
+            if key not in valid:
+                suggestion = difflib.get_close_matches(key, valid, n=1)
+                hint = f" did you mean {suggestion[0]!r}?" if suggestion else ""
+                raise ValueError(f"unknown engine_args key {key!r}.{hint}")
+        engine_kwargs.update(extra)
+        return engine_kwargs
+
     def _messages_to_dicts(self, messages) -> List[dict]:
         result: List[dict] = []
         for msg in messages:
@@ -137,6 +190,16 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         if self.reasoning_parser_name:
             engine_kwargs["reasoning_parser"] = self.reasoning_parser_name
 
+        # engine_args from YAML overrides typed fields above so operators can
+        # tune anything ServerArgs exposes (speculative decoding, attention
+        # backend, MoE, hierarchical cache, …) without waiting on protobuf
+        # changes.
+        try:
+            engine_kwargs = self._apply_engine_args(engine_kwargs, request.EngineArgs)
+        except ValueError as err:
+            print(f"engine_args error: {err}", file=sys.stderr)
+            return backend_pb2.Result(success=False, message=str(err))
+
         try:
             self.llm = Engine(**engine_kwargs)
         except Exception as err:
@@ -221,7 +284,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             "TopP": "top_p",
             "TopK": "top_k",
             "MinP": "min_p",
-            "Seed": "seed",
+            "Seed": _SEED_KEY,
             "StopPrompts": "stop",
             "StopTokenIds": "stop_token_ids",
             "IgnoreEOS": "ignore_eos",

backend/python/sglang/install.sh

@@ -23,17 +23,32 @@ if [ "x${BUILD_PROFILE}" == "xcpu" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
+# cublas12 needs a cu128 torch index (see requirements-cublas12.txt) — without
+# unsafe-best-match uv falls through to default PyPI's cu130 torch wheel and
+# the resulting sgl-kernel can't load on our cu12 host libs.
+if [ "x${BUILD_PROFILE}" == "xcublas12" ]; then
+    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
+fi
+
+# sglang 0.5.11 (Gemma 4 support) declares flash-attn-4 as a hard dep, but
+# upstream only publishes pre-release wheels (4.0.0b*). uv rejects
+# pre-releases by default — opt in for sglang specifically. Drop this once
+# flash-attn-4 4.0 stable lands.
+EXTRA_PIP_INSTALL_FLAGS+=" --prerelease=allow"
+
 # JetPack 7 / L4T arm64 wheels are built for cp312 and shipped via
 # pypi.jetson-ai-lab.io. Bump the venv Python so the prebuilt sglang
-# wheel resolves cleanly. unsafe-best-match is required because the
-# jetson-ai-lab index lists transitive deps (e.g. decord) at older
-# versions only — without it uv refuses to fall through to PyPI for a
-# compatible wheel and resolution fails.
+# wheel resolves cleanly. The actual install on l4t13 goes through
+# pyproject.toml (see the elif branch below) so [tool.uv.sources] can
+# pin only torch/torchvision/torchaudio/sglang to the jetson-ai-lab
+# index — leaving PyPI as the path for transitive deps like
+# markdown-it-py / anthropic / propcache that the L4T mirror's proxy
+# 503s on. No --index-strategy flag here: the explicit index keeps the
+# scoping clean.
 if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
     PYTHON_VERSION="3.12"
     PYTHON_PATCH="12"
     PY_STANDALONE_TAG="20251120"
-    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
 # sglang's CPU path has no prebuilt wheel on PyPI — upstream publishes
@@ -95,6 +110,27 @@ if [ "x${BUILD_TYPE}" == "x" ] || [ "x${FROM_SOURCE:-}" == "xtrue" ]; then
         fi
         uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} .
     popd
+# L4T arm64 (JetPack 7): drive the install through pyproject.toml so that
+# [tool.uv.sources] can pin torch/torchvision/torchaudio/sglang to the
+# jetson-ai-lab index, while everything else (transitive deps and
+# PyPI-resolvable packages like transformers / accelerate) comes from
+# PyPI. Bypasses installRequirements because uv pip install -r
+# requirements.txt does not honor sources — see
+# backend/python/sglang/pyproject.toml for the rationale. Mirrors the
+# equivalent path in backend/python/vllm/install.sh.
+elif [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    ensureVenv
+    if [ "x${PORTABLE_PYTHON}" == "xtrue" ]; then
+        export C_INCLUDE_PATH="${C_INCLUDE_PATH:-}:$(_portable_dir)/include/python${PYTHON_VERSION}"
+    fi
+    pushd "${backend_dir}"
+        # Build deps first (matches installRequirements' requirements-install.txt
+        # pass — sglang/sgl-kernel sdists need packaging/setuptools-scm in the
+        # venv before they can build under --no-build-isolation).
+        uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} -r requirements-install.txt
+        uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement pyproject.toml
+    popd
+    runProtogen
 else
     installRequirements
 fi

backend/python/sglang/pyproject.toml

@@ -0,0 +1,68 @@
+# L4T arm64 (JetPack 7 / sbsa cu130) install spec for the sglang backend.
+#
+# Why this file exists, and why only the l4t13 BUILD_PROFILE consumes it:
+#
+# pypi.jetson-ai-lab.io hosts the L4T-specific torch / sglang / sgl-kernel
+# wheels we need on aarch64 + cuda13, but it ALSO transparently proxies the
+# rest of PyPI through `/+f/<sha>/<filename>` URLs that 503 frequently.
+# With `--extra-index-url` + `--index-strategy=unsafe-best-match` (the
+# historical fix in install.sh) uv would pick those proxy URLs for ordinary
+# PyPI packages — markdown-it-py, anthropic, propcache, etc. — and trip on
+# the 503s. See e.g. CI run 25439791228 (markdown-it-py-4.0.0).
+#
+# `explicit = true` on the index makes uv consult the L4T mirror ONLY for
+# packages mapped under [tool.uv.sources]. Everything else goes to PyPI.
+# This breaks the historical 503 path without losing access to the L4T
+# wheels we actually need from there. Mirrors the equivalent fix already
+# in backend/python/vllm/pyproject.toml.
+#
+# `uv pip install -r requirements.txt` does NOT honor [tool.uv.sources]
+# (sources are project-mode only, not pip-compat mode), so install.sh's
+# l4t13 branch invokes `uv pip install --requirement pyproject.toml`
+# directly. Other BUILD_PROFILEs continue to use the requirements-*.txt
+# pipeline through libbackend.sh's installRequirements and never read
+# this file.
+[project]
+name = "localai-sglang-l4t13"
+version = "0.0.0"
+requires-python = ">=3.12,<3.13"
+dependencies = [
+    # Mirror of requirements.txt — kept in sync manually for now since the
+    # l4t13 path bypasses installRequirements (see install.sh).
+    "grpcio==1.80.0",
+    "protobuf",
+    "certifi",
+    "setuptools",
+    "pillow",
+    # L4T-specific accelerator stack (sourced from jetson-ai-lab below).
+    "torch",
+    "torchvision",
+    "torchaudio",
+    # sglang on jetson — the [all] extra is deliberately omitted because it
+    # pulls outlines/decord, and decord has no aarch64 cp312 wheel anywhere
+    # (PyPI nor the jetson-ai-lab index ships only legacy cp35-cp37). With
+    # [all] uv backtracks through versions trying to satisfy decord and
+    # lands on sglang==0.1.16. The 0.5.0 floor matches the only major
+    # series the jetson-ai-lab sbsa/cu130 mirror currently publishes
+    # (sglang==0.5.1.post2 as of 2026-05-06). Bumping to >=0.5.11 here
+    # would make the build unsatisfiable until the mirror catches up.
+    # Gemma 4 / MTP recipes are therefore not supported on l4t13 — those
+    # features land on cublas12/cublas13 hosts that pull the newer wheel
+    # from PyPI. backend.py keeps backward compat with the 0.5.x SamplingParams
+    # field rename via runtime detection.
+    "sglang>=0.5.0",
+    # PyPI-resolvable packages that complete the runtime.
+    "accelerate",
+    "transformers",
+]
+
+[[tool.uv.index]]
+name = "jetson-ai-lab"
+url = "https://pypi.jetson-ai-lab.io/sbsa/cu130"
+explicit = true
+
+[tool.uv.sources]
+torch = { index = "jetson-ai-lab" }
+torchvision = { index = "jetson-ai-lab" }
+torchaudio = { index = "jetson-ai-lab" }
+sglang = { index = "jetson-ai-lab" }

backend/python/sglang/requirements-cublas12-after.txt

@@ -1,3 +1,4 @@
 # Bump this pin deliberately — sglang releases weekly and API surfaces
 # (FunctionCallParser, ReasoningParser) move between releases.
-sglang[all]>=0.4.0
+# 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
+sglang[all]>=0.5.11

backend/python/sglang/requirements-cublas12.txt

@@ -1,5 +1,12 @@
+# sglang 0.5.11 hard-pins torch==2.9.1. PyPI's default torch 2.9.1 wheel is
+# now the cu130 build, which drags in cu130-flavoured sgl-kernel/sglang-kernel
+# binaries that need libnvrtc.so.13 — incompatible with our cu12 host libs.
+# Pin the cu128 PyTorch index so uv pulls cu12-flavoured torch (and the
+# matching sgl-kernel cu12 wheels). install.sh adds --index-strategy=unsafe-best-match
+# for cublas12 so uv consults this index alongside PyPI.
+--extra-index-url https://download.pytorch.org/whl/cu128
 accelerate
-torch==2.7.1
+torch==2.9.1
 torchvision
-torchaudio==2.7.1
+torchaudio
 transformers

backend/python/sglang/requirements-cublas13-after.txt

@@ -0,0 +1,4 @@
+# Bump this pin deliberately — sglang releases weekly and API surfaces
+# (FunctionCallParser, ReasoningParser) move between releases.
+# 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
+sglang[all]>=0.5.11

backend/python/sglang/requirements-cublas13.txt

@@ -0,0 +1,6 @@
+--extra-index-url https://download.pytorch.org/whl/cu130
+accelerate
+torch
+torchvision
+torchaudio
+transformers

backend/python/sglang/requirements-l4t13.txt

@@ -1,12 +0,0 @@
---extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
-accelerate
-torch
-torchvision
-torchaudio
-transformers
-# Drop the [all] extra: it pulls outlines/decord, and decord has no
-# aarch64 cp312 wheel anywhere (PyPI nor the jetson-ai-lab index ships
-# only legacy cp35-cp37). With [all] uv backtracks through versions
-# trying to satisfy decord and lands on sglang==0.1.16. Floor at 0.5.0
-# so uv can't silently downgrade if a future resolution misfires.
-sglang>=0.5.0

backend/python/sglang/test.py

@@ -0,0 +1,101 @@
+"""Unit tests for the sglang backend.
+
+Helper-level tests run without launching the gRPC server or loading model
+weights — they only exercise the pure-Python helpers on
+``BackendServicer``. They do still require ``sglang`` to be importable
+because ``_apply_engine_args`` validates keys against
+``ServerArgs``'s dataclass fields.
+"""
+import unittest
+
+
+class TestSglangHelpers(unittest.TestCase):
+    """Tests for the pure helpers on BackendServicer (no gRPC, no engine)."""
+
+    def _servicer(self):
+        import sys
+        import os
+        sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+        from backend import BackendServicer  # noqa: E402
+        return BackendServicer()
+
+    def test_parse_options(self):
+        servicer = self._servicer()
+        opts = servicer._parse_options([
+            "tool_parser:hermes",
+            "reasoning_parser:deepseek_r1",
+            "invalid_no_colon",
+            "key_with_colons:a:b:c",
+        ])
+        self.assertEqual(opts["tool_parser"], "hermes")
+        self.assertEqual(opts["reasoning_parser"], "deepseek_r1")
+        self.assertEqual(opts["key_with_colons"], "a:b:c")
+        self.assertNotIn("invalid_no_colon", opts)
+
+    def test_apply_engine_args_known_keys(self):
+        """User-supplied JSON merges into the kwargs dict; pre-set typed
+        fields stay put when not overridden."""
+        import json as _json
+        servicer = self._servicer()
+        base = {
+            "model_path": "facebook/opt-125m",
+            "mem_fraction_static": 0.7,
+        }
+        extras = _json.dumps({
+            "trust_remote_code": True,
+            "speculative_algorithm": "EAGLE",
+            "speculative_num_steps": 1,
+        })
+        out = servicer._apply_engine_args(base, extras)
+        self.assertIs(out, base)  # in-place merge — same dict back
+        self.assertTrue(out["trust_remote_code"])
+        self.assertEqual(out["speculative_algorithm"], "EAGLE")
+        self.assertEqual(out["speculative_num_steps"], 1)
+        self.assertEqual(out["model_path"], "facebook/opt-125m")
+        self.assertEqual(out["mem_fraction_static"], 0.7)
+
+    def test_apply_engine_args_engine_args_overrides_typed_fields(self):
+        """engine_args wins over previously-set typed kwargs (vLLM precedence)."""
+        import json as _json
+        servicer = self._servicer()
+        base = {"model_path": "facebook/opt-125m", "mem_fraction_static": 0.7}
+        out = servicer._apply_engine_args(
+            base, _json.dumps({"mem_fraction_static": 0.5}),
+        )
+        self.assertEqual(out["mem_fraction_static"], 0.5)
+
+    def test_apply_engine_args_unknown_key_raises(self):
+        """Typo'd key raises ValueError with a close-match suggestion."""
+        import json as _json
+        servicer = self._servicer()
+        base = {"model_path": "facebook/opt-125m"}
+        with self.assertRaises(ValueError) as ctx:
+            servicer._apply_engine_args(
+                base, _json.dumps({"trust_remotecode": True}),
+            )
+        msg = str(ctx.exception)
+        self.assertIn("trust_remotecode", msg)
+        self.assertIn("trust_remote_code", msg)
+
+    def test_apply_engine_args_empty_passthrough(self):
+        """Empty / None engine_args returns the kwargs dict untouched."""
+        servicer = self._servicer()
+        base = {"model_path": "facebook/opt-125m"}
+        self.assertIs(servicer._apply_engine_args(base, ""), base)
+        self.assertIs(servicer._apply_engine_args(base, None), base)
+
+    def test_apply_engine_args_invalid_json_raises(self):
+        servicer = self._servicer()
+        with self.assertRaises(ValueError) as ctx:
+            servicer._apply_engine_args({}, "not-json")
+        self.assertIn("not valid JSON", str(ctx.exception))
+
+    def test_apply_engine_args_non_object_raises(self):
+        servicer = self._servicer()
+        with self.assertRaises(ValueError) as ctx:
+            servicer._apply_engine_args({}, "[1,2,3]")
+        self.assertIn("must be a JSON object", str(ctx.exception))
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/sglang/test.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+set -e
+
+backend_dir=$(dirname $0)
+
+if [ -d $backend_dir/common ]; then
+    source $backend_dir/common/libbackend.sh
+else
+    source $backend_dir/../common/libbackend.sh
+fi
+
+runUnittests

backend/python/vllm-omni/install.sh

@@ -79,6 +79,14 @@ fi
 
 cd vllm-omni/
 
+# fa3-fwd ships no aarch64 wheels and there is no source distribution, so on
+# aarch64 (e.g. l4t13 / SBSA cu130) the upstream requirements/cuda.txt is
+# unsatisfiable. Drop it before resolving — vllm-omni does not hard-require
+# the fused FA3 kernel at import time on Jetson/SBSA targets.
+if [ "$(uname -m)" = "aarch64" ] && [ -f requirements/cuda.txt ]; then
+    sed -i '/^fa3-fwd[[:space:]]*==/d' requirements/cuda.txt
+fi
+
 if [ "x${USE_PIP}" == "xtrue" ]; then
     pip install ${EXTRA_PIP_INSTALL_FLAGS:-} -e .
 else

backend/python/vllm/install.sh

@@ -18,12 +18,15 @@ else
     source $backend_dir/../common/libbackend.sh
 fi
 
-# This is here because the Intel pip index is broken and returns 200 status codes for every package name, it just doesn't return any package links.
-# This makes uv think that the package exists in the Intel pip index, and by default it stops looking at other pip indexes once it finds a match.
-# We need uv to continue falling through to the pypi default index to find optimum[openvino] in the pypi index
-# the --upgrade actually allows us to *downgrade* torch to the version provided in the Intel pip index
+# Intel XPU: torch==2.11.0+xpu lives on the PyTorch XPU index, transitive
+# deps on PyPI — unsafe-best-match lets uv mix both. vllm-xpu-kernels only
+# ships a python3.12 wheel per upstream docs, so bump the portable Python
+# before installRequirements (matches the l4t13 pattern below).
+# https://github.com/vllm-project/vllm/blob/main/docs/getting_started/installation/gpu.xpu.inc.md
 if [ "x${BUILD_PROFILE}" == "xintel" ]; then
-    EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
+    PYTHON_VERSION="3.12"
+    PYTHON_PATCH="11"
+    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
 # CPU builds need unsafe-best-match to pull torch==2.10.0+cpu from the
@@ -42,10 +45,12 @@ fi
 
 # JetPack 7 / L4T arm64 wheels (torch, vllm, flash-attn) live on
 # pypi.jetson-ai-lab.io and are built for cp312, so bump the venv Python
-# accordingly. JetPack 6 keeps cp310 + USE_PIP=true. unsafe-best-match
-# is required because the jetson-ai-lab index lists transitive deps at
-# limited versions — without it uv pins to the first matching index and
-# fails to resolve a compatible wheel from PyPI.
+# accordingly. JetPack 6 keeps cp310 + USE_PIP=true.
+#
+# l4t13 uses pyproject.toml (see the elif branch below) to pin only the
+# L4T-specific wheels to the jetson-ai-lab index via [tool.uv.sources].
+# That keeps PyPI as the resolution path for transitive deps like
+# anthropic/openai/propcache, which the L4T mirror's proxy 503s on.
 if [ "x${BUILD_PROFILE}" == "xl4t12" ]; then
     USE_PIP=true
 fi
@@ -53,16 +58,77 @@ if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
     PYTHON_VERSION="3.12"
     PYTHON_PATCH="12"
     PY_STANDALONE_TAG="20251120"
-    EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
 fi
 
+# Intel XPU has no upstream-published vllm wheels, so we always build vllm
+# from source against torch-xpu and replace the default triton with
+# triton-xpu (matching torch 2.11). Mirrors the upstream procedure:
+# https://github.com/vllm-project/vllm/blob/main/docs/getting_started/installation/gpu.xpu.inc.md
+if [ "x${BUILD_TYPE}" == "xintel" ]; then
+    # Hide requirements-intel-after.txt so installRequirements doesn't
+    # try `pip install vllm` (would either fail or grab a non-XPU wheel).
+    _intel_after="${backend_dir}/requirements-intel-after.txt"
+    _intel_after_bak=""
+    if [ -f "${_intel_after}" ]; then
+        _intel_after_bak="${_intel_after}.xpu.bak"
+        mv "${_intel_after}" "${_intel_after_bak}"
+    fi
+    installRequirements
+    if [ -n "${_intel_after_bak}" ]; then
+        mv "${_intel_after_bak}" "${_intel_after}"
+    fi
+
+    # vllm's CMake build needs the Intel oneAPI dpcpp/sycl compiler — the
+    # base image (intel/oneapi-basekit) has it but the env isn't sourced.
+    if [ -f /opt/intel/oneapi/setvars.sh ]; then
+        set +u
+        source /opt/intel/oneapi/setvars.sh --force
+        set -u
+    fi
+
+    _vllm_src=$(mktemp -d)
+    trap 'rm -rf "${_vllm_src}"' EXIT
+    git clone --depth 1 https://github.com/vllm-project/vllm "${_vllm_src}/vllm"
+    pushd "${_vllm_src}/vllm"
+        # Install vllm's own runtime deps (torch-xpu, vllm_xpu_kernels,
+        # pydantic, fastapi, …) from upstream's requirements/xpu.txt — the
+        # canonical source of truth. Avoids re-pinning everything ourselves.
+        uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} -r requirements/xpu.txt
+        # Stock triton (NVIDIA-only) may have come in transitively; replace
+        # with triton-xpu==3.7.0 which matches torch 2.11.
+        uv pip uninstall triton triton-xpu 2>/dev/null || true
+        uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} \
+            --extra-index-url https://download.pytorch.org/whl/xpu \
+            triton-xpu==3.7.0
+        export CMAKE_PREFIX_PATH="$(python -c 'import site; print(site.getsitepackages()[0])'):${CMAKE_PREFIX_PATH:-}"
+        VLLM_TARGET_DEVICE=xpu uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --no-deps .
+    popd
+# L4T arm64 (JetPack 7): drive the install through pyproject.toml so that
+# [tool.uv.sources] can pin torch/vllm/flash-attn/torchvision/torchaudio
+# to the jetson-ai-lab index, while everything else (transitive deps and
+# PyPI-resolvable packages like transformers) comes from PyPI. Bypasses
+# installRequirements because uv pip install -r requirements.txt does not
+# honor sources — see backend/python/vllm/pyproject.toml for the rationale.
+elif [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
+    ensureVenv
+    if [ "x${PORTABLE_PYTHON}" == "xtrue" ]; then
+        export C_INCLUDE_PATH="${C_INCLUDE_PATH:-}:$(_portable_dir)/include/python${PYTHON_VERSION}"
+    fi
+    pushd "${backend_dir}"
+        # Build deps first (matches installRequirements' requirements-install.txt
+        # pass — fastsafetensors and friends need pybind11 in the venv before
+        # their sdists can build under --no-build-isolation).
+        uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} -r requirements-install.txt
+        uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement pyproject.toml
+    popd
+    runProtogen
 # FROM_SOURCE=true on a CPU build skips the prebuilt vllm wheel in
 # requirements-cpu-after.txt and compiles vllm locally against the host's
 # actual CPU. Not used by default because it takes ~30-40 minutes, but
 # kept here for hosts where the prebuilt wheel SIGILLs (CPU without the
 # required SIMD baseline, e.g. AVX-512 VNNI/BF16). Default CI uses a
 # bigger-runner with compatible hardware instead.
-if [ "x${BUILD_TYPE}" == "x" ] && [ "x${FROM_SOURCE:-}" == "xtrue" ]; then
+elif [ "x${BUILD_TYPE}" == "x" ] && [ "x${FROM_SOURCE:-}" == "xtrue" ]; then
     # Temporarily hide the prebuilt wheel so installRequirements doesn't
     # pull it — the rest of the requirements files (base deps, torch,
     # transformers) are still installed normally.

backend/python/vllm/package.sh

@@ -45,5 +45,109 @@ copy_with_symlinks() {
 copy_with_symlinks libnuma.so.1
 copy_with_symlinks libgomp.so.1
 
+# CPU profile only: bundle a g++ toolchain so torch._inductor's
+# ISA probe (always run at vllm engine startup, regardless of
+# enforce_eager) finds a C++ compiler. The LocalAI runtime image
+# is FROM ubuntu:24.04 with a minimal apt list that does not
+# include build-essential, and the backend image itself is FROM
+# scratch -- so without this, cpu-vllm crashes with
+# torch._inductor.exc.InvalidCxxCompiler at first inference
+# unless the operator manually sets TORCH_COMPILE_DISABLE=1.
+#
+# We snapshot every file owned by the toolchain packages, mirroring
+# the /usr/... layout into ${BACKEND}/toolchain/ so g++ can find
+# cc1plus, headers, libs etc. via GCC_EXEC_PREFIX / CPATH /
+# LIBRARY_PATH at runtime (libbackend.sh wires those up). Adds
+# ~400 MB to the cpu-vllm image, which is tolerable -- cpu-vllm is
+# already a niche profile.
+if [ "${BUILD_TYPE:-}" = "" ] && command -v dpkg-query >/dev/null 2>&1; then
+    TOOLCHAIN_DIR="${CURDIR}/toolchain"
+    mkdir -p "${TOOLCHAIN_DIR}"
+    # The unversioned g++/gcc packages on Debian/Ubuntu only ship
+    # symlinks; the actual binaries live in g++-${VER}/gcc-${VER}.
+    # Discover the active version so the symlink targets get bundled
+    # along with their owners.
+    GCC_VER=$(gcc -dumpversion 2>/dev/null | cut -d. -f1 || true)
+    # `g++-${VER}` itself is just another symlink layer on Debian/
+    # Ubuntu — the real binary `x86_64-linux-gnu-g++-${VER}` lives
+    # in `g++-${VER}-x86-64-linux-gnu` (a separate package pulled in
+    # as a dependency). Same story for gcc/cpp. Compute the dpkg
+    # arch-triplet to find the right package name for both amd64 and
+    # arm64 hosts.
+    case "$(dpkg --print-architecture 2>/dev/null)" in
+        amd64) HOST_TRIPLET="x86-64-linux-gnu" ;;
+        arm64) HOST_TRIPLET="aarch64-linux-gnu" ;;
+        *)     HOST_TRIPLET="" ;;
+    esac
+    PKGS=(g++ gcc cpp libstdc++-${GCC_VER}-dev libgcc-${GCC_VER}-dev libc6 libc6-dev binutils binutils-common libbinutils libc-dev-bin linux-libc-dev libcrypt-dev libgomp1 libstdc++6 libgcc-s1 libisl23 libmpc3 libmpfr6 libjansson4 libctf0 libctf-nobfd0 libsframe1)
+    if [ -n "${GCC_VER}" ]; then
+        PKGS+=("g++-${GCC_VER}" "gcc-${GCC_VER}" "cpp-${GCC_VER}" "gcc-${GCC_VER}-base")
+        if [ -n "${HOST_TRIPLET}" ]; then
+            PKGS+=(
+                "g++-${GCC_VER}-${HOST_TRIPLET}"
+                "gcc-${GCC_VER}-${HOST_TRIPLET}"
+                "cpp-${GCC_VER}-${HOST_TRIPLET}"
+                "binutils-${HOST_TRIPLET}"
+            )
+        fi
+    fi
+    for pkg in "${PKGS[@]}"; do
+        if ! dpkg-query -W "${pkg}" >/dev/null 2>&1; then
+            continue
+        fi
+        # Copy each owned path, preserving symlinks and mode. We
+        # tolerate dpkg listing directories alongside files.
+        dpkg -L "${pkg}" | while IFS= read -r path; do
+            if [ -L "${path}" ] || [ -f "${path}" ]; then
+                mkdir -p "${TOOLCHAIN_DIR}$(dirname "${path}")"
+                cp -aP "${path}" "${TOOLCHAIN_DIR}${path}" 2>/dev/null || true
+            fi
+        done
+    done
+    # Ubuntu's filesystem layout has /lib -> /usr/lib (UsrMerge) and
+    # /lib64 -> /usr/lib64. ld scripts (e.g. libm.so) hardcode
+    # `/lib/x86_64-linux-gnu/libm.so.6`; with --sysroot the linker
+    # looks for that path under the sysroot, which means we need
+    # the same symlinks under TOOLCHAIN_DIR.
+    [ -e "${TOOLCHAIN_DIR}/lib" ]   || ln -s usr/lib   "${TOOLCHAIN_DIR}/lib"
+    [ -e "${TOOLCHAIN_DIR}/lib64" ] || ln -s usr/lib64 "${TOOLCHAIN_DIR}/lib64"
+
+    # Replace the unversioned g++/gcc/cpp symlinks with wrapper
+    # scripts that pass --sysroot=<toolchain> and -B <gcc-exec-prefix>.
+    # Without these flags gcc would fall back to its compiled-in
+    # /usr search and fail to find headers (the runtime image has no
+    # libc6-dev) or fail to invoke `as`/`ld` (binutils not on PATH at
+    # /usr/bin). Wrappers self-resolve their location at runtime so
+    # they work from any BackendsPath.
+    BIN_DIR="${TOOLCHAIN_DIR}/usr/bin"
+    if [ -n "${GCC_VER}" ] && [ -n "${HOST_TRIPLET}" ]; then
+        # HOST_TRIPLET in package names uses dashes ("x86-64-linux-gnu");
+        # the binary suffix uses underscores in the arch part
+        # ("x86_64-linux-gnu-g++-13"). Translate.
+        BIN_TRIPLET=${HOST_TRIPLET//x86-64/x86_64}
+        for tool in g++ gcc cpp; do
+            real="${BIN_DIR}/${BIN_TRIPLET}-${tool}-${GCC_VER}"
+            if [ -x "${real}" ]; then
+                rm -f "${BIN_DIR}/${tool}" "${BIN_DIR}/${tool}-${GCC_VER}"
+                cat > "${BIN_DIR}/${tool}" <<EOF
+#!/bin/bash
+# Auto-generated by package.sh. Passes --sysroot and -B so the
+# bundled toolchain works from any BackendsPath without depending
+# on libc6-dev / binutils being installed at /usr in the runtime
+# image. See backend/python/vllm/package.sh.
+DIR="\$(dirname "\$(readlink -f "\$0")")"     # …/toolchain/usr/bin
+SYSROOT="\$(dirname "\$(dirname "\${DIR}")")" # …/toolchain
+exec "\${DIR}/${BIN_TRIPLET}-${tool}-${GCC_VER}" \\
+    -B "\${SYSROOT}/usr/lib/gcc/${BIN_TRIPLET}/${GCC_VER}/" \\
+    --sysroot="\${SYSROOT}" \\
+    "\$@"
+EOF
+                chmod +x "${BIN_DIR}/${tool}"
+            fi
+        done
+    fi
+    echo "Bundled g++ toolchain (gcc-${GCC_VER}) into ${TOOLCHAIN_DIR} ($(du -sh "${TOOLCHAIN_DIR}" | cut -f1))"
+fi
+
 echo "vllm packaging completed successfully"
 ls -liah "${LIB_DIR}/"

backend/python/vllm/pyproject.toml

@@ -0,0 +1,61 @@
+# L4T arm64 (JetPack 7 / sbsa cu130) install spec for the vllm backend.
+#
+# Why this file exists, and why only the l4t13 BUILD_PROFILE consumes it:
+#
+# pypi.jetson-ai-lab.io hosts the L4T-specific torch / vllm / flash-attn
+# wheels we need on aarch64 + cuda13, but it ALSO transparently proxies the
+# rest of PyPI through `/+f/<sha>/<filename>` URLs that 503 frequently. With
+# `--extra-index-url` + `--index-strategy=unsafe-best-match` (the historical
+# fix in install.sh) uv would pick those proxy URLs for ordinary PyPI
+# packages — `anthropic`, `openai`, `propcache`, `annotated-types` — and
+# trip on the 503s. See e.g. CI run 25212201349 (anthropic-0.97.0).
+#
+# `explicit = true` on the index makes uv consult the L4T mirror ONLY for
+# packages mapped under [tool.uv.sources]. Everything else goes to PyPI.
+# This breaks the historical 503 path without losing access to the L4T
+# wheels we actually need from there.
+#
+# `uv pip install -r requirements.txt` does NOT honor [tool.uv.sources]
+# (sources are project-mode only, not pip-compat mode), so install.sh's
+# l4t13 branch invokes `uv pip install --requirement pyproject.toml`
+# directly. Other BUILD_PROFILEs continue to use the requirements-*.txt
+# pipeline through libbackend.sh's installRequirements and never read
+# this file.
+[project]
+name = "localai-vllm-l4t13"
+version = "0.0.0"
+requires-python = ">=3.12,<3.13"
+dependencies = [
+    # Mirror of requirements.txt — kept in sync manually for now since the
+    # l4t13 path bypasses installRequirements (see install.sh).
+    "grpcio==1.80.0",
+    "protobuf",
+    "certifi",
+    "setuptools",
+    "pillow",
+    "charset-normalizer>=3.4.0",
+    "chardet",
+    # L4T-specific accelerator stack (sourced from jetson-ai-lab below).
+    "torch",
+    "torchvision",
+    "torchaudio",
+    "flash-attn",
+    "vllm",
+    # PyPI-resolvable packages that complete the runtime — accelerate,
+    # transformers, bitsandbytes carry their own wheels for aarch64.
+    "accelerate",
+    "transformers",
+    "bitsandbytes",
+]
+
+[[tool.uv.index]]
+name = "jetson-ai-lab"
+url = "https://pypi.jetson-ai-lab.io/sbsa/cu130"
+explicit = true
+
+[tool.uv.sources]
+torch = { index = "jetson-ai-lab" }
+torchvision = { index = "jetson-ai-lab" }
+torchaudio = { index = "jetson-ai-lab" }
+flash-attn = { index = "jetson-ai-lab" }
+vllm = { index = "jetson-ai-lab" }

backend/python/vllm/requirements-cublas13-after.txt

@@ -3,5 +3,5 @@
 # on a cu130 host. Pull the cu130-flavoured wheel from vLLM's per-tag index
 # instead — the cublas13 case in install.sh adds --index-strategy=unsafe-best-match
 # so uv consults this index alongside PyPI.
---extra-index-url https://wheels.vllm.ai/0.20.0/cu130
-vllm==0.20.0
+--extra-index-url https://wheels.vllm.ai/0.20.2/cu130
+vllm==0.20.2

backend/python/vllm/requirements-intel-after.txt

@@ -1 +1,3 @@
-vllm
+# Intel XPU has no upstream-published vllm wheels — install.sh builds vllm
+# from source with VLLM_TARGET_DEVICE=xpu and hides this file during
+# installRequirements. Don't add a `vllm` line here.

backend/python/vllm/requirements-intel.txt

@@ -1,7 +1,8 @@
 --extra-index-url https://download.pytorch.org/whl/xpu
+# vllm's own deps (torch==2.11.0+xpu, vllm_xpu_kernels, pydantic, …) are
+# installed from upstream's requirements/xpu.txt during the source build —
+# see install.sh. Only list what LocalAI's vllm backend.py needs directly.
 accelerate
-torch
 transformers
-optimum[openvino]
+bitsandbytes
 setuptools
-bitsandbytes

backend/python/vllm/requirements-l4t13-after.txt

@@ -1,2 +0,0 @@
---extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
-vllm

backend/python/vllm/requirements-l4t13.txt

@@ -1,8 +0,0 @@
---extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
-accelerate
-torch
-torchvision
-torchaudio
-transformers
-bitsandbytes
-flash-attn

backend/python/vllm/requirements.txt

@@ -1,4 +1,7 @@
 grpcio==1.80.0
 protobuf
 certifi
-setuptools
+setuptools
+pillow
+charset-normalizer>=3.4.0
+chardet