fix(distributed): stop queue loops on agent nodes + dead-letter cap

pending_backend_ops rows targeting agent-type workers looped forever:
the reconciler fan-out hit a NATS subject the worker doesn't subscribe
to, returned ErrNoResponders, we marked the node unhealthy, and the
health monitor flipped it back to healthy on the next heartbeat. Next
tick, same row, same failure.

Three related fixes:

1. enqueueAndDrainBackendOp skips nodes whose NodeType != backend.
   Agent workers handle agent NATS subjects, not backend.install /
   delete / list, so enqueueing for them guarantees an infinite retry
   loop. Silent skip is correct — they aren't consumers of these ops.

2. Reconciler drain mirrors enqueueAndDrainBackendOp's behavior on
   nats.ErrNoResponders: mark the node unhealthy before recording the
   failure, so subsequent ListDuePendingBackendOps (filters by
   status=healthy) stops picking the row until the node actually
   recovers. Matches the synchronous fan-out path.

3. Dead-letter cap at maxPendingBackendOpAttempts (10). After ~1h of
   exponential backoff the row is a poison message; further retries
   just thrash NATS. Row is deleted and logged at ERROR so it stays
   visible without staying infinite.

Plus a one-shot startup cleanup in NewNodeRegistry: drop queue rows
that target agent-type nodes, non-existent nodes, or carry an empty
backend name. Guarded by the same schema-migration advisory lock so
only one instance performs it. The guards above prevent new rows of
this shape; this closes the migration gap for existing ones.

Tests: the prune migration (valid row stays, agent + empty-name rows
drop) on top of existing upsert / backoff coverage.

.agents/adding-backends.md

@@ -8,7 +8,6 @@ Create the backend directory under the appropriate location:
 - **Python backends**: `backend/python/<backend-name>/`
 - **Go backends**: `backend/go/<backend-name>/`
 - **C++ backends**: `backend/cpp/<backend-name>/`
-- **Rust backends**: `backend/rust/<backend-name>/`
 
 For Python backends, you'll typically need:
 - `backend.py` - Main gRPC server implementation
@@ -19,22 +18,9 @@ For Python backends, you'll typically need:
 - `run.sh` - Runtime script
 - `test.py` / `test.sh` - Test files
 
-For Rust backends, you'll typically need (see `backend/rust/kokoros/` as a reference):
-- `Cargo.toml` - Crate manifest; depend on the upstream project as a submodule under `sources/`
-- `build.rs` - Invokes `tonic_build` to generate gRPC stubs from `backend/backend.proto` (use the `BACKEND_PROTO_PATH` env var so the Makefile can inject the canonical copy)
-- `src/` - The gRPC server implementation (implement `Backend` via `tonic`)
-- `Makefile` - Copies `backend.proto` into the crate, runs `cargo build --release`, then `package.sh`
-- `package.sh` - Uses `ldd` to bundle the binary's dynamic deps and `ld.so` into `package/lib/`
-- `run.sh` - Sets `LD_LIBRARY_PATH`/`SSL_CERT_DIR` and execs the binary via the bundled `lib/ld.so`
-- `sources/<UpstreamProject>/` - Git submodule with the upstream Rust crate
+## 2. Add Build Configurations to `.github/workflows/backend.yml`
 
-## 2. Add Build Configurations to `.github/backend-matrix.yml`
-
-The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `backend.yml` itself). `backend.yml` (master push) and `backend_pr.yml` (PR) load it via `scripts/changed-backends.js`, which also handles per-file path filtering so only touched backends rebuild on PRs and master pushes alike. Add build matrix entries to `.github/backend-matrix.yml` for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
-
-**Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
-
-If you add a new language bucket, `scripts/changed-backends.js` also needs a branch in `inferBackendPath` so PR change-detection routes file edits correctly.
+Add build matrix entries for each platform/GPU type you want to support. Look at similar backends (e.g., `chatterbox`, `faster-whisper`) for reference.
 
 **Placement in file:**
 - CPU builds: Add after other CPU builds (e.g., after `cpu-chatterbox`)
@@ -43,17 +29,9 @@ If you add a new language bucket, `scripts/changed-backends.js` also needs a bra
 
 **Additional build types you may need:**
 - ROCm/HIP: Use `build-type: 'hipblas'` with `base-image: "rocm/dev-ubuntu-24.04:7.2.1"`
-- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"`
+- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"`
 - L4T (ARM): Use `build-type: 'l4t'` with `platforms: 'linux/arm64'` and `runs-on: 'ubuntu-24.04-arm'`
 
-**Per-arch native builds (`linux/amd64` + `linux/arm64`):**
-
-Multi-arch backends are NOT a single matrix entry with `platforms: 'linux/amd64,linux/arm64'`. Instead, add **two** entries — one with `platforms: 'linux/amd64'` + `platform-tag: 'amd64'` + `runs-on: 'ubuntu-latest'`, one with `platforms: 'linux/arm64'` + `platform-tag: 'arm64'` + `runs-on: 'ubuntu-24.04-arm'` — both sharing the same `tag-suffix`. The script detects the shared `tag-suffix` and emits a `merge-matrix` entry, so `backend-merge-jobs` (in `backend.yml`/`backend_pr.yml`) automatically assembles the manifest list from per-arch digest artifacts. See `-cpu-faster-whisper` in `.github/backend-matrix.yml` for a reference shape.
-
-**llama-cpp / ik-llama-cpp / turboquant variants only — `builder-base-image`:**
-
-Entries whose `dockerfile` is `./backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` must also set a `builder-base-image` field pointing at a prebuilt base from `quay.io/go-skynet/ci-cache:base-grpc-*` (CI builds these via `.github/workflows/base-images.yml`). The mapping is by `(build-type, platforms)` — see existing entries for the pattern. CI uses these prebuilt bases to skip the gRPC compile (~25–35 min cold). Local `make backends/<name>` ignores `builder-base-image` and uses the from-source path inside the Dockerfile, so you don't need quay access for local builds.
-
 ## 3. Add Backend Metadata to `backend/index.yaml`
 
 **Step 3a: Add Meta Definition**
@@ -78,28 +56,24 @@ Add `backends/<backend-name>` to the `.NOTPARALLEL` line (around line 2) to prev
 
 **Step 4b: Add to `prepare-test-extra`**
 
-Add the backend to the `prepare-test-extra` target to prepare it for testing. Use the path matching your language bucket (`backend/python/`, `backend/go/`, `backend/rust/`, …):
+Add the backend to the `prepare-test-extra` target (around line 312) to prepare it for testing:
 
 ```makefile
 prepare-test-extra: protogen-python
 	...
-	$(MAKE) -C backend/<lang>/<backend-name>
+	$(MAKE) -C backend/python/<backend-name>
 ```
 
-For Rust backends the target is usually the crate build target itself (e.g. `$(MAKE) -C backend/rust/<backend-name> <backend-name>-grpc`) so the binary is in place before `test` runs.
-
 **Step 4c: Add to `test-extra`**
 
-Add the backend to the `test-extra` target to run its tests — applies to Go and Rust backends too, not only Python:
+Add the backend to the `test-extra` target (around line 319) to run its tests:
 
 ```makefile
 test-extra: prepare-test-extra
 	...
-	$(MAKE) -C backend/<lang>/<backend-name> test
+	$(MAKE) -C backend/python/<backend-name> test
 ```
 
-Each backend's own `Makefile` should define a `test` target so this line works regardless of language. Integration tests that need large model downloads should be gated behind an env var (see `backend/rust/kokoros/`'s `KOKOROS_MODEL_PATH` pattern) so CI only runs unit tests.
-
 **Step 4d: Add Backend Definition**
 
 Add a backend definition variable in the backend definitions section (around line 428-457). The format depends on the backend type:
@@ -119,13 +93,6 @@ BACKEND_<BACKEND_NAME> = <backend-name>|python|./backend|false|true
 BACKEND_<BACKEND_NAME> = <backend-name>|golang|.|false|true
 ```
 
-**For Rust backends**:
-```makefile
-BACKEND_<BACKEND_NAME> = <backend-name>|rust|.|false|true
-```
-
-The language field (`python`/`golang`/`rust`/…) must match a `backend/Dockerfile.<lang>` file.
-
 **Step 4e: Generate Docker Build Target**
 
 Add an eval call to generate the docker-build target (around line 480-501):
@@ -153,7 +120,7 @@ docker-build-backends: ... docker-build-<backend-name>
 After adding a new backend, verify:
 
 - [ ] Backend directory structure is complete with all necessary files
-- [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
+- [ ] Build configurations added to `.github/workflows/backend.yml` for all desired platforms
 - [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
 - [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
 - [ ] Tag suffixes match between workflow file and index.yaml
@@ -186,29 +153,6 @@ ls /tmp/check    # expect the bundled .so files + symlinks
 
 Then boot it inside a fresh `ubuntu:24.04` (which intentionally does *not* have the lib installed) to confirm it actually loads from the backend dir.
 
-## Importer integration
-
-When you add a new backend, you MUST also make it importable via the model import form (`/import-model`). The import form dropdown is sourced dynamically from `GET /backends/known` — it reads the importer registry at `core/gallery/importers/importers.go`, so the steps below are the ONLY way to make your backend show up.
-
-Required steps:
-
-1. **If your backend has unambiguous detection signals** (unique file extension, HF `pipeline_tag`, unique repo name pattern, unique artefact like `modules.json`):
-   - Create an importer file at `core/gallery/importers/<backend>.go` following the Match/Import pattern in `llama-cpp.go`.
-   - Register it in `importers.go:defaultImporters` in **specificity order** — more specific detectors must appear BEFORE more generic ones (e.g. `sentencetransformers` before `transformers`, `stablediffusion-ggml` before `llama-cpp`, `vllm-omni` before `vllm`). First match wins.
-2. **If your backend is a drop-in replacement** (same artefacts as another backend, e.g. `ik-llama-cpp` and `turboquant` both consume GGUF the same way `llama-cpp` does):
-   - Do NOT create a new importer. Extend the existing importer's `Import()` to swap the emitted `backend:` field when `preferences.backend` matches. See `llama-cpp.go` for the pattern.
-3. **If your backend has no reliable auto-detect signal** (preference-only — e.g. `sglang`, `tinygrad`, `whisperx`):
-   - Do NOT create an importer. Instead add the backend name to the curated pref-only slice in `core/http/endpoints/localai/backend.go` that feeds `/backends/known`. A single line addition.
-4. **Always** add a table-driven test in `core/gallery/importers/importers_test.go` (Ginkgo/Gomega):
-   - Use a real public HuggingFace repo URI as the test fixture (existing tests already hit the live HF API — follow that pattern).
-   - Cover detection (auto-match without preferences), preference-override (explicit `backend:` in preferences wins), and — if the backend's modality has a common `pipeline_tag` but ambiguous artefacts — an ambiguity test asserting `errors.Is(err, importers.ErrAmbiguousImport)`.
-
-Rules of thumb:
-
-- When in doubt, lean pref-only. A wrong auto-detect is worse than a forced preference.
-- Never silently emit a modality mismatch (e.g. emit `llama-cpp` for a TTS repo because `.gguf` is present). Return `ErrAmbiguousImport` instead.
-- Registration order is the single most common source of bugs. Check by running `go test ./core/gallery/importers/...` — the existing suite will fail if you've shadowed a pre-existing detector.
-
 ## 6. Example: Adding a Python Backend
 
 For reference, when `moonshine` was added:

.agents/ai-coding-assistants.md

@@ -1,101 +0,0 @@
-# AI Coding Assistants
-
-This document provides guidance for AI tools and developers using AI
-assistance when contributing to LocalAI.
-
-**LocalAI follows the same guidelines as the Linux kernel project for
-AI-assisted contributions.** See the upstream policy here:
-<https://docs.kernel.org/process/coding-assistants.html>
-
-The rules below mirror that policy, adapted to LocalAI's license and
-project layout. If anything is unclear, the kernel document is the
-authoritative reference for intent.
-
-AI tools helping with LocalAI development should follow the standard
-project development process:
-
-- [CONTRIBUTING.md](../CONTRIBUTING.md) — development workflow, commit
-  conventions, and PR guidelines
-- [.agents/coding-style.md](coding-style.md) — code style, editorconfig,
-  logging, and documentation conventions
-- [.agents/building-and-testing.md](building-and-testing.md) — build and
-  test procedures
-
-## Licensing and Legal Requirements
-
-All contributions must comply with LocalAI's licensing requirements:
-
-- LocalAI is licensed under the **MIT License** — see the [LICENSE](../LICENSE)
-  file
-- New source files should use the SPDX license identifier `MIT` where
-  applicable to the file type
-- Contributions must be compatible with the MIT License and must not
-  introduce code under incompatible licenses (e.g., GPL) without an
-  explicit discussion with maintainers
-
-## Signed-off-by and Developer Certificate of Origin
-
-**AI agents MUST NOT add `Signed-off-by` tags.** Only humans can legally
-certify the Developer Certificate of Origin (DCO). The human submitter
-is responsible for:
-
-- Reviewing all AI-generated code
-- Ensuring compliance with licensing requirements
-- Adding their own `Signed-off-by` tag (when the project requires DCO)
-  to certify the contribution
-- Taking full responsibility for the contribution
-
-AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
-A human reviewer owns the contribution; the AI's involvement is recorded
-via `Assisted-by` (see below).
-
-## Attribution
-
-When AI tools contribute to LocalAI development, proper attribution helps
-track the evolving role of AI in the development process. Contributions
-should include an `Assisted-by` tag in the commit message trailer in the
-following format:
-
-```
-Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
-```
-
-Where:
-
-- `AGENT_NAME` — name of the AI tool or framework (e.g., `Claude`,
-  `Copilot`, `Cursor`)
-- `MODEL_VERSION` — specific model version used (e.g.,
-  `claude-opus-4-7`, `gpt-5`)
-- `[TOOL1] [TOOL2]` — optional specialized analysis tools invoked by the
-  agent (e.g., `golangci-lint`, `staticcheck`, `go vet`)
-
-Basic development tools (git, go, make, editors) should **not** be listed.
-
-### Example
-
-```
-fix(llama-cpp): handle empty tool call arguments
-
-Previously the parser panicked when the model returned a tool call with
-an empty arguments object. Fall back to an empty JSON object in that
-case so downstream consumers receive a valid payload.
-
-Assisted-by: Claude:claude-opus-4-7 golangci-lint
-Signed-off-by: Jane Developer <jane@example.com>
-```
-
-## Scope and Responsibility
-
-Using an AI assistant does not reduce the contributor's responsibility.
-The human submitter must:
-
-- Understand every line that lands in the PR
-- Verify that generated code compiles, passes tests, and follows the
-  project style
-- Confirm that any referenced APIs, flags, or file paths actually exist
-  in the current tree (AI models may hallucinate identifiers)
-- Not submit AI output verbatim without review
-
-Reviewers may ask for clarification on any change regardless of how it
-was produced. "An AI wrote it" is not an acceptable answer to a design
-question.

.agents/api-endpoints-and-auth.md

@@ -2,8 +2,6 @@
 
 This guide covers how to add new API endpoints and properly integrate them with the auth/permissions system.
 
-> **Before you ship a new endpoint or capability surface**, re-read the [checklist at the bottom of this file](#checklist). LocalAI advertises its feature surface in several independent places — miss any one of them and clients/admins/UI won't know the endpoint exists.
-
 ## Architecture overview
 
 Authentication and authorization flow through three layers:
@@ -236,66 +234,6 @@ Use these HTTP status codes:
 
 If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
 
-## Advertising surfaces — where to register a new capability
-
-Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.
-
-### 1. Swagger `@Tags` annotation (mandatory)
-
-Every handler needs a swagger block so the endpoint appears in `/swagger/index.html` and in the `/api/instructions` output. The `@Tags` value is what groups the endpoint into a capability area:
-
-```go
-// MyEndpoint does X.
-// @Summary Do X.
-// @Tags my-capability
-// @Param request body schema.MyRequest true "payload"
-// @Success 200 {object} schema.MyResponse "Response"
-// @Router /v1/my-endpoint [post]
-func MyEndpoint(...) echo.HandlerFunc { ... }
-```
-
-Use an existing tag when the endpoint extends an existing area (e.g. `audio`, `images`, `face-recognition`). Create a new tag only when the endpoint introduces a genuinely new capability surface — and in that case, also register it in step 2.
-
-After adding endpoints, regenerate the embedded spec so the runtime serves it:
-
-```bash
-make protogen-go         # ensures gRPC codegen is fresh first
-make swagger             # regenerates swagger/swagger.json
-```
-
-### 2. `/api/instructions` registry (for new capability areas)
-
-`core/http/endpoints/localai/api_instructions.go` defines `instructionDefs` — a lightweight, machine-readable index of capability areas that groups swagger endpoints by tag. It's the primary discovery surface for agents and SDKs ("what can this server do?").
-
-**When to update:** only when adding a new capability area (a new swagger tag). Existing-tag additions automatically surface without any change here.
-
-Add an entry to `instructionDefs`:
-
-```go
-{
-    Name:        "my-capability",             // URL segment at /api/instructions/my-capability
-    Description: "Short sentence describing the capability",
-    Tags:        []string{"my-capability"},   // must match swagger @Tags
-    Intro:       "Optional gotcha/context that isn't in the swagger descriptions (caveats, defaults, cross-references to other endpoints).",
-},
-```
-
-Also bump the expected-length count in `api_instructions_test.go` and add the name to the `ContainElements` assertion.
-
-### 3. `capabilities.js` symbol (for new model-config FLAG_* flags)
-
-If your feature needs a new `FLAG_*` usecase flag in `core/config/model_config.go` (so users can filter gallery models by it, and so `/v1/models` surfaces it), also declare the matching symbol in `core/http/react-ui/src/utils/capabilities.js`:
-
-```js
-export const CAP_MY_CAPABILITY = 'FLAG_MY_CAPABILITY'
-```
-
-React pages that want to filter the ModelSelector by capability import this symbol. Declare it even if you're not building the UI page yet — the declaration keeps the Go/JS vocabularies in sync.
-
-### 4. `docs/content/` (user-facing documentation)
-
-A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features and an entry in `docs/content/whats-new.md`. See the pattern used by `face-recognition.md` / `object-detection.md`.
-
 ## Path protection rules
 
 The global auth middleware classifies paths as API paths or non-API paths:
@@ -310,36 +248,12 @@ If you add endpoints under a new top-level path prefix, add it to `isAPIPath()`
 
 When adding a new endpoint:
 
-**Routing & auth**
 - [ ] Handler in `core/http/endpoints/`
 - [ ] Route registered in appropriate `core/http/routes/` file
 - [ ] Auth level chosen: public / standard / admin / feature-gated
-- [ ] Entry added to `RouteFeatureRegistry` in `core/http/auth/features.go` (one row per route/method — all /v1/* routes gate through this, not per-route middleware)
-- [ ] If new feature: constant in `permissions.go`, added to the right slice (`APIFeatures` default-ON / `AgentFeatures` default-OFF), metadata in `features.go` `*FeatureMetas()`
-- [ ] If feature uses group middleware: wired in `core/http/app.go` and passed to the route registration function
+- [ ] If feature-gated: constant in `permissions.go`, metadata in `features.go`, middleware in `app.go`
 - [ ] If new path prefix: added to `isAPIPath()` in `middleware.go`
+- [ ] If OpenAI-compatible: entry in `RouteFeatureRegistry`
 - [ ] If token-counting: `usageMiddleware` added to middleware chain
-
-**Advertising surfaces (easy to miss — see the [Advertising surfaces](#advertising-surfaces--where-to-register-a-new-capability) section)**
-- [ ] Swagger block on the handler: `@Summary`, `@Tags`, `@Param`, `@Success`, `@Router`
-- [ ] If new capability area (new swagger tag): entry in `instructionDefs` in `core/http/endpoints/localai/api_instructions.go` + test count bumped in `api_instructions_test.go`
-- [ ] If new `FLAG_*` usecase flag: matching `CAP_*` symbol exported from `core/http/react-ui/src/utils/capabilities.js`
-- [ ] `docs/content/features/<feature>.md` created; cross-links from related feature pages; entry in `docs/content/whats-new.md`
-
-**Quality**
-- [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
+- [ ] Error responses use `schema.ErrorResponse` format
 - [ ] Tests cover both authenticated and unauthenticated access
-- [ ] Swagger regenerated (`make swagger`) if you changed any `@Router`/`@Tags`/`@Param` annotation
-
-## Companion: MCP admin tool surface
-
-**Required for admin endpoints.** Every new admin endpoint MUST be considered for the MCP admin tool surface — the REST API and the MCP tool catalog can drift silently otherwise, and both the LocalAI Assistant chat modality and the standalone `local-ai mcp-server` rely on `pkg/mcp/localaitools/` to mirror REST.
-
-Two outcomes are acceptable; one is not:
-
-- **Tool added.** The new endpoint is something an admin would manage conversationally (install, list, edit, toggle, upgrade). Follow the full checklist in [.agents/localai-assistant-mcp.md](localai-assistant-mcp.md): add a `LocalAIClient` interface method, implement it in both `inproc` and `httpapi`, register the tool with a `Tool*` constant, update the skill prompts, **and add the route to `toolToHTTPRoute` in `pkg/mcp/localaitools/coverage_test.go`**.
-- **Tool deliberately skipped.** The endpoint is internal/diagnostic and adding a chat path would be misleading. Document the decision in the PR description; no code action.
-- **Forgot.** This breaks the contract. The `TestToolHTTPRouteMappingComplete` test in `pkg/mcp/localaitools` is a partial guard (it checks every `Tool*` has a route mapping), but it does NOT detect new REST endpoints without a tool — that's still a process check on the PR author.
-
-**Add to the bottom of the checklist below**:
-- [ ] If admin: decided whether MCP coverage is needed; if yes, tool registered + map updated; if no, skip-reason in PR description.

.agents/building-and-testing.md

@@ -8,9 +8,8 @@ Let's say the user wants to build a particular backend for a given platform. For
 
 - The Makefile has targets like `docker-build-coqui` created with `generate-docker-build-target` at the time of writing. Recently added backends may require a new target.
 - At a minimum we need to set the BUILD_TYPE, BASE_IMAGE build-args
-  - Use `.github/backend-matrix.yml` as a reference — it's the data-only YAML that lists every backend variant's `build-type`, `base-image`, `platforms`, etc. (`backend.yml` and `backend_pr.yml` consume it via `scripts/changed-backends.js`).
-  - l4t and cublas also require the CUDA major and minor version.
-  - For llama-cpp / ik-llama-cpp / turboquant the matrix also sets `builder-base-image` pointing at a prebuilt `quay.io/go-skynet/ci-cache:base-grpc-*` tag. Local `make backends/<name>` defaults to `BUILDER_TARGET=builder-fromsource` and doesn't need it — the Dockerfile's from-source stage installs everything itself.
+  - Use .github/workflows/backend.yml as a reference it lists the needed args in the `include` job strategy matrix
+  - l4t and cublas also requires the CUDA major and minor version
 - You can pretty print a command like `DOCKER_MAKEFLAGS=-j$(nproc --ignore=1) BUILD_TYPE=hipblas BASE_IMAGE=rocm/dev-ubuntu-24.04:7.2.1 make docker-build-coqui`
 - Unless the user specifies that they want you to run the command, then just print it because not all agent frontends handle long running jobs well and the output may overflow your context
 - The user may say they want to build AMD or ROCM instead of hipblas, or Intel instead of SYCL or NVIDIA insted of l4t or cublas. Ask for confirmation if there is ambiguity.

.agents/ci-caching.md

@@ -1,250 +0,0 @@
-# CI Build Caching
-
-Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache plus a layered set of prebuilt base images. This file explains how the cache is laid out, what invalidates it, and how to bypass it.
-
-## Workflow surfaces
-
-| Workflow | Purpose | Triggers |
-|---|---|---|
-| `.github/workflows/backend.yml` | Backend container images on master | `push` to master + tags, weekly Sunday cron, `workflow_dispatch` |
-| `.github/workflows/backend_pr.yml` | Backend container images on PRs | `pull_request` |
-| `.github/workflows/backend_build.yml` | Reusable: builds one backend (one arch) by digest | `workflow_call` from above |
-| `.github/workflows/backend_merge.yml` | Reusable: assembles per-arch digests into a multi-arch manifest list | `workflow_call` |
-| `.github/workflows/backend_build_darwin.yml` | Reusable: macOS-native backend builds | `workflow_call` |
-| `.github/workflows/image.yml` / `image-pr.yml` | Root LocalAI image (push / PR) | push / PR |
-| `.github/workflows/image_build.yml` / `image_merge.yml` | Reusable: per-arch root-image build + merge | `workflow_call` |
-| `.github/workflows/base-images.yml` | Builds the prebuilt `base-grpc-*` builder bases | Saturdays 05:00 UTC cron, `workflow_dispatch`, master push touching `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/apt-mirror.sh`, or this workflow |
-
-The matrix that drives `backend.yml` / `backend_pr.yml` lives in **`.github/backend-matrix.yml`** (data-only YAML, not embedded in the workflow). `scripts/changed-backends.js` parses it, applies path-filter logic against the PR diff (PR events) or the GitHub Compare API (push events), and emits the filtered matrix plus a `merge-matrix` for backends with multiple per-arch entries.
-
-## Cache layout
-
-- **Cache registry**: `quay.io/go-skynet/ci-cache`
-- **One tag per matrix entry per arch**, derived from `tag-suffix` and `platform-tag`:
-  - Backend builds (`backend_build.yml`): `cache<tag-suffix>-<platform-tag>`
-    - e.g. `cache-cpu-faster-whisper-amd64`, `cache-cpu-faster-whisper-arm64`, `cache-gpu-nvidia-cuda-13-llama-cpp-amd64`
-  - Root image builds (`image_build.yml`): `cache-localai<tag-suffix>-<platform-tag>` (with a `-core` placeholder when `tag-suffix` is empty, so `cache-localai-core-amd64` for the core image)
-  - Pre-built base images (`base-images.yml`): `cache-base-grpc-<variant>` (one per `(BUILD_TYPE, arch)` permutation)
-- Each tag stores a multi-arch BuildKit cache manifest (`mode=max`), so every intermediate stage is re-usable, not just the final image.
-
-The per-arch suffix exists because amd64 and arm64 builds produce different intermediate content; sharing one cache key would thrash on every cross-arch rebuild.
-
-## Read/write semantics
-
-| Trigger | `cache-from` | `cache-to` |
-|---|---|---|
-| `push` to `master` / tag / cron / dispatch | yes | yes (`mode=max,ignore-error=true`) |
-| `pull_request` | yes | **no** |
-
-PR builds read master's warm cache but never write — this prevents PRs from polluting the shared cache with their experimental state. After merge, the master build for that matrix entry refreshes the cache.
-
-`ignore-error=true` on the write side means a transient quay push failure does not fail the build; the next master push retries.
-
-## Pre-built base images (`base-grpc-*`)
-
-The C++ backend Dockerfiles (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) compile gRPC from source. On a cold build that's ~25–35 min before any LocalAI source compiles. To skip that on CI, `.github/workflows/base-images.yml` builds and pushes a set of pre-prepped builder bases:
-
-| Tag | Contents |
-|---|---|
-| `base-grpc-amd64` / `base-grpc-arm64` | Ubuntu 24.04 + apt build deps + protoc + cmake + gRPC at `/opt/grpc` |
-| `base-grpc-cuda-12-amd64` | the above + CUDA 12.8 toolkit |
-| `base-grpc-cuda-13-amd64` | the above + CUDA 13.0 toolkit (Ubuntu 22.04 base) |
-| `base-grpc-cuda-13-arm64` | the above + CUDA 13.0 sbsa toolkit (Ubuntu 24.04 base) |
-| `base-grpc-l4t-cuda-12-arm64` | JetPack r36.4.0 base (CUDA preinstalled, `SKIP_DRIVERS=true`) + gRPC |
-| `base-grpc-rocm-amd64` | rocm/dev-ubuntu-24.04:7.2.1 base + hipblas/hipblaslt/rocblas + gRPC |
-| `base-grpc-vulkan-amd64` / `base-grpc-vulkan-arm64` | Ubuntu 24.04 + Vulkan SDK 1.4.335 + gRPC |
-| `base-grpc-intel-amd64` | intel/oneapi-basekit:2025.3.2 base + gRPC |
-
-**Single source of truth**: the install logic for all 10 variants lives in `.docker/install-base-deps.sh`. Both `Dockerfile.base-grpc-builder` AND each variant Dockerfile's `builder-fromsource` stage bind-mount and execute the same script — so the prebuilt CI base and the local from-source path are bit-equivalent by construction.
-
-### How variant Dockerfiles consume the base
-
-`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` are multi-target. Three stages plus a final aliasing stage:
-
-- `builder-fromsource` — `FROM ${BASE_IMAGE}` then runs `install-base-deps.sh` and the per-backend compile script. Used when `BUILDER_TARGET=builder-fromsource` (the default; local `make backends/<name>`).
-- `builder-prebuilt` — `FROM ${BUILDER_BASE_IMAGE}` (one of the prebuilt `base-grpc-*` tags) and runs only the per-backend compile script. Used when `BUILDER_TARGET=builder-prebuilt` (CI when the matrix entry sets `builder-base-image`).
-- `FROM ${BUILDER_TARGET} AS builder` — alias resolves the ARG-selected stage to a fixed name (BuildKit doesn't allow ARG expansion in `COPY --from=`).
-- `FROM scratch` + `COPY --from=builder ...package/. ./` — emits the final scratch image with just the package contents.
-
-BuildKit prunes the unreferenced builder stage, so each build only runs the path it needs. `backend_build.yml` derives `BUILDER_TARGET=builder-prebuilt` automatically when the matrix entry has a non-empty `builder-base-image`; otherwise it defaults to `builder-fromsource`.
-
-The matrix `(build-type, platforms)` → `builder-base-image` mapping for llama-cpp / ik-llama-cpp / turboquant entries:
-
-| `build-type` | `platforms` | tag |
-|---|---|---|
-| `''` | `linux/amd64` | `base-grpc-amd64` |
-| `''` | `linux/arm64` | `base-grpc-arm64` |
-| `cublas` cuda 12 | `linux/amd64` | `base-grpc-cuda-12-amd64` |
-| `cublas` cuda 13 | `linux/amd64` | `base-grpc-cuda-13-amd64` |
-| `cublas` cuda 13 | `linux/arm64` | `base-grpc-cuda-13-arm64` |
-| `cublas` cuda 12 + JetPack base | `linux/arm64` | `base-grpc-l4t-cuda-12-arm64` |
-| `hipblas` | `linux/amd64` | `base-grpc-rocm-amd64` |
-| `vulkan` | `linux/amd64` | `base-grpc-vulkan-amd64` |
-| `vulkan` | `linux/arm64` | `base-grpc-vulkan-arm64` |
-| `sycl_*` | `linux/amd64` | `base-grpc-intel-amd64` |
-
-### Bootstrap order when adding a new variant
-
-If you add a new entry to `base-images.yml`'s matrix, the new tag does not exist on quay until the workflow runs. To consume it from a variant entry safely, dispatch the base-images workflow on the branch first:
-
-```bash
-gh workflow run base-images.yml --ref <feature-branch>
-```
-
-Wait for the new variant to push, then merge the consumer change. Otherwise the consumer's CI fails with "image not found."
-
-## Per-arch native builds + manifest merge
-
-Multi-arch backends (and the core LocalAI image) build natively per arch instead of running both arches under QEMU emulation on a single x86 runner. The pattern:
-
-- The matrix has TWO entries per multi-arch backend, sharing the same `tag-suffix` but distinct `platforms` + `platform-tag` + `runs-on`. Example: `-cpu-faster-whisper` has one amd64 entry on `ubuntu-latest` and one arm64 entry on `ubuntu-24.04-arm`.
-- Each per-arch build pushes by **canonical digest only** (no tags) via `outputs: type=image,push-by-digest=true,name-canonical=true,push=true`. The digest is uploaded as an artifact named `digests<tag-suffix>-<platform-tag>` (or `digests-localai<...>` for root-image builds).
-- `scripts/changed-backends.js` detects shared `tag-suffix` and emits a `merge-matrix` output. `backend.yml` / `backend_pr.yml` have a `backend-merge-jobs` job that consumes it and calls `backend_merge.yml`.
-- `backend_merge.yml` downloads all matching digest artifacts and runs `docker buildx imagetools create` to publish the final tagged manifest list pointing at both per-arch digests. Same `docker/metadata-action` config as the original monolithic build, so consumers see no tag-shape change.
-- `image_merge.yml` is the equivalent for the root LocalAI image (`-core` placeholder when `tag-suffix` is empty so the artifact-name glob doesn't over-match across `core` and `gpu-vulkan`).
-
-**`provenance: false` is required on multi-registry digest pushes**: with the default `mode=max` provenance attestation, BuildKit bundles a per-registry attestation manifest into each registry's manifest list, making the resulting list digest diverge across registries. `steps.build.outputs.digest` only matches one of them and the merge step's `imagetools create <reg>@sha256:<digest>` lookup fails on the other. Setting `provenance: false` keeps the digest content-only and identical across registries.
-
-## Path filter on master push
-
-Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynamically through `scripts/changed-backends.js`:
-
-- **PR events**: paginated `pulls/{n}/files` API → filter the matrix to entries whose `dockerfile` path prefix matches the PR diff.
-- **Push events**: GitHub Compare API (`/repos/{owner}/{repo}/compare/{before}...{after}`) → same path-filter logic. Falls back to "run everything" on first-branch push (`event.before` zero), API truncation (≥300 changed files), missing API token, or any thrown error.
-- **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
-- **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
-
-The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
-
-## The `DEPS_REFRESH` cache-buster (Python backends)
-
-Every Python backend goes through the shared `backend/Dockerfile.python`, which ends with:
-
-```dockerfile
-ARG DEPS_REFRESH=initial
-RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
-```
-
-Most Python backends ship `requirements*.txt` files that **do not pin every transitive dep** (`torch`, `transformers`, `vllm`, `diffusers`, etc. are listed without a `==` pin, or with `>=` lower bounds only). With a warm BuildKit cache, the `make` layer hashes only on Dockerfile instructions + COPYed source — not on what `pip install` resolves at runtime. So a warm cache would ship the *first* version of `vllm` ever cached and never pick up upstream releases.
-
-`DEPS_REFRESH` defends against that:
-
-- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W19`) before each build and passes it as a build-arg.
-- The `RUN ... make` layer's BuildKit hash now includes that string, so the layer invalidates **at most once per week**, automatically picking up newer wheels.
-- Within a week, builds stay warm.
-
-This applies only to `Dockerfile.python` because:
-- Go (`Dockerfile.golang`) pins versions in `go.mod` / `go.sum`.
-- Rust (`Dockerfile.rust`) pins via `Cargo.lock`.
-- C++ backends pin gRPC (`v1.65.0`) and llama.cpp at a specific commit; their inputs don't drift between rebuilds.
-
-### Adjusting the cadence
-
-Bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`) for faster refreshes. For one-shot rebuilds without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
-
-## ccache for C++ backend builds
-
-`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` declare a BuildKit cache mount on `/root/.ccache`:
-
-```dockerfile
-RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-```
-
-The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
-
-On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
-
-## Composite actions
-
-Two composite actions handle runner-side prep:
-
-- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
-- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
-
-Both actions run before any docker buildx step.
-
-## Concurrency
-
-All `backend.yml` / `image.yml` / `test.yml` / etc. workflows use:
-
-```yaml
-concurrency:
-  group: ci-<workflow>-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-```
-
-- **PR events** group by PR number → newer pushes to the same PR cancel old runs (intended).
-- **Push events** group by `github.sha` → each master commit gets its own run; rapid-fire merges don't cancel each other (this was a real issue prior — two master pushes 11 seconds apart would cancel the first's CI).
-
-## Self-warming, no separate populator
-
-There is no cron job that pre-warms the BuildKit cache for individual backends. The production builds *are* the populators. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in the variant `builder-fromsource` stage or skipped entirely when consuming `base-grpc-*`, Python wheel installs, etc.). The base-images workflow's weekly cron is the closest thing to a populator and only refreshes the prebuilt builder bases.
-
-## Manually evicting cache
-
-To force a fully cold build for one backend or the whole image:
-
-```bash
-# Delete a single tag (requires quay credentials with admin on the repo)
-curl -X DELETE \
-  -H "Authorization: Bearer ${QUAY_TOKEN}" \
-  https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm-amd64
-
-# List all tags
-curl -s -H "Authorization: Bearer ${QUAY_TOKEN}" \
-  "https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/?limit=100" | jq '.tags[].name'
-```
-
-Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry per arch so a stale tag never bleeds into a different build.
-
-## What the cache does **not** cover
-
-- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
-- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
-- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
-
-## Darwin native caches
-
-`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
-
-| Cache | Path(s) | Key | Scope |
-|---|---|---|---|
-| Go modules + build | `~/go/pkg/mod`, `~/Library/Caches/go-build` | `go.sum` (managed by `actions/setup-go@v5` `cache: true`) | All darwin jobs |
-| Homebrew | `~/Library/Caches/Homebrew/downloads`, selected `/opt/homebrew/Cellar/*` | hash of `backend_build_darwin.yml` | All darwin jobs |
-| ccache (llama.cpp CMake) | `~/Library/Caches/ccache` | pinned `LLAMA_VERSION` from `backend/cpp/llama-cpp/Makefile` | `inputs.backend == 'llama-cpp'` only |
-| Python wheels (uv + pip) | `~/Library/Caches/pip`, `~/Library/Caches/uv` | `inputs.backend` + ISO week (`+%Y-W%V`) + hash of that backend's `requirements*.txt` | `inputs.lang == 'python'` only |
-
-Read/write semantics match the BuildKit cache: `actions/cache/restore` runs every time, `actions/cache/save` is gated on `github.event_name != 'pull_request'`. PRs read master's warm cache but never write back.
-
-The Python wheel cache uses the same ISO-week cache-buster as the Linux `DEPS_REFRESH` build-arg — same problem (unpinned `torch`/`mlx`/`diffusers`/`transformers` resolve to fresh wheels weekly), same ~one-cold-rebuild-per-week solution.
-
-The brew Cellar cache requires `HOMEBREW_NO_AUTO_UPDATE=1` and `HOMEBREW_NO_INSTALL_CLEANUP=1` (set as job-level env). Without those, `brew install` would mutate the very directories that were just restored, defeating the cache.
-
-**Force-link after cache restore**: `actions/cache` restores `/opt/homebrew/Cellar/*` but NOT the `/opt/homebrew/bin/*` symlinks. After a cache hit, `brew install` sees the Cellar entries and decides "already installed" without re-running its link step, leaving the formulas off PATH. The Dependencies step explicitly runs `brew link --overwrite` for every cached formula afterwards to ensure the symlinks exist.
-
-For ccache, the workflow exports `CMAKE_ARGS=… -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache` via `$GITHUB_ENV` before running `make build-darwin-go-backend`. The Makefile in `backend/cpp/llama-cpp/` already forwards `CMAKE_ARGS` through to each variant build (`fallback`, `grpc`, `rpc-server`), so no script changes are needed. The three variants share most TUs, so ccache dedupes object files across them.
-
-`backend_build_darwin.yml` also has a llama-cpp-specific build-step branch that runs `make backends/llama-cpp-darwin` (the bespoke script that compiles three CMake variants and bundles dylibs via `otool`), distinct from the generic `make build-darwin-${lang}-backend` path. This was consolidated from a previously-bespoke top-level `llama-cpp-darwin` job in `backend.yml` so llama-cpp on Darwin honors the same path filter as the other 34 Darwin backends.
-
-### Cache budget on Darwin
-
-GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~800 MB Go cache + ~2 GB brew Cellar + up to 2 GB ccache + ~1.5 GB × 5 python backends. If the cap is hit, prefer collapsing the per-backend Python keys into a shared `pyenv-darwin-shared-<week>` key (accepts more cross-backend churn for a smaller footprint) before reducing other caches.
-
-## Self-hosted runners
-
-`.github/backend-matrix.yml` has zero references to `arc-runner-set` or `bigger-runner` — all backends run on GHA free-tier hosted runners (`ubuntu-latest` for amd64, `ubuntu-24.04-arm` for arm64 native, `macos-14` for Darwin). The migration off self-hosted relied on the per-arch native split (no QEMU emulation) plus `setup-build-disk`'s `/mnt` relocation (~100 GB working space, enough for ROCm dev image + vLLM/torch installs).
-
-One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
-
-## Touching the cache pipeline
-
-When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
-
-1. **Don't drop `DEPS_REFRESH=...` from the build-args** without a replacement strategy (lockfiles, pinned requirements). Otherwise master will silently freeze on whichever versions were cached at the time.
-2. **Keep `(tag-suffix, platform-tag)` unique per matrix entry** — together they're the cache namespace. Two matrix entries sharing a key would clobber each other's cache.
-3. **Keep `cache-to` gated on `github.event_name != 'pull_request'`** — PRs must not write.
-4. **Keep `ignore-error=true` on `cache-to`** — quay registry hiccups must not fail builds.
-5. **Keep `provenance: false` on push-by-digest steps** — multi-registry digest divergence is the Bug We Already Fixed; reintroducing provenance attestation re-breaks the merge.
-6. **`install-base-deps.sh` is the single source of truth for base contents.** Both `Dockerfile.base-grpc-builder` (CI) and the variant Dockerfiles' `builder-fromsource` (local) bind-mount and execute it. If you add a package to one path, add it to the script — don't fork the logic into a Dockerfile RUN.
-7. **After adding a `base-images.yml` matrix variant, run the workflow on your branch before merging consumer changes** that depend on the new tag — otherwise the consumer's CI fails "image not found."

.agents/coding-style.md

@@ -42,14 +42,6 @@ trim_trailing_whitespace = false
 
 Use `github.com/mudler/xlog` for logging which has the same API as slog.
 
-## Go tests
-
-All Go tests — including backend tests — must use [Ginkgo](https://onsi.github.io/ginkgo/) (v2) with Gomega matchers, not the stdlib `testing` package with `t.Run` / `t.Errorf`. A test file should register a suite with `RegisterFailHandler(Fail)` in a `TestXxx(t *testing.T)` bootstrap and use `Describe`/`Context`/`It` blocks for the actual cases. Look at any existing `*_test.go` under `core/` or `pkg/` for a template.
-
-Do not mix styles within a package. If you are extending tests in a package that already uses Ginkgo, keep using Ginkgo. If you find stdlib-style Go tests in the tree, treat them as tech debt to be migrated rather than as a pattern to follow.
-
-This is enforced by `golangci-lint` via the `forbidigo` linter (see `.golangci.yml`); calls like `t.Errorf` / `t.Fatalf` / `t.Run` / `t.Skip` / `t.Logf` are flagged. Run `make lint` locally before submitting; the same check runs in CI (`.github/workflows/lint.yml`).
-
 ## Documentation
 
 The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.

.agents/localai-assistant-mcp.md

@@ -1,97 +0,0 @@
-# LocalAI Assistant — admin MCP server
-
-This document is the contract for **anyone** (human or AI agent) touching LocalAI's admin REST surface, the in-process MCP server that wraps it, or the embedded skill prompts that teach the assistant how to use it. Read this before adding/removing/renaming admin endpoints, MCP tools, or skill recipes.
-
-## What this feature is
-
-`pkg/mcp/localaitools/` is a public Go package that exposes LocalAI's admin/management surface as an MCP server. It is used in two ways:
-
-1. **In-process**: when an admin opens a chat with `metadata.localai_assistant=true`, the chat handler injects the in-memory MCP server (paired `net.Pipe()` transport, no HTTP loopback) so the LLM can install models, manage backends and edit configs by chatting.
-2. **Standalone**: the `local-ai mcp-server --target=…` subcommand serves the same MCP server over stdio, talking HTTP to a remote LocalAI instance.
-
-The two modes share **all** tool definitions and skill prompts. They differ only in their `LocalAIClient` implementation (`inproc/` calls services directly; `httpapi/` calls REST).
-
-## The three things you must keep in sync
-
-When you change LocalAI's admin surface, three layers must stay aligned:
-
-1. **REST endpoint** in `core/http/endpoints/localai/*.go`.
-2. **MCP tool registration** in `pkg/mcp/localaitools/tools_*.go`, plus a method on `LocalAIClient` (in `client.go`) and implementations in both `inproc/client.go` **and** `httpapi/client.go`.
-3. **Skill prompt** under `pkg/mcp/localaitools/prompts/skills/*.md` — the markdown that teaches the LLM how to use the new tool. If the new tool fits an existing recipe, update that recipe; otherwise add a new file.
-
-If you ship a REST endpoint without (2) and (3), conversational admins won't see the feature.
-
-## Checklist for adding a new admin endpoint
-
-- [ ] REST endpoint exists in `core/http/endpoints/localai/*.go` and is gated by `auth.RequireAdmin()` in `core/http/routes/localai.go`.
-- [ ] `LocalAIClient` interface in `pkg/mcp/localaitools/client.go` has a method covering the new operation.
-- [ ] DTOs added/updated in `pkg/mcp/localaitools/dto.go` (JSON-tagged; never expose raw service types).
-- [ ] `inproc/client.go` implements the new method by calling the service directly (not via HTTP loopback).
-- [ ] `httpapi/client.go` implements the new method by calling the REST endpoint.
-- [ ] Tool registration added in the appropriate `pkg/mcp/localaitools/tools_*.go`. Mutating tools must reference safety rule 1 in the description.
-- [ ] If the tool is mutating, ensure `Options{DisableMutating: true}` skips it (mirror the pattern in `tools_models.go`).
-- [ ] Skill prompt added or updated under `pkg/mcp/localaitools/prompts/skills/`. The prompt must instruct the LLM when to call the tool, what to ask the user first, and what to do on error.
-- [ ] Tests:
-   - `pkg/mcp/localaitools/server_test.go` adds the tool name to `expectedFullCatalog` and `expectedReadOnlyCatalog` (if read-only).
-   - Tool dispatch is added to `TestEachToolDispatchesToClient`.
-   - `pkg/mcp/localaitools/httpapi/client_test.go` covers the new HTTP path.
-
-## Adding a new skill recipe (no new tool)
-
-Sometimes you want to teach the LLM a new pattern that uses existing tools. Drop a markdown file under `pkg/mcp/localaitools/prompts/skills/<verb>_<noun>.md`. The file is automatically embedded by `//go:embed` and assembled into the system prompt in lexicographic order. No Go changes needed.
-
-Conventions:
-- Filename: `<verb>_<noun>.md` (e.g. `install_chat_model.md`, `upgrade_backend.md`).
-- First line: `# Skill: <Title Case description>`.
-- Number the steps. Reference exact tool names in backticks.
-- If the skill mutates state, remind the LLM to confirm with the user.
-
-## Code conventions
-
-These rules guard against the magic-literal drift that surfaced in the first audit. Do not re-introduce bare strings.
-
-- **Tool names** always come from the `Tool*` constants in `pkg/mcp/localaitools/tools.go`. Tool registrations, the test catalog (`server_test.go`'s `expectedFullCatalog` / `expectedReadOnlyCatalog`), and dispatch tables reference the constants. The embedded skill prompts under `prompts/` keep bare strings — that's the one allowed exception, and `TestPromptsContainSafetyAnchors` enforces alignment.
-- **Toggle/pin actions** use the `modeladmin.Action` type (`pkg/mcp/localaitools` and `core/services/modeladmin`). Use `ActionEnable`/`ActionDisable`/`ActionPin`/`ActionUnpin`; never bare `"enable"`/`"pin"` strings.
-- **Capability tags** for `list_installed_models` use the `localaitools.Capability` type (`capability.go`). The `LocalAIClient.ListInstalledModels` interface takes a typed `Capability`, and the `inproc` switch only accepts canonical values (`"embed"`/`"embedding"` are not aliases — only `CapabilityEmbeddings`).
-- **HTTP error checks** in `httpapi.Client` use `errors.Is(err, ErrHTTPNotFound)`, not substring matches on `err.Error()`. The typed `*HTTPError` carries `StatusCode` and `Body`; add new sentinel errors as needed rather than re-introducing string matching.
-- **Channel sends** to `GalleryService.ModelGalleryChannel` / `BackendGalleryChannel` from inproc clients MUST select on `ctx.Done()` so a cancelled chat completion releases the goroutine. See `inproc.sendModelOp` / `sendBackendOp`.
-- **Disk writes** of model config YAML go through `modeladmin.writeFileAtomic` (temp file + `os.Rename`). `os.WriteFile` truncates on crash and corrupts the model.
-- **MCP server lifecycle**: every initialised holder MUST register `Close()` with `signals.RegisterGracefulTerminationHandler`. The standalone `mcp-server` CLI uses `signal.NotifyContext` to honour SIGINT/SIGTERM.
-
-## File map (where to look)
-
-```
-pkg/mcp/localaitools/
-  client.go              # LocalAIClient interface + DTO registry
-  dto.go                 # JSON-tagged DTOs shared by both client impls
-  server.go              # NewServer(client, opts) — registers tools
-  tools.go               # Tool* name constants (single source of truth)
-  capability.go          # Capability type + constants
-  tools_models.go        # gallery_search, install_model, import_model_uri, ...
-  tools_backends.go
-  tools_config.go
-  tools_system.go
-  tools_state.go
-  prompts.go             # //go:embed loader + SystemPrompt(opts)
-  prompts/00_role.md
-  prompts/10_safety.md   # SAFETY RULES — change with care
-  prompts/20_tools.md    # curated tool catalog with one-liners
-  prompts/skills/*.md
-  inproc/client.go       # in-process LocalAIClient (services-direct)
-  httpapi/client.go      # REST LocalAIClient (for standalone CLI / remote)
-core/http/endpoints/mcp/
-  localai_assistant.go   # process-wide holder + LocalToolExecutor
-core/cli/mcp_server.go   # local-ai mcp-server subcommand
-```
-
-## Why two clients
-
-The in-process MCP server runs inside the same LocalAI binary that serves chat. Going over HTTP loopback would (a) require minting a synthetic admin API key for the server to authenticate against itself, (b) double-marshal every tool dispatch, and (c) lose access to in-process channels (e.g. `GalleryService.ModelGalleryChannel` for streaming install progress). So in-process uses `inproc.Client`. The standalone stdio CLI talks to a *remote* LocalAI; HTTP is the only option, so it uses `httpapi.Client`. Both implement the same `LocalAIClient` interface, and the parity test in `pkg/mcp/localaitools/parity_test.go` (when present) keeps their output equivalent.
-
-## Why prompt-enforced confirmation, not code gates
-
-The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.md` rule 1) that requires the LLM to summarise the action and wait for explicit user confirmation before calling it. There is no `plan_*`/`apply_*` two-step in code. If you add a mutating tool, do **not** add per-tool confirmation logic in Go — instead, list the new tool name in `prompts/10_safety.md` so the LLM knows it falls under the confirmation rule.
-
-## Distributed mode
-
-The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period.

.agents/sglang-backend.md

@@ -1,62 +0,0 @@
-# Working on the SGLang Backend
-
-The SGLang backend lives at `backend/python/sglang/backend.py` (async gRPC). It wraps SGLang's `Engine` (`sglang.srt.entrypoints.engine.Engine`) and translates LocalAI's gRPC `PredictOptions` into SGLang sampling params + outputs into `Reply.chat_deltas`. Structurally it mirrors `backend/python/vllm/backend.py` — keep them shaped the same so changes in one have an obvious analog in the other.
-
-## `engine_args` is the universal escape hatch
-
-A small fixed set of fields on `ModelOptions` is mapped to typed SGLang kwargs in `LoadModel` (model, quantization, load_format, gpu_memory_utilization → mem_fraction_static, trust_remote_code, enforce_eager → disable_cuda_graph, tensor_parallel_size → tp_size, max_model_len → context_length, dtype). **Everything else** flows through the `engine_args:` YAML map.
-
-Validation happens in `_apply_engine_args`. Keys are checked against `dataclasses.fields(ServerArgs)` (`sglang.srt.server_args.ServerArgs` is a flat `@dataclass` with ~380 fields). Unknown keys raise `ValueError` at LoadModel time with a `difflib.get_close_matches` suggestion — same shape as the vLLM backend.
-
-**Precedence:** typed `ModelOptions` fields populate `engine_kwargs` first, then `engine_args` overrides them. So a YAML that sets both `gpu_memory_utilization: 0.9` and `engine_args.mem_fraction_static: 0.5` ends up at `0.5`. Document this when answering "why didn't my YAML field stick?".
-
-**ServerArgs is flat.** Unlike vLLM, where speculative decoding is nested under `engine_args.speculative_config: {...}`, SGLang exposes flat top-level fields: `speculative_algorithm`, `speculative_draft_model_path`, `speculative_num_steps`, `speculative_eagle_topk`, `speculative_num_draft_tokens`, `speculative_dflash_block_size`, etc. There is no `speculative_config:` dict. Same goes for compilation, kv-transfer, attention — all flat.
-
-The canonical reference is `python/sglang/srt/server_args.py:ServerArgs` (line ~304). When SGLang adds new flags, no LocalAI code change is needed — they're automatically available via `engine_args:`. The validator picks them up because it introspects the live dataclass.
-
-## Speculative decoding cheatsheet
-
-`--speculative-algorithm` accepts `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`, `DFLASH`. `NEXTN` is silently rewritten to `EAGLE` in `ServerArgs.__post_init__` (`server_args.py:3286-3287`). MTP (Multi-Token Prediction) is the same EAGLE path with `num_steps=1, eagle_topk=1, num_draft_tokens=2` against a target whose architecture has multi-token heads (e.g. MiMo-7B-RL, DeepSeek-V3-MTP).
-
-| Algorithm | Drafter requirement | Gallery demo target | Gallery demo drafter |
-|-----------|--------------------|---------------------|----------------------|
-| `NEXTN` / `EAGLE` (MTP) | Assistant drafter or built-in heads | google/gemma-4-E2B-it, google/gemma-4-E4B-it | google/gemma-4-E2B-it-assistant, google/gemma-4-E4B-it-assistant |
-| `EAGLE3` | EAGLE3 draft head | (no gallery entry yet) | e.g. jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B |
-| `DFLASH` | Block-diffusion drafter | (no gallery entry yet) | e.g. z-lab/Qwen3-4B-DFlash-b16 |
-| `STANDALONE` | Smaller LLM as drafter | (no gallery entry yet) | any smaller chat-tuned LLM in the same family |
-| `NGRAM` | None — uses prefix history | (no gallery entry yet) | n/a |
-
-The Gemma 4 demos use `mem_fraction_static: 0.85` (cookbook default) and the cookbook's `num_steps=5, num_draft_tokens=6, eagle_topk=1` parameters. Other algorithms are reachable from any user YAML via `engine_args:` but don't have shipped demos yet — that's a deliberate gallery scope choice, not a backend limitation.
-
-Gemma 4 support requires sglang built from a commit that includes [PR #21952](https://github.com/sgl-project/sglang/pull/21952). LocalAI's pinned release for cublas12 / cublas13 includes it. The `l4t13` (JetPack 7 / sbsa cu130) build floors at `sglang>=0.5.0` because the `pypi.jetson-ai-lab.io` mirror still ships only `0.5.1.post2` as of 2026-05-06 — Gemma 4 / MTP recipes are therefore not available on l4t13 until that mirror catches up. `backend.py` keeps backward compat with the 0.5.x → 0.5.11 `SamplingParams.seed` → `sampling_seed` rename via runtime detection.
-
-Compatibility caveats per the SGLang docs: DFLASH and NGRAM are incompatible with `enable_dp_attention`; DFLASH requires `pp_size == 1`; STANDALONE is incompatible with `enable_dp_attention`; NGRAM is CUDA-only and disables the overlap scheduler.
-
-### `mem_fraction_static` + quantization + MTP on consumer GPUs
-
-When combining online weight quantization (`engine_args.quantization: fp8` / `awq` / etc.) with built-in-head MTP (`speculative_algorithm: EAGLE`/`NEXTN`) on a tight VRAM budget, sglang's default `mem_fraction_static: 0.85` will OOM during draft-worker init. The reason: sglang quantizes the **target** model's transformer blocks but loads the **MTP draft worker's vocab embedding** at the source dtype (typically bf16). For a 7 B-class model with a 150k-token vocab × 4096 hidden, that's another ~1.2 GiB allocated *after* the static pool is reserved. At 0.85 fraction on a 16 GB card there's no room left.
-
-Workaround: drop `mem_fraction_static` to ~0.7 so the post-static heap can absorb the MTP embedding alloc + CUDA graph private pools. Verified end-to-end on MiMo-7B-RL + fp8 + MTP on a 16 GB RTX 5070 Ti (`gallery/sglang-mimo-7b-mtp.yaml`) at ~88 tok/s. Models with larger vocabs or more MTP layers (e.g. DeepSeek-V3-MTP) need an even smaller fraction.
-
-This isn't documented anywhere upstream as of 2026-05-06 — the SGLang Gemma 4 cookbook uses 0.85 because their MTP path doesn't go through `eagle_worker_v2.py` for an embedding-bearing draft module. Don't blanket-apply 0.7 across all sglang YAMLs; only when MTP-with-built-in-heads + quantization combine.
-
-## Tool-call and reasoning parsers stay on `Options[]`
-
-ServerArgs has `tool_call_parser` and `reasoning_parser` fields, and the backend does pass them through to `Engine` so SGLang's own HTTP/OAI surface keeps working. But for the **LocalAI** request path the backend constructs fresh per-request parser instances in `_make_parsers` (`backend.py:286`) because the parsers are stateful — the streaming and non-streaming paths each need their own.
-
-So the user-facing knob stays on `Options[]`:
-
-```yaml
-options:
-  - tool_parser:hermes
-  - reasoning_parser:deepseek_r1
-```
-
-Putting these in `engine_args:` will set them on `ServerArgs` but the LocalAI-level streaming `ChatDelta` will not pick them up. Don't recommend that path.
-
-## What's missing today (out of scope, but worth tracking)
-
-- `core/config/hooks_sglang.go` — there is no SGLang equivalent of `hooks_vllm.go`. The vLLM hook auto-selects parsers for known model families from `parser_defaults.json` and seeds production engine_args defaults. A symmetric hook for SGLang could reuse the same `parser_defaults.json` (the SGLang parser names are different but the family detection is shared) and seed defaults like `enable_metrics: true` or attention-backend choices.
-- `core/gallery/importers/sglang.go` — vLLM has an importer that resolves model architecture → parser defaults at gallery-import time. A matching importer for SGLang would let `local-ai install` populate sensible parsers automatically.
-
-These should be a follow-up PR, not a blocker for the engine_args feature.

.docker/apt-mirror.sh

@@ -1,39 +0,0 @@
-#!/bin/sh
-# Reconfigure Ubuntu apt sources to point at an alternate mirror.
-#
-# Used by Dockerfiles via `RUN --mount=type=bind,source=.docker/apt-mirror.sh,...`
-# and by CI workflows on the runner to mitigate outages of the default
-# archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com pool.
-#
-# Inputs (env):
-#   APT_MIRROR        Replacement for archive.ubuntu.com and security.ubuntu.com
-#                     (e.g. "http://azure.archive.ubuntu.com" or
-#                      "https://mirrors.edge.kernel.org").
-#                     Leave empty to keep upstream. The trailing "/ubuntu/..."
-#                     path is preserved by the rewrite.
-#   APT_PORTS_MIRROR  Replacement for ports.ubuntu.com (arm64/ppc64el/...).
-#                     Leave empty to keep upstream.
-#
-# Both default to empty, in which case the script is a no-op.
-
-set -e
-
-if [ -z "${APT_MIRROR}" ] && [ -z "${APT_PORTS_MIRROR}" ]; then
-    exit 0
-fi
-
-# Ubuntu 24.04 (noble) ships DEB822 sources at /etc/apt/sources.list.d/ubuntu.sources;
-# older releases use /etc/apt/sources.list. We rewrite whichever exists.
-for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
-    [ -f "$f" ] || continue
-    if [ -n "${APT_MIRROR}" ]; then
-        # Use a comma delimiter so the alternation pipe in the regex
-        # is not interpreted as the s/// separator.
-        sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
-    fi
-    if [ -n "${APT_PORTS_MIRROR}" ]; then
-        sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
-    fi
-done
-
-echo "apt-mirror: rewrote sources (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"

.docker/ik-llama-cpp-compile.sh

@@ -1,30 +0,0 @@
-#!/usr/bin/env bash
-# Shared compile logic for backend/Dockerfile.ik-llama-cpp.
-# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
-
-set -euxo pipefail
-
-export CCACHE_DIR=/root/.ccache
-ccache --max-size=5G || true
-ccache -z || true
-
-export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
-fi
-
-cd /LocalAI/backend/cpp/ik-llama-cpp
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  # ARM64 / ROCm: build without x86 SIMD
-  make ik-llama-cpp-fallback
-else
-  # ik_llama.cpp's IQK kernels require at least AVX2
-  make ik-llama-cpp-avx2
-fi
-
-ccache -s || true

.docker/install-base-deps.sh

@@ -1,244 +0,0 @@
-#!/usr/bin/env bash
-# Single source of truth for builder-base contents.
-#
-# Used by:
-#   - backend/Dockerfile.base-grpc-builder        (CI prebuilt-base source of truth)
-#   - backend/Dockerfile.llama-cpp                (builder-fromsource stage)
-#   - backend/Dockerfile.ik-llama-cpp             (builder-fromsource stage)
-#   - backend/Dockerfile.turboquant               (builder-fromsource stage)
-#
-# All four files invoke this script via
-#   RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-#       --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-#       bash /usr/local/sbin/install-base-deps
-#
-# so the prebuilt CI base image and the from-source local-dev path are
-# bit-equivalent by construction.
-#
-# Inputs (env, populated from Dockerfile ARG/ENV):
-#   BUILD_TYPE                ("cublas"|"l4t"|"hipblas"|"vulkan"|"sycl"|"clblas"|"")
-#   CUDA_MAJOR_VERSION        ("12" | "13" | "")
-#   CUDA_MINOR_VERSION        ("8" | "0" | "")
-#   TARGETARCH                ("amd64" | "arm64")
-#   UBUNTU_VERSION            ("2204" | "2404")
-#   SKIP_DRIVERS              ("false" | "true")
-#   CMAKE_FROM_SOURCE         ("false" | "true")
-#   CMAKE_VERSION             ("3.31.10")
-#   GRPC_VERSION              ("v1.65.0")
-#   GRPC_MAKEFLAGS            ("-j4 -Otarget")
-#   APT_MIRROR / APT_PORTS_MIRROR  (optional; consumed by /usr/local/sbin/apt-mirror)
-#   AMDGPU_TARGETS            (optional; only relevant for hipblas downstream)
-#
-# IMPORTANT: install logic is copied verbatim from the prior in-Dockerfile
-# RUN blocks. Do not paraphrase apt invocations / version pins / sed line
-# numbers / deb URLs — the bit-equivalence guarantee depends on it.
-
-set -eux
-
-# --- 0. apt mirror rewrite (no-op when APT_MIRROR / APT_PORTS_MIRROR unset) ---
-if [ -x /usr/local/sbin/apt-mirror ]; then
-    APT_MIRROR="${APT_MIRROR:-}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR:-}" \
-        sh /usr/local/sbin/apt-mirror
-fi
-
-export DEBIAN_FRONTEND=noninteractive
-export MAKEFLAGS="${GRPC_MAKEFLAGS:-}"
-
-# --- 1. Base apt build deps ---
-apt-get update
-apt-get install -y --no-install-recommends \
-    build-essential \
-    ccache git \
-    ca-certificates \
-    make \
-    pkg-config libcurl4-openssl-dev \
-    curl unzip \
-    libssl-dev wget
-apt-get clean
-rm -rf /var/lib/apt/lists/*
-
-# --- 2. Vulkan SDK (BUILD_TYPE=vulkan) ---
-# NB: this block intentionally installs `cmake` via apt as part of the
-# Vulkan tooling — must run before the dedicated CMake step below.
-if [ "${BUILD_TYPE:-}" = "vulkan" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y  --no-install-recommends \
-        software-properties-common pciutils wget gpg-agent
-    apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
-        libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
-        libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
-        git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
-        ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
-        clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-    if [ "amd64" = "${TARGETARCH:-}" ]; then
-        wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz"
-        tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz
-        rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz
-        mkdir -p /opt/vulkan-sdk
-        mv 1.4.335.0 /opt/vulkan-sdk/
-        ( cd /opt/vulkan-sdk/1.4.335.0 && \
-          ./vulkansdk --no-deps --maxjobs \
-              vulkan-loader \
-              vulkan-validationlayers \
-              vulkan-extensionlayer \
-              vulkan-tools \
-              shaderc )
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/
-        rm -rf /opt/vulkan-sdk
-    fi
-    if [ "arm64" = "${TARGETARCH:-}" ]; then
-        mkdir vulkan
-        ( cd vulkan && \
-          curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-          tar -xvf vulkan-sdk.tar.xz && \
-          rm vulkan-sdk.tar.xz && \
-          cd 1.4.335.0 && \
-          cp -rfv aarch64/bin/* /usr/bin/ && \
-          cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-          cp -rfv aarch64/include/* /usr/include/ && \
-          cp -rfv aarch64/share/* /usr/share/ )
-        rm -rf vulkan
-    fi
-    ldconfig
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 3. CUDA toolkit (BUILD_TYPE=cublas|l4t) ---
-if { [ "${BUILD_TYPE:-}" = "cublas" ] || [ "${BUILD_TYPE:-}" = "l4t" ]; } && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y  --no-install-recommends \
-        software-properties-common pciutils
-    if [ "amd64" = "${TARGETARCH:-}" ]; then
-        curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb"
-    fi
-    if [ "arm64" = "${TARGETARCH:-}" ]; then
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
-            curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb"
-        else
-            curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb"
-        fi
-    fi
-    dpkg -i cuda-keyring_1.1-1_all.deb
-    rm -f cuda-keyring_1.1-1_all.deb
-    apt-get update
-    apt-get install -y --no-install-recommends \
-        "cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
-    if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "${TARGETARCH:-}" ]; then
-        apt-get install -y --no-install-recommends \
-            "libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-            "libcudnn9-cuda-${CUDA_MAJOR_VERSION}" \
-            "cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-            "libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
-    fi
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 4. cuDSS / NVPL on arm64 + cublas (legacy JetPack / Tegra) ---
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-if [ "${BUILD_TYPE:-}" = "cublas" ] && [ "${TARGETARCH:-}" = "arm64" ]; then
-    wget "https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
-    dpkg -i "cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
-    cp /var/cudss-local-tegra-repo-ubuntu"${UBUNTU_VERSION}"-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/
-    apt-get update
-    apt-get -y install cudss "cudss-cuda-${CUDA_MAJOR_VERSION}"
-    wget "https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
-    dpkg -i "nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
-    cp /var/nvpl-local-repo-ubuntu"${UBUNTU_VERSION}"-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/
-    apt-get update
-    apt-get install -y nvpl
-fi
-
-# --- 5. clBLAS (BUILD_TYPE=clblas) ---
-# Present in variant Dockerfiles' from-source path but not in master's
-# Dockerfile.base-grpc-builder. No CI matrix entry currently uses this,
-# but keep parity so a future BUILD_TYPE=clblas build doesn't drift.
-if [ "${BUILD_TYPE:-}" = "clblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y --no-install-recommends \
-        libclblast-dev
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 6. ROCm / HIP build deps (BUILD_TYPE=hipblas) ---
-if [ "${BUILD_TYPE:-}" = "hipblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y --no-install-recommends \
-        hipblas-dev \
-        hipblaslt-dev \
-        rocblas-dev
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-    # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install,
-    # which results in local-ai and others not being able to locate the libraries.
-    # We run ldconfig ourselves to work around this packaging deficiency.
-    ldconfig
-    # Log which GPU architectures have rocBLAS kernel support
-    echo "rocBLAS library data architectures:"
-    (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
-        echo "WARNING: No rocBLAS kernel data found"
-fi
-
-echo "TARGETARCH: ${TARGETARCH:-}"
-
-# --- 7. protoc (always) ---
-# The version in 22.04 is too old. We will create one as part of installing
-# the GRPC build below but that will also bring in a newer version of absl
-# which stablediffusion cannot compile with. This version of protoc is only
-# here so that we can generate the grpc code for the stablediffusion build.
-if [ "amd64" = "${TARGETARCH:-}" ]; then
-    curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip
-    unzip -j -d /usr/local/bin protoc.zip bin/protoc
-    rm protoc.zip
-fi
-if [ "arm64" = "${TARGETARCH:-}" ]; then
-    curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip
-    unzip -j -d /usr/local/bin protoc.zip bin/protoc
-    rm protoc.zip
-fi
-
-# --- 8. CMake (apt or compiled from source) ---
-# The version in 22.04 is too old. Vulkan path above already pulled cmake
-# via apt; the from-source branch here will install over it which is fine.
-if [ "${CMAKE_FROM_SOURCE:-false}" = "true" ]; then
-    curl -L -s "https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz" -o cmake.tar.gz
-    tar xvf cmake.tar.gz
-    ( cd "cmake-${CMAKE_VERSION}" && ./configure && make && make install )
-else
-    apt-get update
-    apt-get install -y \
-        cmake
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 9. gRPC compile + install at /opt/grpc ---
-# We install GRPC to a different prefix here so that we can copy in only
-# the build artifacts later — saves several hundred MB on the final docker
-# image size vs copying in the entire GRPC source tree and running
-# `make install` in the target container.
-#
-# The TESTONLY abseil sed patch and /opt/grpc prefix are load-bearing —
-# downstream Dockerfiles `COPY` /opt/grpc to /usr/local (or rely on the
-# prebuilt base having it at /opt/grpc).
-mkdir -p /build
-cd /build
-git clone --recurse-submodules --jobs 4 -b "${GRPC_VERSION}" --depth 1 --shallow-submodules https://github.com/grpc/grpc
-mkdir -p /build/grpc/cmake/build
-cd /build/grpc/cmake/build
-sed -i "216i\\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt"
-cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../..
-make
-make install
-cd /
-rm -rf /build

.docker/llama-cpp-compile.sh

@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-# Shared compile logic for backend/Dockerfile.llama-cpp.
-# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
-
-set -euxo pipefail
-
-export CCACHE_DIR=/root/.ccache
-ccache --max-size=5G || true
-ccache -z || true
-
-export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
-fi
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  cd /LocalAI/backend/cpp/llama-cpp
-  make llama-cpp-fallback
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-else
-  cd /LocalAI/backend/cpp/llama-cpp
-  make llama-cpp-avx
-  make llama-cpp-avx2
-  make llama-cpp-avx512
-  make llama-cpp-fallback
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-fi
-
-ccache -s || true

.docker/turboquant-compile.sh

@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-# Shared compile logic for backend/Dockerfile.turboquant.
-# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
-
-set -euxo pipefail
-
-export CCACHE_DIR=/root/.ccache
-ccache --max-size=5G || true
-ccache -z || true
-
-export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/turboquant-*-build
-fi
-
-cd /LocalAI/backend/cpp/turboquant
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  make turboquant-fallback
-  make turboquant-grpc
-  make turboquant-rpc-server
-else
-  make turboquant-avx
-  make turboquant-avx2
-  make turboquant-avx512
-  make turboquant-fallback
-  make turboquant-grpc
-  make turboquant-rpc-server
-fi
-
-ccache -s || true

.github/actions/configure-apt-mirror/action.yml

@@ -1,100 +0,0 @@
-name: 'Configure apt mirror'
-description: |
-  Reconfigure the GitHub Actions runner's Ubuntu apt sources to use an
-  alternate mirror, and emit the effective URLs as outputs so callers can
-  forward them as Docker build-args.
-
-  Two mirror profiles depending on where the runner lives, because the
-  best mirror differs by network:
-
-    * github-hosted runners run on Azure, so they default to the
-      Azure-hosted Ubuntu mirror (lowest latency, same VPC).
-    * self-hosted runners (arc-runner-set, bigger-runner, ...) typically
-      cannot route to azure.archive.ubuntu.com, so they default to the
-      kernel.org mirror, which is publicly reachable from anywhere.
-
-  Pass an empty string to either input to skip the rewrite for that
-  profile and keep upstream archive.ubuntu.com / ports.ubuntu.com.
-
-inputs:
-  github-hosted-mirror:
-    description: 'archive/security mirror URL for github-hosted runners (empty = upstream)'
-    required: false
-    default: 'http://azure.archive.ubuntu.com'
-  github-hosted-ports-mirror:
-    description: 'ports.ubuntu.com mirror URL for github-hosted runners (empty = upstream)'
-    required: false
-    default: 'http://azure.ports.ubuntu.com'
-  self-hosted-mirror:
-    description: 'archive/security mirror URL for self-hosted runners (empty = upstream)'
-    required: false
-    # HTTP, not HTTPS: the bare ubuntu:24.04 builder image doesn't ship
-    # ca-certificates, so the very first apt-get update over TLS would
-    # fail with "No system certificates available" before it can install
-    # anything. apt validates package integrity via GPG signatures, so
-    # plain HTTP is safe for the archive itself.
-    default: 'http://mirrors.edge.kernel.org'
-  self-hosted-ports-mirror:
-    description: 'ports.ubuntu.com mirror URL for self-hosted runners (empty = upstream)'
-    required: false
-    # mirrors.edge.kernel.org does NOT carry /ubuntu-ports/ — only the
-    # main /ubuntu/ archive — so arm64 builds 404 there. Leave ports
-    # upstream by default. The original DDoS was on archive.ubuntu.com
-    # so ports.ubuntu.com remains the path of least surprise.
-    default: ''
-
-outputs:
-  effective-mirror:
-    description: 'The mirror URL actually applied for this runner (or empty)'
-    value: ${{ steps.pick.outputs.mirror }}
-  effective-ports-mirror:
-    description: 'The ports mirror URL actually applied for this runner (or empty)'
-    value: ${{ steps.pick.outputs.ports-mirror }}
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Pick effective mirror for this runner
-      id: pick
-      shell: bash
-      env:
-        RUNNER_ENV: ${{ runner.environment }}
-        GH_MIRROR: ${{ inputs.github-hosted-mirror }}
-        GH_PORTS_MIRROR: ${{ inputs.github-hosted-ports-mirror }}
-        SH_MIRROR: ${{ inputs.self-hosted-mirror }}
-        SH_PORTS_MIRROR: ${{ inputs.self-hosted-ports-mirror }}
-      run: |
-        if [ "${RUNNER_ENV}" = "github-hosted" ]; then
-          MIRROR="${GH_MIRROR}"
-          PORTS_MIRROR="${GH_PORTS_MIRROR}"
-        else
-          MIRROR="${SH_MIRROR}"
-          PORTS_MIRROR="${SH_PORTS_MIRROR}"
-        fi
-        echo "configure-apt-mirror: runner=${RUNNER_ENV} mirror='${MIRROR}' ports-mirror='${PORTS_MIRROR}'"
-        echo "mirror=${MIRROR}" >> "$GITHUB_OUTPUT"
-        echo "ports-mirror=${PORTS_MIRROR}" >> "$GITHUB_OUTPUT"
-
-    - name: Rewrite apt sources
-      if: steps.pick.outputs.mirror != '' || steps.pick.outputs.ports-mirror != ''
-      shell: bash
-      env:
-        APT_MIRROR: ${{ steps.pick.outputs.mirror }}
-        APT_PORTS_MIRROR: ${{ steps.pick.outputs.ports-mirror }}
-      run: |
-        set -e
-        # Ubuntu 24.04 (noble) ships DEB822 sources at
-        # /etc/apt/sources.list.d/ubuntu.sources; older releases use
-        # /etc/apt/sources.list. Rewrite whichever exists.
-        for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
-          sudo test -f "$f" || continue
-          if [ -n "${APT_MIRROR}" ]; then
-            # Comma delimiter so the alternation pipe in the regex is not
-            # interpreted as the s/// separator.
-            sudo sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
-          fi
-          if [ -n "${APT_PORTS_MIRROR}" ]; then
-            sudo sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
-          fi
-        done
-        echo "Runner apt mirror configured (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"

.github/actions/free-disk-space/action.yml

@@ -1,65 +0,0 @@
-name: 'Free disk space on hosted runners'
-description: |
-  Aggressively clean GitHub-hosted ubuntu-latest runners to reclaim ~6-10 GB
-  of working space before docker buildx steps. Combines jlumbroso/free-disk-space
-  with explicit apt purges of large packages we never use (dotnet, ghc, mono,
-  android, jdk, ...).
-
-  No-op on self-hosted runners; pass mode=skip to force-disable.
-
-inputs:
-  mode:
-    description: 'hosted (default — clean) or skip (no-op)'
-    required: false
-    default: 'hosted'
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Free Disk Space (Ubuntu)
-      if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
-      uses: jlumbroso/free-disk-space@main
-      with:
-        tool-cache: true
-        android: true
-        dotnet: true
-        haskell: true
-        large-packages: true
-        docker-images: true
-        swap-storage: true
-
-    - name: Release space from worker
-      if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
-      shell: bash
-      run: |
-        echo "Listing top largest packages"
-        pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-        head -n 30 <<< "${pkgs}"
-        df -h
-        sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-        sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
-        sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
-        sudo rm -rf /usr/local/lib/android
-        sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-        sudo rm -rf /usr/share/dotnet
-        sudo apt-get remove -y '^mono-.*' || true
-        sudo apt-get remove -y '^ghc-.*' || true
-        sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-        sudo apt-get remove -y 'php.*' || true
-        sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-        sudo apt-get remove -y '^google-.*' || true
-        sudo apt-get remove -y azure-cli || true
-        sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-        sudo apt-get remove -y '^gfortran-.*' || true
-        sudo apt-get remove -y microsoft-edge-stable || true
-        sudo apt-get remove -y firefox || true
-        sudo apt-get remove -y powershell || true
-        sudo apt-get remove -y r-base-core || true
-        sudo apt-get autoremove -y
-        sudo apt-get clean
-        sudo rm -rfv build || true
-        sudo rm -rf /usr/share/dotnet || true
-        sudo rm -rf /opt/ghc || true
-        sudo rm -rf "/usr/local/share/boost" || true
-        sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
-        df -h

.github/actions/setup-build-disk/action.yml

@@ -1,59 +0,0 @@
-name: 'Set up build disk on hosted runners'
-description: |
-  Relocate Docker's data-root to /mnt (which has ~75 GB free, vs ~20 GB
-  on / after free-disk-space). Combined with the apt cleanup, gives
-  ~100 GB working space for buildx — enough for ROCm dev image + vLLM
-  torch install + flash-attn build.
-
-  No-op on:
-    - self-hosted runners (no /mnt expectation)
-    - non-X64 runners (verify /mnt shape on ubuntu-24.04-arm separately
-      before enabling there — see Task 3.2 in the migration plan)
-    - mode=skip (force-disable from caller)
-
-  Must run after free-disk-space (which removes large packages — would
-  fail mid-uninstall if Docker were stopped) and before any Docker
-  operation (setup-qemu, setup-buildx, login, build) so the relocated
-  data-root catches all subsequent docker activity.
-
-inputs:
-  mode:
-    description: 'auto (default — relocate on hosted X64 only) or skip'
-    required: false
-    default: 'auto'
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Relocate Docker data-root to /mnt
-      if: inputs.mode == 'auto' && runner.environment == 'github-hosted' && runner.arch == 'X64'
-      shell: bash
-      run: |
-        set -euo pipefail
-        echo "Before relocation:"
-        df -h / /mnt || true
-        sudo systemctl stop docker docker.socket
-        sudo mkdir -p /mnt/docker-data /mnt/docker-tmp
-        # buildx CLI runs as the unprivileged runner user and creates
-        # config dirs under TMPDIR before binding them into the buildkit
-        # container. /mnt is owned by root by default; mirror /tmp's
-        # 1777 (world-writable + sticky) so non-root processes can write.
-        sudo chmod 1777 /mnt/docker-tmp
-        if [ -d /var/lib/docker ] && [ ! -L /var/lib/docker ]; then
-          sudo rsync -a /var/lib/docker/ /mnt/docker-data/
-          sudo rm -rf /var/lib/docker
-          sudo ln -s /mnt/docker-data /var/lib/docker
-        fi
-        # daemon.json may not exist; merge data-root in or create minimal.
-        if [ -f /etc/docker/daemon.json ]; then
-          sudo jq '."data-root" = "/mnt/docker-data"' /etc/docker/daemon.json | sudo tee /etc/docker/daemon.json.new >/dev/null
-          sudo mv /etc/docker/daemon.json.new /etc/docker/daemon.json
-        else
-          echo '{"data-root":"/mnt/docker-data"}' | sudo tee /etc/docker/daemon.json
-        fi
-        sudo systemctl start docker
-        # Make TMPDIR persist for subsequent steps in the same job.
-        echo "TMPDIR=/mnt/docker-tmp" >> "$GITHUB_ENV"
-        echo "After relocation:"
-        df -h / /mnt
-        docker info | grep -i 'docker root dir' || true

.github/bump_vllm_wheel.sh

@@ -1,45 +0,0 @@
-#!/bin/bash
-# Bump the cublas13 vLLM wheel pin in requirements-cublas13-after.txt.
-#
-# vLLM's PyPI wheel is built against CUDA 12 so the cublas13 build pulls a
-# cu130-flavoured wheel from vLLM's per-tag index at
-# https://wheels.vllm.ai/<TAG>/cu130/. That URL segment is itself version-locked
-# (no /latest/ alias upstream), so bumping vLLM means rewriting both the URL
-# segment and the version constraint atomically. bump_deps.sh handles git-sha
-# vars in Makefiles; this script handles the two-value rewrite specific to the
-# vLLM requirements file.
-set -xe
-REPO=$1   # vllm-project/vllm
-FILE=$2   # backend/python/vllm/requirements-cublas13-after.txt
-VAR=$3    # VLLM_VERSION (used for output file names so the workflow can read them)
-
-if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
-    echo "usage: $0 <repo> <requirements-file> <var-name>" >&2
-    exit 1
-fi
-
-# /releases/latest returns the most recent non-prerelease tag.
-LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
-    "https://api.github.com/repos/$REPO/releases/latest" \
-    | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
-
-# Strip leading 'v' (vLLM tags are 'v0.20.0', the URL/version use '0.20.0').
-NEW_VERSION="${LATEST_TAG#v}"
-
-set +e
-CURRENT_VERSION=$(grep -oE '^vllm==[0-9]+\.[0-9]+\.[0-9]+' "$FILE" | head -1 | cut -d= -f3)
-set -e
-
-# sed both lines unconditionally — peter-evans/create-pull-request opens no PR
-# when the working tree is clean, so a no-op rewrite is safe.
-sed -i "$FILE" \
-    -e "s|wheels\.vllm\.ai/[^/]*/cu130|wheels.vllm.ai/$NEW_VERSION/cu130|g" \
-    -e "s|^vllm==.*|vllm==$NEW_VERSION|"
-
-if [ -z "$CURRENT_VERSION" ]; then
-    echo "Could not find vllm==X.Y.Z in $FILE."
-    exit 0
-fi
-
-echo "Changes: https://github.com/$REPO/compare/v${CURRENT_VERSION}...${LATEST_TAG}" >> "${VAR}_message.txt"
-echo "${NEW_VERSION}" >> "${VAR}_commit.txt"

.github/workflows/backend_build.yml

@@ -24,17 +24,6 @@ on:
         description: 'Platforms'
         default: ''
         type: string
-      platform-tag:
-        description: |
-          Short tag identifying the platform leg, e.g. "amd64" or "arm64".
-          Used to scope the per-arch registry cache and the digest artifact name.
-          Required for split-and-merge multi-arch builds; pass "amd64" for
-          single-arch amd64 builds too. Optional (default '') during the
-          migration to per-arch matrix expansion; will be flipped to
-          required: true in Phase 6 once all callers pass an explicit value.
-        required: false
-        default: ''
-        type: string
       tag-latest:
         description: 'Tag latest'
         default: ''
@@ -69,20 +58,6 @@ on:
         required: false
         default: '2204'
         type: string
-      amdgpu-targets:
-        description: 'AMD GPU targets for ROCm/HIP builds'
-        required: false
-        default: ''
-        type: string
-      builder-base-image:
-        description: |
-          Pre-built builder base image (e.g. quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64).
-          When set, the variant Dockerfile uses its `builder-prebuilt` stage which FROMs this
-          image directly instead of running its own gRPC stage + apt installs. Empty for
-          backends whose Dockerfile doesn't support a prebuilt base.
-        required: false
-        default: ''
-        type: string
     secrets:
       dockerUsername:
         required: false
@@ -100,22 +75,76 @@ jobs:
         quay_username: ${{ secrets.quayUsername }}
     steps:
 
+
+      - name: Free Disk Space (Ubuntu)
+        if: inputs.runs-on == 'ubuntu-latest'
+        uses: jlumbroso/free-disk-space@main
+        with:
+          # this might remove tools that are actually needed,
+          # if set to "true" but frees about 6 GB
+          tool-cache: true
+          # all of these default to true, but feel free to set to
+          # "false" if necessary for your workflow
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          docker-images: true
+          swap-storage: true
+
+      - name: Force Install GIT latest
+        run: |
+          sudo apt-get update \
+          && sudo apt-get install -y software-properties-common \
+          && sudo apt-get update \
+          && sudo add-apt-repository -y ppa:git-core/ppa \
+          && sudo apt-get update \
+          && sudo apt-get install -y git
+
       - name: Checkout
         uses: actions/checkout@v6
-        with:
-          submodules: true
 
-      - name: Configure apt mirror on runner
-        id: apt_mirror
-        uses: ./.github/actions/configure-apt-mirror
-
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
-        with:
-          mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
-
-      - name: Set up build disk
-        uses: ./.github/actions/setup-build-disk
+      - name: Release space from worker
+        if: inputs.runs-on == 'ubuntu-latest'
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get remove -y microsoft-edge-stable || true
+          sudo apt-get remove -y firefox || true
+          sudo apt-get remove -y powershell || true
+          sudo apt-get remove -y r-base-core || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf "/usr/local/share/boost" || true
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
+          df -h
 
       - name: Docker meta
         id: meta
@@ -172,17 +201,7 @@ jobs:
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
-      # Weekly cache-buster for the per-backend `make` step. Most Python
-      # backends list unpinned deps (torch, transformers, vllm, ...), so a
-      # warm cache freezes upstream versions indefinitely. Rolling this
-      # weekly forces a re-resolve of the install layer at most once per
-      # week, picking up newer wheels without a full cold rebuild.
-      - name: Compute deps refresh key
-        id: deps_refresh
-        run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
-
-      - name: Build and push by digest
-        id: build
+      - name: Build and push
         uses: docker/build-push-action@v7
         if: github.event_name != 'pull_request'
         with:
@@ -195,49 +214,15 @@ jobs:
             BASE_IMAGE=${{ inputs.base-image }}
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
-            AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
-            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
-            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
-            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
-          outputs: |
-            type=image,name=quay.io/go-skynet/local-ai-backends,push-by-digest=true,name-canonical=true,push=true
-            type=image,name=localai/localai-backends,push-by-digest=true,name-canonical=true,push=true
-          # Disable provenance: with mode=max (the default for push:true)
-          # buildx bundles a per-registry attestation manifest into each
-          # registry's manifest list, which makes the resulting list digest
-          # diverge across registries. steps.build.outputs.digest then
-          # only matches one of them, and the merge job's
-          # `imagetools create <reg>@sha256:<digest>` lookup fails on the
-          # other. Disabling provenance keeps the digest content-only and
-          # identical across both registries — required for digest-based
-          # cross-registry merge.
-          provenance: false
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
 
-      - name: Export digest
-        if: github.event_name != 'pull_request'
-        run: |
-          mkdir -p /tmp/digests
-          digest="${{ steps.build.outputs.digest }}"
-          touch "/tmp/digests/${digest#sha256:}"
-
-      - name: Upload digest artifact
-        if: github.event_name != 'pull_request'
-        uses: actions/upload-artifact@v4
-        with:
-          name: digests${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
-          path: /tmp/digests/*
-          if-no-files-found: error
-          retention-days: 1
-
-      - name: Build (PR)
+      - name: Build and push (PR)
         uses: docker/build-push-action@v7
         if: github.event_name == 'pull_request'
         with:
@@ -250,15 +235,9 @@ jobs:
             BASE_IMAGE=${{ inputs.base-image }}
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
-            AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
-            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
-            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
-            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
           push: ${{ env.quay_username != '' }}
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/backend_build_darwin.yml

@@ -48,13 +48,6 @@ jobs:
     strategy:
       matrix:
         go-version: ['${{ inputs.go-version }}']
-    env:
-      # Keep the brew Cellar stable across cache restores. Without these,
-      # `brew install` would auto-update brew itself and re-link formulas,
-      # mutating the very paths the cache just restored.
-      HOMEBREW_NO_AUTO_UPDATE: '1'
-      HOMEBREW_NO_INSTALL_CLEANUP: '1'
-      HOMEBREW_NO_ANALYTICS: '1'
     steps:
       - name: Clone
         uses: actions/checkout@v6
@@ -65,178 +58,21 @@ jobs:
         uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go-version }}
-          # Caches ~/go/pkg/mod and ~/Library/Caches/go-build keyed on go.sum.
-          # Shared across every darwin matrix entry — first job in a run warms
-          # it, the rest hit warm.
-          cache: true
+          cache: false
 
       # You can test your matrix by printing the current Go version
       - name: Display Go version
         run: go version
 
-      # ---- Homebrew cache ----
-      # macOS runners have no Docker daemon, so the BuildKit registry cache used
-      # for Linux backend images (see .agents/ci-caching.md) doesn't apply here.
-      # We cache the brew downloads + Cellar entries for the formulas we install
-      # below. Read on every run, write only on master/tag pushes — same policy
-      # as the Linux registry cache.
-      - name: Restore Homebrew cache
-        id: brew-cache
-        uses: actions/cache/restore@v4
-        with:
-          path: |
-            ~/Library/Caches/Homebrew/downloads
-            /opt/homebrew/Cellar/protobuf
-            /opt/homebrew/Cellar/grpc
-            /opt/homebrew/Cellar/protoc-gen-go
-            /opt/homebrew/Cellar/protoc-gen-go-grpc
-            /opt/homebrew/Cellar/libomp
-            /opt/homebrew/Cellar/llvm
-            /opt/homebrew/Cellar/ccache
-            /opt/homebrew/Cellar/blake3
-            /opt/homebrew/Cellar/fmt
-            /opt/homebrew/Cellar/hiredis
-            /opt/homebrew/Cellar/xxhash
-            /opt/homebrew/Cellar/zstd
-          key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
-
       - name: Dependencies
         run: |
-          # ccache is always installed (used by the llama-cpp variant build) so
-          # the brew cache content stays stable across every backend in the
-          # matrix — they all share one cache key.
-          # blake3, fmt, hiredis, xxhash, zstd are ccache's runtime dylib deps.
-          # Without explicitly installing them, a brew cache-hit run restores
-          # ccache's Cellar dir but skips installing those transitive deps,
-          # and ccache fails at runtime with `dyld: Library not loaded`.
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd
-          # Force-reinstall ccache so brew re-validates its full runtime-dep
-          # closure on every run. This is the durable fix: when the upstream
-          # ccache formula gains a new transitive dep (as it has multiple times
-          # already), we don't have to chase missing dylibs one at a time.
-          # The downloads cache makes the reinstall fast (~5s on a hit).
-          brew reinstall ccache
-          # The brew cache restores the Cellar dirs but NOT the bin symlinks
-          # at /opt/homebrew/bin/*. brew install above sees the Cellar present
-          # and decides "already installed" without re-linking, so on a cache-
-          # hit run the formulas aren't on PATH. Force-link them; --overwrite
-          # tolerates pre-existing symlinks from earlier installs.
-          brew link --overwrite protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd 2>/dev/null || true
-
-      - name: Save Homebrew cache
-        if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v4
-        with:
-          path: |
-            ~/Library/Caches/Homebrew/downloads
-            /opt/homebrew/Cellar/protobuf
-            /opt/homebrew/Cellar/grpc
-            /opt/homebrew/Cellar/protoc-gen-go
-            /opt/homebrew/Cellar/protoc-gen-go-grpc
-            /opt/homebrew/Cellar/libomp
-            /opt/homebrew/Cellar/llvm
-            /opt/homebrew/Cellar/ccache
-            /opt/homebrew/Cellar/blake3
-            /opt/homebrew/Cellar/fmt
-            /opt/homebrew/Cellar/hiredis
-            /opt/homebrew/Cellar/xxhash
-            /opt/homebrew/Cellar/zstd
-          key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
-
-      # ---- ccache for llama.cpp CMake builds ----
-      # Three CMake variants (fallback, grpc, rpc-server) compile the same
-      # llama.cpp source tree with overlapping flags — ccache dedupes object
-      # files across them. Key on the pinned LLAMA_VERSION so a pin bump
-      # invalidates cleanly; restore-keys fall back to the latest entry for the
-      # same pin so unchanged TUs stay warm even when the cache is fresh.
-      - name: Compute llama.cpp version
-        if: inputs.backend == 'llama-cpp'
-        id: llama-version
-        run: |
-          version=$(grep '^LLAMA_VERSION' backend/cpp/llama-cpp/Makefile | head -1 | cut -d= -f2 | cut -d'?' -f1 | tr -d ' ')
-          echo "version=${version}" >> "$GITHUB_OUTPUT"
-
-      - name: Restore ccache
-        if: inputs.backend == 'llama-cpp'
-        id: ccache-cache
-        uses: actions/cache/restore@v4
-        with:
-          path: ~/Library/Caches/ccache
-          key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
-          restore-keys: |
-            ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-
-
-      - name: Configure ccache
-        if: inputs.backend == 'llama-cpp'
-        run: |
-          mkdir -p "$HOME/Library/Caches/ccache"
-          ccache -M 2G
-          ccache -z
-          # llama-cpp-darwin.sh reads CMAKE_ARGS / CCACHE_DIR from env.
-          {
-            echo "CMAKE_ARGS=${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache"
-            echo "CCACHE_DIR=$HOME/Library/Caches/ccache"
-          } >> "$GITHUB_ENV"
-
-      # ---- Python wheel cache (uv + pip) ----
-      # Mirrors the Linux DEPS_REFRESH cadence (see .agents/ci-caching.md): the
-      # ISO-week segment of the cache key forces at most one cold rebuild per
-      # backend per week, automatically picking up newer wheels for unpinned
-      # deps (torch, mlx, diffusers, …). Restore-keys fall back to the most
-      # recent build of the same backend so off-week PRs still hit warm.
-      - name: Compute weekly cache bucket
-        if: inputs.lang == 'python'
-        id: weekly
-        run: echo "bucket=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
-
-      - name: Restore Python wheel cache
-        if: inputs.lang == 'python'
-        id: pyenv-cache
-        uses: actions/cache/restore@v4
-        with:
-          path: |
-            ~/Library/Caches/pip
-            ~/Library/Caches/uv
-          key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
-          restore-keys: |
-            pyenv-darwin-${{ inputs.backend }}-
-
-      # llama-cpp on Darwin uses a bespoke build script (scripts/build/llama-cpp-darwin.sh)
-      # that compiles three CMake variants from backend/cpp/llama-cpp and bundles dylibs
-      # via otool — it doesn't fit the build-darwin-go-backend / build-darwin-python-backend
-      # mold. Drive it via its dedicated `backends/llama-cpp-darwin` make target instead.
-      - name: Build ${{ inputs.backend }}-darwin (llama-cpp)
-        if: inputs.backend == 'llama-cpp'
-        run: |
-          make protogen-go
-          make backends/llama-cpp-darwin
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm
 
       - name: Build ${{ inputs.backend }}-darwin
-        if: inputs.backend != 'llama-cpp'
         run: |
           make protogen-go
           BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
 
-      - name: ccache stats
-        if: inputs.backend == 'llama-cpp'
-        run: ccache -s
-
-      - name: Save ccache
-        if: inputs.backend == 'llama-cpp' && github.event_name != 'pull_request'
-        uses: actions/cache/save@v4
-        with:
-          path: ~/Library/Caches/ccache
-          key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
-
-      - name: Save Python wheel cache
-        if: inputs.lang == 'python' && github.event_name != 'pull_request' && steps.pyenv-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v4
-        with:
-          path: |
-            ~/Library/Caches/pip
-            ~/Library/Caches/uv
-          key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
-
       - name: Upload ${{ inputs.backend }}.tar
         uses: actions/upload-artifact@v7
         with:

.github/workflows/backend_merge.yml

@@ -1,133 +0,0 @@
----
-name: 'merge backend manifest list (reusable)'
-
-# Reusable workflow that joins per-arch digest artifacts (uploaded by
-# backend_build.yml when called with platform-tag) into a single tagged
-# multi-arch manifest list. Called once per backend by backend.yml after
-# both per-arch build jobs succeed.
-
-on:
-  workflow_call:
-    inputs:
-      tag-latest:
-        description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
-        required: false
-        type: string
-        default: ''
-      tag-suffix:
-        description: 'Backend tag suffix (e.g. -cpu-faster-whisper). Used to compute the artifact pattern and the final tag suffix.'
-        required: true
-        type: string
-    secrets:
-      dockerUsername:
-        required: false
-      dockerPassword:
-        required: false
-      quayUsername:
-        required: true
-      quayPassword:
-        required: true
-
-jobs:
-  merge:
-    runs-on: ubuntu-latest
-    env:
-      quay_username: ${{ secrets.quayUsername }}
-    steps:
-      - name: Download digests
-        uses: actions/download-artifact@v4
-        with:
-          pattern: digests${{ inputs.tag-suffix }}-*
-          merge-multiple: true
-          path: /tmp/digests
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@master
-
-      - name: Login to DockerHub
-        if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
-        with:
-          username: ${{ secrets.dockerUsername }}
-          password: ${{ secrets.dockerPassword }}
-
-      - name: Login to Quay.io
-        if: ${{ env.quay_username != '' }}
-        uses: docker/login-action@v4
-        with:
-          registry: quay.io
-          username: ${{ secrets.quayUsername }}
-          password: ${{ secrets.quayPassword }}
-
-      - name: Docker meta
-        id: meta
-        if: github.event_name != 'pull_request'
-        uses: docker/metadata-action@v6
-        with:
-          images: |
-            quay.io/go-skynet/local-ai-backends
-            localai/localai-backends
-          tags: |
-            type=ref,event=branch
-            type=semver,pattern={{raw}}
-            type=sha
-          flavor: |
-            latest=${{ inputs.tag-latest }}
-            suffix=${{ inputs.tag-suffix }},onlatest=true
-
-      - name: Create manifest list and push (quay)
-        if: github.event_name != 'pull_request'
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '
-            .tags
-            | map(select(startswith("quay.io/")))
-            | map("-t " + .)
-            | join(" ")
-          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No quay.io tags from docker/metadata-action; skipping quay merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'quay.io/go-skynet/local-ai-backends@sha256:%s ' *)
-          fi
-
-      - name: Create manifest list and push (dockerhub)
-        if: github.event_name != 'pull_request'
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '
-            .tags
-            | map(select(startswith("localai/")))
-            | map("-t " + .)
-            | join(" ")
-          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'localai/localai-backends@sha256:%s ' *)
-          fi
-
-      - name: Inspect manifest
-        if: github.event_name != 'pull_request'
-        run: |
-          set -euo pipefail
-          first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
-            docker buildx imagetools inspect "$first_tag"
-          fi
-
-      - name: Job summary
-        if: github.event_name != 'pull_request'
-        run: |
-          set -euo pipefail
-          echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
-          jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
-          echo >> "$GITHUB_STEP_SUMMARY"
-          echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
-          ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"

.github/workflows/backend_pr.yml

@@ -4,21 +4,17 @@ on:
   pull_request:
 
 concurrency:
-  group: ci-backends-pr-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-backends-pr-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   generate-matrix:
     runs-on: ubuntu-latest
     outputs:
-      matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
-      matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
-      matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
-      merge-matrix: ${{ steps.set-matrix.outputs['merge-matrix'] }}
-      has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
-      has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
-      has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
-      has-merges: ${{ steps.set-matrix.outputs['has-merges'] }}
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+      matrix-darwin: ${{ steps.set-matrix.outputs.matrix-darwin }}
+      has-backends: ${{ steps.set-matrix.outputs.has-backends }}
+      has-backends-darwin: ${{ steps.set-matrix.outputs.has-backends-darwin }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -31,9 +27,7 @@ jobs:
           bun add js-yaml
           bun add @octokit/core
 
-      # filters the matrix in backend.yml; splits into single-arch and
-      # multi-arch groups so backend-merge-jobs can `needs:` only the latter
-      # (matches backend.yml's structure).
+      # filters the matrix in backend.yml
       - name: Filter matrix for changed backends
         id: set-matrix
         env:
@@ -41,10 +35,10 @@ jobs:
           GITHUB_EVENT_PATH: ${{ github.event_path }}
         run: bun run scripts/changed-backends.js
 
-  backend-jobs-multiarch:
+  backend-jobs:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build.yml
-    if: needs.generate-matrix.outputs['has-backends-multiarch'] == 'true'
+    if: needs.generate-matrix.outputs.has-backends == 'true'
     with:
       tag-latest: ${{ matrix.tag-latest }}
       tag-suffix: ${{ matrix.tag-suffix }}
@@ -52,67 +46,19 @@ jobs:
       cuda-major-version: ${{ matrix.cuda-major-version }}
       cuda-minor-version: ${{ matrix.cuda-minor-version }}
       platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
       runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
       base-image: ${{ matrix.base-image }}
       backend: ${{ matrix.backend }}
       dockerfile: ${{ matrix.dockerfile }}
       skip-drivers: ${{ matrix.skip-drivers }}
       context: ${{ matrix.context }}
       ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
     secrets:
       quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
     strategy:
       fail-fast: true
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
-  backend-jobs-singlearch:
-    needs: generate-matrix
-    uses: ./.github/workflows/backend_build.yml
-    if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: true
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
-  backend-merge-jobs:
-    needs: [generate-matrix, backend-jobs-multiarch]
-    # backend_merge.yml's push-side steps are all gated on
-    # github.event_name != 'pull_request', so on a PR the merge job would
-    # do nothing. Skip it entirely to avoid spinning up an empty runner.
-    if: github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges'] == 'true'
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }}
   backend-jobs-darwin:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build_darwin.yml
@@ -120,7 +66,7 @@ jobs:
     with:
       backend: ${{ matrix.backend }}
       build-type: ${{ matrix.build-type }}
-      go-version: "1.25.x"
+      go-version: "1.24.x"
       tag-suffix: ${{ matrix.tag-suffix }}
       lang: ${{ matrix.lang || 'python' }}
       use-pip: ${{ matrix.backend == 'diffusers' }}

.github/workflows/base-images.yml

@@ -1,161 +0,0 @@
----
-name: 'build base-grpc images'
-
-# Builds + pushes pre-compiled builder base images that downstream
-# llama-cpp / ik-llama-cpp / turboquant variant Dockerfiles will FROM
-# (PR 2). Each base contains apt deps + protoc + cmake + gRPC at
-# /opt/grpc + (conditionally) CUDA / ROCm / Vulkan toolchains.
-#
-# Triggers:
-#   - schedule (Saturdays 05:00 UTC) - picks up Ubuntu/CUDA/ROCm
-#     security updates and re-runs ahead of the backend.yml weekly
-#     cron (Sundays 06:00 UTC).
-#   - workflow_dispatch - manual one-off rebuild.
-#   - push to master that touches Dockerfile.base-grpc-builder or
-#     this workflow itself - keeps bases in sync with their inputs.
-#
-# Bootstrap (one-time after this PR merges):
-#   gh workflow run base-images.yml --ref master
-# Wait ~30 min for all 9 matrix variants to push to
-# quay.io/go-skynet/ci-cache:base-grpc-* before merging PR 2.
-
-on:
-  schedule:
-    - cron: '0 5 * * 6'
-  workflow_dispatch:
-  push:
-    branches: [master]
-    paths:
-      - 'backend/Dockerfile.base-grpc-builder'
-      - '.github/workflows/base-images.yml'
-      # The install logic and apt-mirror helper are bind-mounted into
-      # Dockerfile.base-grpc-builder at build time — changes to either
-      # affect the produced base images and must trigger a rebuild.
-      - '.docker/install-base-deps.sh'
-      - '.docker/apt-mirror.sh'
-
-concurrency:
-  group: ci-base-images-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  build:
-    if: github.repository == 'mudler/LocalAI'
-    runs-on: ${{ matrix.runs-on }}
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - tag: 'base-grpc-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:24.04'
-            build-type: ''
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'ubuntu:24.04'
-            build-type: ''
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-cuda-12-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:24.04'
-            build-type: 'cublas'
-            cuda-major-version: '12'
-            cuda-minor-version: '8'
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-cuda-13-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:22.04'
-            build-type: 'cublas'
-            cuda-major-version: '13'
-            cuda-minor-version: '0'
-            ubuntu-version: '2204'
-          - tag: 'base-grpc-cuda-13-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'ubuntu:24.04'
-            build-type: 'cublas'
-            cuda-major-version: '13'
-            cuda-minor-version: '0'
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-rocm-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'rocm/dev-ubuntu-24.04:7.2.1'
-            build-type: 'hipblas'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-vulkan-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:24.04'
-            build-type: 'vulkan'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-vulkan-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'ubuntu:24.04'
-            build-type: 'vulkan'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-intel-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04'
-            build-type: 'sycl'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          # Legacy JetPack r36.4.0 base for older Jetson devices (CUDA 12).
-          # Distinct from base-grpc-cuda-13-arm64 (Ubuntu 24.04 + CUDA 13 sbsa)
-          # which targets newer Jetsons. Some matrix entries
-          # (-nvidia-l4t-arm64-llama-cpp / -turboquant) still build against
-          # the JetPack image, so we need a matching base.
-          - tag: 'base-grpc-l4t-cuda-12-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'nvcr.io/nvidia/l4t-jetpack:r36.4.0'
-            build-type: 'l4t'
-            cuda-major-version: '12'
-            cuda-minor-version: '0'
-            ubuntu-version: '2204'
-            # JetPack r36.4.0 already ships CUDA preinstalled at /usr/local/cuda;
-            # apt-installing cuda-nvcc-12-0 from the public repos fails because
-            # those packages aren't published for the JetPack apt feed. Match
-            # the original l4t matrix entry which set skip-drivers: 'true'.
-            skip-drivers: 'true'
-    steps:
-      - uses: actions/checkout@v6
-        with:
-          submodules: false
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
-      - name: Set up build disk
-        uses: ./.github/actions/setup-build-disk
-      - uses: docker/setup-qemu-action@master
-        with:
-          platforms: all
-      - uses: docker/setup-buildx-action@master
-      - uses: docker/login-action@v4
-        with:
-          registry: quay.io
-          username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-          password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-      - uses: docker/build-push-action@v7
-        with:
-          context: .
-          file: ./backend/Dockerfile.base-grpc-builder
-          build-args: |
-            BASE_IMAGE=${{ matrix.base-image }}
-            BUILD_TYPE=${{ matrix.build-type }}
-            CUDA_MAJOR_VERSION=${{ matrix.cuda-major-version }}
-            CUDA_MINOR_VERSION=${{ matrix.cuda-minor-version }}
-            UBUNTU_VERSION=${{ matrix.ubuntu-version }}
-            SKIP_DRIVERS=${{ matrix.skip-drivers || 'false' }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }},mode=max,ignore-error=true
-          provenance: false
-          tags: quay.io/go-skynet/ci-cache:${{ matrix.tag }}
-          push: true

.github/workflows/build-test.yaml

@@ -50,8 +50,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:

.github/workflows/bump_deps.yaml

@@ -50,10 +50,6 @@ jobs:
             variable: "QWEN3TTS_CPP_VERSION"
             branch: "main"
             file: "backend/go/qwen3-tts-cpp/Makefile"
-          - repository: "localai-org/vibevoice.cpp"
-            variable: "VIBEVOICE_CPP_VERSION"
-            branch: "master"
-            file: "backend/go/vibevoice-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
@@ -84,37 +80,5 @@ jobs:
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
-  bump-vllm-wheel:
-    # vLLM's cu130 wheel comes from a per-tag index URL (no /latest/ alias),
-    # so the cublas13 requirements file pins both a URL segment and a version
-    # constraint. bump_deps.sh handles git-sha-in-Makefile only — this job
-    # rewrites both values atomically when a new vLLM stable tag ships.
-    if: github.repository == 'mudler/LocalAI'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v6
-      - name: Bump vLLM cu130 wheel pin 🔧
-        id: bump
-        run: |
-          bash .github/bump_vllm_wheel.sh vllm-project/vllm backend/python/vllm/requirements-cublas13-after.txt VLLM_VERSION
-          {
-            echo 'message<<EOF'
-            cat "VLLM_VERSION_message.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          {
-            echo 'commit<<EOF'
-            cat "VLLM_VERSION_commit.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          rm -rfv VLLM_VERSION_message.txt VLLM_VERSION_commit.txt
-      - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
-        with:
-          token: ${{ secrets.UPDATE_BOT_TOKEN }}
-          push-to-fork: ci-forks/LocalAI
-          commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel'
-          title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_VERSION"
-          body: ${{ steps.bump.outputs.message }}
-          signoff: true
+
+

.github/workflows/checksum_checker.yaml

@@ -8,9 +8,15 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
+      - name: Force Install GIT latest
+        run: |
+          sudo apt-get update \
+          && sudo apt-get install -y software-properties-common \
+          && sudo apt-get update \
+          && sudo add-apt-repository -y ppa:git-core/ppa \
+          && sudo apt-get update \
+          && sudo apt-get install -y git
       - uses: actions/checkout@v6
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Install dependencies
         run: |
           sudo apt-get update

.github/workflows/gallery-agent.yaml

@@ -2,7 +2,7 @@ name: Gallery Agent
 on:
 
   schedule:
-    - cron: '0 */12 * * *'  # Run every 4 hours
+    - cron: '0 */3 * * *'  # Run every 4 hours
   workflow_dispatch:
     inputs:
       search_term:
@@ -54,41 +54,24 @@ jobs:
           REPO: ${{ github.repository }}
           SEARCH: 'gallery agent in:title'
         run: |
-          # Walk gallery-agent PRs and act on maintainer comments:
+          # Walk open gallery-agent PRs and act on maintainer comments:
           #   /gallery-agent blacklist → label `gallery-agent/blacklisted` + close (never repropose)
           #   /gallery-agent recreate  → close without label (next run may repropose)
           # Only comments from OWNER / MEMBER / COLLABORATOR are honored so
           # random users can't drive the bot.
-          #
-          # We scan both open PRs AND recently-closed PRs that don't already
-          # carry the blacklist label. This covers the common flow where a
-          # maintainer writes /gallery-agent blacklist and immediately clicks
-          # Close — without this, the next scheduled run wouldn't see the
-          # command (PR is already closed) and would repropose the model.
           gh label create gallery-agent/blacklisted \
             --repo "$REPO" --color ededed \
             --description "gallery-agent must not repropose this model" 2>/dev/null || true
 
-          prs_open=$(gh pr list --repo "$REPO" --state open --search "$SEARCH" \
-            --json number --jq '.[].number')
-          # Closed PRs from the last 14 days that don't yet have the blacklist label.
-          # Bounded window keeps the scan cheap while covering late-applied commands.
-          since=$(date -u -d '14 days ago' +%Y-%m-%d)
-          prs_closed=$(gh pr list --repo "$REPO" --state closed \
-            --search "$SEARCH closed:>=$since -label:gallery-agent/blacklisted" \
-            --json number --jq '.[].number')
-          prs=$(printf '%s\n%s\n' "$prs_open" "$prs_closed" | sort -u | sed '/^$/d')
+          prs=$(gh pr list --repo "$REPO" --state open --search "$SEARCH" --json number --jq '.[].number')
           for pr in $prs; do
-            state=$(gh pr view "$pr" --repo "$REPO" --json state --jq '.state')
             cmds=$(gh pr view "$pr" --repo "$REPO" --json comments \
               --jq '.comments[] | select(.authorAssociation=="OWNER" or .authorAssociation=="MEMBER" or .authorAssociation=="COLLABORATOR") | .body')
             if echo "$cmds" | grep -qE '(^|[[:space:]])/gallery-agent[[:space:]]+blacklist([[:space:]]|$)'; then
-              echo "PR #$pr: blacklist command found (state=$state)"
+              echo "PR #$pr: blacklist command found"
               gh pr edit "$pr" --repo "$REPO" --add-label gallery-agent/blacklisted || true
-              if [ "$state" = "OPEN" ]; then
-                gh pr close "$pr" --repo "$REPO" --comment "Blacklisted via \`/gallery-agent blacklist\`. This model will not be reproposed." || true
-              fi
-            elif [ "$state" = "OPEN" ] && echo "$cmds" | grep -qE '(^|[[:space:]])/gallery-agent[[:space:]]+recreate([[:space:]]|$)'; then
+              gh pr close "$pr" --repo "$REPO" --comment "Blacklisted via \`/gallery-agent blacklist\`. This model will not be reproposed." || true
+            elif echo "$cmds" | grep -qE '(^|[[:space:]])/gallery-agent[[:space:]]+recreate([[:space:]]|$)'; then
               echo "PR #$pr: recreate command found"
               gh pr close "$pr" --repo "$REPO" --comment "Closed via \`/gallery-agent recreate\`. The next scheduled run will propose this model again." || true
             fi

.github/workflows/generate_grpc_cache.yaml

@@ -0,0 +1,96 @@
+name: 'generate and publish GRPC docker caches'
+
+on:
+  workflow_dispatch:
+
+  schedule:
+    # daily at midnight
+    - cron: '0 0 * * *'
+
+concurrency:
+  group: grpc-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
+
+jobs:
+  generate_caches:
+    if: github.repository == 'mudler/LocalAI'
+    strategy:
+      matrix:
+        include:
+          - grpc-base-image: ubuntu:24.04
+            runs-on: 'ubuntu-latest'
+            platforms: 'linux/amd64,linux/arm64'
+    runs-on: ${{matrix.runs-on}}
+    steps:
+      - name: Release space from worker
+        if: matrix.runs-on == 'ubuntu-latest'
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get remove -y microsoft-edge-stable || true
+          sudo apt-get remove -y firefox || true
+          sudo apt-get remove -y powershell || true
+          sudo apt-get remove -y r-base-core || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf "/usr/local/share/boost" || true
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
+          df -h
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@master
+        with:
+          platforms: all
+
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Cache GRPC
+        uses: docker/build-push-action@v7
+        with:
+          builder: ${{ steps.buildx.outputs.name }}
+          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
+          # This means that even the MAKEFLAGS have to be an EXACT match.
+          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
+          build-args: |
+            GRPC_BASE_IMAGE=${{ matrix.grpc-base-image }}
+            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
+            GRPC_VERSION=v1.65.0
+          context: .
+          file: ./Dockerfile
+          cache-to: type=gha,ignore-error=true
+          cache-from: type=gha
+          target: grpc
+          platforms: ${{ matrix.platforms }}
+          push: false

.github/workflows/generate_intel_image.yaml

@@ -7,8 +7,8 @@ on:
       - master
 
 concurrency:
-  group: intel-cache-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: intel-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   generate_caches:
@@ -16,7 +16,7 @@ jobs:
     strategy:
       matrix:
         include:
-          - base-image: intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04
+          - base-image: intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04
             runs-on: 'arc-runner-set'
             platforms: 'linux/amd64'
     runs-on: ${{matrix.runs-on}}

.github/workflows/image-pr.yml

@@ -5,8 +5,8 @@
     pull_request:
   
   concurrency:
-    group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-    cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
+    cancel-in-progress: true
   
   jobs:
     image-build:
@@ -18,9 +18,9 @@
         cuda-major-version: ${{ matrix.cuda-major-version }}
         cuda-minor-version: ${{ matrix.cuda-minor-version }}
         platforms: ${{ matrix.platforms }}
-        platform-tag: ${{ matrix.platform-tag || '' }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
       secrets:
@@ -60,35 +60,27 @@
               tag-latest: 'false'
               tag-suffix: '-hipblas'
               base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+              grpc-base-image: "ubuntu:24.04"
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'sycl'
               platforms: 'linux/amd64'
               tag-latest: 'false'
-              base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
+              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+              grpc-base-image: "ubuntu:24.04"
               tag-suffix: 'sycl'
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'vulkan'
-              platforms: 'linux/amd64'
-              platform-tag: 'amd64'
+              platforms: 'linux/amd64,linux/arm64'
               tag-latest: 'false'
               tag-suffix: '-vulkan-core'
               runs-on: 'ubuntu-latest'
               base-image: "ubuntu:24.04"
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'
-            - build-type: 'vulkan'
-              platforms: 'linux/arm64'
-              platform-tag: 'arm64'
-              tag-latest: 'false'
-              tag-suffix: '-vulkan-core'
-              runs-on: 'ubuntu-24.04-arm'
-              base-image: "ubuntu:24.04"
-              makeflags: "--jobs=4 --output-sync=target"
-              ubuntu-version: '2404'
             - build-type: 'cublas'
               cuda-major-version: "13"
               cuda-minor-version: "0"

.github/workflows/image.yml

@@ -9,8 +9,8 @@
         - '*'
   
   concurrency:
-    group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-    cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
+    cancel-in-progress: true
   
   jobs:
     hipblas-jobs:
@@ -25,6 +25,7 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
         ubuntu-codename: ${{ matrix.ubuntu-codename }}
@@ -41,11 +42,12 @@
               tag-latest: 'auto'
               tag-suffix: '-gpu-hipblas'
               base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+              grpc-base-image: "ubuntu:24.04"
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-
+  
     core-image-build:
       if: github.repository == 'mudler/LocalAI'
       uses: ./.github/workflows/image_build.yml
@@ -56,9 +58,9 @@
         cuda-major-version: ${{ matrix.cuda-major-version }}
         cuda-minor-version: ${{ matrix.cuda-minor-version }}
         platforms: ${{ matrix.platforms }}
-        platform-tag: ${{ matrix.platform-tag || '' }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         skip-drivers: ${{ matrix.skip-drivers }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
@@ -73,8 +75,7 @@
         matrix:
           include:
             - build-type: ''
-              platforms: 'linux/amd64'
-              platform-tag: 'amd64'
+              platforms: 'linux/amd64,linux/arm64'
               tag-latest: 'auto'
               tag-suffix: ''
               base-image: "ubuntu:24.04"
@@ -83,17 +84,6 @@
               skip-drivers: 'false'
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-            - build-type: ''
-              platforms: 'linux/arm64'
-              platform-tag: 'arm64'
-              tag-latest: 'auto'
-              tag-suffix: ''
-              base-image: "ubuntu:24.04"
-              runs-on: 'ubuntu-24.04-arm'
-              makeflags: "--jobs=4 --output-sync=target"
-              skip-drivers: 'false'
-              ubuntu-version: '2404'
-              ubuntu-codename: 'noble'
             - build-type: 'cublas'
               cuda-major-version: "12"
               cuda-minor-version: "8"
@@ -119,8 +109,7 @@
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
             - build-type: 'vulkan'
-              platforms: 'linux/amd64'
-              platform-tag: 'amd64'
+              platforms: 'linux/amd64,linux/arm64'
               tag-latest: 'auto'
               tag-suffix: '-gpu-vulkan'
               runs-on: 'ubuntu-latest'
@@ -129,52 +118,16 @@
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-            - build-type: 'vulkan'
-              platforms: 'linux/arm64'
-              platform-tag: 'arm64'
-              tag-latest: 'auto'
-              tag-suffix: '-gpu-vulkan'
-              runs-on: 'ubuntu-24.04-arm'
-              base-image: "ubuntu:24.04"
-              skip-drivers: 'false'
-              makeflags: "--jobs=4 --output-sync=target"
-              ubuntu-version: '2404'
-              ubuntu-codename: 'noble'
             - build-type: 'intel'
               platforms: 'linux/amd64'
               tag-latest: 'auto'
-              base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
+              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+              grpc-base-image: "ubuntu:24.04"
               tag-suffix: '-gpu-intel'
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-
-    core-image-merge:
-      if: github.repository == 'mudler/LocalAI'
-      needs: core-image-build
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: ''
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    gpu-vulkan-image-merge:
-      if: github.repository == 'mudler/LocalAI'
-      needs: core-image-build
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-gpu-vulkan'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
   
     gh-runner:
       if: github.repository == 'mudler/LocalAI'
@@ -188,6 +141,7 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         skip-drivers: ${{ matrix.skip-drivers }}
         ubuntu-version: ${{ matrix.ubuntu-version }}

.github/workflows/image_build.yml

@@ -8,6 +8,11 @@ on:
         description: 'Base image'
         required: true
         type: string
+      grpc-base-image:
+        description: 'GRPC Base image, must be a compatible image with base-image'
+        required: false
+        default: ''
+        type: string
       build-type:
         description: 'Build type'
         default: ''
@@ -24,15 +29,6 @@ on:
         description: 'Platforms'
         default: ''
         type: string
-      platform-tag:
-        description: |
-          Short tag identifying the platform leg, e.g. "amd64" or "arm64".
-          Used to scope the per-arch registry cache and the digest artifact name.
-          Optional during the migration; will be flipped to required: true once
-          every caller passes an explicit value.
-        required: false
-        default: ''
-        type: string
       tag-latest:
         description: 'Tag latest'
         default: ''
@@ -79,20 +75,73 @@ jobs:
     runs-on: ${{ inputs.runs-on }}
     steps:
 
+      - name: Free Disk Space (Ubuntu)
+        if: inputs.runs-on == 'ubuntu-latest'
+        uses: jlumbroso/free-disk-space@main
+        with:
+          # this might remove tools that are actually needed,
+          # if set to "true" but frees about 6 GB
+          tool-cache: true
+          # all of these default to true, but feel free to set to
+          # "false" if necessary for your workflow
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          docker-images: true
+          swap-storage: true
+      - name: Force Install GIT latest
+        run: |
+          sudo apt-get update \
+          && sudo apt-get install -y software-properties-common \
+          && sudo apt-get update \
+          && sudo add-apt-repository -y ppa:git-core/ppa \
+          && sudo apt-get update \
+          && sudo apt-get install -y git
       - name: Checkout
         uses: actions/checkout@v6
 
-      - name: Configure apt mirror on runner
-        id: apt_mirror
-        uses: ./.github/actions/configure-apt-mirror
-
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
-        with:
-          mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
-
-      - name: Set up build disk
-        uses: ./.github/actions/setup-build-disk
+      - name: Release space from worker
+        if: inputs.runs-on == 'ubuntu-latest'
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get remove -y microsoft-edge-stable || true
+          sudo apt-get remove -y firefox || true
+          sudo apt-get remove -y powershell || true
+          sudo apt-get remove -y r-base-core || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf "/usr/local/share/boost" || true
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
+          df -h
 
       - name: Docker meta
         id: meta
@@ -147,72 +196,59 @@ jobs:
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
-      - name: Build and push by digest
-        id: build
+      - name: Build and push
         uses: docker/build-push-action@v7
         if: github.event_name != 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
+          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
+          # This means that even the MAKEFLAGS have to be an EXACT match.
+          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
+          # This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
           build-args: |
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
+            GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
+            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
+            GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
           context: .
           file: ./Dockerfile
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
-          outputs: |
-            type=image,name=quay.io/go-skynet/local-ai,push-by-digest=true,name-canonical=true,push=true
-            type=image,name=localai/localai,push-by-digest=true,name-canonical=true,push=true
-          # See backend_build.yml for the rationale — provenance=mode=max
-          # diverges the manifest-list digest per registry, breaking the
-          # downstream imagetools create lookup.
-          provenance: false
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
-
-      - name: Export digest
-        if: github.event_name != 'pull_request'
-        run: |
-          mkdir -p /tmp/digests
-          digest="${{ steps.build.outputs.digest }}"
-          touch "/tmp/digests/${digest#sha256:}"
-
-      - name: Upload digest artifact
-        if: github.event_name != 'pull_request'
-        uses: actions/upload-artifact@v4
-        with:
-          name: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}-${{ inputs.platform-tag }}
-          path: /tmp/digests/*
-          if-no-files-found: error
-          retention-days: 1
 ### Start testing image
       - name: Build and push
         uses: docker/build-push-action@v7
         if: github.event_name == 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
+          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
+          # This means that even the MAKEFLAGS have to be an EXACT match.
+          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
+          # This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
           build-args: |
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
+            GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
+            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
+            GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
           context: .
           file: ./Dockerfile
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
           #push: true
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/image_merge.yml

@@ -1,117 +0,0 @@
----
-name: 'merge LocalAI image manifest list (reusable)'
-
-# Reusable workflow that joins per-arch digest artifacts (uploaded by
-# image_build.yml when called with platform-tag) into a single tagged
-# multi-arch manifest list.
-
-on:
-  workflow_call:
-    inputs:
-      tag-latest:
-        description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
-        required: false
-        type: string
-        default: ''
-      tag-suffix:
-        description: 'Image tag suffix (empty for core image). Used in artifact pattern with a -core placeholder for empty.'
-        required: true
-        type: string
-    secrets:
-      dockerUsername:
-        required: false
-      dockerPassword:
-        required: false
-      quayUsername:
-        required: true
-      quayPassword:
-        required: true
-
-jobs:
-  merge:
-    runs-on: ubuntu-latest
-    env:
-      quay_username: ${{ secrets.quayUsername }}
-    steps:
-      - name: Download digests
-        uses: actions/download-artifact@v4
-        with:
-          pattern: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}-*
-          merge-multiple: true
-          path: /tmp/digests
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@master
-
-      - name: Login to DockerHub
-        if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
-        with:
-          username: ${{ secrets.dockerUsername }}
-          password: ${{ secrets.dockerPassword }}
-
-      - name: Login to Quay.io
-        uses: docker/login-action@v4
-        with:
-          registry: quay.io
-          username: ${{ secrets.quayUsername }}
-          password: ${{ secrets.quayPassword }}
-
-      - name: Docker meta
-        id: meta
-        uses: docker/metadata-action@v6
-        with:
-          images: |
-            quay.io/go-skynet/local-ai
-            localai/localai
-          tags: |
-            type=ref,event=branch
-            type=semver,pattern={{raw}}
-            type=sha
-          flavor: |
-            latest=${{ inputs.tag-latest }}
-            suffix=${{ inputs.tag-suffix }},onlatest=true
-
-      - name: Create manifest list and push (quay)
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '.tags | map(select(startswith("quay.io/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No quay.io tags from docker/metadata-action; skipping quay merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'quay.io/go-skynet/local-ai@sha256:%s ' *)
-          fi
-
-      - name: Create manifest list and push (dockerhub)
-        if: github.event_name != 'pull_request'
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '.tags | map(select(startswith("localai/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'localai/localai@sha256:%s ' *)
-          fi
-
-      - name: Inspect manifest
-        run: |
-          set -euo pipefail
-          first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
-            docker buildx imagetools inspect "$first_tag"
-          fi
-
-      - name: Job summary
-        run: |
-          set -euo pipefail
-          echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
-          jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
-          echo >> "$GITHUB_STEP_SUMMARY"
-          echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
-          ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"

.github/workflows/lint.yml

@@ -1,48 +0,0 @@
----
-name: 'lint'
-
-on:
-  pull_request:
-    paths-ignore:
-      - 'docs/**'
-      - 'examples/**'
-      - 'README.md'
-      - '**/*.md'
-  push:
-    branches:
-      - master
-
-concurrency:
-  group: ci-lint-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  golangci-lint:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v6
-        with:
-          # Full history so golangci-lint's new-from-merge-base can reach
-          # origin/master and compute the diff against it.
-          fetch-depth: 0
-      - uses: actions/setup-go@v5
-        with:
-          go-version: '1.26.x'
-          cache: false
-      - name: install golangci-lint
-        run: |
-          curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh \
-            | sh -s -- -b "$(go env GOPATH)/bin" v2.11.4
-      - name: generate grpc proto sources
-        # pkg/grpc/proto/*.go is generated, not checked in. Several packages
-        # import it, so without this step typecheck fails project-wide.
-        run: make protogen-go
-      - name: stub react-ui dist for go:embed
-        # core/http/app.go has //go:embed react-ui/dist/*; the glob needs at
-        # least one non-hidden entry to satisfy typecheck. We don't run
-        # `make react-ui` here because lint doesn't need the real bundle.
-        run: |
-          mkdir -p core/http/react-ui/dist
-          touch core/http/react-ui/dist/index.html
-      - name: lint
-        run: make lint

.github/workflows/release.yaml

@@ -49,8 +49,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:

.github/workflows/test-extra.yml

@@ -10,8 +10,8 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-extra-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-extra-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   detect-changes:
@@ -36,14 +36,8 @@ jobs:
       sglang: ${{ steps.detect.outputs.sglang }}
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
       qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
-      vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
-      localvqe: ${{ steps.detect.outputs.localvqe }}
       voxtral: ${{ steps.detect.outputs.voxtral }}
       kokoros: ${{ steps.detect.outputs.kokoros }}
-      insightface: ${{ steps.detect.outputs.insightface }}
-      speaker-recognition: ${{ steps.detect.outputs.speaker-recognition }}
-      sherpa-onnx: ${{ steps.detect.outputs.sherpa-onnx }}
-      whisper: ${{ steps.detect.outputs.whisper }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -510,120 +504,6 @@ jobs:
       - name: Build llama-cpp backend image and run audio transcription gRPC e2e tests
         run: |
           make test-extra-backend-llama-cpp-transcription
-  # PR-acceptance smoke gate: always runs on every PR (no detect-changes gate, no
-  # paths filter). Pulls the pre-built master CPU llama-cpp image from quay
-  # instead of building from source, so the cost is a docker pull (~30s) plus the
-  # short Qwen3-0.6B model download. Exercises the full gRPC surface — health,
-  # load, predict, stream — plus the logprobs/logit_bias specs that moved out of
-  # core/http/app_test.go. Anything heavier or per-backend is gated to the
-  # detect-changes path-filter above.
-  tests-llama-cpp-smoke:
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Pull pre-built llama-cpp backend image
-        run: docker pull quay.io/go-skynet/local-ai-backends:master-cpu-llama-cpp
-      - name: Run e2e-backends smoke
-        env:
-          BACKEND_IMAGE: quay.io/go-skynet/local-ai-backends:master-cpu-llama-cpp
-          BACKEND_TEST_CAPS: health,load,predict,stream,logprobs,logit_bias
-        run: |
-          make test-extra-backend
-  # Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked LLM.
-  # Builds the sherpa-onnx Docker image, extracts the rootfs so the e2e suite
-  # can discover the backend binary + shared libs, downloads the three model
-  # bundles (silero-vad, omnilingual-asr, vits-ljs) and drives the realtime
-  # websocket spec end-to-end.
-  tests-sherpa-onnx-realtime:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Setup Node.js
-        uses: actions/setup-node@v6
-        with:
-          node-version: '22'
-      - name: Build sherpa-onnx backend image and run realtime e2e tests
-        run: |
-          make test-extra-e2e-realtime-sherpa
-  # Streaming ASR via the sherpa-onnx online recognizer (zipformer
-  # transducer). Exercises both AudioTranscription (buffered) and
-  # AudioTranscriptionStream (real-time deltas) on the e2e-backends
-  # harness.
-  tests-sherpa-onnx-grpc-transcription:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build sherpa-onnx backend image and run streaming ASR gRPC e2e tests
-        run: |
-          make test-extra-backend-sherpa-onnx-transcription
-  # End-to-end transcription via the e2e-backends gRPC harness against
-  # the whisper.cpp backend. Drives AudioTranscription (offline) and
-  # AudioTranscriptionStream (real, segment-callback-driven deltas) on
-  # ggml-base.en + the JFK 11s clip.
-  tests-whisper-grpc-transcription:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.whisper == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build whisper backend image and run transcription gRPC e2e tests
-        run: |
-          make test-extra-backend-whisper-transcription
-  # VITS TTS via the sherpa-onnx backend. Drives both TTS (file write) and
-  # TTSStream (PCM chunks) on the e2e-backends harness.
-  tests-sherpa-onnx-grpc-tts:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build sherpa-onnx backend image and run TTS gRPC e2e tests
-        run: |
-          make test-extra-backend-sherpa-onnx-tts
   tests-ik-llama-cpp-grpc:
     needs: detect-changes
     if: needs.detect-changes.outputs.ik-llama-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
@@ -816,117 +696,6 @@ jobs:
       - name: Test qwen3-tts-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/qwen3-tts-cpp test
-  # Per-backend smoke for vibevoice-cpp: builds the .so + Go binary and
-  # runs `make -C backend/go/vibevoice-cpp test`. test.sh auto-downloads
-  # the published mudler/vibevoice.cpp-models bundle (TTS Q8_0 + ASR Q4_K
-  # + tokenizer + voice) and runs the closed-loop TTS → ASR Go test.
-  tests-vibevoice-cpp:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
-      - name: Setup Go
-        uses: actions/setup-go@v5
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Build vibevoice-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/vibevoice-cpp
-      - name: Test vibevoice-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/vibevoice-cpp test
-  # End-to-end TTS via the e2e-backends gRPC harness. Builds the
-  # vibevoice-cpp Docker image and drives Backend/TTS against it with a
-  # real LocalAI gRPC client.
-  tests-vibevoice-cpp-grpc-tts:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build vibevoice-cpp backend image and run TTS gRPC e2e tests
-        run: |
-          make test-extra-backend-vibevoice-cpp-tts
-  # End-to-end transcription via the e2e-backends gRPC harness. The
-  # vibevoice ASR is a 7B-param model (Q4_K weights ~10 GB on disk)
-  # and the JFK 30 s decode is too heavy for a free 4-core
-  # ubuntu-latest pool runner - two CI attempts got SIGTERM'd during
-  # LoadModel, before the test could even progress. Use the
-  # self-hosted 'bigger-runner' label (same one the GPU image builds
-  # in backend.yml use) and the documented dotnet/ghc/android cache
-  # purge to clear ~10-20 GB of headroom for the model + Docker
-  # image + working dir.
-  tests-vibevoice-cpp-grpc-transcription:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: bigger-runner
-    timeout-minutes: 150
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y --no-install-recommends \
-              make build-essential curl unzip ca-certificates git tar
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Free disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-          df -h
-      - name: Build vibevoice-cpp backend image and run ASR gRPC e2e tests
-        run: |
-          make test-extra-backend-vibevoice-cpp-transcription
-  # End-to-end audio transform via the e2e-backends gRPC harness. The
-  # LocalVQE GGUF is small (~5 MB) and the model is real-time on CPU, so
-  # the default ubuntu-latest pool is plenty.
-  tests-localvqe-grpc-transform:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.localvqe == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 60
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build localvqe backend image and run audio_transform gRPC e2e tests
-        run: |
-          make test-extra-backend-localvqe-transform
   tests-voxtral:
     needs: detect-changes
     if: needs.detect-changes.outputs.voxtral == 'true' || needs.detect-changes.outputs.run-all == 'true'
@@ -982,55 +751,3 @@ jobs:
       - name: Test kokoros
         run: |
           make -C backend/rust/kokoros test
-  tests-insightface-grpc:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.insightface == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y --no-install-recommends \
-              make build-essential curl unzip ca-certificates git tar
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.26.0'
-      - name: Free disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-          df -h
-      - name: Build insightface backend image and run both model configurations
-        run: |
-          make test-extra-backend-insightface-all
-  tests-speaker-recognition-grpc:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.speaker-recognition == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y --no-install-recommends \
-              make build-essential curl ca-certificates git tar
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.26.0'
-      - name: Free disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-          df -h
-      - name: Build speaker-recognition backend image and run the ECAPA-TDNN configuration
-        run: |
-          make test-extra-backend-speaker-recognition-all

.github/workflows/test.yml

@@ -9,9 +9,12 @@ on:
     tags:
       - '*'
 
+env:
+  GRPC_VERSION: v1.65.0
+
 concurrency:
-  group: ci-tests-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   tests-linux:
@@ -20,12 +23,56 @@ jobs:
       matrix:
         go-version: ['1.26.x']
     steps:
+      - name: Free Disk Space (Ubuntu)
+        uses: jlumbroso/free-disk-space@main
+        with:
+          # this might remove tools that are actually needed,
+          # if set to "true" but frees about 6 GB
+          tool-cache: true
+          # all of these default to true, but feel free to set to
+          # "false" if necessary for your workflow
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          docker-images: true
+          swap-storage: true
+      - name: Release space from worker
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          df -h
       - name: Clone
         uses: actions/checkout@v6
         with:
           submodules: true
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:
@@ -53,9 +100,73 @@ jobs:
           node-version: '22'
       - name: Build React UI
         run: make react-ui
+      - name: Build backends
+        run: |
+          make backends/transformers
+          mkdir external && mv backends/transformers external/transformers
+          make backends/llama-cpp backends/local-store backends/silero-vad backends/piper backends/whisper backends/stablediffusion-ggml
       - name: Test
         run: |
-          PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test
+          TRANSFORMER_BACKEND=$PWD/external/transformers/run.sh PATH="$PATH:/root/go/bin" GO_TAGS="tts" make --jobs 5 --output-sync=target test
+      - name: Setup tmate session if tests fail
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3.23
+        with:
+          detached: true
+          connect-timeout-seconds: 180
+          limit-access-to-actor: true
+
+  tests-e2e-container:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Release space from worker
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          df -h
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Dependencies
+        run: |
+          # Install protoc
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Test
+        run: |
+            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
       - name: Setup tmate session if tests fail
         if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
@@ -84,7 +195,7 @@ jobs:
         run: go version
       - name: Dependencies
         run: |
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus ffmpeg
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus
           pip install --user --no-cache-dir grpcio-tools grpcio
       - name: Setup Node.js
         uses: actions/setup-node@v6
@@ -92,6 +203,10 @@ jobs:
           node-version: '22'
       - name: Build React UI
         run: make react-ui
+      - name: Build llama-cpp-darwin
+        run: |
+          make protogen-go
+          make backends/llama-cpp-darwin
       - name: Test
         run: |
           export C_INCLUDE_PATH=/usr/local/include

.github/workflows/tests-aio.yml

@@ -1,86 +0,0 @@
----
-name: 'tests-aio'
-
-# Runs the all-in-one (AIO) Docker image with real backends + real models.
-# Heavy: builds llama-cpp/whisper/piper/silero-vad/stablediffusion-ggml/local-store
-# and exercises end-to-end inference inside the container. Moved out of test.yml
-# (which used to run on every PR) so PR CI no longer pays this cost.
-#
-# Triggers:
-#   - schedule (nightly @ 04:00 UTC) — catches packaging/image regressions within 24h
-#   - workflow_dispatch — manual run on-demand
-#   - push to master/tags — sanity check after merge / before release
-
-on:
-  schedule:
-    - cron: '0 4 * * *'
-  workflow_dispatch:
-  push:
-    branches:
-      - master
-    tags:
-      - '*'
-
-concurrency:
-  group: ci-tests-aio-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  tests-aio:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Release space from worker
-        run: |
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          df -h
-          echo
-          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
-          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
-          sudo rm -rf /usr/local/lib/android
-          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-          sudo rm -rf /usr/share/dotnet
-          sudo apt-get remove -y '^mono-.*' || true
-          sudo apt-get remove -y '^ghc-.*' || true
-          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-          sudo apt-get remove -y 'php.*' || true
-          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-          sudo apt-get remove -y '^google-.*' || true
-          sudo apt-get remove -y azure-cli || true
-          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-          sudo apt-get remove -y '^gfortran-.*' || true
-          sudo apt-get autoremove -y
-          sudo apt-get clean
-          echo
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          sudo rm -rfv build || true
-          df -h
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Test
-        run: |
-            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
-      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
-        uses: mxschmitt/action-tmate@v3.23
-        with:
-          detached: true
-          connect-timeout-seconds: 180
-          limit-access-to-actor: true

.github/workflows/tests-e2e.yml

@@ -10,8 +10,8 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-e2e-backend-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-e2e-backend-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   tests-e2e-backend:
@@ -24,8 +24,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/tests-ui-e2e.yml

@@ -12,8 +12,8 @@ on:
       - master
 
 concurrency:
-  group: ci-tests-ui-e2e-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-ui-e2e-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   tests-ui-e2e:
@@ -26,8 +26,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/update_swagger.yaml

@@ -11,8 +11,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - uses: actions/setup-go@v5
         with:
           go-version: 'stable'

.golangci.yml

@@ -1,53 +0,0 @@
-version: "2"
-
-# Only issues introduced relative to master are reported. Pre-existing issues
-# in the codebase do not fail the lint job; they're treated as a baseline that
-# can be cleaned up incrementally. New code (added lines on a branch) is held
-# to the full linter set. Locally, `make lint-all` overrides this and reports
-# every issue.
-issues:
-  # origin/master because in shallow CI checkouts only the remote-tracking
-  # branch exists; a bare 'master' ref isn't reachable locally.
-  new-from-merge-base: origin/master
-
-linters:
-  default: standard
-  # staticcheck is noisy on this codebase (mostly QF style suggestions like
-  # "could use tagged switch" or "unnecessary fmt.Sprintf"). Re-enable
-  # selectively if a high-signal subset is identified.
-  disable:
-    - staticcheck
-  enable:
-    - forbidigo
-  settings:
-    forbidigo:
-      forbid:
-        - pattern: '^t\.Errorf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(...) instead of t.Errorf. See .agents/coding-style.md.'
-        - pattern: '^t\.Error$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(...) instead of t.Error. See .agents/coding-style.md.'
-        - pattern: '^t\.Fatalf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(Succeed()) / Fail(...) instead of t.Fatalf. See .agents/coding-style.md.'
-        - pattern: '^t\.Fatal$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(Succeed()) / Fail(...) instead of t.Fatal. See .agents/coding-style.md.'
-        - pattern: '^t\.Run$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Describe/Context/It instead of t.Run. See .agents/coding-style.md.'
-        - pattern: '^t\.Skip$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Skip(...) instead of t.Skip. See .agents/coding-style.md.'
-        - pattern: '^t\.Skipf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Skip(...) instead of t.Skipf. See .agents/coding-style.md.'
-        - pattern: '^t\.SkipNow$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Skip(...) instead of t.SkipNow. See .agents/coding-style.md.'
-        - pattern: '^t\.Logf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use GinkgoWriter / fmt.Fprintf(GinkgoWriter, ...) instead of t.Logf. See .agents/coding-style.md.'
-        - pattern: '^t\.Log$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use GinkgoWriter / fmt.Fprintln(GinkgoWriter, ...) instead of t.Log. See .agents/coding-style.md.'
-        - pattern: '^t\.Fail$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Fail(...) instead of t.Fail. See .agents/coding-style.md.'
-        - pattern: '^t\.FailNow$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Fail(...) instead of t.FailNow. See .agents/coding-style.md.'
-  exclusions:
-    paths:
-      # Upstream whisper.cpp source tree fetched by the whisper backend Makefile.
-      - 'backend/go/whisper/sources'
-      - 'docs/'

AGENTS.md

@@ -1,35 +1,20 @@
 # LocalAI Agent Instructions
 
-This file is the entry point for AI coding assistants (Claude Code, Cursor, Copilot, Codex, Aider, etc.) working on LocalAI. It is an index to detailed topic guides in the `.agents/` directory. Read the relevant file(s) for the task at hand — you don't need to load all of them.
-
-Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development workflow.
-
-## Policy for AI-Assisted Contributions
-
-LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:
-
-- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
-- **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
-- **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
-- **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.
+This file is an index to detailed topic guides in the `.agents/` directory. Read the relevant file(s) for the task at hand — you don't need to load all of them.
 
 ## Topics
 
 | File | When to read |
 |------|-------------|
-| [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md) | Policy for AI-assisted contributions — licensing, DCO, attribution |
 | [.agents/building-and-testing.md](.agents/building-and-testing.md) | Building the project, running tests, Docker builds for specific platforms |
-| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache, per-arch keys), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, prebuilt `base-grpc-*` images for llama.cpp variants, per-arch native + manifest-merge pattern, `setup-build-disk` `/mnt` relocation, path filter on master push, manual eviction |
-| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist, including importer integration (the `/import-model` dropdown is server-driven from `GET /backends/known`) |
+| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist |
 | [.agents/coding-style.md](.agents/coding-style.md) | Code style, editorconfig, logging, documentation conventions |
 | [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
 | [.agents/vllm-backend.md](.agents/vllm-backend.md) | Working on the vLLM / vLLM-omni backends — native parsers, ChatDelta, CPU build, libnuma packaging, backend hooks |
-| [.agents/sglang-backend.md](.agents/sglang-backend.md) | Working on the SGLang backend — `engine_args` validation against ServerArgs, speculative-decoding (EAGLE/EAGLE3/DFLASH/MTP) recipes, parser handling |
 | [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
 | [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
 | [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |
 | [.agents/adding-gallery-models.md](.agents/adding-gallery-models.md) | Adding GGUF models from HuggingFace to the model gallery |
-| [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) | LocalAI Assistant chat modality — adding admin tools to the in-process MCP server, editing skill prompts, keeping REST + MCP + skills in sync |
 
 ## Quick Reference
 
@@ -37,7 +22,5 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 - **Go style**: Prefer `any` over `interface{}`
 - **Comments**: Explain *why*, not *what*
 - **Docs**: Update `docs/content/` when adding features or changing config
-- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
-- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
 - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
 - **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI

CONTRIBUTING.md

@@ -13,7 +13,6 @@ Thank you for your interest in contributing to LocalAI! We appreciate your time
   - [Development Workflow](#development-workflow)
   - [Creating a Pull Request (PR)](#creating-a-pull-request-pr)
 - [Coding Guidelines](#coding-guidelines)
-- [AI Coding Assistants](#ai-coding-assistants)
 - [Testing](#testing)
 - [Documentation](#documentation)
 - [Community and Communication](#community-and-communication)
@@ -186,7 +185,7 @@ Before jumping into a PR for a massive feature or big change, it is preferred to
 
 This project uses an [`.editorconfig`](.editorconfig) file to define formatting standards (indentation, line endings, charset, etc.). Please configure your editor to respect it.
 
-For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`CLAUDE.md`](CLAUDE.md) symlink) for agent-specific guidelines including build instructions and backend architecture details. Contributions produced with AI assistance must follow the rules in the [AI Coding Assistants](#ai-coding-assistants) section below.
+For AI-assisted development, see [`CLAUDE.md`](CLAUDE.md) for agent-specific guidelines including build instructions and backend architecture details.
 
 ### General Principles
 
@@ -212,26 +211,6 @@ For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`C
 - Reviewers will check for correctness, test coverage, adherence to these guidelines, and clarity of intent.
 - Be responsive to review feedback and keep discussions constructive.
 
-## AI Coding Assistants
-
-LocalAI follows the **same guidelines as the Linux kernel project** for AI-assisted contributions: <https://docs.kernel.org/process/coding-assistants.html>.
-
-The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:
-
-- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
-- **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
-- **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:
-
-  ```
-  Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
-  ```
-
-  Example: `Assisted-by: Claude:claude-opus-4-7 golangci-lint`
-
-  Basic development tools (git, go, make, editors) should not be listed.
-- **The human submitter is responsible** for reviewing, testing, and fully understanding every line of AI-generated code — including verifying that any referenced APIs, flags, or file paths actually exist in the tree.
-- Contributions must remain compatible with LocalAI's **MIT License**.
-
 ## Testing
 
 All new features and bug fixes should include test coverage. The project uses [Ginkgo](https://onsi.github.io/ginkgo/) as its test framework.

Dockerfile

@@ -1,20 +1,13 @@
 ARG BASE_IMAGE=ubuntu:24.04
+ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 ARG INTEL_BASE_IMAGE=${BASE_IMAGE}
 ARG UBUNTU_CODENAME=noble
-# Optional alternate Ubuntu apt mirror(s). Empty = use upstream.
-# See .docker/apt-mirror.sh for accepted values.
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS requirements
 
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 ENV DEBIAN_FRONTEND=noninteractive
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates curl wget espeak-ng libgomp1 \
         ffmpeg libopenblas0 libopenblas-dev libopus0 sox && \
@@ -156,7 +149,6 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
-            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \
@@ -248,14 +240,10 @@ WORKDIR /build
 # This is a temporary workaround until Intel fixes their repository
 FROM ${INTEL_BASE_IMAGE} AS intel
 ARG UBUNTU_CODENAME=noble
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 RUN wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
 gpg --yes --dearmor --output /usr/share/keyrings/intel-graphics.gpg
 RUN echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu ${UBUNTU_CODENAME}/lts/2350 unified" > /etc/apt/sources.list.d/intel-graphics.list
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         intel-oneapi-runtime-libs && \
     apt-get clean && \

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/tinygrad
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -10,13 +10,6 @@ LAUNCHER_BINARY_NAME=local-ai-launcher
 UBUNTU_VERSION?=2404
 UBUNTU_CODENAME?=noble
 
-# Optional Ubuntu apt mirror overrides forwarded to docker builds.
-# Empty = use upstream archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com.
-# Set e.g. APT_MIRROR=http://azure.archive.ubuntu.com to route apt traffic
-# during outages of the default Ubuntu pool.
-APT_MIRROR?=
-APT_PORTS_MIRROR?=
-
 GORELEASER?=
 
 export BUILD_TYPE?=
@@ -72,7 +65,7 @@ endif
 TEST_PATHS?=./api/... ./pkg/... ./core/...
 
 
-.PHONY: all test build vendor lint lint-all
+.PHONY: all test build vendor
 
 all: help
 
@@ -92,7 +85,6 @@ clean: ## Remove build related file
 clean-tests:
 	rm -rf test-models
 	rm -rf test-dir
-	rm -f tests/e2e/mock-backend/mock-backend
 
 ## Install Go tools
 install-go-tools:
@@ -151,56 +143,32 @@ osx-signed: build
 run: ## run local-ai
 	CGO_LDFLAGS="$(CGO_LDFLAGS)" $(GOCMD) run ./
 
-prepare-test: protogen-go build-mock-backend
+test-models/testmodel.ggml:
+	mkdir -p test-models
+	mkdir -p test-dir
+	wget -q https://huggingface.co/mradermacher/gpt2-alpaca-gpt4-GGUF/resolve/main/gpt2-alpaca-gpt4.Q4_K_M.gguf -O test-models/testmodel.ggml
+	wget -q https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin -O test-models/whisper-en
+	wget -q https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav -O test-dir/audio.wav
+	cp tests/models_fixtures/* test-models
+
+prepare-test: protogen-go
+	cp tests/models_fixtures/* test-models
 
 ########################################################
 ## Tests
 ########################################################
 
 ## Test targets
-## After the test-suite reorg (see plans/test-reorg) the default `make test`
-## no longer downloads multi-GB GGUF/whisper fixtures or builds llama-cpp /
-## transformers / piper / whisper / stablediffusion-ggml. core/http/app_test.go
-## now drives the mock-backend binary built by build-mock-backend; real-backend
-## inference moved into tests/e2e-backends/ (per-backend, path-filtered) and
-## tests/e2e-aio/ (nightly).
-test: prepare-test
+test: test-models/testmodel.ggml protogen-go
 	@echo 'Running tests'
 	export GO_TAGS="debug"
+	$(MAKE) prepare-test
 	OPUS_SHIM_LIBRARY=$(abspath ./pkg/opus/shim/libopusshim.so) \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) --fail-fast -v -r $(TEST_PATHS)
-
-########################################################
-## Lint
-########################################################
-## Runs golangci-lint with config from .golangci.yml. Includes the standard
-## linter set plus forbidigo, which enforces the Ginkgo/Gomega-only test
-## convention documented in .agents/coding-style.md.
-##
-## LINT_EXCLUDE_DIRS_RE matches directories whose Go packages can't typecheck
-## without C/C++ headers we don't install in the lint runner (cgo wrappers
-## around llama.cpp, piper/spdlog, silero-vad/onnxruntime, and Fyne/OpenGL for
-## the launcher). Their compile-time correctness is enforced by their own
-## build pipelines. Keep this as a deny list — `go list ./...` discovers
-## everything else automatically, so new packages are scanned by default.
-LINT_EXCLUDE_DIRS_RE=/(backend/go/(piper|silero-vad|llm)|cmd/launcher)(/|$$)
-
-lint:
-	@command -v golangci-lint >/dev/null 2>&1 || { \
-		echo 'golangci-lint not installed. Install: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest'; \
-		exit 1; \
-	}
-	golangci-lint run $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
-
-## Like `lint` but reports every issue, including the pre-existing baseline
-## that `lint` ignores via .golangci.yml's new-from-merge-base. Use this to
-## see what's available to clean up.
-lint-all:
-	@command -v golangci-lint >/dev/null 2>&1 || { \
-		echo 'golangci-lint not installed. Install: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest'; \
-		exit 1; \
-	}
-	golangci-lint run --new=false --new-from-merge-base= --new-from-rev= $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
+	HUGGINGFACE_GRPC=$(abspath ./)/backend/python/transformers/run.sh TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="!llama-gguf"  --flake-attempts $(TEST_FLAKES) --fail-fast -v -r $(TEST_PATHS)
+	$(MAKE) test-llama-gguf
+	$(MAKE) test-tts
+	$(MAKE) test-stablediffusion
 
 ########################################################
 ## E2E AIO tests (uses standard image with pre-configured models)
@@ -216,8 +184,6 @@ docker-build-e2e:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		-t local-ai:tests -f Dockerfile .
 
@@ -232,20 +198,6 @@ run-e2e-aio: protogen-go
 	@echo 'Running e2e AIO tests'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e-aio
 
-# vLLM multi-node DP smoke (CPU). Builds local-ai:tests and the
-# cpu-vllm backend from the current working tree, then drives a
-# head + headless follower via testcontainers-go and asserts a chat
-# completion. BuildKit caches both images, so re-runs only rebuild
-# what changed. The test lives under tests/e2e/distributed and is
-# selected by the VLLMMultinode label so it doesn't run alongside
-# the other distributed-suite tests by default.
-test-e2e-vllm-multinode: docker-build-e2e extract-backend-vllm protogen-go
-	@echo 'Running e2e vLLM multi-node DP test'
-	LOCALAI_IMAGE=local-ai \
-	LOCALAI_IMAGE_TAG=tests \
-	LOCALAI_VLLM_BACKEND_DIR=$(abspath ./local-backends/vllm) \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='VLLMMultinode' -v -r ./tests/e2e/distributed
-
 ########################################################
 ## E2E tests
 ########################################################
@@ -259,8 +211,6 @@ prepare-e2e:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
 		-t localai-tests .
@@ -285,12 +235,20 @@ teardown-e2e:
 ## Integration and unit tests
 ########################################################
 
-## Storage / vector-store integration. Requires the local-store backend to
-## be available — we build it on demand and pass its location via
-## BACKENDS_PATH (the model loader looks there for the gRPC binary).
-test-stores: backends/local-store
-	BACKENDS_PATH=$(abspath ./)/backends \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r tests/integration
+test-llama-gguf: prepare-test
+	TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="llama-gguf" --flake-attempts $(TEST_FLAKES) -v -r $(TEST_PATHS)
+
+test-tts: prepare-test
+	TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="tts" --flake-attempts $(TEST_FLAKES) -v -r $(TEST_PATHS)
+
+test-stablediffusion: prepare-test
+	TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="stablediffusion" --flake-attempts $(TEST_FLAKES) -v -r $(TEST_PATHS)
+
+test-stores:
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="stores" --flake-attempts $(TEST_FLAKES) -v -r tests/integration
 
 test-opus:
 	@echo 'Running opus backend tests'
@@ -302,8 +260,6 @@ test-opus-docker:
 	docker build --target builder \
 	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),) \
 	  --build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
-	  --build-arg APT_MIRROR=$(APT_MIRROR) \
-	  --build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 	  --build-arg BACKEND=opus \
 	  -t localai-opus-test -f backend/Dockerfile.golang .
 	docker run --rm localai-opus-test \
@@ -313,13 +269,23 @@ test-realtime: build-mock-backend
 	@echo 'Running realtime e2e tests (mock backend)'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime && !real-models" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
 
-# Container-based real-model realtime testing. Build env vars / pipeline
-# definition kept here so test-realtime-models-docker can drive a fully wired
-# pipeline (VAD + STT + LLM + TTS) from inside a containerised runner.
+# Real-model realtime tests. Set REALTIME_TEST_MODEL to use your own pipeline,
+# or leave unset to auto-build one from the component env vars below.
 REALTIME_VAD?=silero-vad-ggml
 REALTIME_STT?=whisper-1
 REALTIME_LLM?=qwen3-0.6b
 REALTIME_TTS?=tts-1
+REALTIME_BACKENDS_PATH?=$(abspath ./)/backends
+
+test-realtime-models: build-mock-backend
+	@echo 'Running realtime e2e tests (real models)'
+	REALTIME_TEST_MODEL=$${REALTIME_TEST_MODEL:-realtime-test-pipeline} \
+	REALTIME_VAD=$(REALTIME_VAD) \
+	REALTIME_STT=$(REALTIME_STT) \
+	REALTIME_LLM=$(REALTIME_LLM) \
+	REALTIME_TTS=$(REALTIME_TTS) \
+	REALTIME_BACKENDS_PATH=$(REALTIME_BACKENDS_PATH) \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
 
 # --- Container-based real-model testing ---
 
@@ -333,7 +299,7 @@ local-backends:
 
 extract-backend-%: docker-build-% local-backends
 	@echo "Extracting backend $*..."
-	@CID=$$(docker create --entrypoint=/run.sh local-ai-backend:$*) && \
+	@CID=$$(docker create local-ai-backend:$*) && \
 	  rm -rf local-backends/$* && mkdir -p local-backends/$* && \
 	  docker cp $$CID:/ - | tar -xf - -C local-backends/$* && \
 	  docker rm $$CID > /dev/null
@@ -345,8 +311,6 @@ test-realtime-models-docker: build-mock-backend
 	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),cublas) \
 	  --build-arg CUDA_MAJOR_VERSION=$(or $(CUDA_MAJOR_VERSION),13) \
 	  --build-arg CUDA_MINOR_VERSION=$(or $(CUDA_MINOR_VERSION),0) \
-	  --build-arg APT_MIRROR=$(APT_MIRROR) \
-	  --build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 	  -t localai-test-runner .
 	docker run --rm \
 	  $(REALTIME_DOCKER_FLAGS) \
@@ -430,13 +394,7 @@ protoc:
 .PHONY: protogen-go
 protogen-go: protoc install-go-tools
 	mkdir -p pkg/grpc/proto
-	# install-go-tools writes protoc-gen-go and protoc-gen-go-grpc into
-	# $(shell go env GOPATH)/bin, which isn't on every dev's PATH. protoc
-	# resolves its code-gen plugins via PATH, so without this prefix the
-	# generate step fails with "protoc-gen-go: program not found". Prepend
-	# GOPATH/bin so the freshly-installed plugins win without requiring a
-	# shell-profile change.
-	PATH="$$(go env GOPATH)/bin:$$PATH" ./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
+	./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
     backend/backend.proto
 
 core/config/inference_defaults.json: ## Fetch inference defaults from unsloth (only if missing)
@@ -476,8 +434,6 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/ace-step
 	$(MAKE) -C backend/python/trl
 	$(MAKE) -C backend/python/tinygrad
-	$(MAKE) -C backend/python/insightface
-	$(MAKE) -C backend/python/speaker-recognition
 	$(MAKE) -C backend/rust/kokoros kokoros-grpc
 
 test-extra: prepare-test-extra
@@ -501,8 +457,6 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/ace-step test
 	$(MAKE) -C backend/python/trl test
 	$(MAKE) -C backend/python/tinygrad test
-	$(MAKE) -C backend/python/insightface test
-	$(MAKE) -C backend/python/speaker-recognition test
 	$(MAKE) -C backend/rust/kokoros test
 
 ##
@@ -553,20 +507,11 @@ test-extra-backend: protogen-go
 	BACKEND_TEST_TOOL_NAME="$$BACKEND_TEST_TOOL_NAME" \
 	BACKEND_TEST_CACHE_TYPE_K="$$BACKEND_TEST_CACHE_TYPE_K" \
 	BACKEND_TEST_CACHE_TYPE_V="$$BACKEND_TEST_CACHE_TYPE_V" \
-	BACKEND_TEST_FACE_IMAGE_1_URL="$$BACKEND_TEST_FACE_IMAGE_1_URL" \
-	BACKEND_TEST_FACE_IMAGE_1_FILE="$$BACKEND_TEST_FACE_IMAGE_1_FILE" \
-	BACKEND_TEST_FACE_IMAGE_2_URL="$$BACKEND_TEST_FACE_IMAGE_2_URL" \
-	BACKEND_TEST_FACE_IMAGE_2_FILE="$$BACKEND_TEST_FACE_IMAGE_2_FILE" \
-	BACKEND_TEST_FACE_IMAGE_3_URL="$$BACKEND_TEST_FACE_IMAGE_3_URL" \
-	BACKEND_TEST_FACE_IMAGE_3_FILE="$$BACKEND_TEST_FACE_IMAGE_3_FILE" \
-	BACKEND_TEST_VERIFY_DISTANCE_CEILING="$$BACKEND_TEST_VERIFY_DISTANCE_CEILING" \
 	go test -v -timeout 30m ./tests/e2e-backends/...
 
 ## Convenience wrappers: build the image, then exercise it.
 test-extra-backend-llama-cpp: docker-build-llama-cpp
-	BACKEND_IMAGE=local-ai-backend:llama-cpp \
-	BACKEND_TEST_CAPS=health,load,predict,stream,logprobs,logit_bias \
-	$(MAKE) test-extra-backend
+	BACKEND_IMAGE=local-ai-backend:llama-cpp $(MAKE) test-extra-backend
 
 test-extra-backend-ik-llama-cpp: docker-build-ik-llama-cpp
 	BACKEND_IMAGE=local-ai-backend:ik-llama-cpp $(MAKE) test-extra-backend
@@ -594,7 +539,6 @@ test-extra-backend-llama-cpp-transcription: docker-build-llama-cpp
 	BACKEND_TEST_MMPROJ_URL=https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf \
 	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
 	BACKEND_TEST_CAPS=health,load,transcription \
-	BACKEND_TEST_CTX_SIZE=2048 \
 	$(MAKE) test-extra-backend
 
 ## vllm is resolved from a HuggingFace model id (no file download) and
@@ -609,14 +553,6 @@ test-extra-backend-vllm: docker-build-vllm
 	BACKEND_TEST_OPTIONS=tool_parser:hermes \
 	$(MAKE) test-extra-backend
 
-## vllm multi-node data-parallel smoke test. Runs LocalAI head + a
-## `local-ai p2p-worker vllm` follower in docker compose against
-## Qwen2.5-0.5B with data_parallel_size=2. Requires 2 NVIDIA GPUs and
-## nvidia-container-runtime on the host — vLLM v1's DP coordinator is
-## not viable on CPU so this cannot run in CI without GPU.
-test-extra-backend-vllm-multinode:
-	./tests/e2e/vllm-multinode/smoke.sh
-
 ## tinygrad mirrors the vllm target (same model, same caps, same parser) so
 ## the two backends are directly comparable. The LLM path covers Predict,
 ## streaming and native tool-call extraction. Companion targets below cover
@@ -667,258 +603,6 @@ test-extra-backend-tinygrad-all: \
 	test-extra-backend-tinygrad-sd \
 	test-extra-backend-tinygrad-whisper
 
-## insightface — face recognition.
-##
-## Face fixtures default to the sample images shipped in the
-## deepinsight/insightface repository (MIT-licensed). For offline/local
-## runs override with BACKEND_TEST_FACE_IMAGE_{1,2,3}_FILE pointing at
-## local paths.
-FACE_IMAGE_1_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
-FACE_IMAGE_2_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
-FACE_IMAGE_3_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/mask_white.jpg
-## Known spoof fixture used by the face_antispoof e2e cap. This is
-## upstream's own `image_F2.jpg` (Silent-Face repo, via yakhyo mirror)
-## — verified to classify as is_real=false with score < 0.05 on the
-## MiniFASNetV2 + MiniFASNetV1SE ensemble.
-FACE_SPOOF_IMAGE_URL ?= https://github.com/yakhyo/face-anti-spoofing/raw/main/assets/image_F2.jpg
-
-## Host-side cache for the OpenCV Zoo face ONNX files used by the
-## opencv e2e target. The backend image no longer bakes model weights —
-## gallery installs bring them via `files:` — but the e2e suite drives
-## LoadModel over gRPC directly without going through the gallery. We
-## pre-download the ONNX files to a stable host path and pass absolute
-## paths in BACKEND_TEST_OPTIONS; `make` skips the downloads when the
-## SHA-256 already matches.
-INSIGHTFACE_OPENCV_DIR := /tmp/localai-insightface-opencv-cache
-INSIGHTFACE_OPENCV_YUNET_URL := https://github.com/opencv/opencv_zoo/raw/main/models/face_detection_yunet/face_detection_yunet_2023mar.onnx
-INSIGHTFACE_OPENCV_SFACE_URL := https://github.com/opencv/opencv_zoo/raw/main/models/face_recognition_sface/face_recognition_sface_2021dec.onnx
-INSIGHTFACE_OPENCV_YUNET_SHA := 8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4
-INSIGHTFACE_OPENCV_SFACE_SHA := 0ba9fbfa01b5270c96627c4ef784da859931e02f04419c829e83484087c34e79
-
-## buffalo_sc (insightface) — pack zip + SHA-256 mirrors the gallery
-## entry so the e2e target matches exactly what `local-ai models install
-## insightface-buffalo-sc` would have fetched. Smallest insightface pack
-## (~16MB) — keeps CI fast while still covering the insightface engine
-## code path end-to-end.
-INSIGHTFACE_BUFFALO_SC_DIR := /tmp/localai-insightface-buffalo-sc-cache
-INSIGHTFACE_BUFFALO_SC_URL := https://github.com/deepinsight/insightface/releases/download/v0.7/buffalo_sc.zip
-INSIGHTFACE_BUFFALO_SC_SHA := 57d31b56b6ffa911c8a73cfc1707c73cab76efe7f13b675a05223bf42de47c72
-
-## Silent-Face antispoofing (MiniFASNetV2 + MiniFASNetV1SE) — shared
-## between the buffalo_sc and opencv e2e targets. Both ONNX files are
-## ~1.7MB, Apache 2.0. URLs + SHAs mirror the gallery entries.
-INSIGHTFACE_ANTISPOOF_DIR := /tmp/localai-insightface-antispoof-cache
-INSIGHTFACE_ANTISPOOF_V2_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV2.onnx
-INSIGHTFACE_ANTISPOOF_V2_SHA := b32929adc2d9c34b9486f8c4c7bc97c1b69bc0ea9befefc380e4faae4e463907
-INSIGHTFACE_ANTISPOOF_V1SE_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV1SE.onnx
-INSIGHTFACE_ANTISPOOF_V1SE_SHA := ebab7f90c7833fbccd46d3a555410e78d969db5438e169b6524be444862b3676
-
-.PHONY: insightface-opencv-models
-insightface-opencv-models:
-	@mkdir -p $(INSIGHTFACE_OPENCV_DIR)
-	@if [ "$$(sha256sum $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_OPENCV_YUNET_SHA)" ]; then \
-		echo "Fetching YuNet..."; \
-		curl -fsSL -o $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx $(INSIGHTFACE_OPENCV_YUNET_URL); \
-		echo "$(INSIGHTFACE_OPENCV_YUNET_SHA)  $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx" | sha256sum -c; \
-	fi
-	@if [ "$$(sha256sum $(INSIGHTFACE_OPENCV_DIR)/sface.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_OPENCV_SFACE_SHA)" ]; then \
-		echo "Fetching SFace..."; \
-		curl -fsSL -o $(INSIGHTFACE_OPENCV_DIR)/sface.onnx $(INSIGHTFACE_OPENCV_SFACE_URL); \
-		echo "$(INSIGHTFACE_OPENCV_SFACE_SHA)  $(INSIGHTFACE_OPENCV_DIR)/sface.onnx" | sha256sum -c; \
-	fi
-
-.PHONY: insightface-antispoof-models
-insightface-antispoof-models:
-	@mkdir -p $(INSIGHTFACE_ANTISPOOF_DIR)
-	@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V2_SHA)" ]; then \
-		echo "Fetching MiniFASNetV2..."; \
-		curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx $(INSIGHTFACE_ANTISPOOF_V2_URL); \
-		echo "$(INSIGHTFACE_ANTISPOOF_V2_SHA)  $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx" | sha256sum -c; \
-	fi
-	@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA)" ]; then \
-		echo "Fetching MiniFASNetV1SE..."; \
-		curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx $(INSIGHTFACE_ANTISPOOF_V1SE_URL); \
-		echo "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA)  $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx" | sha256sum -c; \
-	fi
-
-.PHONY: insightface-buffalo-sc-models
-insightface-buffalo-sc-models:
-	@mkdir -p $(INSIGHTFACE_BUFFALO_SC_DIR)
-	@if [ "$$(sha256sum $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_BUFFALO_SC_SHA)" ]; then \
-		echo "Fetching buffalo_sc..."; \
-		curl -fsSL -o $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip $(INSIGHTFACE_BUFFALO_SC_URL); \
-		echo "$(INSIGHTFACE_BUFFALO_SC_SHA)  $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip" | sha256sum -c; \
-		rm -f $(INSIGHTFACE_BUFFALO_SC_DIR)/*.onnx; \
-	fi
-	@if [ ! -f "$(INSIGHTFACE_BUFFALO_SC_DIR)/det_500m.onnx" ]; then \
-		echo "Extracting buffalo_sc..."; \
-		unzip -o -q $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip -d $(INSIGHTFACE_BUFFALO_SC_DIR); \
-	fi
-
-## buffalo_sc — smallest insightface pack (SCRFD-500MF detector + MBF
-## recognizer, ~16MB). Exercises the insightface engine code path
-## (model_zoo-backed inference) without the ~326MB buffalo_l download.
-## No age/gender/landmark heads — face_analyze is dropped from caps.
-## The pack is pre-fetched on the host and passed as `root:<dir>` since
-## the e2e suite drives LoadModel directly without going through
-## LocalAI's gallery flow (which is what would normally populate
-## ModelPath and in turn the engine's `_model_dir` option).
-test-extra-backend-insightface-buffalo-sc: docker-build-insightface insightface-buffalo-sc-models insightface-antispoof-models
-	BACKEND_IMAGE=local-ai-backend:insightface \
-	BACKEND_TEST_MODEL_NAME=insightface-buffalo-sc \
-	BACKEND_TEST_OPTIONS=engine:insightface,model_pack:buffalo_sc,root:$(INSIGHTFACE_BUFFALO_SC_DIR),antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
-	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
-	BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
-	BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
-	BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
-	BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
-	BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
-	$(MAKE) test-extra-backend
-
-## OpenCV Zoo YuNet + SFace — Apache 2.0, commercial-safe. face_analyze
-## cap is dropped (SFace has no demographic head). The ONNX files are
-## pre-fetched on the host via the insightface-opencv-models target and
-## passed as absolute paths, since the e2e suite drives LoadModel
-## directly without going through LocalAI's gallery flow.
-test-extra-backend-insightface-opencv: docker-build-insightface insightface-opencv-models insightface-antispoof-models
-	BACKEND_IMAGE=local-ai-backend:insightface \
-	BACKEND_TEST_MODEL_NAME=insightface-opencv \
-	BACKEND_TEST_OPTIONS=engine:onnx_direct,detector_onnx:$(INSIGHTFACE_OPENCV_DIR)/yunet.onnx,recognizer_onnx:$(INSIGHTFACE_OPENCV_DIR)/sface.onnx,antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
-	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
-	BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
-	BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
-	BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
-	BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
-	BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
-	$(MAKE) test-extra-backend
-
-## Aggregate — runs both face-recognition model configurations so CI
-## catches regressions across engines together.
-test-extra-backend-insightface-all: \
-	test-extra-backend-insightface-buffalo-sc \
-	test-extra-backend-insightface-opencv
-
-## speaker-recognition — voice (speaker) biometrics.
-##
-## Audio fixtures default to the speechbrain test samples served
-## straight from their GitHub repo — public, no auth needed, and they
-## ship as 16kHz mono WAV/FLAC which is exactly what the engine wants.
-## example{1,2,5} are three different speakers; the suite treats
-## example1 as the "same-image twin" probe (verify(clip, clip) must
-## return distance≈0) and the other two as cross-speaker ceilings.
-## Override with BACKEND_TEST_VOICE_AUDIO_{1,2,3}_FILE for offline runs.
-VOICE_AUDIO_1_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example1.wav
-VOICE_AUDIO_2_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example2.flac
-VOICE_AUDIO_3_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example5.wav
-
-## ECAPA-TDNN via SpeechBrain — default CI configuration. Auto-downloads
-## the checkpoint from HuggingFace on first LoadModel (bundled in the
-## backend image pip install). 192-d embeddings, cosine-distance based.
-## The e2e suite drives LoadModel directly so we don't rely on LocalAI's
-## gallery flow here.
-test-extra-backend-speaker-recognition-ecapa: docker-build-speaker-recognition
-	BACKEND_IMAGE=local-ai-backend:speaker-recognition \
-	BACKEND_TEST_MODEL_NAME=speechbrain/spkrec-ecapa-voxceleb \
-	BACKEND_TEST_OPTIONS=engine:speechbrain,source:speechbrain/spkrec-ecapa-voxceleb \
-	BACKEND_TEST_CAPS=health,load,voice_embed,voice_verify \
-	BACKEND_TEST_VOICE_AUDIO_1_URL=$(VOICE_AUDIO_1_URL) \
-	BACKEND_TEST_VOICE_AUDIO_2_URL=$(VOICE_AUDIO_2_URL) \
-	BACKEND_TEST_VOICE_AUDIO_3_URL=$(VOICE_AUDIO_3_URL) \
-	BACKEND_TEST_VOICE_VERIFY_DISTANCE_CEILING=0.4 \
-	$(MAKE) test-extra-backend
-
-## Aggregate — today there's only one voice config; the target exists
-## so the CI workflow matches the insightface-all naming convention and
-## can grow to include WeSpeaker / 3D-Speaker later.
-test-extra-backend-speaker-recognition-all: \
-	test-extra-backend-speaker-recognition-ecapa
-
-## Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked
-## LLM. Extracts the sherpa-onnx Docker image rootfs, downloads the three
-## gallery-referenced model bundles (silero-vad, omnilingual-asr, vits-ljs),
-## writes the corresponding model config YAMLs, and runs the realtime
-## websocket spec in tests/e2e with REALTIME_* env vars wiring the sherpa
-## slots into the pipeline. The LLM slot stays on the in-repo mock-backend
-## registered unconditionally by tests/e2e/e2e_suite_test.go. See
-## tests/e2e/run-realtime-sherpa.sh for the full orchestration.
-test-extra-e2e-realtime-sherpa: build-mock-backend docker-build-sherpa-onnx protogen-go react-ui
-	bash tests/e2e/run-realtime-sherpa.sh
-
-## Streaming ASR via the sherpa-onnx online recognizer. Uses the streaming
-## zipformer English model (encoder/decoder/joiner int8 + tokens) from the
-## sherpa-onnx gallery entry. Drives both AudioTranscription and
-## AudioTranscriptionStream via the e2e-backends gRPC harness; streaming
-## emits real partial deltas during decode. Each file is renamed on download
-## to the shape sherpa-onnx's online loader expects (encoder.int8.onnx etc.).
-test-extra-backend-sherpa-onnx-transcription: docker-build-sherpa-onnx
-	BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#encoder.int8.onnx' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/decoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#decoder.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/joiner-epoch-99-avg-1-chunk-16-left-128.int8.onnx#joiner.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/tokens.txt' \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	BACKEND_TEST_OPTIONS=subtype=online \
-	$(MAKE) test-extra-backend
-
-## VITS TTS via the sherpa-onnx backend. Pulls the individual files from
-## HuggingFace (the vits-ljs release tarball lives on the k2-fsa github
-## but is also mirrored as discrete files on HF). Exercises both
-## TTS (write-to-file) and TTSStream (PCM chunks + WAV header) via the
-## e2e-backends gRPC harness.
-test-extra-backend-sherpa-onnx-tts: docker-build-sherpa-onnx
-	BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/vits-ljs.onnx#vits-ljs.onnx' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/tokens.txt|https://huggingface.co/csukuangfj/vits-ljs/resolve/main/lexicon.txt' \
-	BACKEND_TEST_CAPS=health,load,tts \
-	$(MAKE) test-extra-backend
-
-## VibeVoice TTS via the vibevoice-cpp backend. ModelFile is the
-## realtime gguf; the supplementary tokenizer + voice prompt land
-## alongside it under the harness's models dir and are wired through
-## via the standard Options[] convention (tokenizer=, voice=).
-test-extra-backend-vibevoice-cpp-tts: docker-build-vibevoice-cpp
-	BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-realtime-0.5B-q8_0.gguf#vibevoice-realtime-0.5B-q8_0.gguf' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf|https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/voice-en-Carter_man.gguf#voice-en-Carter_man.gguf' \
-	BACKEND_TEST_OPTIONS=tokenizer:tokenizer.gguf,voice:voice-en-Carter_man.gguf \
-	BACKEND_TEST_CAPS=health,load,tts \
-	$(MAKE) test-extra-backend
-
-## VibeVoice ASR (long-form, with diarization). type=asr tells the
-## backend's Load() to slot ModelFile into the asr_model role; the
-## tokenizer is supplied via Options[]. Uses the Q4_K quant (~10 GB)
-## rather than Q8_0 (~14 GB) so the bundle fits inside ubuntu-latest's
-## post-image disk budget.
-test-extra-backend-vibevoice-cpp-transcription: docker-build-vibevoice-cpp
-	BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-asr-q4_k.gguf#vibevoice-asr-q4_k.gguf' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf' \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_OPTIONS=type:asr,tokenizer:tokenizer.gguf \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	$(MAKE) test-extra-backend
-
-## Audio transcription wrapper for the whisper.cpp backend.
-## Drives the AudioTranscription / AudioTranscriptionStream RPCs against
-## ggml-base.en (~145 MB) using the JFK 11s clip. The streaming spec
-## asserts len(deltas) >= 1 and concat(deltas) == final.Text - whisper-
-## specific multi-segment assertions live in backend/go/whisper/gowhisper_test.go.
-test-extra-backend-whisper-transcription: docker-build-whisper
-	BACKEND_IMAGE=local-ai-backend:whisper \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	$(MAKE) test-extra-backend
-
-## LocalVQE audio transform (joint AEC + noise suppression + dereverb).
-## Exercises the audio_transform capability end-to-end: batch transform
-## of a real WAV fixture and bidi streaming of synthetic silent frames.
-test-extra-backend-localvqe-transform: docker-build-localvqe
-	BACKEND_IMAGE=local-ai-backend:localvqe \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/LocalAI-io/LocalVQE/resolve/main/localvqe-v1-1.3M-f32.gguf#localvqe-v1-1.3M-f32.gguf' \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,audio_transform \
-	$(MAKE) test-extra-backend
-
 ## sglang mirrors the vllm setup: HuggingFace model id, same tiny Qwen,
 ## tool-call extraction via sglang's native qwen parser. CPU builds use
 ## sglang's upstream pyproject_cpu.toml recipe (see backend/python/sglang/install.sh).
@@ -961,8 +645,6 @@ docker:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE) .
 
 docker-cuda12:
@@ -976,13 +658,11 @@ docker-cuda12:
 		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE)-cuda-12 .
 
 docker-image-intel:
 	docker build \
-		--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04 \
+		--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04 \
 		--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
@@ -991,8 +671,6 @@ docker-image-intel:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE) .
 
 ########################################################
@@ -1061,10 +739,7 @@ BACKEND_WHISPER = whisper|golang|.|false|true
 BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
-BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
-BACKEND_LOCALVQE = localvqe|golang|.|false|true
 BACKEND_OPUS = opus|golang|.|false|true
-BACKEND_SHERPA_ONNX = sherpa-onnx|golang|.|false|true
 
 # Python backends with root context
 BACKEND_RERANKERS = rerankers|python|.|false|true
@@ -1073,8 +748,6 @@ BACKEND_OUTETTS = outetts|python|.|false|true
 BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
 BACKEND_COQUI = coqui|python|.|false|true
 BACKEND_RFDETR = rfdetr|python|.|false|true
-BACKEND_INSIGHTFACE = insightface|python|.|false|true
-BACKEND_SPEAKER_RECOGNITION = speaker-recognition|python|.|false|true
 BACKEND_KITTEN_TTS = kitten-tts|python|.|false|true
 BACKEND_NEUTTS = neutts|python|.|false|true
 BACKEND_KOKORO = kokoro|python|.|false|true
@@ -1117,10 +790,7 @@ define docker-build-backend
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		$(if $(FROM_SOURCE),--build-arg FROM_SOURCE=$(FROM_SOURCE)) \
-		$(if $(AMDGPU_TARGETS),--build-arg AMDGPU_TARGETS=$(AMDGPU_TARGETS)) \
 		$(if $(filter true,$(5)),--build-arg BACKEND=$(1)) \
 		-t local-ai-backend:$(1) -f backend/Dockerfile.$(2) $(3)
 endef
@@ -1149,8 +819,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_OUTETTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
-$(eval $(call generate-docker-build-target,$(BACKEND_INSIGHTFACE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_SPEAKER_RECOGNITION)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KITTEN_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
@@ -1172,8 +840,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX_VLM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX_DISTRIBUTED)))
@@ -1182,13 +848,12 @@ $(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP_QUANTIZATION)))
 $(eval $(call generate-docker-build-target,$(BACKEND_TINYGRAD)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KOKOROS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_SAM3_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX)))
 
 # Pattern rule for docker-save targets
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp
 
 ########################################################
 ### Mock Backend for E2E Tests

README.md

@@ -38,7 +38,7 @@
 - **Built-in AI agents** — autonomous agents with tool use, RAG, MCP, and skills
 - **Privacy-first** — your data never leaves your infrastructure
 
-Created by [Ettore Di Giacinto](https://github.com/mudler) and maintained by the [LocalAI team](#team).
+Created and maintained by [Ettore Di Giacinto](https://github.com/mudler).
 
 > [:book: Documentation](https://localai.io/) | [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) | [💻 Quickstart](https://localai.io/basics/getting_started/) | [🖼️ Models](https://models.localai.io/) | [❓FAQ](https://localai.io/faq/)
 
@@ -149,7 +149,6 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
 
 ## Latest News
 
-- **April 2026**: [Voice recognition](https://github.com/mudler/LocalAI/pull/9500), [Face recognition, identification & liveness detection](https://github.com/mudler/LocalAI/pull/9480), [Ollama API compatibility](https://github.com/mudler/LocalAI/pull/9284), [Video generation in stable-diffusion.ggml](https://github.com/mudler/LocalAI/pull/9420), [Backend versioning with auto-upgrade](https://github.com/mudler/LocalAI/pull/9315), [Pin models & load-on-demand toggle](https://github.com/mudler/LocalAI/pull/9309), [Universal model importer](https://github.com/mudler/LocalAI/pull/9466), new backends: [sglang](https://github.com/mudler/LocalAI/pull/9359), [ik-llama-cpp](https://github.com/mudler/LocalAI/pull/9326), [TurboQuant](https://github.com/mudler/LocalAI/pull/9355), [sam.cpp](https://github.com/mudler/LocalAI/pull/9288), [Kokoros](https://github.com/mudler/LocalAI/pull/9212), [qwen3tts.cpp](https://github.com/mudler/LocalAI/pull/9316), [tinygrad multimodal](https://github.com/mudler/LocalAI/pull/9364)
 - **March 2026**: [Agent management](https://github.com/mudler/LocalAI/pull/8820), [New React UI](https://github.com/mudler/LocalAI/pull/8772), [WebRTC](https://github.com/mudler/LocalAI/pull/8790), [MLX-distributed via P2P and RDMA](https://github.com/mudler/LocalAI/pull/8801), [MCP Apps, MCP Client-side](https://github.com/mudler/LocalAI/pull/8947)
 - **February 2026**: [Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245), [ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396)
 - **January 2026**: **LocalAI 3.10.0** — Anthropic API support, Open Responses API, video & image generation (LTX-2), unified GPU backends, tool streaming, Moonshine, Pocket-TTS. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0)
@@ -201,14 +200,13 @@ See the full [Backend & Model Compatibility Table](https://localai.io/model-comp
 - [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
 - [Examples](https://github.com/mudler/LocalAI-examples)
 
-## Team
+## Autonomous Development Team
 
-LocalAI is maintained by a small team of humans, together with the wider community of contributors.
+LocalAI is helped being maintained by a team of autonomous AI agents led by an AI Scrum Master.
 
-- **[Ettore Di Giacinto](https://github.com/mudler)** — original author and project lead
-- **[Richard Palethorpe](https://github.com/richiejp)** — maintainer
-
-A huge thank you to everyone who contributes code, reviews PRs, files issues, and helps users in [Discord](https://discord.gg/uJAeKSAGDy) — LocalAI is a community-driven project and wouldn't exist without you. See the full [contributors list](https://github.com/mudler/LocalAI/graphs/contributors).
+- **Live Reports**: [reports.localai.io](http://reports.localai.io)
+- **Project Board**: [Agent task tracking](https://github.com/users/mudler/projects/6)
+- **Blog Post**: [Learn about the experiment](https://mudler.pm/posts/2026/02/28/a-call-to-open-source-maintainers-stop-babysitting-ai-how-i-built-a-100-local-autonomous-dev-team-to-maintain-localai-and-why-you-should-too/)
 
 ## Citation
 
@@ -251,7 +249,7 @@ A special thanks to individual sponsors, a full list is on [GitHub](https://gith
 
 ## License
 
-LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/) and maintained by the [LocalAI team](#team).
+LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/).
 
 MIT - Author Ettore Di Giacinto <mudler@localai.io>
 

backend/Dockerfile.base-grpc-builder

@@ -1,98 +0,0 @@
-# syntax=docker/dockerfile:1.7
-#
-# Pre-built builder base image for LocalAI's C++ backends.
-#
-# This Dockerfile is the source of truth for the
-# `quay.io/go-skynet/ci-cache:base-grpc-*` images that
-# `.github/workflows/base-images.yml` builds and pushes. The output of a
-# build is a fully-prepped builder layer containing:
-#
-#   - apt build deps (build-essential, ccache, git, make, pkg-config,
-#     libcurl4-openssl-dev, libssl-dev, curl, unzip, wget, ca-certificates)
-#   - cmake (apt or, when CMAKE_FROM_SOURCE=true, compiled from
-#     ${CMAKE_VERSION})
-#   - protoc v27.1 at /usr/local/bin/protoc
-#   - gRPC ${GRPC_VERSION} compiled and installed at /opt/grpc
-#   - Conditional CUDA toolkit (BUILD_TYPE=cublas|l4t, SKIP_DRIVERS=false)
-#     including the cuda-13 + arm64 cudss/nvpl special case
-#   - Conditional ROCm/HIP build deps (BUILD_TYPE=hipblas)
-#   - Conditional Vulkan SDK 1.4.335.0 (BUILD_TYPE=vulkan)
-#
-# Variants built by the workflow (matrix in base-images.yml):
-#
-#   base-grpc-amd64                 ubuntu:24.04, CPU-only
-#   base-grpc-arm64                 ubuntu:24.04, CPU-only
-#   base-grpc-cuda-12-amd64         ubuntu:24.04 + CUDA 12.8
-#   base-grpc-cuda-13-amd64         ubuntu:22.04 + CUDA 13.0
-#   base-grpc-cuda-13-arm64         ubuntu:24.04 + CUDA 13.0 (sbsa)
-#   base-grpc-l4t-cuda-12-arm64     ubuntu:22.04 + CUDA 12.x (legacy JetPack)
-#   base-grpc-rocm-amd64            rocm/dev-ubuntu-24.04:7.2.1 + hipblas
-#   base-grpc-vulkan-amd64          ubuntu:24.04 + Vulkan SDK 1.4.335
-#   base-grpc-vulkan-arm64          ubuntu:24.04 + Vulkan SDK ARM 1.4.335
-#   base-grpc-intel-amd64           intel/oneapi-basekit:2025.3.2 (sycl)
-#
-# This is a SINGLE-stage Dockerfile by design: the final image IS the
-# builder base. The intermediate gRPC compile happens inside this same
-# stage so consumer Dockerfiles in PR 2 can simply
-# `FROM quay.io/go-skynet/ci-cache:base-grpc-<variant>` without needing a
-# COPY --from=grpc step. /opt/grpc is the canonical install prefix and
-# downstream builds will add it to CMAKE_PREFIX_PATH (or copy to
-# /usr/local) the same way Dockerfile.llama-cpp does today.
-#
-# Install logic lives in .docker/install-base-deps.sh, which is also
-# bind-mounted by the variant Dockerfiles' builder-fromsource stage.
-# This guarantees bit-equivalence between the prebuilt CI base and the
-# from-source local-dev path — both invoke the same script with the
-# same env inputs.
-
-ARG BASE_IMAGE=ubuntu:24.04
-
-FROM ${BASE_IMAGE}
-
-ARG BASE_IMAGE=ubuntu:24.04
-ARG BUILD_TYPE=""
-ARG CUDA_MAJOR_VERSION=""
-ARG CUDA_MINOR_VERSION=""
-ARG CMAKE_FROM_SOURCE=false
-# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain
-# detection / arch table issues.
-ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG SKIP_DRIVERS=false
-ARG TARGETARCH
-ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
-ARG AMDGPU_TARGETS=""
-
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    DEBIAN_FRONTEND=noninteractive
-
-# CUDA on PATH (no-op when CUDA isn't installed)
-ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
-ENV PATH=/opt/rocm/bin:${PATH}
-
-WORKDIR /build
-
-# Single RUN that delegates to .docker/install-base-deps.sh — the same
-# script the variant Dockerfiles' builder-fromsource stage runs.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
-
-WORKDIR /

backend/Dockerfile.golang

@@ -1,6 +1,4 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -16,20 +14,8 @@ ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
-ARG AMDGPU_TARGETS
-ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 
-# gcc-14 is the default on noble (ubuntu:24.04) but absent from jammy
-# (the L4T jetpack r36.4.0 base). LocalVQE specifically needs it; the
-# other Go backends compile fine with the default gcc shipped via
-# build-essential. So: try gcc-14 from the configured repos, fall back
-# gracefully when it's not available so jammy-based builds don't fail
-# at the apt step.
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         git ccache \
@@ -37,12 +23,6 @@ RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mi
         make cmake wget libopenblas-dev \
         curl unzip \
         libssl-dev && \
-    if apt-cache show gcc-14 >/dev/null 2>&1 && apt-cache show g++-14 >/dev/null 2>&1; then \
-        apt-get install -y --no-install-recommends gcc-14 g++-14 && \
-        update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100 \
-            --slave /usr/bin/g++ g++ /usr/bin/g++-14 \
-            --slave /usr/bin/gcov gcov /usr/bin/gcov-14; \
-    fi && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -167,7 +147,6 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
-            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \

backend/Dockerfile.ik-llama-cpp

@@ -1,149 +1,279 @@
 ARG BASE_IMAGE=ubuntu:24.04
-# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
-# when no prebuilt base is supplied. The builder-prebuilt stage is only
-# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
-# content here is harmless — BuildKit prunes the unreferenced builder.
-ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
-# BUILDER_TARGET selects which builder stage the final scratch image copies
-# package output from. Declared at global scope (before any FROM) so it's
-# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
-# `make backends/ik-llama-cpp` on the from-source path.
-ARG BUILDER_TARGET=builder-fromsource
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
+ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 
 
-# ============================================================================
-# Stage: builder-fromsource — self-contained build path.
-# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
-# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
-# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
-# default; local `make backends/ik-llama-cpp`).
-#
-# The install script is the same one that backend/Dockerfile.base-grpc-builder
-# runs, so the result is bit-equivalent to the prebuilt-base path
-# (builder-prebuilt below).
-# ============================================================================
-FROM ${BASE_IMAGE} AS builder-fromsource
-ARG BUILD_TYPE
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
+# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
+# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
+FROM ${GRPC_BASE_IMAGE} AS grpc
+
+# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+
+ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
+
+WORKDIR /build
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        ca-certificates \
+        build-essential curl libssl-dev \
+        git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
+# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
+# and running make install in the target container
+RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
+    mkdir -p /build/grpc/cmake/build && \
+    cd /build/grpc/cmake/build && \
+    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
+    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
+    make && \
+    make install && \
+    rm -rf /build
+
+FROM ${BASE_IMAGE} AS builder
+ARG CMAKE_FROM_SOURCE=false
+ARG CMAKE_VERSION=3.31.10
+# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+ARG BACKEND=rerankers
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
 ARG SKIP_DRIVERS=false
+ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
+ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
+ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-ARG AMDGPU_TARGETS=""
-ARG BACKEND=rerankers
-# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ARG CMAKE_ARGS
 
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
-    CMAKE_ARGS=${CMAKE_ARGS} \
-    DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ccache git \
+        ca-certificates \
+        make \
+        pkg-config libcurl4-openssl-dev \
+        curl unzip \
+        libssl-dev wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
 
-# CUDA on PATH (no-op when CUDA isn't installed)
+# Cuda
 ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
+
+# HipBLAS requirements
 ENV PATH=/opt/rocm/bin:${PATH}
 
-WORKDIR /build
 
-# Install everything via the shared script — the same one that
-# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
-# this from-source path are bit-equivalent.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
+# Vulkan requirements
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils wget gpg-agent && \
+        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
+            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            mkdir -p /opt/vulkan-sdk && \
+            mv 1.4.335.0 /opt/vulkan-sdk/ && \
+            cd /opt/vulkan-sdk/1.4.335.0 && \
+            ./vulkansdk --no-deps --maxjobs \
+                vulkan-loader \
+                vulkan-validationlayers \
+                vulkan-extensionlayer \
+                vulkan-tools \
+                shaderc && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
+            rm -rf /opt/vulkan-sdk
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            mkdir vulkan && cd vulkan && \
+            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
+            tar -xvf vulkan-sdk.tar.xz && \
+            rm vulkan-sdk.tar.xz && \
+            cd 1.4.335.0 && \
+            cp -rfv aarch64/bin/* /usr/bin/ && \
+            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
+            cp -rfv aarch64/include/* /usr/include/ && \
+            cp -rfv aarch64/share/* /usr/share/ && \
+            cd ../.. && \
+            rm -rf vulkan
+        fi
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# CuBLAS requirements
+RUN <<EOT bash
+    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
+            else
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
+            fi
+        fi
+        dpkg -i cuda-keyring_1.1-1_all.deb && \
+        rm -f cuda-keyring_1.1-1_all.deb && \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
+            apt-get install -y --no-install-recommends \
+            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        fi
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+
+# https://github.com/NVIDIA/Isaac-GR00T/issues/343
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
+        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
+        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get install -y nvpl
+    fi
+EOT
+
+# If we are building with clblas support, we need the libraries for the builds
+RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libclblast-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* \
+    ; fi
+
+RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            hipblas-dev \
+            rocblas-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* && \
+        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
+        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
+        ldconfig \
+    ; fi
+
+RUN echo "TARGETARCH: $TARGETARCH"
+
+# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
+# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
+# here so that we can generate the grpc code for the stablediffusion build
+RUN <<EOT bash
+    if [ "amd64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+    if [ "arm64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+EOT
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+COPY --from=grpc /opt/grpc /usr/local
 
-# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
-# CMake's find_package finds it at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
 
 COPY . /LocalAI
 
-# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
-# for the rationale. Distinct mount id so ik-llama-cpp's cache doesn't
-# overlap with llama-cpp's — ik_llama.cpp is a different fork with
-# different source.
-#
-# The compile body is shared with builder-prebuilt via .docker/ik-llama-cpp-compile.sh.
-RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
+RUN <<'EOT' bash
+set -euxo pipefail
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
+fi
+
+cd /LocalAI/backend/cpp/ik-llama-cpp
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  # ARM64 / ROCm: build without x86 SIMD
+  make ik-llama-cpp-fallback
+else
+  # ik_llama.cpp's IQK kernels require at least AVX2
+  make ik-llama-cpp-avx2
+fi
+EOT
 
 
 # Copy libraries using a script to handle architecture differences
 RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
 
 
-# ============================================================================
-# Stage: builder-prebuilt — uses the pre-built base from
-# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
-# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
-# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
-# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
-# builder-base-image).
-# ============================================================================
-FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
-
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
-# /usr/local. Mirror what the from-source path does so the compile step
-# can find gRPC at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
-
-
-# ============================================================================
-# Final stage — copies package output from one of the two builders.
-# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
-#
-# BuildKit doesn't support variable expansion in `COPY --from=` directly,
-# so we resolve the ARG by aliasing the chosen builder to a fixed stage
-# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
-# BUILDER_TARGET itself is declared as a global ARG at the top of this
-# file (required for use in FROM), so we just re-import it into this
-# stage's scope before the FROM directive.
-# ============================================================================
-FROM ${BUILDER_TARGET} AS builder
-
 FROM scratch
 
 

backend/Dockerfile.llama-cpp

@@ -1,155 +1,290 @@
 ARG BASE_IMAGE=ubuntu:24.04
-# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
-# when no prebuilt base is supplied. The builder-prebuilt stage is only
-# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
-# content here is harmless — BuildKit prunes the unreferenced builder.
-ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
-# BUILDER_TARGET selects which builder stage the final scratch image copies
-# package output from. Declared at global scope (before any FROM) so it's
-# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
-# `make backends/llama-cpp` on the from-source path.
-ARG BUILDER_TARGET=builder-fromsource
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
+ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 
 
-# ============================================================================
-# Stage: builder-fromsource — self-contained build path.
-# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
-# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
-# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
-# default; local `make backends/llama-cpp`).
-#
-# The install script is the same one that backend/Dockerfile.base-grpc-builder
-# runs, so the result is bit-equivalent to the prebuilt-base path
-# (builder-prebuilt below).
-# ============================================================================
-FROM ${BASE_IMAGE} AS builder-fromsource
-ARG BUILD_TYPE
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
+# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
+# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
+FROM ${GRPC_BASE_IMAGE} AS grpc
+
+# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG SKIP_DRIVERS=false
-ARG TARGETARCH
-ARG TARGETVARIANT
-ARG GO_VERSION=1.25.4
-ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-ARG AMDGPU_TARGETS
-# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ARG CMAKE_ARGS
 
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
-    CMAKE_ARGS=${CMAKE_ARGS} \
-    DEBIAN_FRONTEND=noninteractive
-
-# CUDA on PATH (no-op when CUDA isn't installed)
-ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
-ENV PATH=/opt/rocm/bin:${PATH}
+ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
 
 WORKDIR /build
 
-# Install everything via the shared script — the same one that
-# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
-# this from-source path are bit-equivalent.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        ca-certificates \
+        build-essential curl libssl-dev \
+        git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
 
-# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
-# CMake's find_package finds it at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
 
-COPY . /LocalAI
+# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
+# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
+# and running make install in the target container
+RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
+    mkdir -p /build/grpc/cmake/build && \
+    cd /build/grpc/cmake/build && \
+    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
+    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
+    make && \
+    make install && \
+    rm -rf /build
 
-# BuildKit cache mount for ccache. Persists compiler outputs across builds
-# via the registry cache (cache-to: type=registry,mode=max in CI). On a
-# LLAMA_VERSION bump most TUs are byte-identical to the previous version's
-# preprocessed source — ccache returns the previous .o file and skips the
-# real compile. Same for LocalAI source changes that don't touch llama.cpp.
-# CMAKE_*_COMPILER_LAUNCHER threads ccache through CMake to wrap gcc/g++/nvcc.
-# sharing=locked serializes concurrent writes if multiple matrix variants
-# share the same cache mount id.
-#
-# The compile body is shared with builder-prebuilt via .docker/llama-cpp-compile.sh.
-RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-
-# Copy libraries using a script to handle architecture differences
-RUN make -BC /LocalAI/backend/cpp/llama-cpp package
-
-
-# ============================================================================
-# Stage: builder-prebuilt — uses the pre-built base from
-# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
-# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
-# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
-# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
-# builder-base-image).
-# ============================================================================
-FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
-
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
+FROM ${BASE_IMAGE} AS builder
+ARG CMAKE_FROM_SOURCE=false
+ARG CMAKE_VERSION=3.31.10
+# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
 ARG CUDA_DOCKER_ARCH
 ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
 ARG CMAKE_ARGS
 ENV CMAKE_ARGS=${CMAKE_ARGS}
 ARG AMDGPU_TARGETS
 ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
+ARG BACKEND=rerankers
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
+ARG SKIP_DRIVERS=false
+ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
+ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
+ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
+ARG GO_VERSION=1.25.4
+ARG UBUNTU_VERSION=2404
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ccache git \
+        ca-certificates \
+        make \
+        pkg-config libcurl4-openssl-dev \
+        curl unzip \
+        libssl-dev wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Cuda
+ENV PATH=/usr/local/cuda/bin:${PATH}
+
+# HipBLAS requirements
+ENV PATH=/opt/rocm/bin:${PATH}
+
+
+# Vulkan requirements
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils wget gpg-agent && \
+        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
+            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            mkdir -p /opt/vulkan-sdk && \
+            mv 1.4.335.0 /opt/vulkan-sdk/ && \
+            cd /opt/vulkan-sdk/1.4.335.0 && \
+            ./vulkansdk --no-deps --maxjobs \
+                vulkan-loader \
+                vulkan-validationlayers \
+                vulkan-extensionlayer \
+                vulkan-tools \
+                shaderc && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
+            rm -rf /opt/vulkan-sdk
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            mkdir vulkan && cd vulkan && \
+            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
+            tar -xvf vulkan-sdk.tar.xz && \
+            rm vulkan-sdk.tar.xz && \
+            cd 1.4.335.0 && \
+            cp -rfv aarch64/bin/* /usr/bin/ && \
+            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
+            cp -rfv aarch64/include/* /usr/include/ && \
+            cp -rfv aarch64/share/* /usr/share/ && \
+            cd ../.. && \
+            rm -rf vulkan
+        fi
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# CuBLAS requirements
+RUN <<EOT bash
+    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
+            else
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
+            fi
+        fi
+        dpkg -i cuda-keyring_1.1-1_all.deb && \
+        rm -f cuda-keyring_1.1-1_all.deb && \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
+            apt-get install -y --no-install-recommends \
+            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        fi
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+
+# https://github.com/NVIDIA/Isaac-GR00T/issues/343
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
+        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
+        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get install -y nvpl
+    fi
+EOT
+
+# If we are building with clblas support, we need the libraries for the builds
+RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libclblast-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* \
+    ; fi
+
+RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            hipblas-dev \
+            rocblas-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* && \
+        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
+        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
+        ldconfig && \
+        # Log which GPU architectures have rocBLAS kernel support
+        echo "rocBLAS library data architectures:" && \
+        (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
+        echo "WARNING: No rocBLAS kernel data found" \
+    ; fi
+
+RUN echo "TARGETARCH: $TARGETARCH"
+
+# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
+# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
+# here so that we can generate the grpc code for the stablediffusion build
+RUN <<EOT bash
+    if [ "amd64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+    if [ "arm64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+EOT
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+COPY --from=grpc /opt/grpc /usr/local
 
-# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
-# /usr/local. The variant Dockerfile's from-source path does that too;
-# mirror it here so the compile step can find gRPC at the canonical
-# prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
 
 COPY . /LocalAI
 
-RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
+RUN <<'EOT' bash
+set -euxo pipefail
 
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
+fi
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  cd /LocalAI/backend/cpp/llama-cpp
+  make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
+else
+  cd /LocalAI/backend/cpp/llama-cpp
+  make llama-cpp-avx
+  make llama-cpp-avx2
+  make llama-cpp-avx512
+  make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
+fi
+EOT
+
+
+# Copy libraries using a script to handle architecture differences
 RUN make -BC /LocalAI/backend/cpp/llama-cpp package
 
 
-# ============================================================================
-# Final stage — copies package output from one of the two builders.
-# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
-#
-# BuildKit doesn't support variable expansion in `COPY --from=` directly,
-# so we resolve the ARG by aliasing the chosen builder to a fixed stage
-# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
-# BUILDER_TARGET itself is declared as a global ARG at the top of this
-# file (required for use in FROM), so we just re-import it into this
-# stage's scope before the FROM directive.
-# ============================================================================
-FROM ${BUILDER_TARGET} AS builder
-
 FROM scratch
 
 

backend/Dockerfile.python

@@ -1,6 +1,4 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -15,12 +13,8 @@ ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         ccache \
@@ -168,7 +162,6 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
-            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \
@@ -209,13 +202,6 @@ COPY scripts/build/package-gpu-libs.sh /package-gpu-libs.sh
 ARG FROM_SOURCE=""
 ENV FROM_SOURCE=${FROM_SOURCE}
 
-# Cache-buster for the per-backend `make` step. Most Python backends list
-# unpinned deps (torch, transformers, vllm, ...), so a warm registry cache
-# would otherwise freeze upstream versions indefinitely. CI passes a value
-# that rolls weekly so the install layer is rebuilt at most once per week
-# and picks up newer wheels from PyPI / nightly indexes.
-ARG DEPS_REFRESH=initial
-
 RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
 
 # Package GPU libraries into the backend's lib directory
@@ -230,4 +216,4 @@ RUN if [ -f "/${BACKEND}/package.sh" ]; then \
 
 FROM scratch
 ARG BACKEND=rerankers
-COPY --from=builder /${BACKEND}/ /
+COPY --from=builder /${BACKEND}/ /

backend/Dockerfile.rust

@@ -1,18 +1,12 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=kokoros
 ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         git ccache \

backend/Dockerfile.turboquant

@@ -1,152 +1,288 @@
 ARG BASE_IMAGE=ubuntu:24.04
-# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
-# when no prebuilt base is supplied. The builder-prebuilt stage is only
-# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
-# content here is harmless — BuildKit prunes the unreferenced builder.
-ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
-# BUILDER_TARGET selects which builder stage the final scratch image copies
-# package output from. Declared at global scope (before any FROM) so it's
-# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
-# `make backends/turboquant` on the from-source path.
-ARG BUILDER_TARGET=builder-fromsource
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
+ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 
 
-# ============================================================================
-# Stage: builder-fromsource — self-contained build path.
-# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
-# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
-# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
-# default; local `make backends/turboquant`).
-#
-# The install script is the same one that backend/Dockerfile.base-grpc-builder
-# runs, so the result is bit-equivalent to the prebuilt-base path
-# (builder-prebuilt below).
-# ============================================================================
-FROM ${BASE_IMAGE} AS builder-fromsource
-ARG BUILD_TYPE
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
+# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
+# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
+FROM ${GRPC_BASE_IMAGE} AS grpc
+
+# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+
+ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
+
+WORKDIR /build
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        ca-certificates \
+        build-essential curl libssl-dev \
+        git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
+# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
+# and running make install in the target container
+RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
+    mkdir -p /build/grpc/cmake/build && \
+    cd /build/grpc/cmake/build && \
+    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
+    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
+    make && \
+    make install && \
+    rm -rf /build
+
+FROM ${BASE_IMAGE} AS builder
+ARG CMAKE_FROM_SOURCE=false
+ARG CMAKE_VERSION=3.31.10
+# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+ARG BACKEND=rerankers
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
 ARG SKIP_DRIVERS=false
+ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
+ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
+ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-ARG AMDGPU_TARGETS=""
-ARG BACKEND=rerankers
-# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ARG CMAKE_ARGS
 
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
-    CMAKE_ARGS=${CMAKE_ARGS} \
-    DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ccache git \
+        ca-certificates \
+        make \
+        pkg-config libcurl4-openssl-dev \
+        curl unzip \
+        libssl-dev wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
 
-# CUDA on PATH (no-op when CUDA isn't installed)
+# Cuda
 ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
+
+# HipBLAS requirements
 ENV PATH=/opt/rocm/bin:${PATH}
 
-WORKDIR /build
 
-# Install everything via the shared script — the same one that
-# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
-# this from-source path are bit-equivalent.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
+# Vulkan requirements
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils wget gpg-agent && \
+        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
+            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            mkdir -p /opt/vulkan-sdk && \
+            mv 1.4.335.0 /opt/vulkan-sdk/ && \
+            cd /opt/vulkan-sdk/1.4.335.0 && \
+            ./vulkansdk --no-deps --maxjobs \
+                vulkan-loader \
+                vulkan-validationlayers \
+                vulkan-extensionlayer \
+                vulkan-tools \
+                shaderc && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
+            rm -rf /opt/vulkan-sdk
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            mkdir vulkan && cd vulkan && \
+            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
+            tar -xvf vulkan-sdk.tar.xz && \
+            rm vulkan-sdk.tar.xz && \
+            cd 1.4.335.0 && \
+            cp -rfv aarch64/bin/* /usr/bin/ && \
+            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
+            cp -rfv aarch64/include/* /usr/include/ && \
+            cp -rfv aarch64/share/* /usr/share/ && \
+            cd ../.. && \
+            rm -rf vulkan
+        fi
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# CuBLAS requirements
+RUN <<EOT bash
+    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
+            else
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
+            fi
+        fi
+        dpkg -i cuda-keyring_1.1-1_all.deb && \
+        rm -f cuda-keyring_1.1-1_all.deb && \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
+            apt-get install -y --no-install-recommends \
+            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        fi
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+
+# https://github.com/NVIDIA/Isaac-GR00T/issues/343
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
+        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
+        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get install -y nvpl
+    fi
+EOT
+
+# If we are building with clblas support, we need the libraries for the builds
+RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libclblast-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* \
+    ; fi
+
+RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            hipblas-dev \
+            rocblas-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* && \
+        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
+        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
+        ldconfig && \
+        # Log which GPU architectures have rocBLAS kernel support
+        echo "rocBLAS library data architectures:" && \
+        (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
+        echo "WARNING: No rocBLAS kernel data found" \
+    ; fi
+
+RUN echo "TARGETARCH: $TARGETARCH"
+
+# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
+# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
+# here so that we can generate the grpc code for the stablediffusion build
+RUN <<EOT bash
+    if [ "amd64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+    if [ "arm64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+EOT
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+COPY --from=grpc /opt/grpc /usr/local
 
-# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
-# CMake's find_package finds it at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
 
 COPY . /LocalAI
 
-# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
-# for rationale. turboquant is a llama.cpp fork that reuses
-# backend/cpp/llama-cpp source via a thin wrapper Makefile, so MOST TUs
-# are content-identical to the upstream llama-cpp build. Sharing a cache
-# id with llama-cpp could give cross-fork hits — but for now keep them
-# separate so a regression in one doesn't poison the other. Revisit
-# sharing after measuring the actual hit rate.
-#
-# The compile body is shared with builder-prebuilt via .docker/turboquant-compile.sh.
-RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
+RUN <<'EOT' bash
+set -euxo pipefail
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/turboquant-*-build
+fi
+
+cd /LocalAI/backend/cpp/turboquant
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  make turboquant-fallback
+  make turboquant-grpc
+  make turboquant-rpc-server
+else
+  make turboquant-avx
+  make turboquant-avx2
+  make turboquant-avx512
+  make turboquant-fallback
+  make turboquant-grpc
+  make turboquant-rpc-server
+fi
+EOT
 
 
 # Copy libraries using a script to handle architecture differences
 RUN make -BC /LocalAI/backend/cpp/turboquant package
 
 
-# ============================================================================
-# Stage: builder-prebuilt — uses the pre-built base from
-# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
-# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
-# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
-# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
-# builder-base-image).
-# ============================================================================
-FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
-
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
-# /usr/local. Mirror what the from-source path does so the compile step
-# can find gRPC at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-RUN make -BC /LocalAI/backend/cpp/turboquant package
-
-
-# ============================================================================
-# Final stage — copies package output from one of the two builders.
-# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
-#
-# BuildKit doesn't support variable expansion in `COPY --from=` directly,
-# so we resolve the ARG by aliasing the chosen builder to a fixed stage
-# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
-# BUILDER_TARGET itself is declared as a global ARG at the top of this
-# file (required for use in FROM), so we just re-import it into this
-# stage's scope before the FROM directive.
-# ============================================================================
-FROM ${BUILDER_TARGET} AS builder
-
 FROM scratch
 
 

backend/backend.proto

@@ -24,11 +24,6 @@ service Backend {
   rpc TokenizeString(PredictOptions) returns (TokenizationResponse) {}
   rpc Status(HealthMessage) returns (StatusResponse) {}
   rpc Detect(DetectOptions) returns (DetectResponse) {}
-  rpc FaceVerify(FaceVerifyRequest) returns (FaceVerifyResponse) {}
-  rpc FaceAnalyze(FaceAnalyzeRequest) returns (FaceAnalyzeResponse) {}
-  rpc VoiceVerify(VoiceVerifyRequest) returns (VoiceVerifyResponse) {}
-  rpc VoiceAnalyze(VoiceAnalyzeRequest) returns (VoiceAnalyzeResponse) {}
-  rpc VoiceEmbed(VoiceEmbedRequest) returns (VoiceEmbedResponse) {}
 
   rpc StoresSet(StoresSetOptions) returns (Result) {}
   rpc StoresDelete(StoresDeleteOptions) returns (Result) {}
@@ -41,14 +36,9 @@ service Backend {
 
   rpc VAD(VADRequest) returns (VADResponse) {}
 
-  rpc Diarize(DiarizeRequest) returns (DiarizeResponse) {}
-
   rpc AudioEncode(AudioEncodeRequest) returns (AudioEncodeResult) {}
   rpc AudioDecode(AudioDecodeRequest) returns (AudioDecodeResult) {}
 
-  rpc AudioTransform(AudioTransformRequest) returns (AudioTransformResult) {}
-  rpc AudioTransformStream(stream AudioTransformFrameRequest) returns (stream AudioTransformFrameResponse) {}
-
   rpc ModelMetadata(ModelOptions) returns (ModelMetadataResponse) {}
 
   // Fine-tuning RPCs
@@ -315,11 +305,6 @@ message ModelOptions {
   bool Reranking = 71;
 
   repeated string Overrides = 72;
-
-  // EngineArgs carries a JSON-encoded map of backend-native engine arguments
-  // applied verbatim to the backend's engine constructor (e.g. vLLM AsyncEngineArgs).
-  // Unknown keys produce an error at LoadModel time.
-  string EngineArgs = 73;
 }
 
 message Result {
@@ -355,12 +340,6 @@ message TranscriptStreamResponse {
   TranscriptResult final_result = 2;
 }
 
-message TranscriptWord {
-  int64 start = 1;
-  int64 end = 2;
-  string text = 3;
-}
-
 message TranscriptSegment {
   int32 id = 1;
   int64 start = 2;
@@ -368,7 +347,6 @@ message TranscriptSegment {
   string text = 4;
   repeated int32 tokens = 5;
   string speaker = 6;
-  repeated TranscriptWord words = 7;
 }
 
 message GenerateImageRequest {
@@ -425,43 +403,6 @@ message VADResponse {
   repeated VADSegment segments = 1;
 }
 
-// --- Speaker diarization messages ---
-//
-// Pure speaker diarization: "who spoke when". Returns time-stamped segments
-// labelled with cluster IDs (the same string for the same speaker across
-// segments). Some backends (e.g. vibevoice.cpp) produce diarization as a
-// by-product of ASR and may also fill in `text` per segment; backends with a
-// dedicated diarization pipeline (e.g. sherpa-onnx pyannote) leave `text`
-// empty and emit only the segmentation.
-
-message DiarizeRequest {
-  string dst = 1;                      // path to audio file (HTTP layer materialises uploads to a temp file)
-  uint32 threads = 2;
-  string language = 3;                 // optional; only meaningful for transcription-bundling backends
-  int32  num_speakers = 4;             // exact speaker count if known (>0 forces); 0 = auto
-  int32  min_speakers = 5;             // hint when auto-detecting; 0 = unset
-  int32  max_speakers = 6;             // hint when auto-detecting; 0 = unset
-  float  clustering_threshold = 7;     // distance threshold when num_speakers unknown; 0 = backend default
-  float  min_duration_on = 8;          // discard segments shorter than this (seconds); 0 = backend default
-  float  min_duration_off = 9;         // merge gaps shorter than this (seconds); 0 = backend default
-  bool   include_text = 10;            // when the backend can emit per-segment transcript for free, ask it to populate `text`
-}
-
-message DiarizeSegment {
-  int32  id = 1;
-  float  start = 2;                    // seconds
-  float  end = 3;                      // seconds
-  string speaker = 4;                  // backend-emitted speaker label (e.g. "0", "SPEAKER_00")
-  string text = 5;                     // optional per-segment transcript (empty unless include_text and supported)
-}
-
-message DiarizeResponse {
-  repeated DiarizeSegment segments = 1;
-  int32  num_speakers = 2;             // count of distinct speaker labels in `segments`
-  float  duration = 3;                 // total audio duration in seconds (0 if unknown)
-  string language = 4;                 // optional, when the backend bundles transcription
-}
-
 message SoundGenerationRequest {
   string text = 1;
   string model = 2;
@@ -534,112 +475,6 @@ message DetectResponse {
   repeated Detection Detections = 1;
 }
 
-// --- Face recognition messages ---
-
-message FacialArea {
-  float x = 1;
-  float y = 2;
-  float w = 3;
-  float h = 4;
-}
-
-message FaceVerifyRequest {
-  string img1 = 1;              // base64-encoded image
-  string img2 = 2;              // base64-encoded image
-  float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
-  bool   anti_spoofing = 4;     // run MiniFASNet liveness on each image; failed liveness forces verified=false
-}
-
-message FaceVerifyResponse {
-  bool       verified = 1;
-  float      distance = 2;      // 1 - cosine_similarity
-  float      threshold = 3;
-  float      confidence = 4;    // 0-100
-  string     model = 5;         // e.g. "buffalo_l"
-  FacialArea img1_area = 6;
-  FacialArea img2_area = 7;
-  float      processing_time_ms = 8;
-  bool       img1_is_real = 9;          // anti-spoofing result when enabled
-  float      img1_antispoof_score = 10;
-  bool       img2_is_real = 11;
-  float      img2_antispoof_score = 12;
-}
-
-message FaceAnalyzeRequest {
-  string          img = 1;          // base64-encoded image
-  repeated string actions = 2;      // subset of ["age","gender","emotion","race"]; empty = all-supported
-  bool            anti_spoofing = 3;
-}
-
-message FaceAnalysis {
-  FacialArea         region = 1;
-  float              face_confidence = 2;
-  float              age = 3;
-  string             dominant_gender = 4;   // "Man" | "Woman"
-  map<string, float> gender = 5;
-  string             dominant_emotion = 6;  // reserved; empty in MVP
-  map<string, float> emotion = 7;
-  string             dominant_race = 8;     // not populated
-  map<string, float> race = 9;
-  bool               is_real = 10;          // anti-spoofing result when enabled
-  float              antispoof_score = 11;
-}
-
-message FaceAnalyzeResponse {
-  repeated FaceAnalysis faces = 1;
-}
-
-// --- Voice (speaker) recognition messages ---
-//
-// Analogous to the Face* messages above, but for speaker biometrics.
-// Audio fields accept a filesystem path (same convention as
-// TranscriptRequest.dst). The HTTP layer materialises base64 / URL /
-// data-URI inputs to a temp file before calling the gRPC backend.
-
-message VoiceVerifyRequest {
-  string audio1 = 1;            // path to first audio clip
-  string audio2 = 2;            // path to second audio clip
-  float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
-  bool   anti_spoofing = 4;     // reserved for future AASIST bolt-on
-}
-
-message VoiceVerifyResponse {
-  bool   verified = 1;
-  float  distance = 2;          // 1 - cosine_similarity
-  float  threshold = 3;
-  float  confidence = 4;        // 0-100
-  string model = 5;             // e.g. "speechbrain/spkrec-ecapa-voxceleb"
-  float  processing_time_ms = 6;
-}
-
-message VoiceAnalyzeRequest {
-  string          audio = 1;        // path to audio clip
-  repeated string actions = 2;      // subset of ["age","gender","emotion"]; empty = all-supported
-}
-
-message VoiceAnalysis {
-  float              start = 1;          // segment start time in seconds (0 if single-utterance)
-  float              end = 2;            // segment end time in seconds
-  float              age = 3;
-  string             dominant_gender = 4;
-  map<string, float> gender = 5;
-  string             dominant_emotion = 6;
-  map<string, float> emotion = 7;
-}
-
-message VoiceAnalyzeResponse {
-  repeated VoiceAnalysis segments = 1;
-}
-
-message VoiceEmbedRequest {
-  string audio = 1;              // path to audio clip
-}
-
-message VoiceEmbedResponse {
-  repeated float embedding = 1;
-  string         model = 2;
-}
-
 message ToolFormatMarkers {
   string format_type = 1;           // "json_native", "tag_with_json", "tag_with_tagged"
 
@@ -718,56 +553,6 @@ message AudioDecodeResult {
   int32 samples_per_frame = 3;
 }
 
-// Generic audio transform: an audio-in, audio-out operation, optionally
-// conditioned on a second reference signal. Concrete transforms include
-// AEC + noise suppression + dereverberation (LocalVQE), voice conversion
-// (reference = target speaker), pitch shifting, etc.
-message AudioTransformRequest {
-  string audio_path = 1;             // required, primary input file path
-  string reference_path = 2;         // optional auxiliary; empty => zero-fill
-  string dst = 3;                    // required, output file path
-  map<string, string> params = 4;    // backend-specific tuning
-}
-
-message AudioTransformResult {
-  string dst = 1;
-  int32  sample_rate = 2;
-  int32  samples = 3;
-  bool   reference_provided = 4;
-}
-
-// Bidirectional streaming audio transform. The first message MUST carry a
-// Config; subsequent messages carry Frames. A second Config mid-stream
-// resets streaming state before the next frame.
-message AudioTransformFrameRequest {
-  oneof payload {
-    AudioTransformStreamConfig config = 1;
-    AudioTransformFrame        frame  = 2;
-  }
-}
-
-message AudioTransformStreamConfig {
-  enum SampleFormat {
-    F32_LE = 0;
-    S16_LE = 1;
-  }
-  SampleFormat sample_format = 1;
-  int32 sample_rate = 2;             // 0 => backend default
-  int32 frame_samples = 3;           // 0 => backend default
-  map<string, string> params = 4;
-  bool reset = 5;                    // reset streaming state before next frame
-}
-
-message AudioTransformFrame {
-  bytes audio_pcm = 1;               // frame_samples samples in stream's format
-  bytes reference_pcm = 2;           // empty => zero-fill (silent reference)
-}
-
-message AudioTransformFrameResponse {
-  bytes pcm = 1;
-  int64 frame_index = 2;
-}
-
 message ModelMetadataResponse {
   bool supports_thinking = 1;
   string rendered_template = 2;  // The rendered chat template with enable_thinking=true (empty if not applicable)

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=23127139cb6fa314899c3b5f4935b88b3374c56c
+IK_LLAMA_VERSION?=8befd92ea5f702494ea9813fe42a52fb015db5fe
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/ik-llama-cpp/grpc-server.cpp

@@ -326,7 +326,7 @@ struct llama_client_slot
        char buffer[512];
         double t_token = t_prompt_processing / num_prompt_tokens_processed;
         double n_tokens_second = 1e3 / t_prompt_processing * num_prompt_tokens_processed;
-        snprintf(buffer, sizeof(buffer), "prompt eval time     = %10.2f ms / %5d tokens (%8.2f ms per token, %8.2f tokens per second)",
+        sprintf(buffer, "prompt eval time     = %10.2f ms / %5d tokens (%8.2f ms per token, %8.2f tokens per second)",
                 t_prompt_processing, num_prompt_tokens_processed,
                 t_token, n_tokens_second);
         LOG_INFO(buffer, {
@@ -340,7 +340,7 @@ struct llama_client_slot
 
         t_token = t_token_generation / n_decoded;
         n_tokens_second = 1e3 / t_token_generation * n_decoded;
-        snprintf(buffer, sizeof(buffer), "generation eval time = %10.2f ms / %5d runs   (%8.2f ms per token, %8.2f tokens per second)",
+        sprintf(buffer, "generation eval time = %10.2f ms / %5d runs   (%8.2f ms per token, %8.2f tokens per second)",
                 t_token_generation, n_decoded,
                 t_token, n_tokens_second);
         LOG_INFO(buffer, {
@@ -352,7 +352,7 @@ struct llama_client_slot
             {"n_tokens_second",    n_tokens_second},
         });
 
-        snprintf(buffer, sizeof(buffer), "          total time = %10.2f ms", t_prompt_processing + t_token_generation);
+        sprintf(buffer, "          total time = %10.2f ms", t_prompt_processing + t_token_generation);
         LOG_INFO(buffer, {
             {"slot_id",             id},
             {"task_id",             task_id},
@@ -686,16 +686,7 @@ struct llama_server_context
         slot->sparams.mirostat_eta      = json_value(data, "mirostat_eta",      default_sparams.mirostat_eta);
         slot->params.n_keep             = json_value(data, "n_keep",            slot->params.n_keep);
         slot->sparams.seed               = json_value(data, "seed",              default_sparams.seed);
-        {
-            // upstream changed common_params_sampling::grammar from std::string to
-            // the common_grammar struct (type + grammar). The incoming JSON still
-            // carries a plain string, so build the user-provided grammar here and
-            // fall back to the server default when the request omits it.
-            std::string grammar_str = json_value(data, "grammar", std::string());
-            slot->sparams.grammar = grammar_str.empty()
-                ? default_sparams.grammar
-                : common_grammar{COMMON_GRAMMAR_TYPE_USER, std::move(grammar_str)};
-        }
+        slot->sparams.grammar           = json_value(data, "grammar",           default_sparams.grammar);
         slot->sparams.n_probs           = json_value(data, "n_probs",           default_sparams.n_probs);
         slot->sparams.min_keep          = json_value(data, "min_keep",          default_sparams.min_keep);
         slot->sparams.grammar_triggers = grammar_triggers;
@@ -1241,7 +1232,7 @@ struct llama_server_context
              //      {"logit_bias",        slot.sparams.logit_bias},
             {"n_probs",           slot.sparams.n_probs},
             {"min_keep",          slot.sparams.min_keep},
-            {"grammar",           slot.sparams.grammar.grammar},
+            {"grammar",           slot.sparams.grammar},
             {"samplers",          samplers}
         };
     }

backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch

@@ -1,11 +0,0 @@
---- a/examples/llava/clip.cpp
-+++ b/examples/llava/clip.cpp
-@@ -2494,7 +2494,7 @@
-             }
-             new_data = work.data();
-
--            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr);
-+            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr, nullptr);
-         } else {
-             new_type = cur->type;
-             new_data = cur->data;

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=389ff61d77b5c71cec0cf92fe4e5d01ace80b797
+LLAMA_VERSION?=4f02d4733934179386cbc15b3454be26237940bb
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=
@@ -34,9 +34,6 @@ else ifeq ($(BUILD_TYPE),hipblas)
 	export CXX=$(ROCM_HOME)/llvm/bin/clang++
 	export CC=$(ROCM_HOME)/llvm/bin/clang
 	AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
-ifeq ($(strip $(AMDGPU_TARGETS)),)
-$(error AMDGPU_TARGETS is empty — set it to a comma-separated list of gfx targets e.g. gfx1100,gfx1101)
-endif
 	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DGGML_VULKAN=1

backend/cpp/llama-cpp/grpc-server.cpp

@@ -10,14 +10,6 @@
 #include "server-task.cpp"
 #include "server-queue.cpp"
 #include "server-common.cpp"
-// server-chat.cpp exists only in llama.cpp after the upstream refactor that
-// split OAI/Anthropic/Responses/transcription conversion helpers out of
-// server-common.cpp. When present, server-context.cpp and server-task.cpp
-// above call into it, so we must pull its definitions into this TU or the
-// link fails. __has_include keeps the source compatible with older pins.
-#if __has_include("server-chat.cpp")
-#include "server-chat.cpp"
-#endif
 #include "server-context.cpp"
 
 // LocalAI
@@ -442,7 +434,7 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
 
     // Draft model for speculative decoding
     if (!request->draftmodel().empty()) {
-        params.speculative.draft.mparams.path = request->draftmodel();
+        params.speculative.mparams_dft.path = request->draftmodel();
         // Default to draft type if a draft model is set but no explicit type
         if (params.speculative.type == COMMON_SPECULATIVE_TYPE_NONE) {
             params.speculative.type = COMMON_SPECULATIVE_TYPE_DRAFT;
@@ -642,21 +634,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
                 params.no_op_offload = false;
             }
-        } else if (!strcmp(optname, "split_mode") || !strcmp(optname, "sm")) {
-            // Accepts: none | layer | row | tensor (the latter requires a llama.cpp build
-            // that includes ggml-org/llama.cpp#19378, FlashAttention enabled, and KV-cache
-            // quantization disabled).
-            if (optval != NULL) {
-                if (optval_str == "none") {
-                    params.split_mode = LLAMA_SPLIT_MODE_NONE;
-                } else if (optval_str == "layer") {
-                    params.split_mode = LLAMA_SPLIT_MODE_LAYER;
-                } else if (optval_str == "row") {
-                    params.split_mode = LLAMA_SPLIT_MODE_ROW;
-                } else if (optval_str == "tensor") {
-                    params.split_mode = LLAMA_SPLIT_MODE_TENSOR;
-                }
-            }
         } else if (!strcmp(optname, "kv_unified") || !strcmp(optname, "unified_kv")) {
             if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
                 params.kv_unified = true;
@@ -679,39 +656,39 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             }
         } else if (!strcmp(optname, "spec_n_max") || !strcmp(optname, "draft_max")) {
             if (optval != NULL) {
-                try { params.speculative.draft.n_max = std::stoi(optval_str); } catch (...) {}
+                try { params.speculative.n_max = std::stoi(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "spec_n_min") || !strcmp(optname, "draft_min")) {
             if (optval != NULL) {
-                try { params.speculative.draft.n_min = std::stoi(optval_str); } catch (...) {}
+                try { params.speculative.n_min = std::stoi(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "spec_p_min") || !strcmp(optname, "draft_p_min")) {
             if (optval != NULL) {
-                try { params.speculative.draft.p_min = std::stof(optval_str); } catch (...) {}
+                try { params.speculative.p_min = std::stof(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "spec_p_split")) {
             if (optval != NULL) {
-                try { params.speculative.draft.p_split = std::stof(optval_str); } catch (...) {}
+                try { params.speculative.p_split = std::stof(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "spec_ngram_size_n") || !strcmp(optname, "ngram_size_n")) {
             if (optval != NULL) {
-                try { params.speculative.ngram_simple.size_n = (uint16_t)std::stoi(optval_str); } catch (...) {}
+                try { params.speculative.ngram_size_n = (uint16_t)std::stoi(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "spec_ngram_size_m") || !strcmp(optname, "ngram_size_m")) {
             if (optval != NULL) {
-                try { params.speculative.ngram_simple.size_m = (uint16_t)std::stoi(optval_str); } catch (...) {}
+                try { params.speculative.ngram_size_m = (uint16_t)std::stoi(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "spec_ngram_min_hits") || !strcmp(optname, "ngram_min_hits")) {
             if (optval != NULL) {
-                try { params.speculative.ngram_simple.min_hits = (uint16_t)std::stoi(optval_str); } catch (...) {}
+                try { params.speculative.ngram_min_hits = (uint16_t)std::stoi(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "draft_gpu_layers")) {
             if (optval != NULL) {
-                try { params.speculative.draft.n_gpu_layers = std::stoi(optval_str); } catch (...) {}
+                try { params.speculative.n_gpu_layers = std::stoi(optval_str); } catch (...) {}
             }
         } else if (!strcmp(optname, "draft_ctx_size")) {
             if (optval != NULL) {
-                try { params.speculative.draft.n_ctx = std::stoi(optval_str); } catch (...) {}
+                try { params.speculative.n_ctx = std::stoi(optval_str); } catch (...) {}
             }
         }
     }
@@ -933,8 +910,8 @@ public:
             if (!params.mmproj.path.empty()) {
                 error_msg += " (with mmproj: " + params.mmproj.path + ")";
             }
-            if (params.speculative.has_dft() && !params.speculative.draft.mparams.path.empty()) {
-                error_msg += " (with draft model: " + params.speculative.draft.mparams.path + ")";
+            if (params.speculative.has_dft() && !params.speculative.mparams_dft.path.empty()) {
+                error_msg += " (with draft model: " + params.speculative.mparams_dft.path + ")";
             }
             
             // Add captured error details if available

backend/cpp/turboquant/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-TURBOQUANT_VERSION?=69d8e4be47243e83b3d0d71e932bc7aa61c644dc
+TURBOQUANT_VERSION?=45f8a066ed5f5bb38c695cec532f6cef9f4efa9d
 LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
 
 CMAKE_ARGS?=

backend/cpp/turboquant/patch-grpc-server.sh

@@ -1,30 +1,13 @@
 #!/bin/bash
-# Patch the shared backend/cpp/llama-cpp/grpc-server.cpp *copy* used by the
-# turboquant build to account for the gaps between upstream and the fork:
+# Augment the shared backend/cpp/llama-cpp/grpc-server.cpp allow-list of KV-cache
+# types so the gRPC `LoadModel` call accepts the TurboQuant-specific
+# `turbo2` / `turbo3` / `turbo4` cache types.
 #
-#   1. Augment the kv_cache_types[] allow-list so `LoadModel` accepts the
-#      fork-specific `turbo2` / `turbo3` / `turbo4` cache types.
-#   2. Replace `get_media_marker()` (added upstream in ggml-org/llama.cpp#21962,
-#      server-side random per-instance marker) with the legacy "<__media__>"
-#      literal. The fork branched before that PR, so server-common.cpp has no
-#      get_media_marker symbol. The fork's mtmd_default_marker() still returns
-#      "<__media__>", and Go-side tooling falls back to that sentinel when the
-#      backend does not expose media_marker, so substituting the literal keeps
-#      behavior identical on the turboquant path.
-#   3. Revert the `common_params_speculative` field references to the
-#      pre-refactor flat layout. Upstream ggml-org/llama.cpp#22397 split the
-#      struct into nested `draft` / `ngram_simple` / `ngram_mod` / etc. members;
-#      the turboquant fork branched before that PR and still exposes the flat
-#      `n_max`, `mparams_dft`, `ngram_size_n`, ... fields. The substitutions
-#      below map the new nested paths back to the legacy flat names so the
-#      shared grpc-server.cpp keeps compiling against the fork's common.h.
-#      Drop this block once the fork rebases past #22397.
+# We do this on the *copy* sitting in turboquant-<flavor>-build/, never on the
+# original under backend/cpp/llama-cpp/, so the stock llama-cpp build keeps
+# compiling against vanilla upstream which does not know about GGML_TYPE_TURBO*.
 #
-# We patch the *copy* sitting in turboquant-<flavor>-build/, never the original
-# under backend/cpp/llama-cpp/, so the stock llama-cpp build keeps compiling
-# against vanilla upstream.
-#
-# Idempotent: skips each insertion if its marker is already present (so re-runs
+# Idempotent: skips the insertion if the marker is already present (so re-runs
 # of the same build dir don't double-insert).
 
 set -euo pipefail
@@ -42,70 +25,33 @@ if [[ ! -f "$SRC" ]]; then
 fi
 
 if grep -q 'GGML_TYPE_TURBO2_0' "$SRC"; then
-    echo "==> $SRC already has TurboQuant cache types, skipping KV allow-list patch"
-else
-    echo "==> patching $SRC to allow turbo2/turbo3/turbo4 KV-cache types"
+    echo "==> $SRC already has TurboQuant cache types, skipping"
+    exit 0
+fi
 
-    # Insert the three TURBO entries right after the first `    GGML_TYPE_Q5_1,`
-    # line (the kv_cache_types[] allow-list). Using awk because the builder image
-    # does not ship python3, and GNU sed's multi-line `a\` quoting is awkward.
-    awk '
-        /^    GGML_TYPE_Q5_1,$/ && !done {
-            print
-            print "    // turboquant fork extras — added by patch-grpc-server.sh"
-            print "    GGML_TYPE_TURBO2_0,"
-            print "    GGML_TYPE_TURBO3_0,"
-            print "    GGML_TYPE_TURBO4_0,"
-            done = 1
-            next
+echo "==> patching $SRC to allow turbo2/turbo3/turbo4 KV-cache types"
+
+# Insert the three TURBO entries right after the first `    GGML_TYPE_Q5_1,`
+# line (the kv_cache_types[] allow-list). Using awk because the builder image
+# does not ship python3, and GNU sed's multi-line `a\` quoting is awkward.
+awk '
+    /^    GGML_TYPE_Q5_1,$/ && !done {
+        print
+        print "    // turboquant fork extras — added by patch-grpc-server.sh"
+        print "    GGML_TYPE_TURBO2_0,"
+        print "    GGML_TYPE_TURBO3_0,"
+        print "    GGML_TYPE_TURBO4_0,"
+        done = 1
+        next
+    }
+    { print }
+    END {
+        if (!done) {
+            print "patch-grpc-server.sh: anchor `    GGML_TYPE_Q5_1,` not found" > "/dev/stderr"
+            exit 1
         }
-        { print }
-        END {
-            if (!done) {
-                print "patch-grpc-server.sh: anchor `    GGML_TYPE_Q5_1,` not found" > "/dev/stderr"
-                exit 1
-            }
-        }
-    ' "$SRC" > "$SRC.tmp"
-    mv "$SRC.tmp" "$SRC"
+    }
+' "$SRC" > "$SRC.tmp"
+mv "$SRC.tmp" "$SRC"
 
-    echo "==> KV allow-list patch OK"
-fi
-
-if grep -q 'get_media_marker()' "$SRC"; then
-    echo "==> patching $SRC to replace get_media_marker() with legacy \"<__media__>\" literal"
-    # Only one call site today (ModelMetadata), but replace all occurrences to
-    # stay robust if upstream adds more. Use a temp file to avoid relying on
-    # sed -i portability (the builder image uses GNU sed, but keeping this
-    # consistent with the awk block above).
-    sed 's/get_media_marker()/"<__media__>"/g' "$SRC" > "$SRC.tmp"
-    mv "$SRC.tmp" "$SRC"
-    echo "==> get_media_marker() substitution OK"
-else
-    echo "==> $SRC has no get_media_marker() call, skipping media-marker patch"
-fi
-
-if grep -q 'params\.speculative\.draft\.\|params\.speculative\.ngram_simple\.' "$SRC"; then
-    echo "==> patching $SRC to revert common_params_speculative refs to pre-#22397 flat layout"
-    # Each substitution is the exact post-refactor path → legacy flat field.
-    # Order doesn't matter because the source paths are disjoint, but we keep
-    # the most-specific (mparams.path) first for readability.
-    sed -E \
-        -e 's/params\.speculative\.draft\.mparams\.path/params.speculative.mparams_dft.path/g' \
-        -e 's/params\.speculative\.draft\.n_max/params.speculative.n_max/g' \
-        -e 's/params\.speculative\.draft\.n_min/params.speculative.n_min/g' \
-        -e 's/params\.speculative\.draft\.p_min/params.speculative.p_min/g' \
-        -e 's/params\.speculative\.draft\.p_split/params.speculative.p_split/g' \
-        -e 's/params\.speculative\.draft\.n_gpu_layers/params.speculative.n_gpu_layers/g' \
-        -e 's/params\.speculative\.draft\.n_ctx/params.speculative.n_ctx/g' \
-        -e 's/params\.speculative\.ngram_simple\.size_n/params.speculative.ngram_size_n/g' \
-        -e 's/params\.speculative\.ngram_simple\.size_m/params.speculative.ngram_size_m/g' \
-        -e 's/params\.speculative\.ngram_simple\.min_hits/params.speculative.ngram_min_hits/g' \
-        "$SRC" > "$SRC.tmp"
-    mv "$SRC.tmp" "$SRC"
-    echo "==> speculative field rename OK"
-else
-    echo "==> $SRC has no post-#22397 speculative field refs, skipping spec rename patch"
-fi
-
-echo "==> all patches applied"
+echo "==> patched OK"

backend/cpp/turboquant/patches/0001-server-respect-the-ignore-eos-flag.patch

@@ -0,0 +1,83 @@
+From 660600081fb7b9b769ded5c805a2d39a419f0a0d Mon Sep 17 00:00:00 2001
+From: Yuri Khrustalev <ykhrustalev@users.noreply.github.com>
+Date: Wed, 8 Apr 2026 11:12:15 -0400
+Subject: [PATCH] server: respect the ignore eos flag (#21203)
+
+---
+ tools/server/server-context.cpp | 3 +++
+ tools/server/server-context.h   | 3 +++
+ tools/server/server-task.cpp    | 3 ++-
+ tools/server/server-task.h      | 1 +
+ 4 files changed, 9 insertions(+), 1 deletion(-)
+
+diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
+index 9d3ac538..b31981c5 100644
+--- a/tools/server/server-context.cpp
++++ b/tools/server/server-context.cpp
+@@ -3033,6 +3033,8 @@ server_context_meta server_context::get_meta() const {
+         /* fim_rep_token          */ llama_vocab_fim_rep(impl->vocab),
+         /* fim_sep_token          */ llama_vocab_fim_sep(impl->vocab),
+ 
++        /* logit_bias_eog         */ impl->params_base.sampling.logit_bias_eog,
++
+         /* model_vocab_type       */ llama_vocab_type(impl->vocab),
+         /* model_vocab_n_tokens   */ llama_vocab_n_tokens(impl->vocab),
+         /* model_n_ctx_train      */ llama_model_n_ctx_train(impl->model),
+@@ -3117,6 +3119,7 @@ std::unique_ptr<server_res_generator> server_routes::handle_completions_impl(
+                     ctx_server.vocab,
+                     params,
+                     meta->slot_n_ctx,
++                    meta->logit_bias_eog,
+                     data);
+             task.id_slot = json_value(data, "id_slot", -1);
+ 
+diff --git a/tools/server/server-context.h b/tools/server/server-context.h
+index d7ce8735..6ea9afc0 100644
+--- a/tools/server/server-context.h
++++ b/tools/server/server-context.h
+@@ -39,6 +39,9 @@ struct server_context_meta {
+     llama_token fim_rep_token;
+     llama_token fim_sep_token;
+ 
++    // sampling
++    std::vector<llama_logit_bias> logit_bias_eog;
++
+     // model meta
+     enum llama_vocab_type model_vocab_type;
+     int32_t model_vocab_n_tokens;
+diff --git a/tools/server/server-task.cpp b/tools/server/server-task.cpp
+index 4cc87bc5..856b3f0e 100644
+--- a/tools/server/server-task.cpp
++++ b/tools/server/server-task.cpp
+@@ -239,6 +239,7 @@ task_params server_task::params_from_json_cmpl(
+         const llama_vocab * vocab,
+         const common_params & params_base,
+         const int n_ctx_slot,
++        const std::vector<llama_logit_bias> & logit_bias_eog,
+         const json & data) {
+     task_params params;
+ 
+@@ -562,7 +563,7 @@ task_params server_task::params_from_json_cmpl(
+         if (params.sampling.ignore_eos) {
+             params.sampling.logit_bias.insert(
+                     params.sampling.logit_bias.end(),
+-                    defaults.sampling.logit_bias_eog.begin(), defaults.sampling.logit_bias_eog.end());
++                    logit_bias_eog.begin(), logit_bias_eog.end());
+         }
+     }
+ 
+diff --git a/tools/server/server-task.h b/tools/server/server-task.h
+index d855bf08..243e47a8 100644
+--- a/tools/server/server-task.h
++++ b/tools/server/server-task.h
+@@ -209,6 +209,7 @@ struct server_task {
+         const llama_vocab * vocab,
+         const common_params & params_base,
+         const int n_ctx_slot,
++        const std::vector<llama_logit_bias> & logit_bias_eog,
+         const json & data);
+ 
+     // utility function
+-- 
+2.43.0
+

backend/go/local-store/store.go

@@ -4,6 +4,7 @@ package main
 // It is meant to be used by the main executable that is the server for the specific backend type (falcon, gpt3, etc)
 import (
 	"container/heap"
+	"errors"
 	"fmt"
 	"math"
 	"slices"
@@ -99,16 +100,9 @@ func sortIntoKeySlicese(keys []*pb.StoresKey) [][]float32 {
 }
 
 func (s *Store) Load(opts *pb.ModelOptions) error {
-	// local-store is an in-memory vector store with no on-disk artefact to
-	// load — opts.Model is just a namespace identifier. The old `!= ""` guard
-	// rejected any non-empty model name with "not implemented", which broke
-	// callers that pass a namespace to isolate embedding spaces (face vs.
-	// voice biometrics both go through local-store but need distinct stores
-	// so ArcFace 512-D and ECAPA-TDNN 192-D don't collide). Namespace
-	// isolation is already handled upstream: ModelLoader spawns a fresh
-	// local-store process per (backend, model) tuple, so each namespace is
-	// its own Store{} instance. Nothing to do here beyond accepting the load.
-	_ = opts
+	if opts.Model != "" {
+		return errors.New("not implemented")
+	}
 	return nil
 }
 

backend/go/localvqe/.gitignore

@@ -1,7 +0,0 @@
-sources/
-build/
-package/
-liblocalvqe.so*
-libggml*.so*
-localvqe
-.localvqe-build.stamp

backend/go/localvqe/Makefile

@@ -1,98 +0,0 @@
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-
-GOCMD?=go
-GO_TAGS?=
-JOBS?=$(shell nproc --ignore=1)
-
-# LocalVQE upstream version pin. Bump to a specific commit when picking up
-# a new release; `main` works for development but is not reproducible.
-LOCALVQE_REPO?=https://github.com/localai-org/LocalVQE
-LOCALVQE_VERSION?=72bfb4c6
-
-# LocalVQE handles CPU feature selection internally (it ships the multiple
-# libggml-cpu-*.so variants and its loader picks the best one at runtime
-# via GGML_BACKEND_DL), so we build a single liblocalvqe.so + the per-CPU
-# ggml shared libs and let it sort itself out. No need for a wrapper
-# MODULE library or per-AVX backend variants here.
-
-CMAKE_ARGS+=-DLOCALVQE_BUILD_SHARED=ON
-CMAKE_ARGS+=-DGGML_BUILD_TESTS=OFF
-CMAKE_ARGS+=-DGGML_BUILD_EXAMPLES=OFF
-
-ifeq ($(NATIVE),false)
-	CMAKE_ARGS+=-DGGML_NATIVE=OFF
-endif
-
-# LocalVQE upstream supports CPU + Vulkan only. Other BUILD_TYPE values
-# fall through to the default CPU build — Vulkan is already as fast as the
-# specialised GPU paths would be on this 1.3 M-parameter model.
-ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DGGML_VULKAN=ON -DLOCALVQE_VULKAN=ON
-else ifeq ($(OS),Darwin)
-	CMAKE_ARGS+=-DGGML_METAL=OFF
-endif
-
-# --- Sources ---
-
-sources/LocalVQE:
-	mkdir -p sources/LocalVQE
-	cd sources/LocalVQE && \
-	git init && \
-	git remote add origin $(LOCALVQE_REPO) && \
-	git fetch origin && \
-	git checkout $(LOCALVQE_VERSION) && \
-	git submodule update --init --recursive --depth 1 --single-branch
-
-# --- Native build ---
-#
-# Drives cmake directly against the upstream LocalVQE/ggml CMakeLists.
-# Produces liblocalvqe.so plus the per-CPU libggml-cpu-*.so variants in
-# build/bin/, all of which we copy into the backend directory so package.sh
-# can pick them up. The `liblocalvqe.so` rule deliberately uses a sentinel
-# stamp file because Make's wildcard tracking would otherwise mis-decide
-# about freshness when SOVERSION symlinks are involved.
-
-LIB_SENTINEL=.localvqe-build.stamp
-
-$(LIB_SENTINEL): sources/LocalVQE
-	mkdir -p build && \
-	cd build && \
-	cmake ../sources/LocalVQE/ggml $(CMAKE_ARGS) -DCMAKE_BUILD_TYPE=Release && \
-	cmake --build . --config Release -j$(JOBS)
-	# Upstream's CPU build sets GGML_BACKEND_DL=ON + GGML_CPU_ALL_VARIANTS=ON,
-	# which produces multiple libggml-cpu-*.so files (SSE4.2 / AVX2 / AVX-512)
-	# that the loader picks at runtime. We must build every target — the
-	# default `--target localvqe_shared` drops these. CMAKE_LIBRARY_OUTPUT_DIRECTORY
-	# routes all of them into build/bin; copy them out next to the binary.
-	cp -P build/bin/liblocalvqe.so* . 2>/dev/null || cp -P build/liblocalvqe.so* .
-	cp -P build/bin/libggml*.so* . 2>/dev/null || true
-	touch $(LIB_SENTINEL)
-
-liblocalvqe.so: $(LIB_SENTINEL)
-
-# --- Go binary + packaging ---
-
-localvqe: main.go golocalvqe.go $(LIB_SENTINEL)
-	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o localvqe ./
-
-package: localvqe
-	bash package.sh
-
-build: package
-
-clean: purge
-	rm -rf liblocalvqe.so* libggml*.so* package sources/LocalVQE localvqe $(LIB_SENTINEL)
-
-purge:
-	rm -rf build
-
-test: localvqe
-	@echo "Running localvqe tests..."
-	bash test.sh
-	@echo "localvqe tests completed."
-
-all: localvqe package
-
-.PHONY: build package clean purge test all

backend/go/localvqe/golocalvqe.go

@@ -1,610 +0,0 @@
-package main
-
-import (
-	"encoding/binary"
-	"fmt"
-	"io"
-	"os"
-	"path/filepath"
-	"runtime"
-	"strconv"
-	"strings"
-	"unsafe"
-
-	"github.com/mudler/LocalAI/pkg/grpc/base"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	"github.com/mudler/xlog"
-)
-
-// localvqeSampleRate is the only sample rate currently supported by the
-// upstream LocalVQE model. We assert against it after Load() and reject
-// anything else with a clear error rather than letting the C side return
-// garbage.
-const localvqeSampleRate = 16000
-
-// Param map keys understood by LocalVQE. Keep these strings in sync with
-// schema.AudioTransformParam* (separate package — this is a standalone
-// backend module).
-const (
-	paramNoiseGate          = "noise_gate"
-	paramNoiseGateThreshold = "noise_gate_threshold_dbfs"
-)
-
-// Option keys read from ModelOptions.Options[] at Load() time. The backend
-// + device pair is forwarded to the upstream options builder; everything
-// else is consumed locally (noise gate state, etc.).
-const (
-	optionBackend = "backend"
-	optionDevice  = "device"
-)
-
-// purego-bound entry points from liblocalvqe.
-//
-// uintptr opaque handles model the C `uintptr_t ctx` / `uintptr_t opts`
-// tokens; we never dereference them on the Go side, just hand them
-// straight back to the library on every call. Construction always goes
-// through the options builder (CppOptionsNew + setters + CppNewWithOptions)
-// — the bare localvqe_new path doesn't expose backend / device selection.
-var (
-	CppOptionsNew           func() uintptr
-	CppOptionsFree          func(opts uintptr)
-	CppOptionsSetModelPath  func(opts uintptr, modelPath string) int32
-	CppOptionsSetBackend    func(opts uintptr, backend string) int32
-	CppOptionsSetDevice     func(opts uintptr, device int32) int32
-	CppNewWithOptions       func(opts uintptr) uintptr
-	CppFree                 func(ctx uintptr)
-	CppProcessF32           func(ctx uintptr, mic, ref uintptr, nSamples int32, out uintptr) int32
-	CppProcessS16           func(ctx uintptr, mic, ref uintptr, nSamples int32, out uintptr) int32
-	CppProcessFrameF32      func(ctx uintptr, mic, ref uintptr, hopSamples int32, out uintptr) int32
-	CppProcessFrameS16      func(ctx uintptr, mic, ref uintptr, hopSamples int32, out uintptr) int32
-	CppReset                func(ctx uintptr)
-	CppLastError            func(ctx uintptr) string
-	CppSampleRate           func(ctx uintptr) int32
-	CppHopLength            func(ctx uintptr) int32
-	CppFFTSize              func(ctx uintptr) int32
-	CppSetNoiseGate         func(ctx uintptr, enabled int32, thresholdDBFS float32) int32
-	CppGetNoiseGate         func(ctx uintptr, enabledOut, thresholdDBFSOut uintptr) int32
-)
-
-// LocalVQE speaks gRPC against LocalVQE's flat C ABI. The streaming
-// state is per-context, so we serialize calls through SingleThread —
-// concurrent streams would corrupt the overlap-add buffers.
-type LocalVQE struct {
-	base.SingleThread
-	ctx        uintptr // 0 when unloaded
-	sampleRate int
-	hopLength  int
-	fftSize    int
-
-	// modelRoot resolves relative paths from Options[].
-	modelRoot string
-
-	// Cached gate config so we can re-apply on each AudioTransform call
-	// without paying for a CGo round-trip every time. Sourced from
-	// Options[] at Load() time and overridable per-request via the
-	// gRPC params map.
-	gateEnabled bool
-	gateDbfs    float32
-
-	// Backend / device picked via Options[]. Empty backend leaves the
-	// default (CPU) selection to the upstream options builder.
-	backend string
-	device  int32
-}
-
-// parseOptions reads opts.Options[] for backend-specific tuning. Documented
-// keys: noise_gate=true|false and noise_gate_threshold_dbfs=<float> (also
-// settable per-request via AudioTransformRequest.params), plus backend=<name>
-// and device=<index> which route through the upstream options builder so
-// the user can force a non-default GGML backend (e.g. "Vulkan").
-func (v *LocalVQE) parseOptions(opts []string) {
-	for _, raw := range opts {
-		k, val, ok := strings.Cut(raw, "=")
-		if !ok {
-			k, val, ok = strings.Cut(raw, ":")
-			if !ok {
-				continue
-			}
-		}
-		key := strings.TrimSpace(strings.ToLower(k))
-		val = strings.TrimSpace(val)
-		switch key {
-		case paramNoiseGate:
-			if b, err := strconv.ParseBool(val); err == nil {
-				v.gateEnabled = b
-			}
-		case paramNoiseGateThreshold:
-			if f, err := strconv.ParseFloat(val, 32); err == nil {
-				v.gateDbfs = float32(f)
-			}
-		case optionBackend:
-			v.backend = val
-		case optionDevice:
-			if d, err := strconv.Atoi(val); err == nil && d >= 0 {
-				v.device = int32(d)
-			}
-		}
-	}
-}
-
-// newCtxWithOptions builds a context via the upstream options-builder so we
-// can pass backend / device in addition to the model path. Returns 0 on
-// failure; the caller logs/wraps the error since the C side has no
-// last-error channel for construction failures.
-func newCtxWithOptions(modelPath, backend string, device int32) uintptr {
-	o := CppOptionsNew()
-	if o == 0 {
-		return 0
-	}
-	defer CppOptionsFree(o)
-	if rc := CppOptionsSetModelPath(o, modelPath); rc != 0 {
-		return 0
-	}
-	if backend != "" {
-		if rc := CppOptionsSetBackend(o, backend); rc != 0 {
-			return 0
-		}
-	}
-	if device > 0 {
-		if rc := CppOptionsSetDevice(o, device); rc != 0 {
-			return 0
-		}
-	}
-	return CppNewWithOptions(o)
-}
-
-func (v *LocalVQE) Load(opts *pb.ModelOptions) error {
-	if opts.ModelFile == "" {
-		return fmt.Errorf("localvqe: ModelFile is required")
-	}
-
-	modelFile := opts.ModelFile
-	if !filepath.IsAbs(modelFile) && opts.ModelPath != "" {
-		modelFile = filepath.Join(opts.ModelPath, modelFile)
-	}
-	v.modelRoot = opts.ModelPath
-	if v.modelRoot == "" {
-		v.modelRoot = filepath.Dir(modelFile)
-	}
-
-	// Defaults — gate off, threshold at -45 dBFS as a reasonable starting
-	// point per the upstream localvqe_api.h documentation.
-	v.gateEnabled = false
-	v.gateDbfs = -45.0
-	v.parseOptions(opts.Options)
-
-	// localvqe_new reads GGML_NTHREADS at construction time; without it
-	// the C side falls back to single-threaded compute (~1× realtime
-	// instead of the documented ~9× on a multi-core CPU). Pass the
-	// model config's Threads through, defaulting to min(NumCPU, 4).
-	//
-	// LocalVQE is 1.3M parameters; per the upstream bench sweep 1–4
-	// threads is the sweet spot — beyond ~4 the per-frame budget gets
-	// dominated by sync overhead and p99 latency degrades. We cap at 4
-	// even when the user passes more so a globally-configured
-	// LOCALAI_THREADS=N tuned for a 70B LLM doesn't accidentally
-	// pessimise audio processing.
-	const localvqeMaxThreads = 4
-	threads := int(opts.Threads)
-	if threads <= 0 {
-		threads = runtime.NumCPU()
-	}
-	if threads > localvqeMaxThreads {
-		threads = localvqeMaxThreads
-	}
-	if threads < 1 {
-		threads = 1
-	}
-	if err := os.Setenv("GGML_NTHREADS", fmt.Sprintf("%d", threads)); err != nil {
-		return fmt.Errorf("localvqe: setenv GGML_NTHREADS: %w", err)
-	}
-
-	xlog.Info("[localvqe] loading model", "path", modelFile, "threads", threads, "backend", v.backend, "device", v.device, "noise_gate", v.gateEnabled, "threshold_dbfs", v.gateDbfs)
-
-	ctx := newCtxWithOptions(modelFile, v.backend, v.device)
-	if ctx == 0 {
-		return fmt.Errorf("localvqe: localvqe_new_with_options failed for %q (backend=%q device=%d)", modelFile, v.backend, v.device)
-	}
-	v.ctx = ctx
-
-	v.sampleRate = int(CppSampleRate(ctx))
-	v.hopLength = int(CppHopLength(ctx))
-	v.fftSize = int(CppFFTSize(ctx))
-
-	if v.sampleRate != localvqeSampleRate {
-		CppFree(ctx)
-		v.ctx = 0
-		return fmt.Errorf("localvqe: unsupported sample rate %d (only %d Hz is supported)", v.sampleRate, localvqeSampleRate)
-	}
-	if v.hopLength <= 0 || v.fftSize <= 0 {
-		CppFree(ctx)
-		v.ctx = 0
-		return fmt.Errorf("localvqe: model reports invalid hop=%d fft=%d", v.hopLength, v.fftSize)
-	}
-
-	if v.gateEnabled {
-		if rc := CppSetNoiseGate(ctx, 1, v.gateDbfs); rc != 0 {
-			err := fmt.Errorf("localvqe: localvqe_set_noise_gate failed (rc=%d): %s", rc, CppLastError(ctx))
-			CppFree(ctx)
-			v.ctx = 0
-			return err
-		}
-	}
-
-	return nil
-}
-
-func (v *LocalVQE) Free() error {
-	if v.ctx != 0 {
-		CppFree(v.ctx)
-		v.ctx = 0
-	}
-	return nil
-}
-
-// applyParams forwards backend-specific tuning to the C side per call.
-func (v *LocalVQE) applyParams(params map[string]string) error {
-	if len(params) == 0 {
-		return nil
-	}
-	enabled := v.gateEnabled
-	threshold := v.gateDbfs
-	updated := false
-
-	if val, ok := params[paramNoiseGate]; ok {
-		if b, err := strconv.ParseBool(val); err == nil {
-			enabled = b
-			updated = true
-		}
-	}
-	if val, ok := params[paramNoiseGateThreshold]; ok {
-		if f, err := strconv.ParseFloat(val, 32); err == nil {
-			threshold = float32(f)
-			updated = true
-		}
-	}
-	if !updated {
-		return nil
-	}
-
-	gateOn := int32(0)
-	if enabled {
-		gateOn = 1
-	}
-	if rc := CppSetNoiseGate(v.ctx, gateOn, threshold); rc != 0 {
-		return fmt.Errorf("localvqe_set_noise_gate failed (rc=%d): %s", rc, CppLastError(v.ctx))
-	}
-	v.gateEnabled = enabled
-	v.gateDbfs = threshold
-	return nil
-}
-
-func (v *LocalVQE) AudioTransform(req *pb.AudioTransformRequest) (*pb.AudioTransformResult, error) {
-	if v.ctx == 0 {
-		return nil, fmt.Errorf("localvqe: no model loaded")
-	}
-	if req.AudioPath == "" || req.Dst == "" {
-		return nil, fmt.Errorf("localvqe: audio_path and dst are required")
-	}
-
-	if err := v.applyParams(req.Params); err != nil {
-		return nil, err
-	}
-
-	mic, micRate, err := readMonoWAVf32(req.AudioPath)
-	if err != nil {
-		return nil, fmt.Errorf("read audio: %w", err)
-	}
-	if micRate != v.sampleRate {
-		return nil, fmt.Errorf("localvqe: audio sample rate %d != model %d (resample upstream)", micRate, v.sampleRate)
-	}
-
-	refProvided := req.ReferencePath != ""
-	var ref []float32
-	if refProvided {
-		var refRate int
-		ref, refRate, err = readMonoWAVf32(req.ReferencePath)
-		if err != nil {
-			return nil, fmt.Errorf("read reference: %w", err)
-		}
-		if refRate != v.sampleRate {
-			return nil, fmt.Errorf("localvqe: reference sample rate %d != model %d", refRate, v.sampleRate)
-		}
-		// Length-mismatch policy: zero-pad a short reference (silence past
-		// the mic's tail), truncate a long one (the trailing reference
-		// can't have leaked into a mic that wasn't recording yet).
-		switch {
-		case len(ref) < len(mic):
-			padded := make([]float32, len(mic))
-			copy(padded, ref)
-			ref = padded
-		case len(ref) > len(mic):
-			ref = ref[:len(mic)]
-		}
-	} else {
-		ref = make([]float32, len(mic))
-	}
-
-	if len(mic) < v.fftSize {
-		return nil, fmt.Errorf("localvqe: audio too short (%d samples, need ≥ %d)", len(mic), v.fftSize)
-	}
-
-	out := make([]float32, len(mic))
-	rc := CppProcessF32(v.ctx,
-		uintptr(unsafe.Pointer(&mic[0])),
-		uintptr(unsafe.Pointer(&ref[0])),
-		int32(len(mic)),
-		uintptr(unsafe.Pointer(&out[0])))
-	if rc != 0 {
-		return nil, fmt.Errorf("localvqe_process_f32 failed (rc=%d): %s", rc, CppLastError(v.ctx))
-	}
-
-	if err := writeMonoWAVf32(req.Dst, out, v.sampleRate); err != nil {
-		return nil, fmt.Errorf("write output: %w", err)
-	}
-
-	return &pb.AudioTransformResult{
-		Dst:               req.Dst,
-		SampleRate:        int32(v.sampleRate),
-		Samples:           int32(len(out)),
-		ReferenceProvided: refProvided,
-	}, nil
-}
-
-// AudioTransformStream runs the bidirectional streaming path. The first
-// inbound message MUST be a Config; subsequent messages MUST be Frames.
-// A second Config mid-stream resets the streaming state.
-func (v *LocalVQE) AudioTransformStream(in <-chan *pb.AudioTransformFrameRequest, out chan<- *pb.AudioTransformFrameResponse) error {
-	defer close(out)
-
-	if v.ctx == 0 {
-		return fmt.Errorf("localvqe: no model loaded")
-	}
-
-	first, ok := <-in
-	if !ok {
-		return nil
-	}
-	cfg := first.GetConfig()
-	if cfg == nil {
-		return fmt.Errorf("localvqe: first stream message must be a Config")
-	}
-	if err := v.applyStreamConfig(cfg); err != nil {
-		return err
-	}
-
-	hop := v.hopLength
-	if cfg.FrameSamples != 0 && int(cfg.FrameSamples) != hop {
-		return fmt.Errorf("localvqe: frame_samples=%d != hop_length=%d", cfg.FrameSamples, hop)
-	}
-
-	// Pre-allocated scratch buffers for the C-side process call. The
-	// per-frame output []byte stays a fresh allocation: the response
-	// channel is buffered, so reusing one backing array would race with
-	// the gRPC send goroutine flushing prior queued frames.
-	micF32 := make([]float32, hop)
-	refF32 := make([]float32, hop)
-	outF32 := make([]float32, hop)
-	micS16 := make([]int16, hop)
-	refS16 := make([]int16, hop)
-	outS16 := make([]int16, hop)
-
-	useS16 := cfg.SampleFormat == pb.AudioTransformStreamConfig_S16_LE
-	frameSize := hop * 4
-	if useS16 {
-		frameSize = hop * 2
-	}
-
-	frameIndex := int64(0)
-	for req := range in {
-		switch payload := req.Payload.(type) {
-		case *pb.AudioTransformFrameRequest_Config:
-			if err := v.applyStreamConfig(payload.Config); err != nil {
-				return err
-			}
-			if payload.Config.Reset_ {
-				CppReset(v.ctx)
-				frameIndex = 0
-			}
-			continue
-		case *pb.AudioTransformFrameRequest_Frame:
-			if len(payload.Frame.AudioPcm) != frameSize {
-				return fmt.Errorf("localvqe: frame audio bytes=%d expected=%d", len(payload.Frame.AudioPcm), frameSize)
-			}
-			refBuf := payload.Frame.ReferencePcm
-			if len(refBuf) != 0 && len(refBuf) != frameSize {
-				return fmt.Errorf("localvqe: frame reference bytes=%d expected=%d (or 0)", len(refBuf), frameSize)
-			}
-
-			var outBytes []byte
-			if useS16 {
-				if err := decodeS16LE(payload.Frame.AudioPcm, micS16); err != nil {
-					return err
-				}
-				if len(refBuf) > 0 {
-					if err := decodeS16LE(refBuf, refS16); err != nil {
-						return err
-					}
-				} else {
-					zeroS16(refS16)
-				}
-				rc := CppProcessFrameS16(v.ctx,
-					uintptr(unsafe.Pointer(&micS16[0])),
-					uintptr(unsafe.Pointer(&refS16[0])),
-					int32(hop),
-					uintptr(unsafe.Pointer(&outS16[0])))
-				if rc != 0 {
-					return fmt.Errorf("localvqe_process_frame_s16 (rc=%d): %s", rc, CppLastError(v.ctx))
-				}
-				outBytes = make([]byte, hop*2)
-				encodeS16LE(outS16, outBytes)
-			} else {
-				if err := decodeF32LE(payload.Frame.AudioPcm, micF32); err != nil {
-					return err
-				}
-				if len(refBuf) > 0 {
-					if err := decodeF32LE(refBuf, refF32); err != nil {
-						return err
-					}
-				} else {
-					zeroF32(refF32)
-				}
-				rc := CppProcessFrameF32(v.ctx,
-					uintptr(unsafe.Pointer(&micF32[0])),
-					uintptr(unsafe.Pointer(&refF32[0])),
-					int32(hop),
-					uintptr(unsafe.Pointer(&outF32[0])))
-				if rc != 0 {
-					return fmt.Errorf("localvqe_process_frame_f32 (rc=%d): %s", rc, CppLastError(v.ctx))
-				}
-				outBytes = make([]byte, hop*4)
-				encodeF32LE(outF32, outBytes)
-			}
-			out <- &pb.AudioTransformFrameResponse{Pcm: outBytes, FrameIndex: frameIndex}
-			frameIndex++
-		default:
-			return fmt.Errorf("localvqe: unexpected stream payload %T", payload)
-		}
-	}
-	return nil
-}
-
-func zeroS16(s []int16) {
-	for i := range s {
-		s[i] = 0
-	}
-}
-
-func zeroF32(s []float32) {
-	for i := range s {
-		s[i] = 0
-	}
-}
-
-func (v *LocalVQE) applyStreamConfig(cfg *pb.AudioTransformStreamConfig) error {
-	if cfg.SampleRate != 0 && int(cfg.SampleRate) != v.sampleRate {
-		return fmt.Errorf("localvqe: sample_rate=%d != model %d", cfg.SampleRate, v.sampleRate)
-	}
-	return v.applyParams(cfg.Params)
-}
-
-// ---- WAV I/O ----------------------------------------------------------
-//
-// Minimal mono PCM WAV reader/writer. Only handles the subset LocalVQE
-// cares about (mono, 16-bit signed, no extensible chunks). For broader
-// audio support the HTTP layer's `audio.NormalizeAudioFile` already
-// converts arbitrary input to a canonical WAV before we see it; this
-// reader just decodes the canonical shape.
-
-func readMonoWAVf32(path string) ([]float32, int, error) {
-	f, err := os.Open(path)
-	if err != nil {
-		return nil, 0, err
-	}
-	defer func() { _ = f.Close() }()
-	header := make([]byte, 44)
-	if _, err := io.ReadFull(f, header); err != nil {
-		return nil, 0, err
-	}
-	if string(header[0:4]) != "RIFF" || string(header[8:12]) != "WAVE" {
-		return nil, 0, fmt.Errorf("not a WAV file")
-	}
-	channels := binary.LittleEndian.Uint16(header[22:24])
-	sampleRate := binary.LittleEndian.Uint32(header[24:28])
-	bitsPerSample := binary.LittleEndian.Uint16(header[34:36])
-
-	if channels != 1 {
-		return nil, 0, fmt.Errorf("only mono WAV supported (got %d channels)", channels)
-	}
-	if bitsPerSample != 16 {
-		return nil, 0, fmt.Errorf("only 16-bit PCM supported (got %d bits)", bitsPerSample)
-	}
-
-	rest, err := io.ReadAll(f)
-	if err != nil {
-		return nil, 0, err
-	}
-	n := len(rest) / 2
-	out := make([]float32, n)
-	for i := 0; i < n; i++ {
-		s := int16(binary.LittleEndian.Uint16(rest[i*2 : i*2+2]))
-		out[i] = float32(s) / 32768.0
-	}
-	return out, int(sampleRate), nil
-}
-
-func writeMonoWAVf32(path string, samples []float32, sampleRate int) error {
-	f, err := os.Create(path)
-	if err != nil {
-		return err
-	}
-	defer func() { _ = f.Close() }()
-
-	dataLen := uint32(len(samples) * 2)
-	header := make([]byte, 44)
-	copy(header[0:4], []byte("RIFF"))
-	binary.LittleEndian.PutUint32(header[4:8], 36+dataLen)
-	copy(header[8:12], []byte("WAVE"))
-	copy(header[12:16], []byte("fmt "))
-	binary.LittleEndian.PutUint32(header[16:20], 16)        // fmt chunk size
-	binary.LittleEndian.PutUint16(header[20:22], 1)         // PCM
-	binary.LittleEndian.PutUint16(header[22:24], 1)         // mono
-	binary.LittleEndian.PutUint32(header[24:28], uint32(sampleRate))
-	binary.LittleEndian.PutUint32(header[28:32], uint32(sampleRate*2)) // byte rate
-	binary.LittleEndian.PutUint16(header[32:34], 2)         // block align
-	binary.LittleEndian.PutUint16(header[34:36], 16)        // bits per sample
-	copy(header[36:40], []byte("data"))
-	binary.LittleEndian.PutUint32(header[40:44], dataLen)
-	if _, err := f.Write(header); err != nil {
-		return err
-	}
-
-	body := make([]byte, len(samples)*2)
-	for i, s := range samples {
-		clamped := s * 32768.0
-		if clamped > 32767 {
-			clamped = 32767
-		} else if clamped < -32768 {
-			clamped = -32768
-		}
-		binary.LittleEndian.PutUint16(body[i*2:i*2+2], uint16(int16(clamped)))
-	}
-	_, err = f.Write(body)
-	return err
-}
-
-// ---- PCM endec helpers ------------------------------------------------
-
-func decodeS16LE(buf []byte, out []int16) error {
-	if len(buf) != len(out)*2 {
-		return fmt.Errorf("decodeS16LE: buf=%d out=%d", len(buf), len(out))
-	}
-	for i := range out {
-		out[i] = int16(binary.LittleEndian.Uint16(buf[i*2 : i*2+2]))
-	}
-	return nil
-}
-
-func encodeS16LE(in []int16, out []byte) {
-	for i, s := range in {
-		binary.LittleEndian.PutUint16(out[i*2:i*2+2], uint16(s))
-	}
-}
-
-func decodeF32LE(buf []byte, out []float32) error {
-	if len(buf) != len(out)*4 {
-		return fmt.Errorf("decodeF32LE: buf=%d out=%d", len(buf), len(out))
-	}
-	for i := range out {
-		bits := binary.LittleEndian.Uint32(buf[i*4 : i*4+4])
-		out[i] = *(*float32)(unsafe.Pointer(&bits))
-	}
-	return nil
-}
-
-func encodeF32LE(in []float32, out []byte) {
-	for i, s := range in {
-		bits := *(*uint32)(unsafe.Pointer(&s))
-		binary.LittleEndian.PutUint32(out[i*4:i*4+4], bits)
-	}
-}

backend/go/localvqe/localvqe_test.go

@@ -1,120 +0,0 @@
-package main
-
-import (
-	"os"
-	"testing"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func TestLocalVQE(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "LocalVQE-cpp Backend Suite")
-}
-
-// modelPathOrSkip returns the LocalVQE GGUF path or Skip()s the current
-// spec when LOCALVQE_MODEL_PATH is unset / unreadable.
-func modelPathOrSkip() string {
-	path := os.Getenv("LOCALVQE_MODEL_PATH")
-	if path == "" {
-		Skip("LOCALVQE_MODEL_PATH not set, skipping model-dependent specs")
-	}
-	if _, err := os.Stat(path); err != nil {
-		Skip("LOCALVQE_MODEL_PATH unreadable: " + err.Error())
-	}
-	return path
-}
-
-var _ = Describe("LocalVQE-cpp", func() {
-	Context("backend semantics (no purego load needed)", func() {
-		It("is locking - the engine has per-context streaming state", func() {
-			Expect((&LocalVQE{}).Locking()).To(BeTrue())
-		})
-
-		It("rejects Load with empty ModelFile", func() {
-			err := (&LocalVQE{}).Load(&pb.ModelOptions{})
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("ModelFile"))
-		})
-
-		It("rejects AudioTransform without a loaded model", func() {
-			_, err := (&LocalVQE{}).AudioTransform(&pb.AudioTransformRequest{
-				AudioPath: "/tmp/audio.wav",
-				Dst:       "/tmp/out.wav",
-			})
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("no model loaded"))
-		})
-
-		It("closes the output channel and errors on AudioTransformStream without a loaded model", func() {
-			in := make(chan *pb.AudioTransformFrameRequest, 1)
-			out := make(chan *pb.AudioTransformFrameResponse, 1)
-			close(in)
-			err := (&LocalVQE{}).AudioTransformStream(in, out)
-			Expect(err).To(HaveOccurred())
-			_, ok := <-out
-			Expect(ok).To(BeFalse(), "AudioTransformStream must close results channel even on error")
-		})
-
-		It("rejects AudioTransform with empty audio_path", func() {
-			v := &LocalVQE{ctx: 1, sampleRate: localvqeSampleRate, hopLength: 256, fftSize: 512}
-			_, err := v.AudioTransform(&pb.AudioTransformRequest{Dst: "/tmp/out.wav"})
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("audio_path"))
-		})
-	})
-
-	Context("parseOptions", func() {
-		It("reads noise_gate=true (=)", func() {
-			v := &LocalVQE{}
-			v.parseOptions([]string{"noise_gate=true"})
-			Expect(v.gateEnabled).To(BeTrue())
-		})
-
-		It("reads noise_gate_threshold_dbfs=-50 (:)", func() {
-			v := &LocalVQE{}
-			v.parseOptions([]string{"noise_gate_threshold_dbfs:-50"})
-			Expect(v.gateDbfs).To(BeNumerically("==", -50.0))
-		})
-
-		It("ignores unknown keys without error", func() {
-			v := &LocalVQE{}
-			v.parseOptions([]string{"unknown=value", "another:thing"})
-			Expect(v.gateEnabled).To(BeFalse())
-		})
-
-		It("is case-insensitive on keys", func() {
-			v := &LocalVQE{}
-			v.parseOptions([]string{"NOISE_GATE=true"})
-			Expect(v.gateEnabled).To(BeTrue())
-		})
-	})
-
-	Context("model-gated integration (LOCALVQE_MODEL_PATH)", func() {
-		It("load + sample rate + hop + fft", func() {
-			path := modelPathOrSkip()
-			v := &LocalVQE{}
-			Expect(v.Load(&pb.ModelOptions{ModelFile: path})).To(Succeed())
-			defer func() { _ = v.Free() }()
-			Expect(v.sampleRate).To(Equal(localvqeSampleRate))
-			Expect(v.hopLength).To(Equal(256))
-			Expect(v.fftSize).To(Equal(512))
-		})
-
-		It("sets reference_provided correctly", func() {
-			// This spec is best exercised against a real model + WAV
-			// fixture, which the e2e harness drives separately. Here
-			// we just assert the expectation when ref is empty.
-			path := modelPathOrSkip()
-			v := &LocalVQE{}
-			Expect(v.Load(&pb.ModelOptions{ModelFile: path})).To(Succeed())
-			defer func() { _ = v.Free() }()
-			// Synthetic input; the C side handles a constant-zero ref
-			// just fine. Skip writing the WAV: this spec is a smoke
-			// check — the SNR-improvement assertion lives in the e2e
-			// harness where we have a real fixture.
-		})
-	})
-})

backend/go/localvqe/main.go

@@ -1,62 +0,0 @@
-package main
-
-// Started internally by LocalAI - one gRPC server per loaded model.
-import (
-	"flag"
-	"os"
-
-	"github.com/ebitengine/purego"
-	grpc "github.com/mudler/LocalAI/pkg/grpc"
-)
-
-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
-)
-
-type LibFuncs struct {
-	FuncPtr any
-	Name    string
-}
-
-func main() {
-	libName := os.Getenv("LOCALVQE_LIBRARY")
-	if libName == "" {
-		libName = "./liblocalvqe.so"
-	}
-
-	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-	if err != nil {
-		panic(err)
-	}
-
-	libFuncs := []LibFuncs{
-		{&CppOptionsNew, "localvqe_options_new"},
-		{&CppOptionsFree, "localvqe_options_free"},
-		{&CppOptionsSetModelPath, "localvqe_options_set_model_path"},
-		{&CppOptionsSetBackend, "localvqe_options_set_backend"},
-		{&CppOptionsSetDevice, "localvqe_options_set_device"},
-		{&CppNewWithOptions, "localvqe_new_with_options"},
-		{&CppFree, "localvqe_free"},
-		{&CppProcessF32, "localvqe_process_f32"},
-		{&CppProcessS16, "localvqe_process_s16"},
-		{&CppProcessFrameF32, "localvqe_process_frame_f32"},
-		{&CppProcessFrameS16, "localvqe_process_frame_s16"},
-		{&CppReset, "localvqe_reset"},
-		{&CppLastError, "localvqe_last_error"},
-		{&CppSampleRate, "localvqe_sample_rate"},
-		{&CppHopLength, "localvqe_hop_length"},
-		{&CppFFTSize, "localvqe_fft_size"},
-		{&CppSetNoiseGate, "localvqe_set_noise_gate"},
-		{&CppGetNoiseGate, "localvqe_get_noise_gate"},
-	}
-
-	for _, lf := range libFuncs {
-		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
-	}
-
-	flag.Parse()
-
-	if err := grpc.StartServer(*addr, &LocalVQE{}); err != nil {
-		panic(err)
-	}
-}

backend/go/localvqe/package.sh

@@ -1,61 +0,0 @@
-#!/bin/bash
-
-# Bundle the localvqe binary, the upstream liblocalvqe.so + the per-CPU
-# libggml-*.so runtime variants, the run wrapper, and the runtime libs the
-# binary depends on so the package is self-contained.
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-mkdir -p $CURDIR/package/lib
-
-cp -avf $CURDIR/localvqe $CURDIR/package/
-# liblocalvqe.so* (with SOVERSION symlinks) and the libggml-*.so runtime
-# variants — LocalVQE picks the matching CPU variant at load time.
-cp -P $CURDIR/liblocalvqe.so* $CURDIR/package/ 2>/dev/null || true
-cp -P $CURDIR/libggml*.so* $CURDIR/package/ 2>/dev/null || true
-cp -fv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ $(uname -s) = "Darwin" ]; then
-    echo "Detected Darwin"
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/go/localvqe/run.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-set -ex
-
-CURDIR=$(dirname "$(realpath $0)")
-
-# LocalVQE's runtime CPU-variant loader (ggml_backend_load_all) searches
-# get_executable_path() and current_path() — the second one is what saves us
-# when /proc/self/exe resolves to lib/ld.so under the bundled-loader path.
-# So we cd into $CURDIR (where all the libggml-cpu-*.so files live) before
-# exec'ing the binary.
-cd "$CURDIR"
-
-export LD_LIBRARY_PATH=$CURDIR:$CURDIR/lib:$LD_LIBRARY_PATH
-export LOCALVQE_LIBRARY=$CURDIR/liblocalvqe.so
-
-if [ -f $CURDIR/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using library: $LOCALVQE_LIBRARY"
-	exec $CURDIR/lib/ld.so $CURDIR/localvqe "$@"
-fi
-
-echo "Using library: $LOCALVQE_LIBRARY"
-exec $CURDIR/localvqe "$@"

backend/go/localvqe/test.sh

@@ -1,14 +0,0 @@
-#!/bin/bash
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-cd "$CURDIR"
-
-# The Go test suite uses a built localvqe binary for end-to-end
-# specs. It also opportunistically runs the integration tests when
-# LOCALVQE_MODEL_PATH points at a real GGUF; otherwise those specs Skip().
-
-export LOCALVQE_BINARY="${LOCALVQE_BINARY:-$CURDIR/localvqe}"
-export LD_LIBRARY_PATH="$CURDIR:$LD_LIBRARY_PATH"
-
-go test -v ./...

backend/go/sam3-cpp/CMakeLists.txt

@@ -10,7 +10,7 @@ set(SAM3_BUILD_TESTS OFF CACHE BOOL "Disable sam3.cpp tests" FORCE)
 
 add_subdirectory(./sources/sam3.cpp)
 
-add_library(gosam3 MODULE cpp/gosam3.cpp)
+add_library(gosam3 MODULE gosam3.cpp)
 target_link_libraries(gosam3 PRIVATE sam3 ggml)
 
 if(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9.0)

backend/go/sam3-cpp/Makefile

@@ -111,7 +111,7 @@ libgosam3-fallback.so: sources/sam3.cpp
 	SO_TARGET=libgosam3-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgosam3-custom
 	rm -rfv build*
 
-libgosam3-custom: CMakeLists.txt cpp/gosam3.cpp cpp/gosam3.h
+libgosam3-custom: CMakeLists.txt gosam3.cpp gosam3.h
 	mkdir -p build-$(SO_TARGET) && \
 	cd build-$(SO_TARGET) && \
 	cmake .. $(CMAKE_ARGS) && \

backend/go/sherpa-onnx/.gitignore

@@ -1,11 +0,0 @@
-.cache/
-sources/
-build*/
-package/
-backend-assets/
-sherpa-onnx
-*.so
-compile_commands.json
-sherpa-onnx-whisper-*
-vits-ljs/
-streaming-zipformer-en/

backend/go/sherpa-onnx/Makefile

@@ -1,120 +0,0 @@
-CURRENT_DIR=$(abspath ./)
-GOCMD=go
-
-ONNX_VERSION?=1.24.4
-# v1.12.39 — includes upstream's onnxruntime 1.24.4 bump (#3501). Earlier
-# pinned commits only support onnxruntime 1.23.2, which has no CUDA 13
-# pre-built tarball, blocking the -gpu-nvidia-cuda-13 build matrix entry.
-SHERPA_COMMIT?=7288d15e3e31a7bd589b2ba88828d521e7a6b140
-ONNX_ARCH?=x64
-ONNX_OS?=linux
-
-ifneq (,$(findstring aarch64,$(shell uname -m)))
-	ONNX_ARCH=aarch64
-endif
-
-ifeq ($(OS),Darwin)
-	ONNX_OS=osx
-	ifneq (,$(findstring aarch64,$(shell uname -m)))
-		ONNX_ARCH=arm64
-	else ifneq (,$(findstring arm64,$(shell uname -m)))
-		ONNX_ARCH=arm64
-	else
-		ONNX_ARCH=x86_64
-	endif
-endif
-
-# Upstream onnxruntime ships CUDA 12 and CUDA 13 variants under different
-# names: -gpu-<ver>.tgz for CUDA 12, -gpu_cuda13-<ver>.tgz for CUDA 13
-# (note underscore vs dash). CUDA 13 tarballs only exist from 1.24.x onward.
-ifeq ($(BUILD_TYPE),cublas)
-	SHERPA_GPU=ON
-	ONNX_PROVIDER=cuda
-	ifeq ($(CUDA_MAJOR_VERSION),13)
-		ONNX_VARIANT=-gpu_cuda13
-	else
-		ONNX_VARIANT=-gpu
-	endif
-else
-	ONNX_VARIANT=
-	SHERPA_GPU=OFF
-	ONNX_PROVIDER=cpu
-endif
-
-JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
-
-sources/onnxruntime:
-	mkdir -p sources/onnxruntime
-	curl -L https://github.com/microsoft/onnxruntime/releases/download/v$(ONNX_VERSION)/onnxruntime-$(ONNX_OS)-$(ONNX_ARCH)$(ONNX_VARIANT)-$(ONNX_VERSION).tgz \
-	  -o sources/onnxruntime/onnxruntime.tgz
-	cd sources/onnxruntime && tar -xf onnxruntime.tgz --strip-components=1 && rm onnxruntime.tgz
-
-sources/sherpa-onnx: sources/onnxruntime
-	git clone https://github.com/k2-fsa/sherpa-onnx.git sources/sherpa-onnx
-	cd sources/sherpa-onnx && git checkout $(SHERPA_COMMIT)
-	mkdir -p sources/sherpa-onnx/build
-	# sherpa-onnx's cmake detects a pre-installed onnxruntime via the
-	# SHERPA_ONNXRUNTIME_{INCLUDE,LIB}_DIR env vars (not via -D flags).
-	# Point them at our locally-downloaded Microsoft tarball — without
-	# this, sherpa-onnx falls through to download_onnxruntime() which
-	# fetches from csukuangfj/onnxruntime-libs. For the GPU 1.24.4
-	# build that release mirror publishes `-patched.zip` instead of the
-	# expected `.tgz`, so the download 404s and the build fails.
-	cd sources/sherpa-onnx/build && \
-	SHERPA_ONNXRUNTIME_INCLUDE_DIR=$(CURRENT_DIR)/sources/onnxruntime/include \
-	SHERPA_ONNXRUNTIME_LIB_DIR=$(CURRENT_DIR)/sources/onnxruntime/lib \
-	cmake \
-	  -DCMAKE_BUILD_TYPE=Release \
-	  -DCMAKE_C_FLAGS="-Wno-error=format-security" \
-	  -DCMAKE_CXX_FLAGS="-Wno-error=format-security" \
-	  -DSHERPA_ONNX_ENABLE_GPU=$(SHERPA_GPU) \
-	  -DSHERPA_ONNX_ENABLE_TTS=ON \
-	  -DSHERPA_ONNX_ENABLE_BINARY=OFF \
-	  -DSHERPA_ONNX_ENABLE_PYTHON=OFF \
-	  -DSHERPA_ONNX_ENABLE_TESTS=OFF \
-	  -DSHERPA_ONNX_ENABLE_C_API=ON \
-	  -DBUILD_SHARED_LIBS=ON \
-	  -DSHERPA_ONNX_USE_PRE_INSTALLED_ONNXRUNTIME_IF_AVAILABLE=ON \
-	  ..
-	cd sources/sherpa-onnx/build && make -j$(JOBS)
-
-backend-assets/lib: sources/sherpa-onnx sources/onnxruntime
-	mkdir -p backend-assets/lib
-	cp -rfLv sources/onnxruntime/lib/* backend-assets/lib/
-	cp -rfLv sources/sherpa-onnx/build/lib/*.so* backend-assets/lib/ 2>/dev/null || true
-	cp -rfLv sources/sherpa-onnx/build/lib/*.dylib backend-assets/lib/ 2>/dev/null || true
-
-# libsherpa-shim wraps sherpa-onnx's nested config structs and TTS
-# callback plumbing behind a purego-friendly API: opaque handles plus
-# fixed-signature setters/getters/trampoline. Plain C compile — no cgo.
-SHIM_EXT=so
-ifeq ($(OS),Darwin)
-	SHIM_EXT=dylib
-endif
-
-backend-assets/lib/libsherpa-shim.$(SHIM_EXT): csrc/shim.c csrc/shim.h backend-assets/lib
-	$(CC) -shared -fPIC -O2 \
-	  -I$(CURRENT_DIR)/sources/sherpa-onnx/sherpa-onnx/c-api \
-	  -o $@ csrc/shim.c \
-	  -L$(CURRENT_DIR)/backend-assets/lib \
-	  -lsherpa-onnx-c-api \
-	  -Wl,-rpath,'$$ORIGIN'
-
-sherpa-onnx: backend-assets/lib backend-assets/lib/libsherpa-shim.$(SHIM_EXT)
-	CGO_ENABLED=0 $(GOCMD) build \
-	  -ldflags "$(LD_FLAGS) -X main.onnxProvider=$(ONNX_PROVIDER)" \
-	  -tags "$(GO_TAGS)" -o sherpa-onnx ./
-
-package:
-	bash package.sh
-
-build: sherpa-onnx package
-
-clean:
-	rm -rf sherpa-onnx sources/ backend-assets/ package/ vits-ljs/ sherpa-onnx-whisper-*/
-
-test: sherpa-onnx
-	LD_LIBRARY_PATH=$(CURRENT_DIR)/backend-assets/lib \
-	bash test.sh
-
-.PHONY: build package clean test

backend/go/sherpa-onnx/backend_test.go

@@ -1,170 +0,0 @@
-package main
-
-import (
-	"context"
-	"os"
-	"path/filepath"
-	"testing"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func TestSherpaBackend(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "Sherpa-ONNX Backend Suite")
-}
-
-// Load libsherpa-shim + libsherpa-onnx-c-api via purego before any spec
-// runs — otherwise any Load/TTS/VAD/AudioTranscription call hits a nil
-// function pointer. LD_LIBRARY_PATH must contain the directory holding
-// both .so files; test.sh sets this.
-var _ = BeforeSuite(func() {
-	Expect(loadSherpaLibs()).To(Succeed())
-})
-
-var _ = Describe("Sherpa-ONNX", func() {
-	Context("lifecycle", func() {
-		It("is locking (C API is not thread safe)", func() {
-			Expect((&SherpaBackend{}).Locking()).To(BeTrue())
-		})
-
-		It("errors loading a non-existent model", func() {
-			tmpDir, err := os.MkdirTemp("", "sherpa-test-nonexistent")
-			Expect(err).ToNot(HaveOccurred())
-			defer os.RemoveAll(tmpDir)
-
-			err = (&SherpaBackend{}).Load(&pb.ModelOptions{
-				ModelFile: filepath.Join(tmpDir, "non-existent-model.onnx"),
-			})
-			Expect(err).To(HaveOccurred())
-		})
-
-		It("errors loading a non-existent ASR model", func() {
-			tmpDir, err := os.MkdirTemp("", "sherpa-test-asr")
-			Expect(err).ToNot(HaveOccurred())
-			defer os.RemoveAll(tmpDir)
-
-			err = (&SherpaBackend{}).Load(&pb.ModelOptions{
-				ModelFile: filepath.Join(tmpDir, "model.onnx"),
-				Type:      "asr",
-			})
-			Expect(err).To(HaveOccurred())
-		})
-
-		It("dispatches Load by Type", func() {
-			tmpDir, err := os.MkdirTemp("", "sherpa-test-dispatch")
-			Expect(err).ToNot(HaveOccurred())
-			defer os.RemoveAll(tmpDir)
-
-			modelFile := filepath.Join(tmpDir, "model.onnx")
-			for _, typ := range []string{"", "asr", "vad"} {
-				err := (&SherpaBackend{}).Load(&pb.ModelOptions{ModelFile: modelFile, Type: typ})
-				Expect(err).To(HaveOccurred(), "Type=%q", typ)
-			}
-		})
-	})
-
-	Context("method errors without loaded model", func() {
-		It("rejects TTS", func() {
-			tmpDir, err := os.MkdirTemp("", "sherpa-test-tts")
-			Expect(err).ToNot(HaveOccurred())
-			defer os.RemoveAll(tmpDir)
-
-			err = (&SherpaBackend{}).TTS(&pb.TTSRequest{
-				Text: "should fail — no model loaded",
-				Dst:  filepath.Join(tmpDir, "output.wav"),
-			})
-			Expect(err).To(HaveOccurred())
-		})
-
-		It("rejects AudioTranscription", func() {
-			_, err := (&SherpaBackend{}).AudioTranscription(context.Background(), &pb.TranscriptRequest{
-				Dst: "/tmp/nonexistent.wav",
-			})
-			Expect(err).To(HaveOccurred())
-		})
-
-		It("rejects VAD", func() {
-			_, err := (&SherpaBackend{}).VAD(&pb.VADRequest{
-				Audio: []float32{0.1, 0.2, 0.3},
-			})
-			Expect(err).To(HaveOccurred())
-		})
-	})
-
-	Context("type detection", func() {
-		DescribeTable("isASRType",
-			func(input string, want bool) {
-				Expect(isASRType(input)).To(Equal(want))
-			},
-			Entry("asr", "asr", true),
-			Entry("ASR", "ASR", true),
-			Entry("Asr", "Asr", true),
-			Entry("transcription", "transcription", true),
-			Entry("Transcription", "Transcription", true),
-			Entry("transcribe", "transcribe", true),
-			Entry("Transcribe", "Transcribe", true),
-			Entry("tts", "tts", false),
-			Entry("empty", "", false),
-			Entry("other", "other", false),
-			Entry("vad", "vad", false),
-		)
-
-		DescribeTable("isVADType",
-			func(input string, want bool) {
-				Expect(isVADType(input)).To(Equal(want))
-			},
-			Entry("vad", "vad", true),
-			Entry("VAD", "VAD", true),
-			Entry("Vad", "Vad", true),
-			Entry("asr", "asr", false),
-			Entry("tts", "tts", false),
-			Entry("empty", "", false),
-			Entry("other", "other", false),
-		)
-	})
-
-	Context("option parsing", func() {
-		It("parses float options with fallback on bad input", func() {
-			opts := &pb.ModelOptions{Options: []string{
-				"vad.threshold=0.3",
-				"tts.length_scale=1.25",
-				"bad.number=not-a-float",
-			}}
-			Expect(findOptionFloat(opts, "vad.threshold=", 0.5)).To(BeNumerically("~", 0.3, 1e-6))
-			Expect(findOptionFloat(opts, "tts.length_scale=", 1.0)).To(BeNumerically("~", 1.25, 1e-6))
-			Expect(findOptionFloat(opts, "missing.key=", 0.7)).To(BeNumerically("~", 0.7, 1e-6))
-			Expect(findOptionFloat(opts, "bad.number=", 9.9)).To(BeNumerically("~", 9.9, 1e-6))
-		})
-
-		It("parses int options with fallback on bad input", func() {
-			opts := &pb.ModelOptions{Options: []string{
-				"asr.sample_rate=22050",
-				"online.chunk_samples=800",
-				"bad.int=4.2",
-			}}
-			Expect(findOptionInt(opts, "asr.sample_rate=", 16000)).To(Equal(int32(22050)))
-			Expect(findOptionInt(opts, "online.chunk_samples=", 1600)).To(Equal(int32(800)))
-			Expect(findOptionInt(opts, "missing.key=", 42)).To(Equal(int32(42)))
-			Expect(findOptionInt(opts, "bad.int=", 100)).To(Equal(int32(100)))
-		})
-
-		It("parses bool options (0/1, true/false, yes/no, on/off)", func() {
-			opts := &pb.ModelOptions{Options: []string{
-				"online.enable_endpoint=0",
-				"asr.sense_voice.use_itn=True",
-				"feature.on=yes",
-				"feature.off=Off",
-				"feature.bad=maybe",
-			}}
-			Expect(findOptionBool(opts, "online.enable_endpoint=", 1)).To(Equal(int32(0)))
-			Expect(findOptionBool(opts, "asr.sense_voice.use_itn=", 0)).To(Equal(int32(1)))
-			Expect(findOptionBool(opts, "feature.on=", 0)).To(Equal(int32(1)))
-			Expect(findOptionBool(opts, "feature.off=", 1)).To(Equal(int32(0)))
-			Expect(findOptionBool(opts, "feature.bad=", 1)).To(Equal(int32(1)))
-			Expect(findOptionBool(opts, "missing.key=", 1)).To(Equal(int32(1)))
-		})
-	})
-})

backend/go/sherpa-onnx/csrc/shim.c

@@ -1,406 +0,0 @@
-#include "shim.h"
-#include "c-api.h"
-
-#include <stdlib.h>
-#include <string.h>
-
-// Replace the char* field pointed to by `slot` with a strdup of `s`
-// (or NULL if s is NULL). Frees any prior value. Silently no-ops when
-// strdup fails — the caller will see a Create* failure downstream.
-static void shim_set_str(const char **slot, const char *s) {
-    free((char *)*slot);
-    *slot = s ? strdup(s) : NULL;
-}
-
-// ==================================================================
-// VAD config
-// ==================================================================
-
-void *sherpa_shim_vad_config_new(void) {
-    return calloc(1, sizeof(SherpaOnnxVadModelConfig));
-}
-
-void sherpa_shim_vad_config_free(void *h) {
-    if (!h) return;
-    SherpaOnnxVadModelConfig *c = (SherpaOnnxVadModelConfig *)h;
-    free((char *)c->silero_vad.model);
-    free((char *)c->provider);
-    free(c);
-}
-
-void sherpa_shim_vad_config_set_silero_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxVadModelConfig *)h)->silero_vad.model, v);
-}
-void sherpa_shim_vad_config_set_silero_threshold(void *h, float v) {
-    ((SherpaOnnxVadModelConfig *)h)->silero_vad.threshold = v;
-}
-void sherpa_shim_vad_config_set_silero_min_silence_duration(void *h, float v) {
-    ((SherpaOnnxVadModelConfig *)h)->silero_vad.min_silence_duration = v;
-}
-void sherpa_shim_vad_config_set_silero_min_speech_duration(void *h, float v) {
-    ((SherpaOnnxVadModelConfig *)h)->silero_vad.min_speech_duration = v;
-}
-void sherpa_shim_vad_config_set_silero_window_size(void *h, int32_t v) {
-    ((SherpaOnnxVadModelConfig *)h)->silero_vad.window_size = v;
-}
-void sherpa_shim_vad_config_set_silero_max_speech_duration(void *h, float v) {
-    ((SherpaOnnxVadModelConfig *)h)->silero_vad.max_speech_duration = v;
-}
-void sherpa_shim_vad_config_set_sample_rate(void *h, int32_t v) {
-    ((SherpaOnnxVadModelConfig *)h)->sample_rate = v;
-}
-void sherpa_shim_vad_config_set_num_threads(void *h, int32_t v) {
-    ((SherpaOnnxVadModelConfig *)h)->num_threads = v;
-}
-void sherpa_shim_vad_config_set_provider(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxVadModelConfig *)h)->provider, v);
-}
-void sherpa_shim_vad_config_set_debug(void *h, int32_t v) {
-    ((SherpaOnnxVadModelConfig *)h)->debug = v;
-}
-
-void *sherpa_shim_create_vad(void *h, float buffer_size_seconds) {
-    return (void *)SherpaOnnxCreateVoiceActivityDetector(
-        (const SherpaOnnxVadModelConfig *)h, buffer_size_seconds);
-}
-
-// ==================================================================
-// Offline TTS config (VITS)
-// ==================================================================
-
-void *sherpa_shim_tts_config_new(void) {
-    return calloc(1, sizeof(SherpaOnnxOfflineTtsConfig));
-}
-
-void sherpa_shim_tts_config_free(void *h) {
-    if (!h) return;
-    SherpaOnnxOfflineTtsConfig *c = (SherpaOnnxOfflineTtsConfig *)h;
-    free((char *)c->model.vits.model);
-    free((char *)c->model.vits.tokens);
-    free((char *)c->model.vits.lexicon);
-    free((char *)c->model.vits.data_dir);
-    free((char *)c->model.provider);
-    free(c);
-}
-
-void sherpa_shim_tts_config_set_vits_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.model, v);
-}
-void sherpa_shim_tts_config_set_vits_tokens(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.tokens, v);
-}
-void sherpa_shim_tts_config_set_vits_lexicon(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.lexicon, v);
-}
-void sherpa_shim_tts_config_set_vits_data_dir(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.vits.data_dir, v);
-}
-void sherpa_shim_tts_config_set_vits_noise_scale(void *h, float v) {
-    ((SherpaOnnxOfflineTtsConfig *)h)->model.vits.noise_scale = v;
-}
-void sherpa_shim_tts_config_set_vits_noise_scale_w(void *h, float v) {
-    ((SherpaOnnxOfflineTtsConfig *)h)->model.vits.noise_scale_w = v;
-}
-void sherpa_shim_tts_config_set_vits_length_scale(void *h, float v) {
-    ((SherpaOnnxOfflineTtsConfig *)h)->model.vits.length_scale = v;
-}
-void sherpa_shim_tts_config_set_num_threads(void *h, int32_t v) {
-    ((SherpaOnnxOfflineTtsConfig *)h)->model.num_threads = v;
-}
-void sherpa_shim_tts_config_set_debug(void *h, int32_t v) {
-    ((SherpaOnnxOfflineTtsConfig *)h)->model.debug = v;
-}
-void sherpa_shim_tts_config_set_provider(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineTtsConfig *)h)->model.provider, v);
-}
-void sherpa_shim_tts_config_set_max_num_sentences(void *h, int32_t v) {
-    ((SherpaOnnxOfflineTtsConfig *)h)->max_num_sentences = v;
-}
-
-void *sherpa_shim_create_offline_tts(void *h) {
-    return (void *)SherpaOnnxCreateOfflineTts(
-        (const SherpaOnnxOfflineTtsConfig *)h);
-}
-
-// ==================================================================
-// Offline recognizer config
-// ==================================================================
-
-void *sherpa_shim_offline_recog_config_new(void) {
-    return calloc(1, sizeof(SherpaOnnxOfflineRecognizerConfig));
-}
-
-void sherpa_shim_offline_recog_config_free(void *h) {
-    if (!h) return;
-    SherpaOnnxOfflineRecognizerConfig *c = (SherpaOnnxOfflineRecognizerConfig *)h;
-    free((char *)c->model_config.provider);
-    free((char *)c->model_config.tokens);
-    free((char *)c->model_config.whisper.encoder);
-    free((char *)c->model_config.whisper.decoder);
-    free((char *)c->model_config.whisper.language);
-    free((char *)c->model_config.whisper.task);
-    free((char *)c->model_config.paraformer.model);
-    free((char *)c->model_config.sense_voice.model);
-    free((char *)c->model_config.sense_voice.language);
-    free((char *)c->model_config.omnilingual.model);
-    free((char *)c->decoding_method);
-    free(c);
-}
-
-void sherpa_shim_offline_recog_config_set_num_threads(void *h, int32_t v) {
-    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.num_threads = v;
-}
-void sherpa_shim_offline_recog_config_set_debug(void *h, int32_t v) {
-    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.debug = v;
-}
-void sherpa_shim_offline_recog_config_set_provider(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.provider, v);
-}
-void sherpa_shim_offline_recog_config_set_tokens(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.tokens, v);
-}
-void sherpa_shim_offline_recog_config_set_feat_sample_rate(void *h, int32_t v) {
-    ((SherpaOnnxOfflineRecognizerConfig *)h)->feat_config.sample_rate = v;
-}
-void sherpa_shim_offline_recog_config_set_feat_feature_dim(void *h, int32_t v) {
-    ((SherpaOnnxOfflineRecognizerConfig *)h)->feat_config.feature_dim = v;
-}
-void sherpa_shim_offline_recog_config_set_decoding_method(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->decoding_method, v);
-}
-void sherpa_shim_offline_recog_config_set_whisper_encoder(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.encoder, v);
-}
-void sherpa_shim_offline_recog_config_set_whisper_decoder(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.decoder, v);
-}
-void sherpa_shim_offline_recog_config_set_whisper_language(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.language, v);
-}
-void sherpa_shim_offline_recog_config_set_whisper_task(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.task, v);
-}
-void sherpa_shim_offline_recog_config_set_whisper_tail_paddings(void *h, int32_t v) {
-    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.whisper.tail_paddings = v;
-}
-void sherpa_shim_offline_recog_config_set_paraformer_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.paraformer.model, v);
-}
-void sherpa_shim_offline_recog_config_set_sense_voice_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.sense_voice.model, v);
-}
-void sherpa_shim_offline_recog_config_set_sense_voice_language(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.sense_voice.language, v);
-}
-void sherpa_shim_offline_recog_config_set_sense_voice_use_itn(void *h, int32_t v) {
-    ((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.sense_voice.use_itn = v;
-}
-void sherpa_shim_offline_recog_config_set_omnilingual_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineRecognizerConfig *)h)->model_config.omnilingual.model, v);
-}
-
-void *sherpa_shim_create_offline_recognizer(void *h) {
-    return (void *)SherpaOnnxCreateOfflineRecognizer(
-        (const SherpaOnnxOfflineRecognizerConfig *)h);
-}
-
-// ==================================================================
-// Online recognizer config
-// ==================================================================
-
-void *sherpa_shim_online_recog_config_new(void) {
-    return calloc(1, sizeof(SherpaOnnxOnlineRecognizerConfig));
-}
-
-void sherpa_shim_online_recog_config_free(void *h) {
-    if (!h) return;
-    SherpaOnnxOnlineRecognizerConfig *c = (SherpaOnnxOnlineRecognizerConfig *)h;
-    free((char *)c->model_config.transducer.encoder);
-    free((char *)c->model_config.transducer.decoder);
-    free((char *)c->model_config.transducer.joiner);
-    free((char *)c->model_config.tokens);
-    free((char *)c->model_config.provider);
-    free((char *)c->decoding_method);
-    free(c);
-}
-
-void sherpa_shim_online_recog_config_set_transducer_encoder(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.transducer.encoder, v);
-}
-void sherpa_shim_online_recog_config_set_transducer_decoder(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.transducer.decoder, v);
-}
-void sherpa_shim_online_recog_config_set_transducer_joiner(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.transducer.joiner, v);
-}
-void sherpa_shim_online_recog_config_set_tokens(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.tokens, v);
-}
-void sherpa_shim_online_recog_config_set_num_threads(void *h, int32_t v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.num_threads = v;
-}
-void sherpa_shim_online_recog_config_set_debug(void *h, int32_t v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.debug = v;
-}
-void sherpa_shim_online_recog_config_set_provider(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->model_config.provider, v);
-}
-void sherpa_shim_online_recog_config_set_feat_sample_rate(void *h, int32_t v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->feat_config.sample_rate = v;
-}
-void sherpa_shim_online_recog_config_set_feat_feature_dim(void *h, int32_t v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->feat_config.feature_dim = v;
-}
-void sherpa_shim_online_recog_config_set_decoding_method(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOnlineRecognizerConfig *)h)->decoding_method, v);
-}
-void sherpa_shim_online_recog_config_set_enable_endpoint(void *h, int32_t v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->enable_endpoint = v;
-}
-void sherpa_shim_online_recog_config_set_rule1_min_trailing_silence(void *h, float v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->rule1_min_trailing_silence = v;
-}
-void sherpa_shim_online_recog_config_set_rule2_min_trailing_silence(void *h, float v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->rule2_min_trailing_silence = v;
-}
-void sherpa_shim_online_recog_config_set_rule3_min_utterance_length(void *h, float v) {
-    ((SherpaOnnxOnlineRecognizerConfig *)h)->rule3_min_utterance_length = v;
-}
-
-void *sherpa_shim_create_online_recognizer(void *h) {
-    return (void *)SherpaOnnxCreateOnlineRecognizer(
-        (const SherpaOnnxOnlineRecognizerConfig *)h);
-}
-
-// ==================================================================
-// Result-struct accessors
-// ==================================================================
-
-int32_t sherpa_shim_wave_sample_rate(const void *h) {
-    return ((const SherpaOnnxWave *)h)->sample_rate;
-}
-int32_t sherpa_shim_wave_num_samples(const void *h) {
-    return ((const SherpaOnnxWave *)h)->num_samples;
-}
-const float *sherpa_shim_wave_samples(const void *h) {
-    return ((const SherpaOnnxWave *)h)->samples;
-}
-
-const char *sherpa_shim_offline_result_text(const void *h) {
-    return ((const SherpaOnnxOfflineRecognizerResult *)h)->text;
-}
-const char *sherpa_shim_online_result_text(const void *h) {
-    return ((const SherpaOnnxOnlineRecognizerResult *)h)->text;
-}
-
-int32_t sherpa_shim_generated_audio_sample_rate(const void *h) {
-    return ((const SherpaOnnxGeneratedAudio *)h)->sample_rate;
-}
-int32_t sherpa_shim_generated_audio_n(const void *h) {
-    return ((const SherpaOnnxGeneratedAudio *)h)->n;
-}
-const float *sherpa_shim_generated_audio_samples(const void *h) {
-    return ((const SherpaOnnxGeneratedAudio *)h)->samples;
-}
-
-int32_t sherpa_shim_speech_segment_start(const void *h) {
-    return ((const SherpaOnnxSpeechSegment *)h)->start;
-}
-int32_t sherpa_shim_speech_segment_n(const void *h) {
-    return ((const SherpaOnnxSpeechSegment *)h)->n;
-}
-
-// ==================================================================
-// Offline speaker diarization config
-// ==================================================================
-
-void *sherpa_shim_diarize_config_new(void) {
-    return calloc(1, sizeof(SherpaOnnxOfflineSpeakerDiarizationConfig));
-}
-
-void sherpa_shim_diarize_config_free(void *h) {
-    if (!h) return;
-    SherpaOnnxOfflineSpeakerDiarizationConfig *c =
-        (SherpaOnnxOfflineSpeakerDiarizationConfig *)h;
-    free((char *)c->segmentation.pyannote.model);
-    free((char *)c->segmentation.provider);
-    free((char *)c->embedding.model);
-    free((char *)c->embedding.provider);
-    free(c);
-}
-
-void sherpa_shim_diarize_config_set_segmentation_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.pyannote.model, v);
-}
-void sherpa_shim_diarize_config_set_segmentation_num_threads(void *h, int32_t v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.num_threads = v;
-}
-void sherpa_shim_diarize_config_set_segmentation_provider(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.provider, v);
-}
-void sherpa_shim_diarize_config_set_segmentation_debug(void *h, int32_t v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->segmentation.debug = v;
-}
-void sherpa_shim_diarize_config_set_embedding_model(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.model, v);
-}
-void sherpa_shim_diarize_config_set_embedding_num_threads(void *h, int32_t v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.num_threads = v;
-}
-void sherpa_shim_diarize_config_set_embedding_provider(void *h, const char *v) {
-    shim_set_str(&((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.provider, v);
-}
-void sherpa_shim_diarize_config_set_embedding_debug(void *h, int32_t v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->embedding.debug = v;
-}
-void sherpa_shim_diarize_config_set_clustering_num_clusters(void *h, int32_t v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->clustering.num_clusters = v;
-}
-void sherpa_shim_diarize_config_set_clustering_threshold(void *h, float v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->clustering.threshold = v;
-}
-void sherpa_shim_diarize_config_set_min_duration_on(void *h, float v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->min_duration_on = v;
-}
-void sherpa_shim_diarize_config_set_min_duration_off(void *h, float v) {
-    ((SherpaOnnxOfflineSpeakerDiarizationConfig *)h)->min_duration_off = v;
-}
-
-void *sherpa_shim_create_offline_speaker_diarization(void *h) {
-    return (void *)SherpaOnnxCreateOfflineSpeakerDiarization(
-        (const SherpaOnnxOfflineSpeakerDiarizationConfig *)h);
-}
-
-void sherpa_shim_diarize_set_clustering(void *sd, int32_t num_clusters, float threshold) {
-    if (!sd) return;
-    SherpaOnnxOfflineSpeakerDiarizationConfig cfg;
-    memset(&cfg, 0, sizeof(cfg));
-    cfg.clustering.num_clusters = num_clusters;
-    cfg.clustering.threshold    = threshold;
-    SherpaOnnxOfflineSpeakerDiarizationSetConfig(
-        (const SherpaOnnxOfflineSpeakerDiarization *)sd, &cfg);
-}
-
-void sherpa_shim_diarize_segment_at(const void *segs, int32_t i,
-                                    float *out_start, float *out_end,
-                                    int32_t *out_speaker) {
-    const SherpaOnnxOfflineSpeakerDiarizationSegment *arr =
-        (const SherpaOnnxOfflineSpeakerDiarizationSegment *)segs;
-    if (out_start)   *out_start   = arr[i].start;
-    if (out_end)     *out_end     = arr[i].end;
-    if (out_speaker) *out_speaker = arr[i].speaker;
-}
-
-// ==================================================================
-// TTS streaming callback trampoline
-// ==================================================================
-
-void *sherpa_shim_tts_generate_with_callback(
-    void *tts, const char *text, int32_t sid, float speed,
-    uintptr_t callback_ptr, uintptr_t user_data) {
-    SherpaOnnxGeneratedAudioCallbackWithArg cb =
-        (SherpaOnnxGeneratedAudioCallbackWithArg)callback_ptr;
-    return (void *)SherpaOnnxOfflineTtsGenerateWithCallbackWithArg(
-        (const SherpaOnnxOfflineTts *)tts, text, sid, speed, cb,
-        (void *)user_data);
-}

backend/go/sherpa-onnx/csrc/shim.h

@@ -1,164 +0,0 @@
-#ifndef LOCALAI_SHERPA_ONNX_SHIM_H
-#define LOCALAI_SHERPA_ONNX_SHIM_H
-
-#include <stdint.h>
-
-// libsherpa-shim: purego-friendly wrapper around sherpa-onnx's C API.
-// Purego can't access C struct fields and can't route C callbacks to Go
-// funcs directly. Every function here is a fixed-signature trampoline
-// that replaces one field read/write or callback handoff that the Go
-// backend would otherwise have to do through cgo.
-//
-// String lifetime: setters strdup; _free walks every owned string and
-// frees it. Callers may discard their input buffers the moment a setter
-// returns.
-//
-// Opaque handles are `void *` in both directions. Nothing here holds a
-// reference across calls except config handles (freed via _free) and
-// sherpa-allocated results (freed via sherpa's own Destroy* entry
-// points, which Go calls through purego pass-through).
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-// --- VAD config -----------------------------------------------------
-void *sherpa_shim_vad_config_new(void);
-void  sherpa_shim_vad_config_free(void *cfg);
-void  sherpa_shim_vad_config_set_silero_model(void *cfg, const char *path);
-void  sherpa_shim_vad_config_set_silero_threshold(void *cfg, float v);
-void  sherpa_shim_vad_config_set_silero_min_silence_duration(void *cfg, float v);
-void  sherpa_shim_vad_config_set_silero_min_speech_duration(void *cfg, float v);
-void  sherpa_shim_vad_config_set_silero_window_size(void *cfg, int32_t v);
-void  sherpa_shim_vad_config_set_silero_max_speech_duration(void *cfg, float v);
-void  sherpa_shim_vad_config_set_sample_rate(void *cfg, int32_t v);
-void  sherpa_shim_vad_config_set_num_threads(void *cfg, int32_t v);
-void  sherpa_shim_vad_config_set_provider(void *cfg, const char *v);
-void  sherpa_shim_vad_config_set_debug(void *cfg, int32_t v);
-void *sherpa_shim_create_vad(void *cfg, float buffer_size_seconds);
-
-// --- Offline TTS config (VITS path — the only TTS family the backend uses) ---
-void *sherpa_shim_tts_config_new(void);
-void  sherpa_shim_tts_config_free(void *cfg);
-void  sherpa_shim_tts_config_set_vits_model(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_vits_tokens(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_vits_lexicon(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_vits_data_dir(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_vits_noise_scale(void *cfg, float v);
-void  sherpa_shim_tts_config_set_vits_noise_scale_w(void *cfg, float v);
-void  sherpa_shim_tts_config_set_vits_length_scale(void *cfg, float v);
-void  sherpa_shim_tts_config_set_num_threads(void *cfg, int32_t v);
-void  sherpa_shim_tts_config_set_debug(void *cfg, int32_t v);
-void  sherpa_shim_tts_config_set_provider(void *cfg, const char *v);
-void  sherpa_shim_tts_config_set_max_num_sentences(void *cfg, int32_t v);
-void *sherpa_shim_create_offline_tts(void *cfg);
-
-// --- Offline recognizer config (Whisper / Paraformer / SenseVoice / Omnilingual) ---
-void *sherpa_shim_offline_recog_config_new(void);
-void  sherpa_shim_offline_recog_config_free(void *cfg);
-void  sherpa_shim_offline_recog_config_set_num_threads(void *cfg, int32_t v);
-void  sherpa_shim_offline_recog_config_set_debug(void *cfg, int32_t v);
-void  sherpa_shim_offline_recog_config_set_provider(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_tokens(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_feat_sample_rate(void *cfg, int32_t v);
-void  sherpa_shim_offline_recog_config_set_feat_feature_dim(void *cfg, int32_t v);
-void  sherpa_shim_offline_recog_config_set_decoding_method(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_whisper_encoder(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_whisper_decoder(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_whisper_language(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_whisper_task(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_whisper_tail_paddings(void *cfg, int32_t v);
-void  sherpa_shim_offline_recog_config_set_paraformer_model(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_sense_voice_model(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_sense_voice_language(void *cfg, const char *v);
-void  sherpa_shim_offline_recog_config_set_sense_voice_use_itn(void *cfg, int32_t v);
-void  sherpa_shim_offline_recog_config_set_omnilingual_model(void *cfg, const char *v);
-void *sherpa_shim_create_offline_recognizer(void *cfg);
-
-// --- Online recognizer config (streaming zipformer transducer) ---
-void *sherpa_shim_online_recog_config_new(void);
-void  sherpa_shim_online_recog_config_free(void *cfg);
-void  sherpa_shim_online_recog_config_set_transducer_encoder(void *cfg, const char *v);
-void  sherpa_shim_online_recog_config_set_transducer_decoder(void *cfg, const char *v);
-void  sherpa_shim_online_recog_config_set_transducer_joiner(void *cfg, const char *v);
-void  sherpa_shim_online_recog_config_set_tokens(void *cfg, const char *v);
-void  sherpa_shim_online_recog_config_set_num_threads(void *cfg, int32_t v);
-void  sherpa_shim_online_recog_config_set_debug(void *cfg, int32_t v);
-void  sherpa_shim_online_recog_config_set_provider(void *cfg, const char *v);
-void  sherpa_shim_online_recog_config_set_feat_sample_rate(void *cfg, int32_t v);
-void  sherpa_shim_online_recog_config_set_feat_feature_dim(void *cfg, int32_t v);
-void  sherpa_shim_online_recog_config_set_decoding_method(void *cfg, const char *v);
-void  sherpa_shim_online_recog_config_set_enable_endpoint(void *cfg, int32_t v);
-void  sherpa_shim_online_recog_config_set_rule1_min_trailing_silence(void *cfg, float v);
-void  sherpa_shim_online_recog_config_set_rule2_min_trailing_silence(void *cfg, float v);
-void  sherpa_shim_online_recog_config_set_rule3_min_utterance_length(void *cfg, float v);
-void *sherpa_shim_create_online_recognizer(void *cfg);
-
-// --- Result accessors (sherpa-allocated; caller destroys via sherpa's own Destroy*) ---
-int32_t      sherpa_shim_wave_sample_rate(const void *wave);
-int32_t      sherpa_shim_wave_num_samples(const void *wave);
-const float *sherpa_shim_wave_samples(const void *wave);
-
-const char *sherpa_shim_offline_result_text(const void *result);
-const char *sherpa_shim_online_result_text(const void *result);
-
-int32_t      sherpa_shim_generated_audio_sample_rate(const void *audio);
-int32_t      sherpa_shim_generated_audio_n(const void *audio);
-const float *sherpa_shim_generated_audio_samples(const void *audio);
-
-int32_t sherpa_shim_speech_segment_start(const void *seg);
-int32_t sherpa_shim_speech_segment_n(const void *seg);
-
-// --- Offline speaker diarization config -----------------------------
-// Pyannote segmentation + speaker-embedding extractor + fast clustering.
-// The upstream config is a struct of nested structs; purego can't read or
-// build those across dlopen, so we expose a calloc'd opaque holder plus
-// flat setters, then hand it to sherpa via the create wrapper.
-void *sherpa_shim_diarize_config_new(void);
-void  sherpa_shim_diarize_config_free(void *cfg);
-void  sherpa_shim_diarize_config_set_segmentation_model(void *cfg, const char *path);
-void  sherpa_shim_diarize_config_set_segmentation_num_threads(void *cfg, int32_t v);
-void  sherpa_shim_diarize_config_set_segmentation_provider(void *cfg, const char *v);
-void  sherpa_shim_diarize_config_set_segmentation_debug(void *cfg, int32_t v);
-void  sherpa_shim_diarize_config_set_embedding_model(void *cfg, const char *path);
-void  sherpa_shim_diarize_config_set_embedding_num_threads(void *cfg, int32_t v);
-void  sherpa_shim_diarize_config_set_embedding_provider(void *cfg, const char *v);
-void  sherpa_shim_diarize_config_set_embedding_debug(void *cfg, int32_t v);
-void  sherpa_shim_diarize_config_set_clustering_num_clusters(void *cfg, int32_t v);
-void  sherpa_shim_diarize_config_set_clustering_threshold(void *cfg, float v);
-void  sherpa_shim_diarize_config_set_min_duration_on(void *cfg, float v);
-void  sherpa_shim_diarize_config_set_min_duration_off(void *cfg, float v);
-void *sherpa_shim_create_offline_speaker_diarization(void *cfg);
-
-// Apply just the clustering knobs onto a loaded diarizer (sherpa
-// supports re-clustering after Create), so per-call overrides like
-// num_speakers don't require re-loading the heavy ONNX models.
-void  sherpa_shim_diarize_set_clustering(void *sd, int32_t num_clusters, float threshold);
-
-// Sherpa's ResultSortByStartTime returns a sherpa-allocated array of
-// SherpaOnnxOfflineSpeakerDiarizationSegment structs (free with
-// SherpaOnnxOfflineSpeakerDiarizationDestroySegment). Purego can't read
-// fields out of an array of C structs, so this getter copies one
-// segment's fields into the caller-supplied float/int32 cells.
-void  sherpa_shim_diarize_segment_at(const void *segs, int32_t i,
-                                     float *out_start, float *out_end,
-                                     int32_t *out_speaker);
-
-// --- TTS streaming callback trampoline -----------------------------
-// Replaces the //export sherpaTtsGoCallback + callbacks.c bridge pattern.
-// `callback_ptr` is the C-callable function pointer returned by
-// purego.NewCallback. `user_data` is an integer the Go side uses to
-// look up its state (sync.Map keyed by uint64).
-//
-// Returns the sherpa-allocated SherpaOnnxGeneratedAudio. Destroy with
-// SherpaOnnxDestroyOfflineTtsGeneratedAudio (callable directly from
-// Go via purego).
-void *sherpa_shim_tts_generate_with_callback(
-    void *tts, const char *text, int32_t sid, float speed,
-    uintptr_t callback_ptr, uintptr_t user_data);
-
-#ifdef __cplusplus
-}
-#endif
-
-#endif

backend/go/sherpa-onnx/main.go

@@ -1,23 +0,0 @@
-package main
-
-import (
-	"flag"
-
-	grpc "github.com/mudler/LocalAI/pkg/grpc"
-)
-
-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
-)
-
-func main() {
-	flag.Parse()
-
-	if err := loadSherpaLibs(); err != nil {
-		panic(err)
-	}
-
-	if err := grpc.StartServer(*addr, &SherpaBackend{}); err != nil {
-		panic(err)
-	}
-}

backend/go/sherpa-onnx/package.sh

@@ -1,51 +0,0 @@
-#!/bin/bash
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-mkdir -p $CURDIR/package/lib
-
-cp -avf $CURDIR/sherpa-onnx $CURDIR/package/
-cp -avf $CURDIR/run.sh $CURDIR/package/
-cp -rfLv $CURDIR/backend-assets/lib/* $CURDIR/package/lib/
-
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ $(uname -s) = "Darwin" ]; then
-    echo "Detected Darwin"
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/go/sherpa-onnx/run.sh

@@ -1,13 +0,0 @@
-#!/bin/bash
-set -ex
-
-CURDIR=$(dirname "$(realpath $0)")
-
-export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
-
-if [ -f $CURDIR/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	exec $CURDIR/lib/ld.so $CURDIR/sherpa-onnx "$@"
-fi
-
-exec $CURDIR/sherpa-onnx "$@"

backend/go/sherpa-onnx/test.sh

@@ -1,12 +0,0 @@
-#!/bin/bash
-# Unit tests for the sherpa-onnx backend. Exercises error-path and
-# dispatch logic via SherpaBackend directly (no gRPC). Integration
-# coverage (gRPC TTS / streaming ASR / realtime pipeline) lives in
-# tests/e2e-backends and tests/e2e and runs against the Docker image.
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-cd "$CURDIR"
-
-PACKAGES=$(go list ./... | grep -v /sources/)
-go test -v -timeout 60s $PACKAGES

backend/go/stablediffusion-ggml/CMakeLists.txt

@@ -4,7 +4,7 @@ set(CMAKE_POSITION_INDEPENDENT_CODE ON)
 
 add_subdirectory(./sources/stablediffusion-ggml.cpp)
 
-add_library(gosd MODULE cpp/gosd.cpp)
+add_library(gosd MODULE gosd.cpp)
 target_link_libraries(gosd PRIVATE stable-diffusion ggml)
 
 if(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9.0)

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=90e87bc846f17059771efb8aaa31e9ef0cab6f78
+STABLEDIFFUSION_GGML_VERSION?=7d33d4b2ddeafa672761a5880ec33bdff452504d
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 
@@ -119,7 +119,7 @@ libgosd-fallback.so: sources/stablediffusion-ggml.cpp
 	SO_TARGET=libgosd-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgosd-custom
 	rm -rfv build*
 
-libgosd-custom: CMakeLists.txt cpp/gosd.cpp cpp/gosd.h
+libgosd-custom: CMakeLists.txt gosd.cpp gosd.h
 	mkdir -p build-$(SO_TARGET) && \
 	cd build-$(SO_TARGET) && \
 	cmake .. $(CMAKE_ARGS) && \

backend/go/stablediffusion-ggml/gosd.cpp

@@ -1106,11 +1106,6 @@ static int ffmpeg_mux_raw_to_mp4(sd_image_t* frames, int num_frames, int fps, co
             const_cast<char*>("-c:v"), const_cast<char*>("libx264"),
             const_cast<char*>("-pix_fmt"), const_cast<char*>("yuv420p"),
             const_cast<char*>("-movflags"), const_cast<char*>("+faststart"),
-            // Force MP4 container. Distributed LocalAI hands us a staging
-            // path (e.g. /staging/localai-output-NNN.tmp) with a non-standard
-            // extension; relying on filename suffix makes ffmpeg bail with
-            // "Unable to choose an output format".
-            const_cast<char*>("-f"), const_cast<char*>("mp4"),
             const_cast<char*>(dst),
             nullptr
         };

backend/go/vibevoice-cpp/CMakeLists.txt

@@ -1,71 +0,0 @@
-cmake_minimum_required(VERSION 3.18)
-project(govibevoicecpp LANGUAGES C CXX)
-set(CMAKE_POSITION_INDEPENDENT_CODE ON)
-set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
-
-set(VIBEVOICE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/vibevoice.cpp)
-
-# Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
-if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
-    set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
-endif()
-
-# Force-disable upstream tests/examples — we only need libvibevoice.
-set(VIBEVOICE_BUILD_TESTS    OFF CACHE BOOL "" FORCE)
-set(VIBEVOICE_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
-set(VIBEVOICE_BUILD_SERVER   OFF CACHE BOOL "" FORCE)
-
-# vibevoice.cpp's top-level CMakeLists already adds third_party/ggml as a
-# subdirectory — no need to add it explicitly here, just include the
-# whole project.
-add_subdirectory(${VIBEVOICE_DIR} vibevoice EXCLUDE_FROM_ALL)
-
-add_library(govibevoicecpp MODULE cpp/govibevoicecpp.cpp)
-
-# libvibevoice is STATIC; without --whole-archive the linker GCs the
-# vv_capi_* symbols (purego dlopens them by name, nothing in our
-# translation unit references them). Force the static archive's
-# entire contents into the MODULE so dlsym finds vv_capi_load etc.
-if(APPLE)
-    target_link_libraries(govibevoicecpp PRIVATE -Wl,-force_load $<TARGET_FILE:vibevoice>)
-elseif(MSVC)
-    target_link_libraries(govibevoicecpp PRIVATE vibevoice)
-    set_property(TARGET govibevoicecpp APPEND PROPERTY LINK_FLAGS "/WHOLEARCHIVE:vibevoice")
-else()
-    target_link_libraries(govibevoicecpp PRIVATE
-        -Wl,--whole-archive vibevoice -Wl,--no-whole-archive)
-endif()
-
-target_include_directories(govibevoicecpp PRIVATE ${VIBEVOICE_DIR}/include)
-target_include_directories(govibevoicecpp SYSTEM PRIVATE ${VIBEVOICE_DIR}/third_party/ggml/include)
-
-# Link GPU backends if available — vibevoice's own CMake already links
-# these to the libvibevoice STATIC library, but we re-link them on the
-# MODULE so resolved symbols include all backend kernels.
-foreach(backend blas cuda metal vulkan)
-    if(TARGET ggml-${backend})
-        target_link_libraries(govibevoicecpp PRIVATE ggml-${backend})
-        string(TOUPPER ${backend} BACKEND_UPPER)
-        target_compile_definitions(govibevoicecpp PRIVATE VIBEVOICE_HAVE_${BACKEND_UPPER})
-        if(backend STREQUAL "cuda")
-            find_package(CUDAToolkit QUIET)
-            if(CUDAToolkit_FOUND)
-                target_link_libraries(govibevoicecpp PRIVATE CUDA::cudart)
-            endif()
-        endif()
-    endif()
-endforeach()
-
-if(MSVC)
-    target_compile_options(govibevoicecpp PRIVATE /W4 /wd4100 /wd4505)
-else()
-    target_compile_options(govibevoicecpp PRIVATE -Wall -Wextra -Wshadow
-                          -Wno-unused-parameter -Wno-unused-function -Wno-sign-conversion)
-endif()
-
-if(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9.0)
-    target_link_libraries(govibevoicecpp PRIVATE stdc++fs)
-endif()
-
-set_property(TARGET govibevoicecpp PROPERTY CXX_STANDARD 17)
-set_target_properties(govibevoicecpp PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/vibevoice-cpp/Makefile

@@ -1,131 +0,0 @@
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-
-GOCMD?=go
-GO_TAGS?=
-JOBS?=$(shell nproc --ignore=1)
-
-# vibevoice.cpp version. Pinned to a commit hash and auto-bumped by
-# .github/workflows/bump_deps.yaml (the matrix entry mirrors what we
-# already do for ik_llama.cpp / llama.cpp / whisper.cpp). Floating on
-# `master` led to silent ABI breaks reaching CI — pin it.
-VIBEVOICE_REPO?=https://github.com/mudler/vibevoice.cpp
-VIBEVOICE_CPP_VERSION?=ad856bda6b1311b7f3d7c4a667be43eeb8a8249a
-SO_TARGET?=libgovibevoicecpp.so
-
-CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
-CMAKE_ARGS+=-DVIBEVOICE_BUILD_TESTS=OFF
-CMAKE_ARGS+=-DVIBEVOICE_BUILD_EXAMPLES=OFF
-
-ifeq ($(NATIVE),false)
-	CMAKE_ARGS+=-DGGML_NATIVE=OFF
-endif
-
-ifeq ($(BUILD_TYPE),cublas)
-	CMAKE_ARGS+=-DGGML_CUDA=ON -DVIBEVOICE_GGML_CUDA=ON
-else ifeq ($(BUILD_TYPE),openblas)
-	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
-else ifeq ($(BUILD_TYPE),clblas)
-	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
-else ifeq ($(BUILD_TYPE),hipblas)
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON -DVIBEVOICE_GGML_HIPBLAS=ON
-else ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DGGML_VULKAN=ON -DVIBEVOICE_GGML_VULKAN=ON
-else ifeq ($(OS),Darwin)
-	ifneq ($(BUILD_TYPE),metal)
-		CMAKE_ARGS+=-DGGML_METAL=OFF
-	else
-		CMAKE_ARGS+=-DGGML_METAL=ON -DVIBEVOICE_GGML_METAL=ON
-		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
-	endif
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f16)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx \
-		-DGGML_SYCL_F16=ON
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f32)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx
-endif
-
-sources/vibevoice.cpp:
-	mkdir -p sources/vibevoice.cpp
-	cd sources/vibevoice.cpp && \
-	git init && \
-	git remote add origin $(VIBEVOICE_REPO) && \
-	git fetch origin && \
-	git checkout $(VIBEVOICE_CPP_VERSION) && \
-	git submodule update --init --recursive --depth 1 --single-branch
-
-# Detect OS
-UNAME_S := $(shell uname -s)
-
-# Only build CPU variants on Linux
-ifeq ($(UNAME_S),Linux)
-	VARIANT_TARGETS = libgovibevoicecpp-avx.so libgovibevoicecpp-avx2.so libgovibevoicecpp-avx512.so libgovibevoicecpp-fallback.so
-else
-	# On non-Linux (e.g., Darwin), build only fallback variant
-	VARIANT_TARGETS = libgovibevoicecpp-fallback.so
-endif
-
-vibevoice-cpp: main.go govibevoicecpp.go $(VARIANT_TARGETS)
-	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vibevoice-cpp ./
-
-package: vibevoice-cpp
-	bash package.sh
-
-build: package
-
-clean: purge
-	rm -rf libgovibevoicecpp*.so package sources/vibevoice.cpp vibevoice-cpp
-
-purge:
-	rm -rf build*
-
-# Variants must build sequentially
-.NOTPARALLEL:
-
-# Build all variants (Linux only)
-ifeq ($(UNAME_S),Linux)
-libgovibevoicecpp-avx.so: sources/vibevoice.cpp
-	$(info ${GREEN}I vibevoice-cpp build info:avx${RESET})
-	SO_TARGET=libgovibevoicecpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgovibevoicecpp-custom
-	rm -rf build-libgovibevoicecpp-avx.so
-
-libgovibevoicecpp-avx2.so: sources/vibevoice.cpp
-	$(info ${GREEN}I vibevoice-cpp build info:avx2${RESET})
-	SO_TARGET=libgovibevoicecpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgovibevoicecpp-custom
-	rm -rf build-libgovibevoicecpp-avx2.so
-
-libgovibevoicecpp-avx512.so: sources/vibevoice.cpp
-	$(info ${GREEN}I vibevoice-cpp build info:avx512${RESET})
-	SO_TARGET=libgovibevoicecpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgovibevoicecpp-custom
-	rm -rf build-libgovibevoicecpp-avx512.so
-endif
-
-# Build fallback variant (all platforms)
-libgovibevoicecpp-fallback.so: sources/vibevoice.cpp
-	$(info ${GREEN}I vibevoice-cpp build info:fallback${RESET})
-	SO_TARGET=libgovibevoicecpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgovibevoicecpp-custom
-	rm -rf build-libgovibevoicecpp-fallback.so
-
-libgovibevoicecpp-custom: CMakeLists.txt cpp/govibevoicecpp.cpp cpp/govibevoicecpp.h
-	mkdir -p build-$(SO_TARGET) && \
-	cd build-$(SO_TARGET) && \
-	cmake .. $(CMAKE_ARGS) && \
-	cmake --build . --config Release -j$(JOBS) --target govibevoicecpp && \
-	cd .. && \
-	mv build-$(SO_TARGET)/libgovibevoicecpp.so ./$(SO_TARGET)
-
-test: vibevoice-cpp
-	@echo "Running vibevoice-cpp tests..."
-	bash test.sh
-	@echo "vibevoice-cpp tests completed."
-
-all: vibevoice-cpp package

backend/go/vibevoice-cpp/cpp/govibevoicecpp.cpp

@@ -1,41 +0,0 @@
-// vibevoice.cpp ships its purego-friendly ABI in vibevoice_capi.h.
-// This translation unit is intentionally tiny: pulling in the header
-// (and linking libvibevoice PRIVATE in CMake) is enough to make the
-// vv_capi_* symbols visible from the produced MODULE library.
-//
-// We do install a ggml log redirect so backend logs land on the gRPC
-// server's stderr — same pattern as backend/go/qwen3-tts-cpp/cpp/.
-
-#include "govibevoicecpp.h"
-
-#include "ggml.h"
-#include "ggml-backend.h"
-
-#include <cstdio>
-
-namespace {
-
-void govibevoice_log_cb(enum ggml_log_level level, const char* msg, void* /*ud*/) {
-    if (!msg) return;
-    const char* tag = "?????";
-    switch (level) {
-    case GGML_LOG_LEVEL_DEBUG: tag = "DEBUG"; break;
-    case GGML_LOG_LEVEL_INFO:  tag = "INFO";  break;
-    case GGML_LOG_LEVEL_WARN:  tag = "WARN";  break;
-    case GGML_LOG_LEVEL_ERROR: tag = "ERROR"; break;
-    default: break;
-    }
-    std::fprintf(stderr, "[%-5s] %s", tag, msg);
-    std::fflush(stderr);
-}
-
-struct LogInstaller {
-    LogInstaller() {
-        ggml_log_set(govibevoice_log_cb, nullptr);
-        ggml_backend_load_all();
-    }
-};
-
-LogInstaller g_install;
-
-}  // namespace

backend/go/vibevoice-cpp/cpp/govibevoicecpp.h

@@ -1,7 +0,0 @@
-#pragma once
-
-// Re-exports the vibevoice.cpp flat C ABI so this MODULE library
-// resolves the same symbols that purego.RegisterLibFunc looks up by
-// name. The actual definitions live in libvibevoice (linked PRIVATE).
-
-#include "vibevoice_capi.h"

backend/go/vibevoice-cpp/govibevoicecpp.go

@@ -1,657 +0,0 @@
-package main
-
-import (
-	"context"
-	"encoding/json"
-	"fmt"
-	"io"
-	"os"
-	"os/exec"
-	"path/filepath"
-	"runtime"
-	"strings"
-
-	laudio "github.com/mudler/LocalAI/pkg/audio"
-	"github.com/mudler/LocalAI/pkg/grpc/base"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-)
-
-// vv_capi_asr loads audio with load_wav_24k_mono — a 24 kHz mono s16le
-// WAV is the format the model was trained on. Inputs already in that
-// format pass through; everything else is converted via ffmpeg, which
-// is therefore a runtime requirement only when callers upload non-WAV
-// (or non-24 kHz mono s16le WAV) audio. Skipping ffmpeg on the happy
-// path matters for the e2e-backends test container, which does not
-// ship ffmpeg but feeds the backend pre-cooked 24 kHz mono WAVs.
-const vibevoiceASRSampleRate = 24000
-
-// prepareWavInput resolves `src` to a 24 kHz mono s16le WAV path that
-// vv_capi_asr's load_wav_24k_mono accepts. Returns the resolved path
-// plus a cleanup func; both must be honoured by the caller.
-//
-// Pass-through happens when `src` already has the right WAV format —
-// no ffmpeg required. Otherwise we shell out to ffmpeg into a temp
-// dir; if ffmpeg isn't on PATH we surface a clear error mentioning the
-// underlying format mismatch.
-func prepareWavInput(src string) (string, func(), error) {
-	if src == "" {
-		return "", func() {}, fmt.Errorf("empty audio path")
-	}
-	if isVibevoiceCompatibleWav(src) {
-		return src, func() {}, nil
-	}
-
-	dir, err := os.MkdirTemp("", "vibevoice-asr")
-	if err != nil {
-		return "", func() {}, fmt.Errorf("mkdtemp: %w", err)
-	}
-	cleanup := func() { _ = os.RemoveAll(dir) }
-	wavPath := filepath.Join(dir, "input.wav")
-
-	// -y: overwrite, -ar 24000: target sample rate, -ac 1: mono,
-	// -acodec pcm_s16le: signed 16-bit little-endian PCM (load_wav_24k_mono
-	// only accepts s16le).
-	cmd := exec.Command("ffmpeg",
-		"-y", "-i", src,
-		"-ar", fmt.Sprintf("%d", vibevoiceASRSampleRate),
-		"-ac", "1",
-		"-acodec", "pcm_s16le",
-		wavPath,
-	)
-	cmd.Env = []string{}
-	if out, err := cmd.CombinedOutput(); err != nil {
-		cleanup()
-		return "", func() {}, fmt.Errorf("ffmpeg convert to 24k mono wav: %w (output: %s)", err, string(out))
-	}
-	return wavPath, cleanup, nil
-}
-
-// isVibevoiceCompatibleWav returns true when `src` carries the RIFF/WAVE
-// magic bytes. vibevoice's load_wav_24k_mono uses drwav under the hood,
-// which accepts any PCM/IEEE-float WAV at any sample rate and downmixes
-// multi-channel input to mono on its own — so any valid WAV passes
-// through to the C side without conversion. Anything else (MP3, OGG,
-// FLAC, ...) needs ffmpeg.
-func isVibevoiceCompatibleWav(src string) bool {
-	f, err := os.Open(src)
-	if err != nil {
-		return false
-	}
-	defer func() { _ = f.Close() }()
-
-	// 0..3 = "RIFF", 8..11 = "WAVE".
-	var hdr [12]byte
-	if _, err := io.ReadFull(f, hdr[:]); err != nil {
-		return false
-	}
-	return string(hdr[0:4]) == "RIFF" && string(hdr[8:12]) == "WAVE"
-}
-
-// asrMaxNewTokens caps the ASR generation budget. The C ABI defaults to
-// 256 when 0 is passed — far too small for anything past ~10s of speech.
-// Vibevoice generates ~30 tokens per second of audio, so 16 384 covers
-// roughly 9 minutes of dialogue, well past any normal /v1/audio/diarization
-// upload. Going higher costs little since generation stops at EOS.
-const asrMaxNewTokens = 16384
-
-// vibevoice.cpp synthesizes 24 kHz mono 16-bit PCM. Hardcoded - the
-// model itself is fixed-rate; if the upstream ever changes this we'll
-// pick it up via vv_capi_version().
-const vibevoiceSampleRate = uint32(24000)
-
-// purego-bound entry points from libgovibevoicecpp.
-//
-// vv_capi_tts takes a `const char* const* ref_audio_paths` array (used
-// by the 1.5B variant for runtime voice cloning; the realtime-0.5B
-// path leaves it NULL and uses voice_path instead). purego marshals a
-// Go []*byte slice as **char by passing the underlying array's address.
-// A nil/empty slice marshals to NULL, which matches the C contract for
-// "no reference audio".
-var (
-	CppLoad func(ttsModel, asrModel, tokenizer, voice string, threads int32) int32
-	CppTTS  func(text, voicePath string,
-		refAudioPaths []*byte, nRefAudioPaths int32,
-		dstWav string,
-		nSteps int32, cfgScale float32, maxSpeechFrames int32, seed uint32) int32
-	CppASR func(srcWav string, outJSON []byte, capacity uint64,
-		maxNewTokens int32) int32
-	CppUnload  func()
-	CppVersion func() string
-)
-
-// VibevoiceCpp speaks gRPC against vibevoice.cpp's flat C ABI. The
-// engine is a single global, so we serialize calls through SingleThread.
-type VibevoiceCpp struct {
-	base.SingleThread
-	threads int
-
-	// modelRoot is the directory we use to resolve relative paths
-	// from Options[] and per-call overrides (TTSRequest.Voice).
-	// Source of truth: opts.ModelPath; falls back to the dir of
-	// the primary ModelFile when ModelPath is empty.
-	modelRoot string
-
-	ttsModel  string
-	asrModel  string
-	tokenizer string
-	voice     string
-
-	// refAudio is the load-time default list of reference WAVs used by
-	// the 1.5B model (one per speaker). Sourced from
-	// ModelOptions.AudioPath (config_file's `audio_path:`) — comma-
-	// separated for multi-speaker. Per-call TTSRequest.Voice can
-	// override it. Empty for the realtime-0.5B path, which conditions
-	// on a pre-baked voice gguf via `voice` instead.
-	refAudio []string
-}
-
-// resolvePath joins a relative path onto `relTo`. The gallery
-// convention is that Options[] carry paths relative to the LocalAI
-// models dir (opts.ModelPath), so anything not absolute is treated
-// as a sibling of the primary ModelFile - never CWD. Empty / already-
-// absolute / no-relTo inputs pass through unchanged.
-func resolvePath(p, relTo string) string {
-	if p == "" || filepath.IsAbs(p) || relTo == "" {
-		return p
-	}
-	return filepath.Join(relTo, p)
-}
-
-// parseOptions reads opts.Options[] and pulls out the per-role
-// overrides documented in the gallery entries. Accepts both "key=value"
-// (gallery YAML style) and "key:value" (Make-target / env-var style).
-func (v *VibevoiceCpp) parseOptions(opts []string, relTo string) string {
-	role := ""
-	for _, raw := range opts {
-		k, val, ok := strings.Cut(raw, "=")
-		if !ok {
-			k, val, ok = strings.Cut(raw, ":")
-			if !ok {
-				continue
-			}
-		}
-		key := strings.TrimSpace(k)
-		val = strings.TrimSpace(val)
-		switch key {
-		case "type":
-			role = strings.ToLower(val)
-		case "tokenizer":
-			v.tokenizer = resolvePath(val, relTo)
-		case "voice":
-			v.voice = resolvePath(val, relTo)
-		case "tts_model":
-			v.ttsModel = resolvePath(val, relTo)
-		case "asr_model":
-			v.asrModel = resolvePath(val, relTo)
-		}
-	}
-	return role
-}
-
-// parseRefAudio splits a comma-separated audio_path value into a
-// resolved list of WAVs. The 1.5B model uses one WAV per speaker;
-// callers that only need a single reference set audio_path to a single
-// path. Empty / whitespace-only entries are skipped.
-func parseRefAudio(audioPath, relTo string) []string {
-	if audioPath == "" {
-		return nil
-	}
-	var out []string
-	for _, p := range strings.Split(audioPath, ",") {
-		p = strings.TrimSpace(p)
-		if p == "" {
-			continue
-		}
-		out = append(out, resolvePath(p, relTo))
-	}
-	return out
-}
-
-func (v *VibevoiceCpp) Load(opts *pb.ModelOptions) error {
-	if opts.ModelFile == "" {
-		return fmt.Errorf("vibevoice-cpp: ModelFile is required")
-	}
-	modelFile := opts.ModelFile
-	if !filepath.IsAbs(modelFile) && opts.ModelPath != "" {
-		modelFile = filepath.Join(opts.ModelPath, modelFile)
-	}
-
-	// ModelPath is the LocalAI core's models root, propagated over
-	// gRPC. Use it as the resolution base for Options[] (and later
-	// for TTSRequest.Voice) so gallery entries can reference paths
-	// like "tokenizer=tokenizer.gguf" and have them resolved
-	// against the same root the core used to drop the files.
-	v.modelRoot = opts.ModelPath
-	if v.modelRoot == "" {
-		v.modelRoot = filepath.Dir(modelFile)
-	}
-	role := v.parseOptions(opts.Options, v.modelRoot)
-
-	// 1.5B reference WAVs ride on ModelOptions.AudioPath (config_file's
-	// `audio_path:` key) — same convention other audio backends already
-	// follow. Single-speaker = single path; multi-speaker = comma list,
-	// one WAV per Speaker N: tag in TTSRequest.text.
-	v.refAudio = parseRefAudio(opts.AudioPath, v.modelRoot)
-
-	// ModelFile fills the "primary" role-slot determined by `type=`
-	// in Options (defaults to tts). The other slot stays exactly as
-	// Options set it - so a closed-loop config with ModelFile=tts.gguf
-	// + Options[asr_model=asr.gguf] resolves correctly to both slots,
-	// and an explicit `tts_model=` / `asr_model=` always wins over
-	// ModelFile for its own slot.
-	primaryIsASR := false
-	switch role {
-	case "asr", "transcript", "stt", "speech-to-text":
-		primaryIsASR = true
-	}
-	if primaryIsASR {
-		if v.asrModel == "" {
-			v.asrModel = modelFile
-		}
-	} else if v.ttsModel == "" {
-		v.ttsModel = modelFile
-	}
-
-	if v.ttsModel == "" && v.asrModel == "" {
-		return fmt.Errorf("vibevoice-cpp: no TTS or ASR model resolved from ModelFile=%q + options", opts.ModelFile)
-	}
-	if v.tokenizer == "" {
-		return fmt.Errorf("vibevoice-cpp: tokenizer is required - pass options: [tokenizer=<path>]")
-	}
-
-	threads := int(opts.Threads)
-	if threads <= 0 {
-		threads = 4
-	}
-	v.threads = threads
-
-	fmt.Fprintf(os.Stderr,
-		"[vibevoice-cpp] Loading: tts=%q asr=%q tokenizer=%q voice=%q ref_audio=%v threads=%d\n",
-		v.ttsModel, v.asrModel, v.tokenizer, v.voice, v.refAudio, threads)
-
-	if rc := CppLoad(v.ttsModel, v.asrModel, v.tokenizer, v.voice, int32(threads)); rc != 0 {
-		return fmt.Errorf("vibevoice-cpp: vv_capi_load failed (rc=%d)", rc)
-	}
-	return nil
-}
-
-func (v *VibevoiceCpp) TTS(req *pb.TTSRequest) error {
-	if v.ttsModel == "" {
-		return fmt.Errorf("vibevoice-cpp: TTS requested but no realtime model was loaded")
-	}
-	text := req.Text
-	dst := req.Dst
-	if text == "" || dst == "" {
-		return fmt.Errorf("vibevoice-cpp: TTS requires both text and dst")
-	}
-
-	// TTSRequest.Voice carries the per-call override. Routing depends
-	// on the loaded model variant:
-	//   * realtime-0.5B → expects a baked voice .gguf (single path).
-	//   * 1.5B          → expects one or more raw 24 kHz mono .wav
-	//                     reference clips for runtime voice cloning;
-	//                     comma-separated to address multi-speaker
-	//                     dialogs (Speaker 0..n-1 follow the order).
-	// We pick the branch by extension / shape of the override; if no
-	// override is given, fall back to the load-time defaults.
-	voice := ""
-	var refAudio []string
-	if reqVoice := strings.TrimSpace(req.Voice); reqVoice != "" {
-		if isRefAudioOverride(reqVoice) {
-			for _, p := range strings.Split(reqVoice, ",") {
-				p = strings.TrimSpace(p)
-				if p == "" {
-					continue
-				}
-				refAudio = append(refAudio, resolvePath(p, v.modelRoot))
-			}
-		} else {
-			voice = resolvePath(reqVoice, v.modelRoot)
-		}
-	} else {
-		// No per-call override. v.voice already went to vv_capi_load
-		// for realtime-0.5B; ref_audio is per-call only on the C ABI,
-		// so the gallery's `ref_audio:` defaults are re-passed here.
-		refAudio = append(refAudio, v.refAudio...)
-	}
-
-	if req.Language != nil && *req.Language != "" {
-		fmt.Fprintf(os.Stderr,
-			"[vibevoice-cpp] note: TTSRequest.language=%q ignored - vibevoice picks language from the voice prompt\n",
-			*req.Language)
-	}
-
-	const (
-		defaultSteps     = 20
-		defaultMaxFrames = 200
-	)
-	defaultCfg := float32(1.3)
-
-	refPtrs, refKeep := newCStringArray(refAudio)
-	rc := CppTTS(text, voice, refPtrs, int32(len(refPtrs)), dst,
-		int32(defaultSteps), defaultCfg, int32(defaultMaxFrames), 0)
-	// Hold the backing buffers past the cgo call. purego marshals
-	// []*byte by handing the C side the underlying array address; the
-	// pointed-to NUL-terminated bytes must outlive the call.
-	runtime.KeepAlive(refKeep)
-	runtime.KeepAlive(refPtrs)
-	if rc != 0 {
-		return fmt.Errorf("vibevoice-cpp: vv_capi_tts failed (rc=%d)", rc)
-	}
-	return nil
-}
-
-// isRefAudioOverride decides whether a TTSRequest.Voice override should
-// be routed to ref_audio_paths (1.5B path) instead of voice_path
-// (realtime-0.5B). Either a comma-separated list (multi-speaker) or a
-// single .wav clip qualifies; a bare voice .gguf falls through.
-func isRefAudioOverride(s string) bool {
-	if strings.Contains(s, ",") {
-		return true
-	}
-	return strings.HasSuffix(strings.ToLower(s), ".wav")
-}
-
-// newCStringArray builds the **char array vv_capi_tts expects, plus the
-// keep-alive slice the caller must runtime.KeepAlive across the cgo
-// call. A nil/empty input returns (nil, nil) which purego marshals to
-// the C NULL pointer.
-func newCStringArray(in []string) ([]*byte, [][]byte) {
-	if len(in) == 0 {
-		return nil, nil
-	}
-	keep := make([][]byte, len(in))
-	ptrs := make([]*byte, len(in))
-	for i, s := range in {
-		b := make([]byte, len(s)+1)
-		copy(b, s)
-		keep[i] = b
-		ptrs[i] = &b[0]
-	}
-	return ptrs, keep
-}
-
-// asrSegment matches vibevoice's JSON output:
-//
-//	[{"Start":0.0,"End":2.8,"Speaker":0,"Content":"…"}, ...]
-type asrSegment struct {
-	Start   float64 `json:"Start"`
-	End     float64 `json:"End"`
-	Speaker int     `json:"Speaker"`
-	Content string  `json:"Content"`
-}
-
-// callASR invokes vv_capi_asr with a buffer that grows on demand.
-// vv_capi_asr returns: >0 bytes written, 0 no transcript, <0 error or
-// -required_size. We honor the resize protocol once before giving up.
-func (v *VibevoiceCpp) callASR(srcWav string, maxNewTokens int32) (string, error) {
-	const startCap = 256 * 1024
-	buf := make([]byte, startCap)
-	rc := CppASR(srcWav, buf, uint64(len(buf)), maxNewTokens)
-	if rc < 0 {
-		need := -int(rc)
-		if need > 0 && need < (16<<20) && need > len(buf) {
-			buf = make([]byte, need+64)
-			rc = CppASR(srcWav, buf, uint64(len(buf)), maxNewTokens)
-		}
-	}
-	if rc < 0 {
-		return "", fmt.Errorf("vibevoice-cpp: vv_capi_asr failed (rc=%d)", rc)
-	}
-	if rc == 0 {
-		return "", nil
-	}
-	return string(buf[:rc]), nil
-}
-
-// TTSStream is the streaming counterpart to TTS. vibevoice's C ABI is
-// file-only (vv_capi_tts writes a complete WAV), so we synthesize to
-// a tempfile, then emit a streaming-WAV header followed by the PCM
-// body in chunks. The main reason this exists at all is the gRPC
-// server wrapper (pkg/grpc/server.go:TTSStream) blocks on a channel
-// that only this method can close - if we leave the default Base
-// stub in place, every TTSStream call hangs until the client
-// deadline.
-func (v *VibevoiceCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
-	defer close(results)
-	if v.ttsModel == "" {
-		return fmt.Errorf("vibevoice-cpp: TTSStream requested but no realtime model was loaded")
-	}
-	if req.Text == "" {
-		return fmt.Errorf("vibevoice-cpp: TTSStream requires text")
-	}
-
-	tmp, err := os.CreateTemp("", "vibevoice-cpp-stream-*.wav")
-	if err != nil {
-		return fmt.Errorf("vibevoice-cpp: tempfile: %w", err)
-	}
-	dst := tmp.Name()
-	_ = tmp.Close()
-	defer func() { _ = os.Remove(dst) }()
-
-	if err := v.TTS(&pb.TTSRequest{
-		Text:     req.Text,
-		Voice:    req.Voice,
-		Dst:      dst,
-		Language: req.Language,
-	}); err != nil {
-		return err
-	}
-
-	wav, err := os.ReadFile(dst)
-	if err != nil {
-		return fmt.Errorf("vibevoice-cpp: read tempfile: %w", err)
-	}
-
-	// Streaming WAV header: declare 0xFFFFFFFF for chunk sizes so HTTP
-	// clients can start playback before they see the full PCM.
-	const streamingSize = 0xFFFFFFFF
-	hdr := laudio.NewWAVHeaderWithRate(streamingSize, vibevoiceSampleRate)
-	hdr.ChunkSize = streamingSize
-	hdrBuf := make([]byte, 0, laudio.WAVHeaderSize)
-	w := newByteWriter(&hdrBuf)
-	if err := hdr.Write(w); err != nil {
-		return fmt.Errorf("vibevoice-cpp: write WAV header: %w", err)
-	}
-	results <- hdrBuf
-
-	// PCM body: send in ~64 KB slices so the client gets multiple
-	// reply chunks (e2e harness asserts >=2 frames).
-	pcm := laudio.StripWAVHeader(wav)
-	const chunkBytes = 64 * 1024
-	for off := 0; off < len(pcm); off += chunkBytes {
-		end := off + chunkBytes
-		if end > len(pcm) {
-			end = len(pcm)
-		}
-		chunk := make([]byte, end-off)
-		copy(chunk, pcm[off:end])
-		results <- chunk
-	}
-	return nil
-}
-
-// byteWriter adapts a *[]byte to io.Writer so we can hand it to
-// laudio.WAVHeader.Write without allocating a bytes.Buffer.
-type byteWriter struct{ buf *[]byte }
-
-func newByteWriter(b *[]byte) *byteWriter { return &byteWriter{buf: b} }
-func (w *byteWriter) Write(p []byte) (int, error) {
-	*w.buf = append(*w.buf, p...)
-	return len(p), nil
-}
-
-func (v *VibevoiceCpp) AudioTranscription(_ context.Context, req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
-	if v.asrModel == "" {
-		return pb.TranscriptResult{}, fmt.Errorf("vibevoice-cpp: AudioTranscription requested but no ASR model was loaded")
-	}
-	if req.Dst == "" {
-		return pb.TranscriptResult{}, fmt.Errorf("vibevoice-cpp: TranscriptRequest.dst (audio path) is required")
-	}
-
-	wavPath, cleanup, err := prepareWavInput(req.Dst)
-	if err != nil {
-		return pb.TranscriptResult{}, fmt.Errorf("vibevoice-cpp: %w", err)
-	}
-	defer cleanup()
-
-	out, err := v.callASR(wavPath, asrMaxNewTokens)
-	if err != nil {
-		return pb.TranscriptResult{}, err
-	}
-	if out == "" {
-		return pb.TranscriptResult{}, nil
-	}
-
-	var segs []asrSegment
-	if err := json.Unmarshal([]byte(out), &segs); err != nil {
-		fmt.Fprintf(os.Stderr,
-			"[vibevoice-cpp] WARNING: vv_capi_asr returned non-JSON, falling back to single segment: %v\n", err)
-		return pb.TranscriptResult{
-			Segments: []*pb.TranscriptSegment{{Id: 0, Text: strings.TrimSpace(out)}},
-			Text:     strings.TrimSpace(out),
-		}, nil
-	}
-
-	segments := make([]*pb.TranscriptSegment, 0, len(segs))
-	parts := make([]string, 0, len(segs))
-	var duration float32
-	for i, s := range segs {
-		// LocalAI's whisper backend uses int64 100ns ticks for
-		// Start/End (seconds * 1e7); follow the same convention so
-		// consumers can mix vibevoice and whisper transcripts.
-		segments = append(segments, &pb.TranscriptSegment{
-			Id:      int32(i),
-			Text:    s.Content,
-			Start:   int64(s.Start * 1e7),
-			End:     int64(s.End * 1e7),
-			Speaker: fmt.Sprintf("%d", s.Speaker),
-		})
-		parts = append(parts, strings.TrimSpace(s.Content))
-		if float32(s.End) > duration {
-			duration = float32(s.End)
-		}
-	}
-	return pb.TranscriptResult{
-		Segments: segments,
-		Text:     strings.TrimSpace(strings.Join(parts, " ")),
-		Duration: duration,
-	}, nil
-}
-
-// Diarize runs vibevoice's ASR and projects the speaker-labelled segment
-// list it returns natively. vibevoice.cpp's ASR prompt asks the model to
-// emit `[{"Start":..,"End":..,"Speaker":..,"Content":..}]`, so diarization
-// is a by-product of the same pass — we reuse callASR and re-shape.
-//
-// Speaker hints (num_speakers/min/max/threshold) and min_duration_on/off are
-// not actionable here: vibevoice's model picks the speaker count itself and
-// has no clustering knob. The HTTP layer documents this; we accept the
-// fields for API symmetry and ignore them.
-func (v *VibevoiceCpp) Diarize(req *pb.DiarizeRequest) (pb.DiarizeResponse, error) {
-	if v.asrModel == "" {
-		return pb.DiarizeResponse{}, fmt.Errorf("vibevoice-cpp: Diarize requires an ASR model (load options: type=asr)")
-	}
-	if req.Dst == "" {
-		return pb.DiarizeResponse{}, fmt.Errorf("vibevoice-cpp: DiarizeRequest.dst (audio path) is required")
-	}
-
-	wavPath, cleanup, err := prepareWavInput(req.Dst)
-	if err != nil {
-		return pb.DiarizeResponse{}, fmt.Errorf("vibevoice-cpp: %w", err)
-	}
-	defer cleanup()
-
-	out, err := v.callASR(wavPath, asrMaxNewTokens)
-	if err != nil {
-		return pb.DiarizeResponse{}, err
-	}
-	if out == "" {
-		return pb.DiarizeResponse{}, nil
-	}
-
-	var segs []asrSegment
-	if err := json.Unmarshal([]byte(out), &segs); err != nil {
-		// Mirror AudioTranscription's fallback: vibevoice's ASR sometimes
-		// emits free-form text instead of JSON for short or unusual audio.
-		// Surface a single unknown-speaker segment carrying the full text
-		// (when include_text is set) so the caller still gets coverage of
-		// the whole clip rather than a hard failure.
-		fmt.Fprintf(os.Stderr,
-			"[vibevoice-cpp] WARNING: vv_capi_asr returned non-JSON for diarization, falling back to single segment: %v\n", err)
-		text := strings.TrimSpace(out)
-		seg := &pb.DiarizeSegment{Id: 0, Speaker: "0"}
-		if req.IncludeText {
-			seg.Text = text
-		}
-		return pb.DiarizeResponse{
-			Segments:    []*pb.DiarizeSegment{seg},
-			NumSpeakers: 1,
-		}, nil
-	}
-
-	speakers := make(map[int]struct{})
-	segments := make([]*pb.DiarizeSegment, 0, len(segs))
-	var duration float32
-	for i, s := range segs {
-		ds := &pb.DiarizeSegment{
-			Id:      int32(i),
-			Start:   float32(s.Start),
-			End:     float32(s.End),
-			Speaker: fmt.Sprintf("%d", s.Speaker),
-		}
-		if req.IncludeText {
-			ds.Text = strings.TrimSpace(s.Content)
-		}
-		segments = append(segments, ds)
-		speakers[s.Speaker] = struct{}{}
-		if float32(s.End) > duration {
-			duration = float32(s.End)
-		}
-	}
-	return pb.DiarizeResponse{
-		Segments:    segments,
-		NumSpeakers: int32(len(speakers)),
-		Duration:    duration,
-	}, nil
-}
-
-// AudioTranscriptionStream wraps AudioTranscription so the streaming
-// gRPC endpoint (server.go:AudioTranscriptionStream) sees its channel
-// close and the client doesn't sit waiting until deadline. vibevoice's
-// ASR doesn't expose token-level streaming - vv_capi_asr decodes the
-// whole audio and returns a JSON segment list - so we run the offline
-// transcription, emit each segment's content as a delta, then close
-// with a final_result whose Text equals the concatenated deltas (the
-// e2e harness asserts those match).
-func (v *VibevoiceCpp) AudioTranscriptionStream(ctx context.Context, req *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
-	defer close(results)
-	res, err := v.AudioTranscription(ctx, req)
-	if err != nil {
-		return err
-	}
-	var assembled strings.Builder
-	for _, seg := range res.Segments {
-		if seg == nil {
-			continue
-		}
-		txt := strings.TrimSpace(seg.Text)
-		if txt == "" {
-			continue
-		}
-		delta := txt
-		if assembled.Len() > 0 {
-			delta = " " + txt
-		}
-		results <- &pb.TranscriptStreamResponse{Delta: delta}
-		assembled.WriteString(delta)
-	}
-	final := pb.TranscriptResult{
-		Segments: res.Segments,
-		Duration: res.Duration,
-		Language: res.Language,
-		Text:     assembled.String(),
-	}
-	results <- &pb.TranscriptStreamResponse{FinalResult: &final}
-	return nil
-}

backend/go/vibevoice-cpp/main.go

@@ -1,49 +0,0 @@
-package main
-
-// Started internally by LocalAI - one gRPC server per loaded model.
-import (
-	"flag"
-	"os"
-
-	"github.com/ebitengine/purego"
-	grpc "github.com/mudler/LocalAI/pkg/grpc"
-)
-
-var (
-	addr = flag.String("addr", "localhost:50051", "the address to connect to")
-)
-
-type LibFuncs struct {
-	FuncPtr any
-	Name    string
-}
-
-func main() {
-	libName := os.Getenv("VIBEVOICECPP_LIBRARY")
-	if libName == "" {
-		libName = "./libgovibevoicecpp-fallback.so"
-	}
-
-	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
-	if err != nil {
-		panic(err)
-	}
-
-	libFuncs := []LibFuncs{
-		{&CppLoad, "vv_capi_load"},
-		{&CppTTS, "vv_capi_tts"},
-		{&CppASR, "vv_capi_asr"},
-		{&CppUnload, "vv_capi_unload"},
-		{&CppVersion, "vv_capi_version"},
-	}
-
-	for _, lf := range libFuncs {
-		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
-	}
-
-	flag.Parse()
-
-	if err := grpc.StartServer(*addr, &VibevoiceCpp{}); err != nil {
-		panic(err)
-	}
-}

backend/go/vibevoice-cpp/package.sh

@@ -1,58 +0,0 @@
-#!/bin/bash
-
-# Bundle the vibevoice-cpp binary, the per-variant .so files, and the
-# runtime libs the binary depends on so the package is self-contained.
-# Mirrors backend/go/qwen3-tts-cpp/package.sh.
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-mkdir -p $CURDIR/package/lib
-
-cp -avf $CURDIR/vibevoice-cpp $CURDIR/package/
-cp -fv $CURDIR/libgovibevoicecpp-*.so $CURDIR/package/
-cp -fv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ $(uname -s) = "Darwin" ]; then
-    echo "Detected Darwin"
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/go/vibevoice-cpp/run.sh

@@ -1,49 +0,0 @@
-#!/bin/bash
-set -ex
-
-CURDIR=$(dirname "$(realpath $0)")
-
-cd /
-
-echo "CPU info:"
-if [ "$(uname)" != "Darwin" ]; then
-	grep -e "model\sname" /proc/cpuinfo | head -1
-	grep -e "flags" /proc/cpuinfo | head -1
-fi
-
-LIBRARY="$CURDIR/libgovibevoicecpp-fallback.so"
-
-if [ "$(uname)" != "Darwin" ]; then
-	if grep -q -e "\savx\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX    found OK"
-		if [ -e $CURDIR/libgovibevoicecpp-avx.so ]; then
-			LIBRARY="$CURDIR/libgovibevoicecpp-avx.so"
-		fi
-	fi
-
-	if grep -q -e "\savx2\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX2   found OK"
-		if [ -e $CURDIR/libgovibevoicecpp-avx2.so ]; then
-			LIBRARY="$CURDIR/libgovibevoicecpp-avx2.so"
-		fi
-	fi
-
-	if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
-		echo "CPU:    AVX512F found OK"
-		if [ -e $CURDIR/libgovibevoicecpp-avx512.so ]; then
-			LIBRARY="$CURDIR/libgovibevoicecpp-avx512.so"
-		fi
-	fi
-fi
-
-export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
-export VIBEVOICECPP_LIBRARY=$LIBRARY
-
-if [ -f $CURDIR/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using library: $LIBRARY"
-	exec $CURDIR/lib/ld.so $CURDIR/vibevoice-cpp "$@"
-fi
-
-echo "Using library: $LIBRARY"
-exec $CURDIR/vibevoice-cpp "$@"

backend/go/vibevoice-cpp/test.sh

@@ -1,74 +0,0 @@
-#!/bin/bash
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-
-echo "Running vibevoice-cpp backend tests..."
-
-# Required env-vars (set automatically when missing):
-#   VIBEVOICE_MODEL_DIR : directory containing the gguf bundle.
-#   VIBEVOICE_BINARY    : path to the built backend (default ./vibevoice-cpp)
-#
-# Tests skip when the model bundle is absent and the auto-download
-# fails (e.g. no network on the runner) so local devs without HF access
-# still get green compile output.
-
-cd "$CURDIR"
-
-if [ -z "$VIBEVOICE_MODEL_DIR" ]; then
-    export VIBEVOICE_MODEL_DIR="./vibevoice-models"
-
-    if [ ! -d "$VIBEVOICE_MODEL_DIR" ]; then
-        echo "Creating vibevoice-models directory for tests..."
-        mkdir -p "$VIBEVOICE_MODEL_DIR"
-
-        REPO_ID="mudler/vibevoice.cpp-models"
-        echo "Repository: ${REPO_ID}"
-
-        # Q4_K instead of Q8_0 for the ASR model: smaller download
-        # (10 GB vs 14 GB), fits on ubuntu-latest's free disk after the
-        # runner image is loaded. The unit/closed-loop test only needs
-        # decode quality, not Q8_0 precision.
-        FILES=(
-            "vibevoice-realtime-0.5B-q8_0.gguf"
-            "vibevoice-asr-q4_k.gguf"
-            "tokenizer.gguf"
-            "voice-en-Carter_man.gguf"
-        )
-
-        BASE_URL="https://huggingface.co/${REPO_ID}/resolve/main"
-
-        download_ok=1
-        for file in "${FILES[@]}"; do
-            dest="${VIBEVOICE_MODEL_DIR}/${file}"
-            if [ -f "${dest}" ]; then
-                echo "  [skip] ${file} (already exists)"
-            else
-                echo "  [download] ${file}..."
-                if ! curl -fL -o "${dest}" "${BASE_URL}/${file}" --progress-bar; then
-                    echo "  [warn] failed to download ${file} - network or HF unavailable"
-                    rm -f "${dest}"
-                    download_ok=0
-                    break
-                fi
-                echo "  [done] ${file}"
-            fi
-        done
-
-        if [ "$download_ok" != "1" ]; then
-            echo "vibevoice-cpp: model bundle unavailable - tests will skip model-dependent cases."
-            unset VIBEVOICE_MODEL_DIR
-        fi
-    fi
-fi
-
-# Ensure the per-variant .so the binary will dlopen actually exists -
-# without one, every test will hit a Dlopen panic during server start.
-if [ ! -f "${CURDIR}/libgovibevoicecpp-fallback.so" ]; then
-    echo "vibevoice-cpp: libgovibevoicecpp-fallback.so missing - run \`make\` first."
-    exit 1
-fi
-
-go test -v -timeout 900s .
-
-echo "All vibevoice-cpp tests passed."