Fix tests

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.agents/adding-backends.md

@@ -8,7 +8,6 @@ Create the backend directory under the appropriate location:
 - **Python backends**: `backend/python/<backend-name>/`
 - **Go backends**: `backend/go/<backend-name>/`
 - **C++ backends**: `backend/cpp/<backend-name>/`
-- **Rust backends**: `backend/rust/<backend-name>/`
 
 For Python backends, you'll typically need:
 - `backend.py` - Main gRPC server implementation
@@ -19,70 +18,9 @@ For Python backends, you'll typically need:
 - `run.sh` - Runtime script
 - `test.py` / `test.sh` - Test files
 
-For Rust backends, you'll typically need (see `backend/rust/kokoros/` as a reference):
-- `Cargo.toml` - Crate manifest; depend on the upstream project as a submodule under `sources/`
-- `build.rs` - Invokes `tonic_build` to generate gRPC stubs from `backend/backend.proto` (use the `BACKEND_PROTO_PATH` env var so the Makefile can inject the canonical copy)
-- `src/` - The gRPC server implementation (implement `Backend` via `tonic`)
-- `Makefile` - Copies `backend.proto` into the crate, runs `cargo build --release`, then `package.sh`
-- `package.sh` - Uses `ldd` to bundle the binary's dynamic deps and `ld.so` into `package/lib/`
-- `run.sh` - Sets `LD_LIBRARY_PATH`/`SSL_CERT_DIR` and execs the binary via the bundled `lib/ld.so`
-- `sources/<UpstreamProject>/` - Git submodule with the upstream Rust crate
+## 2. Add Build Configurations to `.github/workflows/backend.yml`
 
-## 2. Add Build Configurations to `.github/backend-matrix.yml`
-
-The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `backend.yml` itself). `backend.yml` (master push) and `backend_pr.yml` (PR) load it via `scripts/changed-backends.js`, which also handles per-file path filtering so only touched backends rebuild on PRs and master pushes alike. Add build matrix entries to `.github/backend-matrix.yml` for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
-
-**Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
-
-**`scripts/changed-backends.js` registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/changed-backends.js:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
-
-```js
-if (item.dockerfile.endsWith("<your-dockerfile-suffix>")) {
-    return `backend/cpp/<your-backend>/`;   // or backend/python|go|rust/...
-}
-```
-
-The `endsWith()` test is against the matrix entry's `dockerfile:` value (e.g. `./backend/Dockerfile.ds4` → `endsWith("ds4")`). Specificity order matters here just like it does for importers: more-specific suffixes go BEFORE more-generic ones (e.g. `ds4` before `llama-cpp` even though both end with letters, because some upstream might one day call itself `super-ds4-llama-cpp`). Verify locally before pushing:
-
-```bash
-# Confirm your dockerfile suffix is unique enough
-node -e "
-const yaml = require('js-yaml'); const fs = require('fs');
-const m = yaml.load(fs.readFileSync('.github/backend-matrix.yml','utf8'));
-for (const e of m.include.filter(e => e.backend === '<your-backend>')) {
-  console.log(e.dockerfile, '->', e.dockerfile.endsWith('<suffix>'));
-}"
-```
-
-A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/changed-backends.js`.
-
-**`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=<sha>` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`):
-
-```yaml
-# .github/workflows/bump_deps.yaml
-matrix:
-  include:
-    - repository: "antirez/ds4"
-      variable: "DS4_VERSION"
-      branch: "main"
-      file: "backend/cpp/ds4/Makefile"
-```
-
-And the corresponding Makefile shape (mirror `backend/cpp/llama-cpp/Makefile`):
-
-```makefile
-DS4_VERSION?=ae302c2fa18cc6d9aefc021d0f27ae03c9ad2fc0
-DS4_REPO?=https://github.com/antirez/ds4
-...
-ds4:
-	mkdir -p ds4
-	cd ds4 && git init -q && \
-	git remote add origin $(DS4_REPO) && \
-	git fetch --depth 1 origin $(DS4_VERSION) && \
-	git checkout FETCH_HEAD
-```
-
-If you have a `prepare.sh` doing the clone, delete it — the recipe belongs in the Makefile target so `make purge && make` works as a clean-and-rebuild and so the bump bot finds the pin.
+Add build matrix entries for each platform/GPU type you want to support. Look at similar backends (e.g., `chatterbox`, `faster-whisper`) for reference.
 
 **Placement in file:**
 - CPU builds: Add after other CPU builds (e.g., after `cpu-chatterbox`)
@@ -90,18 +28,10 @@ If you have a `prepare.sh` doing the clone, delete it — the recipe belongs in
 - CUDA 13 builds: Add after other CUDA 13 builds (e.g., after `gpu-nvidia-cuda-13-chatterbox`)
 
 **Additional build types you may need:**
-- ROCm/HIP: Use `build-type: 'hipblas'` with `base-image: "rocm/dev-ubuntu-24.04:7.2.1"`
-- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"`
+- ROCm/HIP: Use `build-type: 'hipblas'` with `base-image: "rocm/dev-ubuntu-24.04:6.4.4"`
+- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"`
 - L4T (ARM): Use `build-type: 'l4t'` with `platforms: 'linux/arm64'` and `runs-on: 'ubuntu-24.04-arm'`
 
-**Per-arch native builds (`linux/amd64` + `linux/arm64`):**
-
-Multi-arch backends are NOT a single matrix entry with `platforms: 'linux/amd64,linux/arm64'`. Instead, add **two** entries — one with `platforms: 'linux/amd64'` + `platform-tag: 'amd64'` + `runs-on: 'ubuntu-latest'`, one with `platforms: 'linux/arm64'` + `platform-tag: 'arm64'` + `runs-on: 'ubuntu-24.04-arm'` — both sharing the same `tag-suffix`. The script detects the shared `tag-suffix` and emits a `merge-matrix` entry, so `backend-merge-jobs` (in `backend.yml`/`backend_pr.yml`) automatically assembles the manifest list from per-arch digest artifacts. See `-cpu-faster-whisper` in `.github/backend-matrix.yml` for a reference shape.
-
-**llama-cpp / ik-llama-cpp / turboquant variants only — `builder-base-image`:**
-
-Entries whose `dockerfile` is `./backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` must also set a `builder-base-image` field pointing at a prebuilt base from `quay.io/go-skynet/ci-cache:base-grpc-*` (CI builds these via `.github/workflows/base-images.yml`). The mapping is by `(build-type, platforms)` — see existing entries for the pattern. CI uses these prebuilt bases to skip the gRPC compile (~25–35 min cold). Local `make backends/<name>` ignores `builder-base-image` and uses the from-source path inside the Dockerfile, so you don't need quay access for local builds.
-
 ## 3. Add Backend Metadata to `backend/index.yaml`
 
 **Step 3a: Add Meta Definition**
@@ -126,28 +56,24 @@ Add `backends/<backend-name>` to the `.NOTPARALLEL` line (around line 2) to prev
 
 **Step 4b: Add to `prepare-test-extra`**
 
-Add the backend to the `prepare-test-extra` target to prepare it for testing. Use the path matching your language bucket (`backend/python/`, `backend/go/`, `backend/rust/`, …):
+Add the backend to the `prepare-test-extra` target (around line 312) to prepare it for testing:
 
 ```makefile
 prepare-test-extra: protogen-python
 	...
-	$(MAKE) -C backend/<lang>/<backend-name>
+	$(MAKE) -C backend/python/<backend-name>
 ```
 
-For Rust backends the target is usually the crate build target itself (e.g. `$(MAKE) -C backend/rust/<backend-name> <backend-name>-grpc`) so the binary is in place before `test` runs.
-
 **Step 4c: Add to `test-extra`**
 
-Add the backend to the `test-extra` target to run its tests — applies to Go and Rust backends too, not only Python:
+Add the backend to the `test-extra` target (around line 319) to run its tests:
 
 ```makefile
 test-extra: prepare-test-extra
 	...
-	$(MAKE) -C backend/<lang>/<backend-name> test
+	$(MAKE) -C backend/python/<backend-name> test
 ```
 
-Each backend's own `Makefile` should define a `test` target so this line works regardless of language. Integration tests that need large model downloads should be gated behind an env var (see `backend/rust/kokoros/`'s `KOKOROS_MODEL_PATH` pattern) so CI only runs unit tests.
-
 **Step 4d: Add Backend Definition**
 
 Add a backend definition variable in the backend definitions section (around line 428-457). The format depends on the backend type:
@@ -167,13 +93,6 @@ BACKEND_<BACKEND_NAME> = <backend-name>|python|./backend|false|true
 BACKEND_<BACKEND_NAME> = <backend-name>|golang|.|false|true
 ```
 
-**For Rust backends**:
-```makefile
-BACKEND_<BACKEND_NAME> = <backend-name>|rust|.|false|true
-```
-
-The language field (`python`/`golang`/`rust`/…) must match a `backend/Dockerfile.<lang>` file.
-
 **Step 4e: Generate Docker Build Target**
 
 Add an eval call to generate the docker-build target (around line 480-501):
@@ -201,7 +120,7 @@ docker-build-backends: ... docker-build-<backend-name>
 After adding a new backend, verify:
 
 - [ ] Backend directory structure is complete with all necessary files
-- [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
+- [ ] Build configurations added to `.github/workflows/backend.yml` for all desired platforms
 - [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
 - [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
 - [ ] Tag suffixes match between workflow file and index.yaml
@@ -210,53 +129,6 @@ After adding a new backend, verify:
 - [ ] No Makefile syntax errors (check with linter)
 - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
 
-## Bundling runtime shared libraries (`package.sh`)
-
-The final `Dockerfile.python` stage is `FROM scratch` — there is no system `libc`, no `apt`, no fallback library path. Only files explicitly copied from the builder stage end up in the backend image. That means any runtime `dlopen` your backend (or its Python deps) needs **must** be packaged into `${BACKEND}/lib/`.
-
-Pattern:
-
-1. Make sure the library is installed in the builder stage of `backend/Dockerfile.python` (add it to the top-level `apt-get install`).
-2. Drop a `package.sh` in your backend directory that copies the library — and its soname symlinks — into `$(dirname $0)/lib`. See `backend/python/vllm/package.sh` for a reference implementation that walks `/usr/lib/x86_64-linux-gnu`, `/usr/lib/aarch64-linux-gnu`, etc.
-3. `Dockerfile.python` already runs `package.sh` automatically if it exists, after `package-gpu-libs.sh`.
-4. `libbackend.sh` automatically prepends `${EDIR}/lib` to `LD_LIBRARY_PATH` at run time, so anything packaged this way is found by `dlopen`.
-
-How to find missing libs: when a Python module silently fails to register torch ops or you see `AttributeError: '_OpNamespace' '...' object has no attribute '...'`, run the backend image's Python with `LD_DEBUG=libs` to see which `dlopen` failed. The filename in the error message (e.g. `libnuma.so.1`) is what you need to package.
-
-To verify packaging works without trusting the host:
-
-```bash
-make docker-build-<backend>
-CID=$(docker create --entrypoint=/run.sh local-ai-backend:<backend>)
-docker cp $CID:/lib /tmp/check && docker rm $CID
-ls /tmp/check    # expect the bundled .so files + symlinks
-```
-
-Then boot it inside a fresh `ubuntu:24.04` (which intentionally does *not* have the lib installed) to confirm it actually loads from the backend dir.
-
-## Importer integration
-
-When you add a new backend, you MUST also make it importable via the model import form (`/import-model`). The import form dropdown is sourced dynamically from `GET /backends/known` — it reads the importer registry at `core/gallery/importers/importers.go`, so the steps below are the ONLY way to make your backend show up.
-
-Required steps:
-
-1. **If your backend has unambiguous detection signals** (unique file extension, HF `pipeline_tag`, unique repo name pattern, unique artefact like `modules.json`):
-   - Create an importer file at `core/gallery/importers/<backend>.go` following the Match/Import pattern in `llama-cpp.go`.
-   - Register it in `importers.go:defaultImporters` in **specificity order** — more specific detectors must appear BEFORE more generic ones (e.g. `sentencetransformers` before `transformers`, `stablediffusion-ggml` before `llama-cpp`, `vllm-omni` before `vllm`). First match wins.
-2. **If your backend is a drop-in replacement** (same artefacts as another backend, e.g. `ik-llama-cpp` and `turboquant` both consume GGUF the same way `llama-cpp` does):
-   - Do NOT create a new importer. Extend the existing importer's `Import()` to swap the emitted `backend:` field when `preferences.backend` matches. See `llama-cpp.go` for the pattern.
-3. **If your backend has no reliable auto-detect signal** (preference-only — e.g. `sglang`, `tinygrad`, `whisperx`):
-   - Do NOT create an importer. Instead add the backend name to the curated pref-only slice in `core/http/endpoints/localai/backend.go` that feeds `/backends/known`. A single line addition.
-4. **Always** add a table-driven test in `core/gallery/importers/importers_test.go` (Ginkgo/Gomega):
-   - Use a real public HuggingFace repo URI as the test fixture (existing tests already hit the live HF API — follow that pattern).
-   - Cover detection (auto-match without preferences), preference-override (explicit `backend:` in preferences wins), and — if the backend's modality has a common `pipeline_tag` but ambiguous artefacts — an ambiguity test asserting `errors.Is(err, importers.ErrAmbiguousImport)`.
-
-Rules of thumb:
-
-- When in doubt, lean pref-only. A wrong auto-detect is worse than a forced preference.
-- Never silently emit a modality mismatch (e.g. emit `llama-cpp` for a TTS repo because `.gguf` is present). Return `ErrAmbiguousImport` instead.
-- Registration order is the single most common source of bugs. Check by running `go test ./core/gallery/importers/...` — the existing suite will fail if you've shadowed a pre-existing detector.
-
 ## 6. Example: Adding a Python Backend
 
 For reference, when `moonshine` was added:

.agents/adding-gallery-models.md

@@ -1,111 +0,0 @@
-# Adding GGUF Models from HuggingFace to the Gallery
-
-When adding a GGUF model from HuggingFace to the LocalAI model gallery, follow this guide.
-
-## Gallery file
-
-All models are defined in `gallery/index.yaml`. Find the appropriate section (embedding models near other embeddings, chat models near similar chat models) and add a new entry.
-
-## Getting the SHA256
-
-GGUF files on HuggingFace expose their SHA256 via the `x-linked-etag` HTTP header. Fetch it with:
-
-```bash
-curl -sI "https://huggingface.co/<org>/<repo>/resolve/main/<filename>.gguf" | grep -i x-linked-etag
-```
-
-The value (without quotes) is the SHA256 hash. Example:
-
-```bash
-curl -sI "https://huggingface.co/ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/resolve/main/embeddinggemma-300m-qat-Q8_0.gguf" | grep -i x-linked-etag
-# x-linked-etag: "6fa0c02a9c302be6f977521d399b4de3a46310a4f2621ee0063747881b673f67"
-```
-
-**Important**: Pay attention to exact filename casing — HuggingFace filenames are case-sensitive (e.g., `Q8_0` vs `q8_0`). Check the repo's file listing to get the exact name.
-
-## Entry format — Embedding models
-
-Embedding models use `gallery/virtual.yaml` as the base config and set `embeddings: true`:
-
-```yaml
-- name: "model-name"
-  url: github:mudler/LocalAI/gallery/virtual.yaml@master
-  urls:
-    - https://huggingface.co/<original-model-org>/<original-model-name>
-    - https://huggingface.co/<gguf-org>/<gguf-repo-name>
-  description: |
-    Short description of the model, its size, and capabilities.
-  tags:
-    - embeddings
-  overrides:
-    backend: llama-cpp
-    embeddings: true
-    parameters:
-      model: <filename>.gguf
-  files:
-    - filename: <filename>.gguf
-      uri: huggingface://<gguf-org>/<gguf-repo-name>/<filename>.gguf
-      sha256: <sha256-hash>
-```
-
-## Entry format — Chat/LLM models
-
-Chat models typically reference a template config (e.g., `gallery/gemma.yaml`, `gallery/chatml.yaml`) that defines the prompt format. Use YAML anchors (`&name` / `*name`) if adding multiple quantization variants of the same model:
-
-```yaml
-- &model-anchor
-  url: "github:mudler/LocalAI/gallery/<template>.yaml@master"
-  name: "model-name"
-  icon: https://example.com/icon.png
-  license: <license>
-  urls:
-    - https://huggingface.co/<org>/<model>
-    - https://huggingface.co/<gguf-org>/<gguf-repo>
-  description: |
-    Model description.
-  tags:
-    - llm
-    - gguf
-    - gpu
-    - cpu
-  overrides:
-    parameters:
-      model: <filename>-Q4_K_M.gguf
-  files:
-    - filename: <filename>-Q4_K_M.gguf
-      sha256: <sha256>
-      uri: huggingface://<gguf-org>/<gguf-repo>/<filename>-Q4_K_M.gguf
-```
-
-To add a variant (e.g., different quantization), use YAML merge:
-
-```yaml
-- !!merge <<: *model-anchor
-  name: "model-name-q8"
-  overrides:
-    parameters:
-      model: <filename>-Q8_0.gguf
-  files:
-    - filename: <filename>-Q8_0.gguf
-      sha256: <sha256>
-      uri: huggingface://<gguf-org>/<gguf-repo>/<filename>-Q8_0.gguf
-```
-
-## Available template configs
-
-Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture:
-
-- `gemma.yaml` — Gemma-family models (gemma, embeddinggemma, etc.)
-- `chatml.yaml` — ChatML format (many Mistral/OpenHermes models)
-- `deepseek.yaml` — DeepSeek models
-- `virtual.yaml` — Minimal base (good for embedding models that don't need chat templates)
-
-## Checklist
-
-1. **Find the GGUF file** on HuggingFace — note exact filename (case-sensitive)
-2. **Get the SHA256** using the `curl -sI` + `x-linked-etag` method above
-3. **Choose the right template** config from `gallery/` based on model architecture
-4. **Add the entry** to `gallery/index.yaml` near similar models
-5. **Set `embeddings: true`** if it's an embedding model
-6. **Include both URLs** — the original model page and the GGUF repo
-7. **Write a description** — mention model size, capabilities, and quantization type

.agents/ai-coding-assistants.md

@@ -1,101 +0,0 @@
-# AI Coding Assistants
-
-This document provides guidance for AI tools and developers using AI
-assistance when contributing to LocalAI.
-
-**LocalAI follows the same guidelines as the Linux kernel project for
-AI-assisted contributions.** See the upstream policy here:
-<https://docs.kernel.org/process/coding-assistants.html>
-
-The rules below mirror that policy, adapted to LocalAI's license and
-project layout. If anything is unclear, the kernel document is the
-authoritative reference for intent.
-
-AI tools helping with LocalAI development should follow the standard
-project development process:
-
-- [CONTRIBUTING.md](../CONTRIBUTING.md) — development workflow, commit
-  conventions, and PR guidelines
-- [.agents/coding-style.md](coding-style.md) — code style, editorconfig,
-  logging, and documentation conventions
-- [.agents/building-and-testing.md](building-and-testing.md) — build and
-  test procedures
-
-## Licensing and Legal Requirements
-
-All contributions must comply with LocalAI's licensing requirements:
-
-- LocalAI is licensed under the **MIT License** — see the [LICENSE](../LICENSE)
-  file
-- New source files should use the SPDX license identifier `MIT` where
-  applicable to the file type
-- Contributions must be compatible with the MIT License and must not
-  introduce code under incompatible licenses (e.g., GPL) without an
-  explicit discussion with maintainers
-
-## Signed-off-by and Developer Certificate of Origin
-
-**AI agents MUST NOT add `Signed-off-by` tags.** Only humans can legally
-certify the Developer Certificate of Origin (DCO). The human submitter
-is responsible for:
-
-- Reviewing all AI-generated code
-- Ensuring compliance with licensing requirements
-- Adding their own `Signed-off-by` tag (when the project requires DCO)
-  to certify the contribution
-- Taking full responsibility for the contribution
-
-AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
-A human reviewer owns the contribution; the AI's involvement is recorded
-via `Assisted-by` (see below).
-
-## Attribution
-
-When AI tools contribute to LocalAI development, proper attribution helps
-track the evolving role of AI in the development process. Contributions
-should include an `Assisted-by` tag in the commit message trailer in the
-following format:
-
-```
-Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
-```
-
-Where:
-
-- `AGENT_NAME` — name of the AI tool or framework (e.g., `Claude`,
-  `Copilot`, `Cursor`)
-- `MODEL_VERSION` — specific model version used (e.g.,
-  `claude-opus-4-7`, `gpt-5`)
-- `[TOOL1] [TOOL2]` — optional specialized analysis tools invoked by the
-  agent (e.g., `golangci-lint`, `staticcheck`, `go vet`)
-
-Basic development tools (git, go, make, editors) should **not** be listed.
-
-### Example
-
-```
-fix(llama-cpp): handle empty tool call arguments
-
-Previously the parser panicked when the model returned a tool call with
-an empty arguments object. Fall back to an empty JSON object in that
-case so downstream consumers receive a valid payload.
-
-Assisted-by: Claude:claude-opus-4-7 golangci-lint
-Signed-off-by: Jane Developer <jane@example.com>
-```
-
-## Scope and Responsibility
-
-Using an AI assistant does not reduce the contributor's responsibility.
-The human submitter must:
-
-- Understand every line that lands in the PR
-- Verify that generated code compiles, passes tests, and follows the
-  project style
-- Confirm that any referenced APIs, flags, or file paths actually exist
-  in the current tree (AI models may hallucinate identifiers)
-- Not submit AI output verbatim without review
-
-Reviewers may ask for clarification on any change regardless of how it
-was produced. "An AI wrote it" is not an acceptable answer to a design
-question.

.agents/api-endpoints-and-auth.md

@@ -2,8 +2,6 @@
 
 This guide covers how to add new API endpoints and properly integrate them with the auth/permissions system.
 
-> **Before you ship a new endpoint or capability surface**, re-read the [checklist at the bottom of this file](#checklist). LocalAI advertises its feature surface in several independent places — miss any one of them and clients/admins/UI won't know the endpoint exists.
-
 ## Architecture overview
 
 Authentication and authorization flow through three layers:
@@ -236,76 +234,6 @@ Use these HTTP status codes:
 
 If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
 
-## Advertising surfaces — where to register a new capability
-
-Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.
-
-### 1. Swagger `@Tags` annotation (mandatory)
-
-Every handler needs a swagger block so the endpoint appears in `/swagger/index.html` and in the `/api/instructions` output. The `@Tags` value is what groups the endpoint into a capability area:
-
-```go
-// MyEndpoint does X.
-// @Summary Do X.
-// @Tags my-capability
-// @Param request body schema.MyRequest true "payload"
-// @Success 200 {object} schema.MyResponse "Response"
-// @Router /v1/my-endpoint [post]
-func MyEndpoint(...) echo.HandlerFunc { ... }
-```
-
-Use an existing tag when the endpoint extends an existing area (e.g. `audio`, `images`, `face-recognition`). Create a new tag only when the endpoint introduces a genuinely new capability surface — and in that case, also register it in step 2.
-
-After adding endpoints, regenerate the embedded spec so the runtime serves it:
-
-```bash
-make protogen-go         # ensures gRPC codegen is fresh first
-make swagger             # regenerates swagger/swagger.json
-```
-
-### 2. `/api/instructions` registry (for new capability areas)
-
-`core/http/endpoints/localai/api_instructions.go` defines `instructionDefs` — a lightweight, machine-readable index of capability areas that groups swagger endpoints by tag. It's the primary discovery surface for agents and SDKs ("what can this server do?").
-
-**When to update:** only when adding a new capability area (a new swagger tag). Existing-tag additions automatically surface without any change here.
-
-Add an entry to `instructionDefs`:
-
-```go
-{
-    Name:        "my-capability",             // URL segment at /api/instructions/my-capability
-    Description: "Short sentence describing the capability",
-    Tags:        []string{"my-capability"},   // must match swagger @Tags
-    Intro:       "Optional gotcha/context that isn't in the swagger descriptions (caveats, defaults, cross-references to other endpoints).",
-},
-```
-
-Also bump the expected-length count in `api_instructions_test.go` and add the name to the `ContainElements` assertion.
-
-### 3. `capabilities.js` symbol (for new model-config FLAG_* flags)
-
-If your feature needs a new `FLAG_*` usecase flag in `core/config/model_config.go` (so users can filter gallery models by it, and so `/v1/models` surfaces it), you need to update **all** of:
-
-- `Usecase<Name>` string constant in `core/config/backend_capabilities.go`
-- `UsecaseInfoMap` entry mapping the string to its flag + gRPC method
-- `FLAG_<NAME>` bitmask in `core/config/model_config.go`
-- `GetAllModelConfigUsecases()` map entry (otherwise the YAML loader silently ignores the string)
-- `ModalityGroups` membership if the flag should affect `IsMultimodal()` (e.g. realtime_audio is in both speech-input and audio-output groups so a lone flag still reads as multimodal)
-- `GuessUsecases()` branch listing the backends that own this capability
-- `usecaseFilters` in `core/http/routes/ui_api.go` (drives the gallery filter dropdown)
-- `Models.jsx` `FILTERS` array + matching `filters.<camelCase>` i18n key in `core/http/react-ui/public/locales/en/models.json`
-- `core/http/react-ui/src/utils/capabilities.js`:
-
-```js
-export const CAP_MY_CAPABILITY = 'FLAG_MY_CAPABILITY'
-```
-
-React pages that want to filter the ModelSelector by capability import this symbol. Declare it even if you're not building the UI page yet — the declaration keeps the Go/JS vocabularies in sync.
-
-### 4. `docs/content/` (user-facing documentation)
-
-A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features and an entry in `docs/content/whats-new.md`. See the pattern used by `face-recognition.md` / `object-detection.md`.
-
 ## Path protection rules
 
 The global auth middleware classifies paths as API paths or non-API paths:
@@ -320,36 +248,12 @@ If you add endpoints under a new top-level path prefix, add it to `isAPIPath()`
 
 When adding a new endpoint:
 
-**Routing & auth**
 - [ ] Handler in `core/http/endpoints/`
 - [ ] Route registered in appropriate `core/http/routes/` file
 - [ ] Auth level chosen: public / standard / admin / feature-gated
-- [ ] Entry added to `RouteFeatureRegistry` in `core/http/auth/features.go` (one row per route/method — all /v1/* routes gate through this, not per-route middleware)
-- [ ] If new feature: constant in `permissions.go`, added to the right slice (`APIFeatures` default-ON / `AgentFeatures` default-OFF), metadata in `features.go` `*FeatureMetas()`
-- [ ] If feature uses group middleware: wired in `core/http/app.go` and passed to the route registration function
+- [ ] If feature-gated: constant in `permissions.go`, metadata in `features.go`, middleware in `app.go`
 - [ ] If new path prefix: added to `isAPIPath()` in `middleware.go`
+- [ ] If OpenAI-compatible: entry in `RouteFeatureRegistry`
 - [ ] If token-counting: `usageMiddleware` added to middleware chain
-
-**Advertising surfaces (easy to miss — see the [Advertising surfaces](#advertising-surfaces--where-to-register-a-new-capability) section)**
-- [ ] Swagger block on the handler: `@Summary`, `@Tags`, `@Param`, `@Success`, `@Router`
-- [ ] If new capability area (new swagger tag): entry in `instructionDefs` in `core/http/endpoints/localai/api_instructions.go` + test count bumped in `api_instructions_test.go`
-- [ ] If new `FLAG_*` usecase flag: matching `CAP_*` symbol exported from `core/http/react-ui/src/utils/capabilities.js`
-- [ ] `docs/content/features/<feature>.md` created; cross-links from related feature pages; entry in `docs/content/whats-new.md`
-
-**Quality**
-- [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
+- [ ] Error responses use `schema.ErrorResponse` format
 - [ ] Tests cover both authenticated and unauthenticated access
-- [ ] Swagger regenerated (`make swagger`) if you changed any `@Router`/`@Tags`/`@Param` annotation
-
-## Companion: MCP admin tool surface
-
-**Required for admin endpoints.** Every new admin endpoint MUST be considered for the MCP admin tool surface — the REST API and the MCP tool catalog can drift silently otherwise, and both the LocalAI Assistant chat modality and the standalone `local-ai mcp-server` rely on `pkg/mcp/localaitools/` to mirror REST.
-
-Two outcomes are acceptable; one is not:
-
-- **Tool added.** The new endpoint is something an admin would manage conversationally (install, list, edit, toggle, upgrade). Follow the full checklist in [.agents/localai-assistant-mcp.md](localai-assistant-mcp.md): add a `LocalAIClient` interface method, implement it in both `inproc` and `httpapi`, register the tool with a `Tool*` constant, update the skill prompts, **and add the route to `toolToHTTPRoute` in `pkg/mcp/localaitools/coverage_test.go`**.
-- **Tool deliberately skipped.** The endpoint is internal/diagnostic and adding a chat path would be misleading. Document the decision in the PR description; no code action.
-- **Forgot.** This breaks the contract. The `TestToolHTTPRouteMappingComplete` test in `pkg/mcp/localaitools` is a partial guard (it checks every `Tool*` has a route mapping), but it does NOT detect new REST endpoints without a tool — that's still a process check on the PR author.
-
-**Add to the bottom of the checklist below**:
-- [ ] If admin: decided whether MCP coverage is needed; if yes, tool registered + map updated; if no, skip-reason in PR description.

.agents/building-and-testing.md

@@ -8,10 +8,9 @@ Let's say the user wants to build a particular backend for a given platform. For
 
 - The Makefile has targets like `docker-build-coqui` created with `generate-docker-build-target` at the time of writing. Recently added backends may require a new target.
 - At a minimum we need to set the BUILD_TYPE, BASE_IMAGE build-args
-  - Use `.github/backend-matrix.yml` as a reference — it's the data-only YAML that lists every backend variant's `build-type`, `base-image`, `platforms`, etc. (`backend.yml` and `backend_pr.yml` consume it via `scripts/changed-backends.js`).
-  - l4t and cublas also require the CUDA major and minor version.
-  - For llama-cpp / ik-llama-cpp / turboquant the matrix also sets `builder-base-image` pointing at a prebuilt `quay.io/go-skynet/ci-cache:base-grpc-*` tag. Local `make backends/<name>` defaults to `BUILDER_TARGET=builder-fromsource` and doesn't need it — the Dockerfile's from-source stage installs everything itself.
-- You can pretty print a command like `DOCKER_MAKEFLAGS=-j$(nproc --ignore=1) BUILD_TYPE=hipblas BASE_IMAGE=rocm/dev-ubuntu-24.04:7.2.1 make docker-build-coqui`
+  - Use .github/workflows/backend.yml as a reference it lists the needed args in the `include` job strategy matrix
+  - l4t and cublas also requires the CUDA major and minor version
+- You can pretty print a command like `DOCKER_MAKEFLAGS=-j$(nproc --ignore=1) BUILD_TYPE=hipblas BASE_IMAGE=rocm/dev-ubuntu-24.04:6.4.4 make docker-build-coqui`
 - Unless the user specifies that they want you to run the command, then just print it because not all agent frontends handle long running jobs well and the output may overflow your context
 - The user may say they want to build AMD or ROCM instead of hipblas, or Intel instead of SYCL or NVIDIA insted of l4t or cublas. Ask for confirmation if there is ambiguity.
 - Sometimes the user may need extra parameters to be added to `docker build` (e.g. `--platform` for cross-platform builds or `--progress` to view the full logs), in which case you can generate the `docker build` command directly.

.agents/ci-caching.md

@@ -1,250 +0,0 @@
-# CI Build Caching
-
-Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache plus a layered set of prebuilt base images. This file explains how the cache is laid out, what invalidates it, and how to bypass it.
-
-## Workflow surfaces
-
-| Workflow | Purpose | Triggers |
-|---|---|---|
-| `.github/workflows/backend.yml` | Backend container images on master | `push` to master + tags, weekly Sunday cron, `workflow_dispatch` |
-| `.github/workflows/backend_pr.yml` | Backend container images on PRs | `pull_request` |
-| `.github/workflows/backend_build.yml` | Reusable: builds one backend (one arch) by digest | `workflow_call` from above |
-| `.github/workflows/backend_merge.yml` | Reusable: assembles per-arch digests into a multi-arch manifest list | `workflow_call` |
-| `.github/workflows/backend_build_darwin.yml` | Reusable: macOS-native backend builds | `workflow_call` |
-| `.github/workflows/image.yml` / `image-pr.yml` | Root LocalAI image (push / PR) | push / PR |
-| `.github/workflows/image_build.yml` / `image_merge.yml` | Reusable: per-arch root-image build + merge | `workflow_call` |
-| `.github/workflows/base-images.yml` | Builds the prebuilt `base-grpc-*` builder bases | Saturdays 05:00 UTC cron, `workflow_dispatch`, master push touching `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/apt-mirror.sh`, or this workflow |
-
-The matrix that drives `backend.yml` / `backend_pr.yml` lives in **`.github/backend-matrix.yml`** (data-only YAML, not embedded in the workflow). `scripts/changed-backends.js` parses it, applies path-filter logic against the PR diff (PR events) or the GitHub Compare API (push events), and emits the filtered matrix plus a `merge-matrix` for backends with multiple per-arch entries.
-
-## Cache layout
-
-- **Cache registry**: `quay.io/go-skynet/ci-cache`
-- **One tag per matrix entry per arch**, derived from `tag-suffix` and `platform-tag`:
-  - Backend builds (`backend_build.yml`): `cache<tag-suffix>-<platform-tag>`
-    - e.g. `cache-cpu-faster-whisper-amd64`, `cache-cpu-faster-whisper-arm64`, `cache-gpu-nvidia-cuda-13-llama-cpp-amd64`
-  - Root image builds (`image_build.yml`): `cache-localai<tag-suffix>-<platform-tag>` (with a `-core` placeholder when `tag-suffix` is empty, so `cache-localai-core-amd64` for the core image)
-  - Pre-built base images (`base-images.yml`): `cache-base-grpc-<variant>` (one per `(BUILD_TYPE, arch)` permutation)
-- Each tag stores a multi-arch BuildKit cache manifest (`mode=max`), so every intermediate stage is re-usable, not just the final image.
-
-The per-arch suffix exists because amd64 and arm64 builds produce different intermediate content; sharing one cache key would thrash on every cross-arch rebuild.
-
-## Read/write semantics
-
-| Trigger | `cache-from` | `cache-to` |
-|---|---|---|
-| `push` to `master` / tag / cron / dispatch | yes | yes (`mode=max,ignore-error=true`) |
-| `pull_request` | yes | **no** |
-
-PR builds read master's warm cache but never write — this prevents PRs from polluting the shared cache with their experimental state. After merge, the master build for that matrix entry refreshes the cache.
-
-`ignore-error=true` on the write side means a transient quay push failure does not fail the build; the next master push retries.
-
-## Pre-built base images (`base-grpc-*`)
-
-The C++ backend Dockerfiles (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) compile gRPC from source. On a cold build that's ~25–35 min before any LocalAI source compiles. To skip that on CI, `.github/workflows/base-images.yml` builds and pushes a set of pre-prepped builder bases:
-
-| Tag | Contents |
-|---|---|
-| `base-grpc-amd64` / `base-grpc-arm64` | Ubuntu 24.04 + apt build deps + protoc + cmake + gRPC at `/opt/grpc` |
-| `base-grpc-cuda-12-amd64` | the above + CUDA 12.8 toolkit |
-| `base-grpc-cuda-13-amd64` | the above + CUDA 13.0 toolkit (Ubuntu 22.04 base) |
-| `base-grpc-cuda-13-arm64` | the above + CUDA 13.0 sbsa toolkit (Ubuntu 24.04 base) |
-| `base-grpc-l4t-cuda-12-arm64` | JetPack r36.4.0 base (CUDA preinstalled, `SKIP_DRIVERS=true`) + gRPC |
-| `base-grpc-rocm-amd64` | rocm/dev-ubuntu-24.04:7.2.1 base + hipblas/hipblaslt/rocblas + gRPC |
-| `base-grpc-vulkan-amd64` / `base-grpc-vulkan-arm64` | Ubuntu 24.04 + Vulkan SDK 1.4.335 + gRPC |
-| `base-grpc-intel-amd64` | intel/oneapi-basekit:2025.3.2 base + gRPC |
-
-**Single source of truth**: the install logic for all 10 variants lives in `.docker/install-base-deps.sh`. Both `Dockerfile.base-grpc-builder` AND each variant Dockerfile's `builder-fromsource` stage bind-mount and execute the same script — so the prebuilt CI base and the local from-source path are bit-equivalent by construction.
-
-### How variant Dockerfiles consume the base
-
-`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` are multi-target. Three stages plus a final aliasing stage:
-
-- `builder-fromsource` — `FROM ${BASE_IMAGE}` then runs `install-base-deps.sh` and the per-backend compile script. Used when `BUILDER_TARGET=builder-fromsource` (the default; local `make backends/<name>`).
-- `builder-prebuilt` — `FROM ${BUILDER_BASE_IMAGE}` (one of the prebuilt `base-grpc-*` tags) and runs only the per-backend compile script. Used when `BUILDER_TARGET=builder-prebuilt` (CI when the matrix entry sets `builder-base-image`).
-- `FROM ${BUILDER_TARGET} AS builder` — alias resolves the ARG-selected stage to a fixed name (BuildKit doesn't allow ARG expansion in `COPY --from=`).
-- `FROM scratch` + `COPY --from=builder ...package/. ./` — emits the final scratch image with just the package contents.
-
-BuildKit prunes the unreferenced builder stage, so each build only runs the path it needs. `backend_build.yml` derives `BUILDER_TARGET=builder-prebuilt` automatically when the matrix entry has a non-empty `builder-base-image`; otherwise it defaults to `builder-fromsource`.
-
-The matrix `(build-type, platforms)` → `builder-base-image` mapping for llama-cpp / ik-llama-cpp / turboquant entries:
-
-| `build-type` | `platforms` | tag |
-|---|---|---|
-| `''` | `linux/amd64` | `base-grpc-amd64` |
-| `''` | `linux/arm64` | `base-grpc-arm64` |
-| `cublas` cuda 12 | `linux/amd64` | `base-grpc-cuda-12-amd64` |
-| `cublas` cuda 13 | `linux/amd64` | `base-grpc-cuda-13-amd64` |
-| `cublas` cuda 13 | `linux/arm64` | `base-grpc-cuda-13-arm64` |
-| `cublas` cuda 12 + JetPack base | `linux/arm64` | `base-grpc-l4t-cuda-12-arm64` |
-| `hipblas` | `linux/amd64` | `base-grpc-rocm-amd64` |
-| `vulkan` | `linux/amd64` | `base-grpc-vulkan-amd64` |
-| `vulkan` | `linux/arm64` | `base-grpc-vulkan-arm64` |
-| `sycl_*` | `linux/amd64` | `base-grpc-intel-amd64` |
-
-### Bootstrap order when adding a new variant
-
-If you add a new entry to `base-images.yml`'s matrix, the new tag does not exist on quay until the workflow runs. To consume it from a variant entry safely, dispatch the base-images workflow on the branch first:
-
-```bash
-gh workflow run base-images.yml --ref <feature-branch>
-```
-
-Wait for the new variant to push, then merge the consumer change. Otherwise the consumer's CI fails with "image not found."
-
-## Per-arch native builds + manifest merge
-
-Multi-arch backends (and the core LocalAI image) build natively per arch instead of running both arches under QEMU emulation on a single x86 runner. The pattern:
-
-- The matrix has TWO entries per multi-arch backend, sharing the same `tag-suffix` but distinct `platforms` + `platform-tag` + `runs-on`. Example: `-cpu-faster-whisper` has one amd64 entry on `ubuntu-latest` and one arm64 entry on `ubuntu-24.04-arm`.
-- Each per-arch build pushes by **canonical digest only** (no tags) via `outputs: type=image,push-by-digest=true,name-canonical=true,push=true`. The digest is uploaded as an artifact named `digests<tag-suffix>-<platform-tag>` (or `digests-localai<...>` for root-image builds).
-- `scripts/changed-backends.js` detects shared `tag-suffix` and emits a `merge-matrix` output. `backend.yml` / `backend_pr.yml` have a `backend-merge-jobs` job that consumes it and calls `backend_merge.yml`.
-- `backend_merge.yml` downloads all matching digest artifacts and runs `docker buildx imagetools create` to publish the final tagged manifest list pointing at both per-arch digests. Same `docker/metadata-action` config as the original monolithic build, so consumers see no tag-shape change.
-- `image_merge.yml` is the equivalent for the root LocalAI image (`-core` placeholder when `tag-suffix` is empty so the artifact-name glob doesn't over-match across `core` and `gpu-vulkan`).
-
-**`provenance: false` is required on multi-registry digest pushes**: with the default `mode=max` provenance attestation, BuildKit bundles a per-registry attestation manifest into each registry's manifest list, making the resulting list digest diverge across registries. `steps.build.outputs.digest` only matches one of them and the merge step's `imagetools create <reg>@sha256:<digest>` lookup fails on the other. Setting `provenance: false` keeps the digest content-only and identical across registries.
-
-## Path filter on master push
-
-Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynamically through `scripts/changed-backends.js`:
-
-- **PR events**: paginated `pulls/{n}/files` API → filter the matrix to entries whose `dockerfile` path prefix matches the PR diff.
-- **Push events**: GitHub Compare API (`/repos/{owner}/{repo}/compare/{before}...{after}`) → same path-filter logic. Falls back to "run everything" on first-branch push (`event.before` zero), API truncation (≥300 changed files), missing API token, or any thrown error.
-- **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
-- **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
-
-The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
-
-## The `DEPS_REFRESH` cache-buster (Python backends)
-
-Every Python backend goes through the shared `backend/Dockerfile.python`, which ends with:
-
-```dockerfile
-ARG DEPS_REFRESH=initial
-RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
-```
-
-Most Python backends ship `requirements*.txt` files that **do not pin every transitive dep** (`torch`, `transformers`, `vllm`, `diffusers`, etc. are listed without a `==` pin, or with `>=` lower bounds only). With a warm BuildKit cache, the `make` layer hashes only on Dockerfile instructions + COPYed source — not on what `pip install` resolves at runtime. So a warm cache would ship the *first* version of `vllm` ever cached and never pick up upstream releases.
-
-`DEPS_REFRESH` defends against that:
-
-- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W19`) before each build and passes it as a build-arg.
-- The `RUN ... make` layer's BuildKit hash now includes that string, so the layer invalidates **at most once per week**, automatically picking up newer wheels.
-- Within a week, builds stay warm.
-
-This applies only to `Dockerfile.python` because:
-- Go (`Dockerfile.golang`) pins versions in `go.mod` / `go.sum`.
-- Rust (`Dockerfile.rust`) pins via `Cargo.lock`.
-- C++ backends pin gRPC (`v1.65.0`) and llama.cpp at a specific commit; their inputs don't drift between rebuilds.
-
-### Adjusting the cadence
-
-Bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`) for faster refreshes. For one-shot rebuilds without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
-
-## ccache for C++ backend builds
-
-`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` declare a BuildKit cache mount on `/root/.ccache`:
-
-```dockerfile
-RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-```
-
-The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
-
-On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
-
-## Composite actions
-
-Two composite actions handle runner-side prep:
-
-- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
-- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
-
-Both actions run before any docker buildx step.
-
-## Concurrency
-
-All `backend.yml` / `image.yml` / `test.yml` / etc. workflows use:
-
-```yaml
-concurrency:
-  group: ci-<workflow>-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-```
-
-- **PR events** group by PR number → newer pushes to the same PR cancel old runs (intended).
-- **Push events** group by `github.sha` → each master commit gets its own run; rapid-fire merges don't cancel each other (this was a real issue prior — two master pushes 11 seconds apart would cancel the first's CI).
-
-## Self-warming, no separate populator
-
-There is no cron job that pre-warms the BuildKit cache for individual backends. The production builds *are* the populators. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in the variant `builder-fromsource` stage or skipped entirely when consuming `base-grpc-*`, Python wheel installs, etc.). The base-images workflow's weekly cron is the closest thing to a populator and only refreshes the prebuilt builder bases.
-
-## Manually evicting cache
-
-To force a fully cold build for one backend or the whole image:
-
-```bash
-# Delete a single tag (requires quay credentials with admin on the repo)
-curl -X DELETE \
-  -H "Authorization: Bearer ${QUAY_TOKEN}" \
-  https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm-amd64
-
-# List all tags
-curl -s -H "Authorization: Bearer ${QUAY_TOKEN}" \
-  "https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/?limit=100" | jq '.tags[].name'
-```
-
-Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry per arch so a stale tag never bleeds into a different build.
-
-## What the cache does **not** cover
-
-- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
-- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
-- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
-
-## Darwin native caches
-
-`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
-
-| Cache | Path(s) | Key | Scope |
-|---|---|---|---|
-| Go modules + build | `~/go/pkg/mod`, `~/Library/Caches/go-build` | `go.sum` (managed by `actions/setup-go@v5` `cache: true`) | All darwin jobs |
-| Homebrew | `~/Library/Caches/Homebrew/downloads`, selected `/opt/homebrew/Cellar/*` | hash of `backend_build_darwin.yml` | All darwin jobs |
-| ccache (llama.cpp CMake) | `~/Library/Caches/ccache` | pinned `LLAMA_VERSION` from `backend/cpp/llama-cpp/Makefile` | `inputs.backend == 'llama-cpp'` only |
-| Python wheels (uv + pip) | `~/Library/Caches/pip`, `~/Library/Caches/uv` | `inputs.backend` + ISO week (`+%Y-W%V`) + hash of that backend's `requirements*.txt` | `inputs.lang == 'python'` only |
-
-Read/write semantics match the BuildKit cache: `actions/cache/restore` runs every time, `actions/cache/save` is gated on `github.event_name != 'pull_request'`. PRs read master's warm cache but never write back.
-
-The Python wheel cache uses the same ISO-week cache-buster as the Linux `DEPS_REFRESH` build-arg — same problem (unpinned `torch`/`mlx`/`diffusers`/`transformers` resolve to fresh wheels weekly), same ~one-cold-rebuild-per-week solution.
-
-The brew Cellar cache requires `HOMEBREW_NO_AUTO_UPDATE=1` and `HOMEBREW_NO_INSTALL_CLEANUP=1` (set as job-level env). Without those, `brew install` would mutate the very directories that were just restored, defeating the cache.
-
-**Force-link after cache restore**: `actions/cache` restores `/opt/homebrew/Cellar/*` but NOT the `/opt/homebrew/bin/*` symlinks. After a cache hit, `brew install` sees the Cellar entries and decides "already installed" without re-running its link step, leaving the formulas off PATH. The Dependencies step explicitly runs `brew link --overwrite` for every cached formula afterwards to ensure the symlinks exist.
-
-For ccache, the workflow exports `CMAKE_ARGS=… -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache` via `$GITHUB_ENV` before running `make build-darwin-go-backend`. The Makefile in `backend/cpp/llama-cpp/` already forwards `CMAKE_ARGS` through to each variant build (`fallback`, `grpc`, `rpc-server`), so no script changes are needed. The three variants share most TUs, so ccache dedupes object files across them.
-
-`backend_build_darwin.yml` also has a llama-cpp-specific build-step branch that runs `make backends/llama-cpp-darwin` (the bespoke script that compiles three CMake variants and bundles dylibs via `otool`), distinct from the generic `make build-darwin-${lang}-backend` path. This was consolidated from a previously-bespoke top-level `llama-cpp-darwin` job in `backend.yml` so llama-cpp on Darwin honors the same path filter as the other 34 Darwin backends.
-
-### Cache budget on Darwin
-
-GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~800 MB Go cache + ~2 GB brew Cellar + up to 2 GB ccache + ~1.5 GB × 5 python backends. If the cap is hit, prefer collapsing the per-backend Python keys into a shared `pyenv-darwin-shared-<week>` key (accepts more cross-backend churn for a smaller footprint) before reducing other caches.
-
-## Self-hosted runners
-
-`.github/backend-matrix.yml` has zero references to `arc-runner-set` or `bigger-runner` — all backends run on GHA free-tier hosted runners (`ubuntu-latest` for amd64, `ubuntu-24.04-arm` for arm64 native, `macos-14` for Darwin). The migration off self-hosted relied on the per-arch native split (no QEMU emulation) plus `setup-build-disk`'s `/mnt` relocation (~100 GB working space, enough for ROCm dev image + vLLM/torch installs).
-
-One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
-
-## Touching the cache pipeline
-
-When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
-
-1. **Don't drop `DEPS_REFRESH=...` from the build-args** without a replacement strategy (lockfiles, pinned requirements). Otherwise master will silently freeze on whichever versions were cached at the time.
-2. **Keep `(tag-suffix, platform-tag)` unique per matrix entry** — together they're the cache namespace. Two matrix entries sharing a key would clobber each other's cache.
-3. **Keep `cache-to` gated on `github.event_name != 'pull_request'`** — PRs must not write.
-4. **Keep `ignore-error=true` on `cache-to`** — quay registry hiccups must not fail builds.
-5. **Keep `provenance: false` on push-by-digest steps** — multi-registry digest divergence is the Bug We Already Fixed; reintroducing provenance attestation re-breaks the merge.
-6. **`install-base-deps.sh` is the single source of truth for base contents.** Both `Dockerfile.base-grpc-builder` (CI) and the variant Dockerfiles' `builder-fromsource` (local) bind-mount and execute it. If you add a package to one path, add it to the script — don't fork the logic into a Dockerfile RUN.
-7. **After adding a `base-images.yml` matrix variant, run the workflow on your branch before merging consumer changes** that depend on the new tag — otherwise the consumer's CI fails "image not found."

.agents/coding-style.md

@@ -42,14 +42,6 @@ trim_trailing_whitespace = false
 
 Use `github.com/mudler/xlog` for logging which has the same API as slog.
 
-## Go tests
-
-All Go tests — including backend tests — must use [Ginkgo](https://onsi.github.io/ginkgo/) (v2) with Gomega matchers, not the stdlib `testing` package with `t.Run` / `t.Errorf`. A test file should register a suite with `RegisterFailHandler(Fail)` in a `TestXxx(t *testing.T)` bootstrap and use `Describe`/`Context`/`It` blocks for the actual cases. Look at any existing `*_test.go` under `core/` or `pkg/` for a template.
-
-Do not mix styles within a package. If you are extending tests in a package that already uses Ginkgo, keep using Ginkgo. If you find stdlib-style Go tests in the tree, treat them as tech debt to be migrated rather than as a pattern to follow.
-
-This is enforced by `golangci-lint` via the `forbidigo` linter (see `.golangci.yml`); calls like `t.Errorf` / `t.Fatalf` / `t.Run` / `t.Skip` / `t.Logf` are flagged. Run `make lint` locally before submitting; the same check runs in CI (`.github/workflows/lint.yml`).
-
 ## Documentation
 
 The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.
@@ -57,4 +49,3 @@ The project documentation is located in `docs/content`. When adding new features
 - **Feature Documentation**: If you add a new feature (like a new backend or API endpoint), create a new markdown file in `docs/content/features/` explaining what it is, how to configure it, and how to use it.
 - **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
 - **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.
-- **Shortcodes**: Use `{{% notice note %}}`, `{{% notice tip %}}`, or `{{% notice warning %}}` for callout boxes. Do **not** use `{{% alert %}}` — that shortcode does not exist in this project's Hugo theme and will break the docs build.

.agents/ds4-backend.md

@@ -1,84 +0,0 @@
-# Working on the ds4 Backend
-
-`antirez/ds4` is a single-model inference engine for DeepSeek V4 Flash.
-LocalAI wraps the engine's C API (`ds4/ds4.h`) with a fresh C++ gRPC server at
-`backend/cpp/ds4/` - NOT a fork of llama-cpp's grpc-server.cpp.
-
-## Pin
-
-`backend/cpp/ds4/Makefile` pins `DS4_VERSION?=<sha>` at the top. The `ds4`
-target in the Makefile clones `antirez/ds4` at that commit (mirroring the
-llama-cpp / ik-llama-cpp / turboquant pattern). The bump-deps bot
-(`.github/workflows/bump_deps.yaml`) finds this pin via grep and opens a
-daily PR to update it. To bump manually: edit the `DS4_VERSION?=` line,
-then `make purge && make` (or rely on CI's clean build).
-
-## Wire shape
-
-| RPC | Implementation |
-|---|---|
-| Health, Free, Status | Trivial; no engine dependency for Health |
-| LoadModel | `ds4_engine_open` + `ds4_session_create`; backend is compile-time (DS4_NO_GPU → CPU, __APPLE__ → Metal, otherwise CUDA) |
-| TokenizeString | `ds4_tokenize_text` |
-| Predict | `ds4_engine_generate_argmax` + `DsmlParser` → one ChatDelta with content / reasoning_content / tool_calls[] |
-| PredictStream | Same, per-token ChatDelta writes |
-
-## DSML
-
-ds4 emits tool calls as literal text markers (`<｜DSML｜tool_calls>` etc.) -
-NOT special tokens. `dsml_parser.{h,cpp}` is our streaming state machine that
-classifies token bytes into CONTENT / REASONING / TOOL_START / TOOL_ARGS / TOOL_END
-events. `dsml_renderer.{h,cpp}` does the prompt direction: turns
-OpenAI tool_calls + role=tool messages back into DSML for the next turn.
-
-## Thinking modes
-
-`PredictOptions.Metadata["enable_thinking"]` gates thinking on/off (default ON).
-`["reasoning_effort"] == "max" | "xhigh"` selects `DS4_THINK_MAX`; anything else
-maps to `DS4_THINK_HIGH`. We pass the chosen mode to `ds4_chat_append_assistant_prefix`.
-
-## Disk KV cache
-
-`kv_cache.{h,cpp}` implements an SHA1-keyed file cache using ds4's public
-`ds4_session_save_payload` / `ds4_session_load_payload` API. Enable per request
-via `ModelOptions.Options[] = "kv_cache_dir:/some/path"`. Format is **our own** -
-NOT bit-compatible with ds4-server's KVC files (interop is a follow-up plan).
-
-## Build matrix
-
-| Build | Where | Notes |
-|---|---|---|
-| `cpu-ds4` (amd64 + arm64) | Linux GHA | ds4 considers CPU debug-only; useful only for wiring tests |
-| `cuda13-ds4` (amd64 + arm64) | Linux GHA + DGX Spark validation | Primary production path on Linux |
-| `ds4-darwin` (arm64) | macOS GHA runners | Metal; uses `scripts/build/ds4-darwin.sh` like llama-cpp-darwin |
-
-cuda12 is intentionally omitted. ROCm / Vulkan / SYCL are not applicable.
-
-## Hardware-gated validation
-
-`tests/e2e-backends/backend_test.go` in `BACKEND_BINARY` mode:
-
-```
-BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \
-BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \
-BACKEND_TEST_CAPS=health,load,predict,stream,tools \
-BACKEND_TEST_TOOL_PROMPT="What's the weather in Paris?" \
-go test -count=1 -timeout=30m -v ./tests/e2e-backends/...
-```
-
-CI does not load the model; the suite is opt-in via env vars.
-
-## Importer
-
-`core/gallery/importers/ds4.go` (`DS4Importer`) auto-detects ds4 weights by
-matching the `antirez/deepseek-v4-gguf` repo URI or the
-`DeepSeek-V4-Flash-*.gguf` filename pattern. **Registered BEFORE
-`LlamaCPPImporter`** in `defaultImporters` - both match `.gguf` but ds4 is more
-specific, and first-match-wins. The importer emits `backend: ds4`, uses
-`ds4flash.gguf` as the local filename (matches ds4's own CLI default), and
-disables the Go-side automatic tool-parsing fallback (the C++ backend emits
-ChatDelta.tool_calls natively via `DsmlParser`).
-
-ds4 is also listed in `core/http/endpoints/localai/backend.go`'s pref-only
-slice so the `/import-model` UI surfaces it as a manual choice for users who
-want to force the backend on a non-canonical URI.

.agents/localai-assistant-mcp.md

@@ -1,97 +0,0 @@
-# LocalAI Assistant — admin MCP server
-
-This document is the contract for **anyone** (human or AI agent) touching LocalAI's admin REST surface, the in-process MCP server that wraps it, or the embedded skill prompts that teach the assistant how to use it. Read this before adding/removing/renaming admin endpoints, MCP tools, or skill recipes.
-
-## What this feature is
-
-`pkg/mcp/localaitools/` is a public Go package that exposes LocalAI's admin/management surface as an MCP server. It is used in two ways:
-
-1. **In-process**: when an admin opens a chat with `metadata.localai_assistant=true`, the chat handler injects the in-memory MCP server (paired `net.Pipe()` transport, no HTTP loopback) so the LLM can install models, manage backends and edit configs by chatting.
-2. **Standalone**: the `local-ai mcp-server --target=…` subcommand serves the same MCP server over stdio, talking HTTP to a remote LocalAI instance.
-
-The two modes share **all** tool definitions and skill prompts. They differ only in their `LocalAIClient` implementation (`inproc/` calls services directly; `httpapi/` calls REST).
-
-## The three things you must keep in sync
-
-When you change LocalAI's admin surface, three layers must stay aligned:
-
-1. **REST endpoint** in `core/http/endpoints/localai/*.go`.
-2. **MCP tool registration** in `pkg/mcp/localaitools/tools_*.go`, plus a method on `LocalAIClient` (in `client.go`) and implementations in both `inproc/client.go` **and** `httpapi/client.go`.
-3. **Skill prompt** under `pkg/mcp/localaitools/prompts/skills/*.md` — the markdown that teaches the LLM how to use the new tool. If the new tool fits an existing recipe, update that recipe; otherwise add a new file.
-
-If you ship a REST endpoint without (2) and (3), conversational admins won't see the feature.
-
-## Checklist for adding a new admin endpoint
-
-- [ ] REST endpoint exists in `core/http/endpoints/localai/*.go` and is gated by `auth.RequireAdmin()` in `core/http/routes/localai.go`.
-- [ ] `LocalAIClient` interface in `pkg/mcp/localaitools/client.go` has a method covering the new operation.
-- [ ] DTOs added/updated in `pkg/mcp/localaitools/dto.go` (JSON-tagged; never expose raw service types).
-- [ ] `inproc/client.go` implements the new method by calling the service directly (not via HTTP loopback).
-- [ ] `httpapi/client.go` implements the new method by calling the REST endpoint.
-- [ ] Tool registration added in the appropriate `pkg/mcp/localaitools/tools_*.go`. Mutating tools must reference safety rule 1 in the description.
-- [ ] If the tool is mutating, ensure `Options{DisableMutating: true}` skips it (mirror the pattern in `tools_models.go`).
-- [ ] Skill prompt added or updated under `pkg/mcp/localaitools/prompts/skills/`. The prompt must instruct the LLM when to call the tool, what to ask the user first, and what to do on error.
-- [ ] Tests:
-   - `pkg/mcp/localaitools/server_test.go` adds the tool name to `expectedFullCatalog` and `expectedReadOnlyCatalog` (if read-only).
-   - Tool dispatch is added to `TestEachToolDispatchesToClient`.
-   - `pkg/mcp/localaitools/httpapi/client_test.go` covers the new HTTP path.
-
-## Adding a new skill recipe (no new tool)
-
-Sometimes you want to teach the LLM a new pattern that uses existing tools. Drop a markdown file under `pkg/mcp/localaitools/prompts/skills/<verb>_<noun>.md`. The file is automatically embedded by `//go:embed` and assembled into the system prompt in lexicographic order. No Go changes needed.
-
-Conventions:
-- Filename: `<verb>_<noun>.md` (e.g. `install_chat_model.md`, `upgrade_backend.md`).
-- First line: `# Skill: <Title Case description>`.
-- Number the steps. Reference exact tool names in backticks.
-- If the skill mutates state, remind the LLM to confirm with the user.
-
-## Code conventions
-
-These rules guard against the magic-literal drift that surfaced in the first audit. Do not re-introduce bare strings.
-
-- **Tool names** always come from the `Tool*` constants in `pkg/mcp/localaitools/tools.go`. Tool registrations, the test catalog (`server_test.go`'s `expectedFullCatalog` / `expectedReadOnlyCatalog`), and dispatch tables reference the constants. The embedded skill prompts under `prompts/` keep bare strings — that's the one allowed exception, and `TestPromptsContainSafetyAnchors` enforces alignment.
-- **Toggle/pin actions** use the `modeladmin.Action` type (`pkg/mcp/localaitools` and `core/services/modeladmin`). Use `ActionEnable`/`ActionDisable`/`ActionPin`/`ActionUnpin`; never bare `"enable"`/`"pin"` strings.
-- **Capability tags** for `list_installed_models` use the `localaitools.Capability` type (`capability.go`). The `LocalAIClient.ListInstalledModels` interface takes a typed `Capability`, and the `inproc` switch only accepts canonical values (`"embed"`/`"embedding"` are not aliases — only `CapabilityEmbeddings`).
-- **HTTP error checks** in `httpapi.Client` use `errors.Is(err, ErrHTTPNotFound)`, not substring matches on `err.Error()`. The typed `*HTTPError` carries `StatusCode` and `Body`; add new sentinel errors as needed rather than re-introducing string matching.
-- **Channel sends** to `GalleryService.ModelGalleryChannel` / `BackendGalleryChannel` from inproc clients MUST select on `ctx.Done()` so a cancelled chat completion releases the goroutine. See `inproc.sendModelOp` / `sendBackendOp`.
-- **Disk writes** of model config YAML go through `modeladmin.writeFileAtomic` (temp file + `os.Rename`). `os.WriteFile` truncates on crash and corrupts the model.
-- **MCP server lifecycle**: every initialised holder MUST register `Close()` with `signals.RegisterGracefulTerminationHandler`. The standalone `mcp-server` CLI uses `signal.NotifyContext` to honour SIGINT/SIGTERM.
-
-## File map (where to look)
-
-```
-pkg/mcp/localaitools/
-  client.go              # LocalAIClient interface + DTO registry
-  dto.go                 # JSON-tagged DTOs shared by both client impls
-  server.go              # NewServer(client, opts) — registers tools
-  tools.go               # Tool* name constants (single source of truth)
-  capability.go          # Capability type + constants
-  tools_models.go        # gallery_search, install_model, import_model_uri, ...
-  tools_backends.go
-  tools_config.go
-  tools_system.go
-  tools_state.go
-  prompts.go             # //go:embed loader + SystemPrompt(opts)
-  prompts/00_role.md
-  prompts/10_safety.md   # SAFETY RULES — change with care
-  prompts/20_tools.md    # curated tool catalog with one-liners
-  prompts/skills/*.md
-  inproc/client.go       # in-process LocalAIClient (services-direct)
-  httpapi/client.go      # REST LocalAIClient (for standalone CLI / remote)
-core/http/endpoints/mcp/
-  localai_assistant.go   # process-wide holder + LocalToolExecutor
-core/cli/mcp_server.go   # local-ai mcp-server subcommand
-```
-
-## Why two clients
-
-The in-process MCP server runs inside the same LocalAI binary that serves chat. Going over HTTP loopback would (a) require minting a synthetic admin API key for the server to authenticate against itself, (b) double-marshal every tool dispatch, and (c) lose access to in-process channels (e.g. `GalleryService.ModelGalleryChannel` for streaming install progress). So in-process uses `inproc.Client`. The standalone stdio CLI talks to a *remote* LocalAI; HTTP is the only option, so it uses `httpapi.Client`. Both implement the same `LocalAIClient` interface, and the parity test in `pkg/mcp/localaitools/parity_test.go` (when present) keeps their output equivalent.
-
-## Why prompt-enforced confirmation, not code gates
-
-The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.md` rule 1) that requires the LLM to summarise the action and wait for explicit user confirmation before calling it. There is no `plan_*`/`apply_*` two-step in code. If you add a mutating tool, do **not** add per-tool confirmation logic in Go — instead, list the new tool name in `prompts/10_safety.md` so the LLM knows it falls under the confirmation rule.
-
-## Distributed mode
-
-The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period.

.agents/sglang-backend.md

@@ -1,62 +0,0 @@
-# Working on the SGLang Backend
-
-The SGLang backend lives at `backend/python/sglang/backend.py` (async gRPC). It wraps SGLang's `Engine` (`sglang.srt.entrypoints.engine.Engine`) and translates LocalAI's gRPC `PredictOptions` into SGLang sampling params + outputs into `Reply.chat_deltas`. Structurally it mirrors `backend/python/vllm/backend.py` — keep them shaped the same so changes in one have an obvious analog in the other.
-
-## `engine_args` is the universal escape hatch
-
-A small fixed set of fields on `ModelOptions` is mapped to typed SGLang kwargs in `LoadModel` (model, quantization, load_format, gpu_memory_utilization → mem_fraction_static, trust_remote_code, enforce_eager → disable_cuda_graph, tensor_parallel_size → tp_size, max_model_len → context_length, dtype). **Everything else** flows through the `engine_args:` YAML map.
-
-Validation happens in `_apply_engine_args`. Keys are checked against `dataclasses.fields(ServerArgs)` (`sglang.srt.server_args.ServerArgs` is a flat `@dataclass` with ~380 fields). Unknown keys raise `ValueError` at LoadModel time with a `difflib.get_close_matches` suggestion — same shape as the vLLM backend.
-
-**Precedence:** typed `ModelOptions` fields populate `engine_kwargs` first, then `engine_args` overrides them. So a YAML that sets both `gpu_memory_utilization: 0.9` and `engine_args.mem_fraction_static: 0.5` ends up at `0.5`. Document this when answering "why didn't my YAML field stick?".
-
-**ServerArgs is flat.** Unlike vLLM, where speculative decoding is nested under `engine_args.speculative_config: {...}`, SGLang exposes flat top-level fields: `speculative_algorithm`, `speculative_draft_model_path`, `speculative_num_steps`, `speculative_eagle_topk`, `speculative_num_draft_tokens`, `speculative_dflash_block_size`, etc. There is no `speculative_config:` dict. Same goes for compilation, kv-transfer, attention — all flat.
-
-The canonical reference is `python/sglang/srt/server_args.py:ServerArgs` (line ~304). When SGLang adds new flags, no LocalAI code change is needed — they're automatically available via `engine_args:`. The validator picks them up because it introspects the live dataclass.
-
-## Speculative decoding cheatsheet
-
-`--speculative-algorithm` accepts `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`, `DFLASH`. `NEXTN` is silently rewritten to `EAGLE` in `ServerArgs.__post_init__` (`server_args.py:3286-3287`). MTP (Multi-Token Prediction) is the same EAGLE path with `num_steps=1, eagle_topk=1, num_draft_tokens=2` against a target whose architecture has multi-token heads (e.g. MiMo-7B-RL, DeepSeek-V3-MTP).
-
-| Algorithm | Drafter requirement | Gallery demo target | Gallery demo drafter |
-|-----------|--------------------|---------------------|----------------------|
-| `NEXTN` / `EAGLE` (MTP) | Assistant drafter or built-in heads | google/gemma-4-E2B-it, google/gemma-4-E4B-it | google/gemma-4-E2B-it-assistant, google/gemma-4-E4B-it-assistant |
-| `EAGLE3` | EAGLE3 draft head | (no gallery entry yet) | e.g. jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B |
-| `DFLASH` | Block-diffusion drafter | (no gallery entry yet) | e.g. z-lab/Qwen3-4B-DFlash-b16 |
-| `STANDALONE` | Smaller LLM as drafter | (no gallery entry yet) | any smaller chat-tuned LLM in the same family |
-| `NGRAM` | None — uses prefix history | (no gallery entry yet) | n/a |
-
-The Gemma 4 demos use `mem_fraction_static: 0.85` (cookbook default) and the cookbook's `num_steps=5, num_draft_tokens=6, eagle_topk=1` parameters. Other algorithms are reachable from any user YAML via `engine_args:` but don't have shipped demos yet — that's a deliberate gallery scope choice, not a backend limitation.
-
-Gemma 4 support requires sglang built from a commit that includes [PR #21952](https://github.com/sgl-project/sglang/pull/21952). LocalAI's pinned release for cublas12 / cublas13 includes it. The `l4t13` (JetPack 7 / sbsa cu130) build floors at `sglang>=0.5.0` because the `pypi.jetson-ai-lab.io` mirror still ships only `0.5.1.post2` as of 2026-05-06 — Gemma 4 / MTP recipes are therefore not available on l4t13 until that mirror catches up. `backend.py` keeps backward compat with the 0.5.x → 0.5.11 `SamplingParams.seed` → `sampling_seed` rename via runtime detection.
-
-Compatibility caveats per the SGLang docs: DFLASH and NGRAM are incompatible with `enable_dp_attention`; DFLASH requires `pp_size == 1`; STANDALONE is incompatible with `enable_dp_attention`; NGRAM is CUDA-only and disables the overlap scheduler.
-
-### `mem_fraction_static` + quantization + MTP on consumer GPUs
-
-When combining online weight quantization (`engine_args.quantization: fp8` / `awq` / etc.) with built-in-head MTP (`speculative_algorithm: EAGLE`/`NEXTN`) on a tight VRAM budget, sglang's default `mem_fraction_static: 0.85` will OOM during draft-worker init. The reason: sglang quantizes the **target** model's transformer blocks but loads the **MTP draft worker's vocab embedding** at the source dtype (typically bf16). For a 7 B-class model with a 150k-token vocab × 4096 hidden, that's another ~1.2 GiB allocated *after* the static pool is reserved. At 0.85 fraction on a 16 GB card there's no room left.
-
-Workaround: drop `mem_fraction_static` to ~0.7 so the post-static heap can absorb the MTP embedding alloc + CUDA graph private pools. Verified end-to-end on MiMo-7B-RL + fp8 + MTP on a 16 GB RTX 5070 Ti (`gallery/sglang-mimo-7b-mtp.yaml`) at ~88 tok/s. Models with larger vocabs or more MTP layers (e.g. DeepSeek-V3-MTP) need an even smaller fraction.
-
-This isn't documented anywhere upstream as of 2026-05-06 — the SGLang Gemma 4 cookbook uses 0.85 because their MTP path doesn't go through `eagle_worker_v2.py` for an embedding-bearing draft module. Don't blanket-apply 0.7 across all sglang YAMLs; only when MTP-with-built-in-heads + quantization combine.
-
-## Tool-call and reasoning parsers stay on `Options[]`
-
-ServerArgs has `tool_call_parser` and `reasoning_parser` fields, and the backend does pass them through to `Engine` so SGLang's own HTTP/OAI surface keeps working. But for the **LocalAI** request path the backend constructs fresh per-request parser instances in `_make_parsers` (`backend.py:286`) because the parsers are stateful — the streaming and non-streaming paths each need their own.
-
-So the user-facing knob stays on `Options[]`:
-
-```yaml
-options:
-  - tool_parser:hermes
-  - reasoning_parser:deepseek_r1
-```
-
-Putting these in `engine_args:` will set them on `ServerArgs` but the LocalAI-level streaming `ChatDelta` will not pick them up. Don't recommend that path.
-
-## What's missing today (out of scope, but worth tracking)
-
-- `core/config/hooks_sglang.go` — there is no SGLang equivalent of `hooks_vllm.go`. The vLLM hook auto-selects parsers for known model families from `parser_defaults.json` and seeds production engine_args defaults. A symmetric hook for SGLang could reuse the same `parser_defaults.json` (the SGLang parser names are different but the family detection is shared) and seed defaults like `enable_metrics: true` or attention-backend choices.
-- `core/gallery/importers/sglang.go` — vLLM has an importer that resolves model architecture → parser defaults at gallery-import time. A matching importer for SGLang would let `local-ai install` populate sensible parsers automatically.
-
-These should be a follow-up PR, not a blocker for the engine_args feature.

.agents/vllm-backend.md

@@ -1,115 +0,0 @@
-# Working on the vLLM Backend
-
-The vLLM backend lives at `backend/python/vllm/backend.py` (async gRPC) and the multimodal variant at `backend/python/vllm-omni/backend.py` (sync gRPC). Both wrap vLLM's `AsyncLLMEngine` / `Omni` and translate the LocalAI gRPC `PredictOptions` into vLLM `SamplingParams` + outputs into `Reply.chat_deltas`.
-
-This file captures the non-obvious bits — most of the bring-up was a single PR (`feat/vllm-parity`) and the things below are easy to get wrong.
-
-## Tool calling and reasoning use vLLM's *native* parsers
-
-Do not write regex-based tool-call extractors for vLLM. vLLM ships:
-
-- `vllm.tool_parsers.ToolParserManager` — 50+ registered parsers (`hermes`, `llama3_json`, `llama4_pythonic`, `mistral`, `qwen3_xml`, `deepseek_v3`, `granite4`, `openai`, `kimi_k2`, `glm45`, …)
-- `vllm.reasoning.ReasoningParserManager` — 25+ registered parsers (`deepseek_r1`, `qwen3`, `mistral`, `gemma4`, …)
-
-Both can be used standalone: instantiate with a tokenizer, call `extract_tool_calls(text, request=None)` / `extract_reasoning(text, request=None)`. The backend stores the parser *classes* on `self.tool_parser_cls` / `self.reasoning_parser_cls` at LoadModel time and instantiates them per request.
-
-**Selection:** vLLM does *not* auto-detect parsers from model name — neither does the LocalAI backend. The user (or `core/config/hooks_vllm.go`) must pick one and pass it via `Options[]`:
-
-```yaml
-options:
-  - tool_parser:hermes
-  - reasoning_parser:qwen3
-```
-
-Auto-defaults for known model families live in `core/config/parser_defaults.json` and are applied:
-- at gallery import time by `core/gallery/importers/vllm.go`
-- at model load time by the `vllm` / `vllm-omni` backend hook in `core/config/hooks_vllm.go`
-
-User-supplied `tool_parser:`/`reasoning_parser:` in the config wins over defaults — the hook checks for existing entries before appending.
-
-**When to update `parser_defaults.json`:** any time vLLM ships a new tool or reasoning parser, or you onboard a new model family that LocalAI users will pull from HuggingFace. The file is keyed by *family pattern* matched against `normalizeModelID(cfg.Model)` (lowercase, org-prefix stripped, `_`→`-`). Patterns are checked **longest-first** — keep `qwen3.5` before `qwen3`, `llama-3.3` before `llama-3`, etc., or the wrong family wins. Add a covering test in `core/config/hooks_test.go`.
-
-**Sister file — `core/config/inference_defaults.json`:** same pattern but for sampling parameters (temperature, top_p, top_k, min_p, repeat_penalty, presence_penalty). Loaded by `core/config/inference_defaults.go` and applied by `ApplyInferenceDefaults()`. The schema is `map[string]float64` only — *strings don't fit*, which is why parser defaults needed their own JSON file. The inference file is **auto-generated from unsloth** via `go generate ./core/config/` (see `core/config/gen_inference_defaults/`) — don't hand-edit it; instead update the upstream source or regenerate. Both files share `normalizeModelID()` and the longest-first pattern ordering.
-
-**Constructor compatibility gotcha:** the abstract `ToolParser.__init__` accepts `tools=`, but several concrete parsers (Hermes2ProToolParser, etc.) override `__init__` and *only* accept `tokenizer`. Always:
-
-```python
-try:
-    tp = self.tool_parser_cls(self.tokenizer, tools=tools)
-except TypeError:
-    tp = self.tool_parser_cls(self.tokenizer)
-```
-
-## ChatDelta is the streaming contract
-
-The Go side (`core/backend/llm.go`, `pkg/functions/chat_deltas.go`) consumes `Reply.chat_deltas` to assemble the OpenAI response. For tool calls to surface in `chat/completions`, the Python backend **must** populate `Reply.chat_deltas[].tool_calls` with `ToolCallDelta{index, id, name, arguments}`. Returning the raw `<tool_call>...</tool_call>` text in `Reply.message` is *not* enough — the Go regex fallback exists for llama.cpp, not for vllm.
-
-Same story for `reasoning_content` — emit it on `ChatDelta.reasoning_content`, not as part of `content`.
-
-## Message conversion to chat templates
-
-`tokenizer.apply_chat_template()` expects a list of dicts, not proto Messages. The shared helper in `backend/python/common/vllm_utils.py` (`messages_to_dicts`) handles the mapping including:
-
-- `tool_call_id` and `name` for `role="tool"` messages
-- `tool_calls` JSON-string field → parsed Python list for `role="assistant"`
-- `reasoning_content` for thinking models
-
-Pass `tools=json.loads(request.Tools)` and (when `request.Metadata.get("enable_thinking") == "true"`) `enable_thinking=True` to `apply_chat_template`. Wrap in `try/except TypeError` because not every tokenizer template accepts those kwargs.
-
-## CPU support and the SIMD/library minefield
-
-vLLM publishes prebuilt CPU wheels at `https://github.com/vllm-project/vllm/releases/...`. The pin lives in `backend/python/vllm/requirements-cpu-after.txt`.
-
-**Version compatibility — important:** newer vllm CPU wheels (≥ 0.15) declare `torch==2.10.0+cpu` as a hard dep, but `torch==2.10.0` only exists on the PyTorch test channel and pulls in an incompatible `torchvision`. Stay on **`vllm 0.14.1+cpu` + `torch 2.9.1+cpu`** until both upstream catch up. Bumping requires verifying torchvision/torchaudio match.
-
-`requirements-cpu.txt` uses `--extra-index-url https://download.pytorch.org/whl/cpu`. `install.sh` adds `--index-strategy=unsafe-best-match` for the `cpu` profile so uv resolves transformers/vllm from PyPI while pulling torch from the PyTorch index.
-
-**SIMD baseline:** the prebuilt CPU wheel is compiled with AVX-512 VNNI/BF16. On a CPU without those instructions, importing `vllm.model_executor.models.registry` SIGILLs at `_run_in_subprocess` time during model inspection. There is no runtime flag to disable it. Workarounds:
-
-1. **Run on a host with the right SIMD baseline** (default — fast)
-2. **Build from source** with `FROM_SOURCE=true` env var. Plumbing exists end-to-end:
-   - `install.sh` hides `requirements-cpu-after.txt`, runs `installRequirements` for the base deps, then clones vllm and `VLLM_TARGET_DEVICE=cpu uv pip install --no-deps .`
-   - `backend/Dockerfile.python` declares `ARG FROM_SOURCE` + `ENV FROM_SOURCE`
-   - `Makefile` `docker-build-backend` macro forwards `--build-arg FROM_SOURCE=$(FROM_SOURCE)` when set
-   - Source build takes 30–50 minutes — too slow for per-PR CI but fine for local.
-
-**Runtime shared libraries:** vLLM's `vllm._C` extension `dlopen`s `libnuma.so.1` at import time. If missing, the C extension silently fails and `torch.ops._C_utils.init_cpu_threads_env` is never registered → `EngineCore` crashes on `init_device` with:
-
-```
-AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env'
-```
-
-`backend/python/vllm/package.sh` bundles `libnuma.so.1` and `libgomp.so.1` into `${BACKEND}/lib/`, which `libbackend.sh` adds to `LD_LIBRARY_PATH` at run time. The builder stage in `backend/Dockerfile.python` installs `libnuma1`/`libgomp1` so package.sh has something to copy. Do *not* assume the production host has these — backend images are `FROM scratch`.
-
-## Backend hook system (`core/config/backend_hooks.go`)
-
-Per-backend defaults that used to be hardcoded in `ModelConfig.Prepare()` now live in `core/config/hooks_*.go` files and self-register via `init()`:
-
-- `hooks_llamacpp.go` → GGUF metadata parsing, context size, GPU layers, jinja template
-- `hooks_vllm.go` → tool/reasoning parser auto-selection from `parser_defaults.json`
-
-Hook keys:
-- `"llama-cpp"`, `"vllm"`, `"vllm-omni"`, … — backend-specific
-- `""` — runs only when `cfg.Backend` is empty (auto-detect case)
-- `"*"` — global catch-all, runs for every backend before specific hooks
-
-Multiple hooks per key are supported and run in registration order. Adding a new backend default:
-
-```go
-// core/config/hooks_<backend>.go
-func init() {
-    RegisterBackendHook("<backend>", myDefaults)
-}
-func myDefaults(cfg *ModelConfig, modelPath string) {
-    // only fill in fields the user didn't set
-}
-```
-
-## The `Messages.ToProto()` fields you need to set
-
-`core/schema/message.go:ToProto()` must serialize:
-- `ToolCallID` → `proto.Message.ToolCallId` (for `role="tool"` messages — links result back to the call)
-- `Reasoning` → `proto.Message.ReasoningContent`
-- `ToolCalls` → `proto.Message.ToolCalls` (JSON-encoded string)
-
-These were originally not serialized and tool-calling conversations broke silently — the C++ llama.cpp backend reads them but always got empty strings. Any new field added to `schema.Message` *and* `proto.Message` needs a matching line in `ToProto()`.

.docker/apt-mirror.sh

@@ -1,39 +0,0 @@
-#!/bin/sh
-# Reconfigure Ubuntu apt sources to point at an alternate mirror.
-#
-# Used by Dockerfiles via `RUN --mount=type=bind,source=.docker/apt-mirror.sh,...`
-# and by CI workflows on the runner to mitigate outages of the default
-# archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com pool.
-#
-# Inputs (env):
-#   APT_MIRROR        Replacement for archive.ubuntu.com and security.ubuntu.com
-#                     (e.g. "http://azure.archive.ubuntu.com" or
-#                      "https://mirrors.edge.kernel.org").
-#                     Leave empty to keep upstream. The trailing "/ubuntu/..."
-#                     path is preserved by the rewrite.
-#   APT_PORTS_MIRROR  Replacement for ports.ubuntu.com (arm64/ppc64el/...).
-#                     Leave empty to keep upstream.
-#
-# Both default to empty, in which case the script is a no-op.
-
-set -e
-
-if [ -z "${APT_MIRROR}" ] && [ -z "${APT_PORTS_MIRROR}" ]; then
-    exit 0
-fi
-
-# Ubuntu 24.04 (noble) ships DEB822 sources at /etc/apt/sources.list.d/ubuntu.sources;
-# older releases use /etc/apt/sources.list. We rewrite whichever exists.
-for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
-    [ -f "$f" ] || continue
-    if [ -n "${APT_MIRROR}" ]; then
-        # Use a comma delimiter so the alternation pipe in the regex
-        # is not interpreted as the s/// separator.
-        sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
-    fi
-    if [ -n "${APT_PORTS_MIRROR}" ]; then
-        sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
-    fi
-done
-
-echo "apt-mirror: rewrote sources (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"

.docker/ik-llama-cpp-compile.sh

@@ -1,30 +0,0 @@
-#!/usr/bin/env bash
-# Shared compile logic for backend/Dockerfile.ik-llama-cpp.
-# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
-
-set -euxo pipefail
-
-export CCACHE_DIR=/root/.ccache
-ccache --max-size=5G || true
-ccache -z || true
-
-export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
-fi
-
-cd /LocalAI/backend/cpp/ik-llama-cpp
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  # ARM64 / ROCm: build without x86 SIMD
-  make ik-llama-cpp-fallback
-else
-  # ik_llama.cpp's IQK kernels require at least AVX2
-  make ik-llama-cpp-avx2
-fi
-
-ccache -s || true

.docker/install-base-deps.sh

@@ -1,244 +0,0 @@
-#!/usr/bin/env bash
-# Single source of truth for builder-base contents.
-#
-# Used by:
-#   - backend/Dockerfile.base-grpc-builder        (CI prebuilt-base source of truth)
-#   - backend/Dockerfile.llama-cpp                (builder-fromsource stage)
-#   - backend/Dockerfile.ik-llama-cpp             (builder-fromsource stage)
-#   - backend/Dockerfile.turboquant               (builder-fromsource stage)
-#
-# All four files invoke this script via
-#   RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-#       --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-#       bash /usr/local/sbin/install-base-deps
-#
-# so the prebuilt CI base image and the from-source local-dev path are
-# bit-equivalent by construction.
-#
-# Inputs (env, populated from Dockerfile ARG/ENV):
-#   BUILD_TYPE                ("cublas"|"l4t"|"hipblas"|"vulkan"|"sycl"|"clblas"|"")
-#   CUDA_MAJOR_VERSION        ("12" | "13" | "")
-#   CUDA_MINOR_VERSION        ("8" | "0" | "")
-#   TARGETARCH                ("amd64" | "arm64")
-#   UBUNTU_VERSION            ("2204" | "2404")
-#   SKIP_DRIVERS              ("false" | "true")
-#   CMAKE_FROM_SOURCE         ("false" | "true")
-#   CMAKE_VERSION             ("3.31.10")
-#   GRPC_VERSION              ("v1.65.0")
-#   GRPC_MAKEFLAGS            ("-j4 -Otarget")
-#   APT_MIRROR / APT_PORTS_MIRROR  (optional; consumed by /usr/local/sbin/apt-mirror)
-#   AMDGPU_TARGETS            (optional; only relevant for hipblas downstream)
-#
-# IMPORTANT: install logic is copied verbatim from the prior in-Dockerfile
-# RUN blocks. Do not paraphrase apt invocations / version pins / sed line
-# numbers / deb URLs — the bit-equivalence guarantee depends on it.
-
-set -eux
-
-# --- 0. apt mirror rewrite (no-op when APT_MIRROR / APT_PORTS_MIRROR unset) ---
-if [ -x /usr/local/sbin/apt-mirror ]; then
-    APT_MIRROR="${APT_MIRROR:-}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR:-}" \
-        sh /usr/local/sbin/apt-mirror
-fi
-
-export DEBIAN_FRONTEND=noninteractive
-export MAKEFLAGS="${GRPC_MAKEFLAGS:-}"
-
-# --- 1. Base apt build deps ---
-apt-get update
-apt-get install -y --no-install-recommends \
-    build-essential \
-    ccache git \
-    ca-certificates \
-    make \
-    pkg-config libcurl4-openssl-dev \
-    curl unzip \
-    libssl-dev wget
-apt-get clean
-rm -rf /var/lib/apt/lists/*
-
-# --- 2. Vulkan SDK (BUILD_TYPE=vulkan) ---
-# NB: this block intentionally installs `cmake` via apt as part of the
-# Vulkan tooling — must run before the dedicated CMake step below.
-if [ "${BUILD_TYPE:-}" = "vulkan" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y  --no-install-recommends \
-        software-properties-common pciutils wget gpg-agent
-    apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
-        libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
-        libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
-        git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
-        ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
-        clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-    if [ "amd64" = "${TARGETARCH:-}" ]; then
-        wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz"
-        tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz
-        rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz
-        mkdir -p /opt/vulkan-sdk
-        mv 1.4.335.0 /opt/vulkan-sdk/
-        ( cd /opt/vulkan-sdk/1.4.335.0 && \
-          ./vulkansdk --no-deps --maxjobs \
-              vulkan-loader \
-              vulkan-validationlayers \
-              vulkan-extensionlayer \
-              vulkan-tools \
-              shaderc )
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/
-        cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/
-        rm -rf /opt/vulkan-sdk
-    fi
-    if [ "arm64" = "${TARGETARCH:-}" ]; then
-        mkdir vulkan
-        ( cd vulkan && \
-          curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-          tar -xvf vulkan-sdk.tar.xz && \
-          rm vulkan-sdk.tar.xz && \
-          cd 1.4.335.0 && \
-          cp -rfv aarch64/bin/* /usr/bin/ && \
-          cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-          cp -rfv aarch64/include/* /usr/include/ && \
-          cp -rfv aarch64/share/* /usr/share/ )
-        rm -rf vulkan
-    fi
-    ldconfig
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 3. CUDA toolkit (BUILD_TYPE=cublas|l4t) ---
-if { [ "${BUILD_TYPE:-}" = "cublas" ] || [ "${BUILD_TYPE:-}" = "l4t" ]; } && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y  --no-install-recommends \
-        software-properties-common pciutils
-    if [ "amd64" = "${TARGETARCH:-}" ]; then
-        curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb"
-    fi
-    if [ "arm64" = "${TARGETARCH:-}" ]; then
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
-            curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb"
-        else
-            curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb"
-        fi
-    fi
-    dpkg -i cuda-keyring_1.1-1_all.deb
-    rm -f cuda-keyring_1.1-1_all.deb
-    apt-get update
-    apt-get install -y --no-install-recommends \
-        "cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-        "libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
-    if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "${TARGETARCH:-}" ]; then
-        apt-get install -y --no-install-recommends \
-            "libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-            "libcudnn9-cuda-${CUDA_MAJOR_VERSION}" \
-            "cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
-            "libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
-    fi
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 4. cuDSS / NVPL on arm64 + cublas (legacy JetPack / Tegra) ---
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-if [ "${BUILD_TYPE:-}" = "cublas" ] && [ "${TARGETARCH:-}" = "arm64" ]; then
-    wget "https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
-    dpkg -i "cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
-    cp /var/cudss-local-tegra-repo-ubuntu"${UBUNTU_VERSION}"-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/
-    apt-get update
-    apt-get -y install cudss "cudss-cuda-${CUDA_MAJOR_VERSION}"
-    wget "https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
-    dpkg -i "nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
-    cp /var/nvpl-local-repo-ubuntu"${UBUNTU_VERSION}"-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/
-    apt-get update
-    apt-get install -y nvpl
-fi
-
-# --- 5. clBLAS (BUILD_TYPE=clblas) ---
-# Present in variant Dockerfiles' from-source path but not in master's
-# Dockerfile.base-grpc-builder. No CI matrix entry currently uses this,
-# but keep parity so a future BUILD_TYPE=clblas build doesn't drift.
-if [ "${BUILD_TYPE:-}" = "clblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y --no-install-recommends \
-        libclblast-dev
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 6. ROCm / HIP build deps (BUILD_TYPE=hipblas) ---
-if [ "${BUILD_TYPE:-}" = "hipblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
-    apt-get update
-    apt-get install -y --no-install-recommends \
-        hipblas-dev \
-        hipblaslt-dev \
-        rocblas-dev
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-    # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install,
-    # which results in local-ai and others not being able to locate the libraries.
-    # We run ldconfig ourselves to work around this packaging deficiency.
-    ldconfig
-    # Log which GPU architectures have rocBLAS kernel support
-    echo "rocBLAS library data architectures:"
-    (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
-        echo "WARNING: No rocBLAS kernel data found"
-fi
-
-echo "TARGETARCH: ${TARGETARCH:-}"
-
-# --- 7. protoc (always) ---
-# The version in 22.04 is too old. We will create one as part of installing
-# the GRPC build below but that will also bring in a newer version of absl
-# which stablediffusion cannot compile with. This version of protoc is only
-# here so that we can generate the grpc code for the stablediffusion build.
-if [ "amd64" = "${TARGETARCH:-}" ]; then
-    curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip
-    unzip -j -d /usr/local/bin protoc.zip bin/protoc
-    rm protoc.zip
-fi
-if [ "arm64" = "${TARGETARCH:-}" ]; then
-    curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip
-    unzip -j -d /usr/local/bin protoc.zip bin/protoc
-    rm protoc.zip
-fi
-
-# --- 8. CMake (apt or compiled from source) ---
-# The version in 22.04 is too old. Vulkan path above already pulled cmake
-# via apt; the from-source branch here will install over it which is fine.
-if [ "${CMAKE_FROM_SOURCE:-false}" = "true" ]; then
-    curl -L -s "https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz" -o cmake.tar.gz
-    tar xvf cmake.tar.gz
-    ( cd "cmake-${CMAKE_VERSION}" && ./configure && make && make install )
-else
-    apt-get update
-    apt-get install -y \
-        cmake
-    apt-get clean
-    rm -rf /var/lib/apt/lists/*
-fi
-
-# --- 9. gRPC compile + install at /opt/grpc ---
-# We install GRPC to a different prefix here so that we can copy in only
-# the build artifacts later — saves several hundred MB on the final docker
-# image size vs copying in the entire GRPC source tree and running
-# `make install` in the target container.
-#
-# The TESTONLY abseil sed patch and /opt/grpc prefix are load-bearing —
-# downstream Dockerfiles `COPY` /opt/grpc to /usr/local (or rely on the
-# prebuilt base having it at /opt/grpc).
-mkdir -p /build
-cd /build
-git clone --recurse-submodules --jobs 4 -b "${GRPC_VERSION}" --depth 1 --shallow-submodules https://github.com/grpc/grpc
-mkdir -p /build/grpc/cmake/build
-cd /build/grpc/cmake/build
-sed -i "216i\\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt"
-cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../..
-make
-make install
-cd /
-rm -rf /build

.docker/llama-cpp-compile.sh

@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-# Shared compile logic for backend/Dockerfile.llama-cpp.
-# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
-
-set -euxo pipefail
-
-export CCACHE_DIR=/root/.ccache
-ccache --max-size=5G || true
-ccache -z || true
-
-export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
-fi
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  cd /LocalAI/backend/cpp/llama-cpp
-  make llama-cpp-fallback
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-else
-  cd /LocalAI/backend/cpp/llama-cpp
-  make llama-cpp-avx
-  make llama-cpp-avx2
-  make llama-cpp-avx512
-  make llama-cpp-fallback
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-fi
-
-ccache -s || true

.docker/turboquant-compile.sh

@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-# Shared compile logic for backend/Dockerfile.turboquant.
-# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
-
-set -euxo pipefail
-
-export CCACHE_DIR=/root/.ccache
-ccache --max-size=5G || true
-ccache -z || true
-
-export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/turboquant-*-build
-fi
-
-cd /LocalAI/backend/cpp/turboquant
-
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
-  make turboquant-fallback
-  make turboquant-grpc
-  make turboquant-rpc-server
-else
-  make turboquant-avx
-  make turboquant-avx2
-  make turboquant-avx512
-  make turboquant-fallback
-  make turboquant-grpc
-  make turboquant-rpc-server
-fi
-
-ccache -s || true

.github/actions/configure-apt-mirror/action.yml

@@ -1,100 +0,0 @@
-name: 'Configure apt mirror'
-description: |
-  Reconfigure the GitHub Actions runner's Ubuntu apt sources to use an
-  alternate mirror, and emit the effective URLs as outputs so callers can
-  forward them as Docker build-args.
-
-  Two mirror profiles depending on where the runner lives, because the
-  best mirror differs by network:
-
-    * github-hosted runners run on Azure, so they default to the
-      Azure-hosted Ubuntu mirror (lowest latency, same VPC).
-    * self-hosted runners (arc-runner-set, bigger-runner, ...) typically
-      cannot route to azure.archive.ubuntu.com, so they default to the
-      kernel.org mirror, which is publicly reachable from anywhere.
-
-  Pass an empty string to either input to skip the rewrite for that
-  profile and keep upstream archive.ubuntu.com / ports.ubuntu.com.
-
-inputs:
-  github-hosted-mirror:
-    description: 'archive/security mirror URL for github-hosted runners (empty = upstream)'
-    required: false
-    default: 'http://azure.archive.ubuntu.com'
-  github-hosted-ports-mirror:
-    description: 'ports.ubuntu.com mirror URL for github-hosted runners (empty = upstream)'
-    required: false
-    default: 'http://azure.ports.ubuntu.com'
-  self-hosted-mirror:
-    description: 'archive/security mirror URL for self-hosted runners (empty = upstream)'
-    required: false
-    # HTTP, not HTTPS: the bare ubuntu:24.04 builder image doesn't ship
-    # ca-certificates, so the very first apt-get update over TLS would
-    # fail with "No system certificates available" before it can install
-    # anything. apt validates package integrity via GPG signatures, so
-    # plain HTTP is safe for the archive itself.
-    default: 'http://mirrors.edge.kernel.org'
-  self-hosted-ports-mirror:
-    description: 'ports.ubuntu.com mirror URL for self-hosted runners (empty = upstream)'
-    required: false
-    # mirrors.edge.kernel.org does NOT carry /ubuntu-ports/ — only the
-    # main /ubuntu/ archive — so arm64 builds 404 there. Leave ports
-    # upstream by default. The original DDoS was on archive.ubuntu.com
-    # so ports.ubuntu.com remains the path of least surprise.
-    default: ''
-
-outputs:
-  effective-mirror:
-    description: 'The mirror URL actually applied for this runner (or empty)'
-    value: ${{ steps.pick.outputs.mirror }}
-  effective-ports-mirror:
-    description: 'The ports mirror URL actually applied for this runner (or empty)'
-    value: ${{ steps.pick.outputs.ports-mirror }}
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Pick effective mirror for this runner
-      id: pick
-      shell: bash
-      env:
-        RUNNER_ENV: ${{ runner.environment }}
-        GH_MIRROR: ${{ inputs.github-hosted-mirror }}
-        GH_PORTS_MIRROR: ${{ inputs.github-hosted-ports-mirror }}
-        SH_MIRROR: ${{ inputs.self-hosted-mirror }}
-        SH_PORTS_MIRROR: ${{ inputs.self-hosted-ports-mirror }}
-      run: |
-        if [ "${RUNNER_ENV}" = "github-hosted" ]; then
-          MIRROR="${GH_MIRROR}"
-          PORTS_MIRROR="${GH_PORTS_MIRROR}"
-        else
-          MIRROR="${SH_MIRROR}"
-          PORTS_MIRROR="${SH_PORTS_MIRROR}"
-        fi
-        echo "configure-apt-mirror: runner=${RUNNER_ENV} mirror='${MIRROR}' ports-mirror='${PORTS_MIRROR}'"
-        echo "mirror=${MIRROR}" >> "$GITHUB_OUTPUT"
-        echo "ports-mirror=${PORTS_MIRROR}" >> "$GITHUB_OUTPUT"
-
-    - name: Rewrite apt sources
-      if: steps.pick.outputs.mirror != '' || steps.pick.outputs.ports-mirror != ''
-      shell: bash
-      env:
-        APT_MIRROR: ${{ steps.pick.outputs.mirror }}
-        APT_PORTS_MIRROR: ${{ steps.pick.outputs.ports-mirror }}
-      run: |
-        set -e
-        # Ubuntu 24.04 (noble) ships DEB822 sources at
-        # /etc/apt/sources.list.d/ubuntu.sources; older releases use
-        # /etc/apt/sources.list. Rewrite whichever exists.
-        for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
-          sudo test -f "$f" || continue
-          if [ -n "${APT_MIRROR}" ]; then
-            # Comma delimiter so the alternation pipe in the regex is not
-            # interpreted as the s/// separator.
-            sudo sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
-          fi
-          if [ -n "${APT_PORTS_MIRROR}" ]; then
-            sudo sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
-          fi
-        done
-        echo "Runner apt mirror configured (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"

.github/actions/free-disk-space/action.yml

@@ -1,65 +0,0 @@
-name: 'Free disk space on hosted runners'
-description: |
-  Aggressively clean GitHub-hosted ubuntu-latest runners to reclaim ~6-10 GB
-  of working space before docker buildx steps. Combines jlumbroso/free-disk-space
-  with explicit apt purges of large packages we never use (dotnet, ghc, mono,
-  android, jdk, ...).
-
-  No-op on self-hosted runners; pass mode=skip to force-disable.
-
-inputs:
-  mode:
-    description: 'hosted (default — clean) or skip (no-op)'
-    required: false
-    default: 'hosted'
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Free Disk Space (Ubuntu)
-      if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
-      uses: jlumbroso/free-disk-space@main
-      with:
-        tool-cache: true
-        android: true
-        dotnet: true
-        haskell: true
-        large-packages: true
-        docker-images: true
-        swap-storage: true
-
-    - name: Release space from worker
-      if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
-      shell: bash
-      run: |
-        echo "Listing top largest packages"
-        pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-        head -n 30 <<< "${pkgs}"
-        df -h
-        sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-        sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
-        sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
-        sudo rm -rf /usr/local/lib/android
-        sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-        sudo rm -rf /usr/share/dotnet
-        sudo apt-get remove -y '^mono-.*' || true
-        sudo apt-get remove -y '^ghc-.*' || true
-        sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-        sudo apt-get remove -y 'php.*' || true
-        sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-        sudo apt-get remove -y '^google-.*' || true
-        sudo apt-get remove -y azure-cli || true
-        sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-        sudo apt-get remove -y '^gfortran-.*' || true
-        sudo apt-get remove -y microsoft-edge-stable || true
-        sudo apt-get remove -y firefox || true
-        sudo apt-get remove -y powershell || true
-        sudo apt-get remove -y r-base-core || true
-        sudo apt-get autoremove -y
-        sudo apt-get clean
-        sudo rm -rfv build || true
-        sudo rm -rf /usr/share/dotnet || true
-        sudo rm -rf /opt/ghc || true
-        sudo rm -rf "/usr/local/share/boost" || true
-        sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
-        df -h

.github/actions/setup-build-disk/action.yml

@@ -1,59 +0,0 @@
-name: 'Set up build disk on hosted runners'
-description: |
-  Relocate Docker's data-root to /mnt (which has ~75 GB free, vs ~20 GB
-  on / after free-disk-space). Combined with the apt cleanup, gives
-  ~100 GB working space for buildx — enough for ROCm dev image + vLLM
-  torch install + flash-attn build.
-
-  No-op on:
-    - self-hosted runners (no /mnt expectation)
-    - non-X64 runners (verify /mnt shape on ubuntu-24.04-arm separately
-      before enabling there — see Task 3.2 in the migration plan)
-    - mode=skip (force-disable from caller)
-
-  Must run after free-disk-space (which removes large packages — would
-  fail mid-uninstall if Docker were stopped) and before any Docker
-  operation (setup-qemu, setup-buildx, login, build) so the relocated
-  data-root catches all subsequent docker activity.
-
-inputs:
-  mode:
-    description: 'auto (default — relocate on hosted X64 only) or skip'
-    required: false
-    default: 'auto'
-
-runs:
-  using: 'composite'
-  steps:
-    - name: Relocate Docker data-root to /mnt
-      if: inputs.mode == 'auto' && runner.environment == 'github-hosted' && runner.arch == 'X64'
-      shell: bash
-      run: |
-        set -euo pipefail
-        echo "Before relocation:"
-        df -h / /mnt || true
-        sudo systemctl stop docker docker.socket
-        sudo mkdir -p /mnt/docker-data /mnt/docker-tmp
-        # buildx CLI runs as the unprivileged runner user and creates
-        # config dirs under TMPDIR before binding them into the buildkit
-        # container. /mnt is owned by root by default; mirror /tmp's
-        # 1777 (world-writable + sticky) so non-root processes can write.
-        sudo chmod 1777 /mnt/docker-tmp
-        if [ -d /var/lib/docker ] && [ ! -L /var/lib/docker ]; then
-          sudo rsync -a /var/lib/docker/ /mnt/docker-data/
-          sudo rm -rf /var/lib/docker
-          sudo ln -s /mnt/docker-data /var/lib/docker
-        fi
-        # daemon.json may not exist; merge data-root in or create minimal.
-        if [ -f /etc/docker/daemon.json ]; then
-          sudo jq '."data-root" = "/mnt/docker-data"' /etc/docker/daemon.json | sudo tee /etc/docker/daemon.json.new >/dev/null
-          sudo mv /etc/docker/daemon.json.new /etc/docker/daemon.json
-        else
-          echo '{"data-root":"/mnt/docker-data"}' | sudo tee /etc/docker/daemon.json
-        fi
-        sudo systemctl start docker
-        # Make TMPDIR persist for subsequent steps in the same job.
-        echo "TMPDIR=/mnt/docker-tmp" >> "$GITHUB_ENV"
-        echo "After relocation:"
-        df -h / /mnt
-        docker info | grep -i 'docker root dir' || true

.github/bump_vllm_wheel.sh

@@ -1,45 +0,0 @@
-#!/bin/bash
-# Bump the cublas13 vLLM wheel pin in requirements-cublas13-after.txt.
-#
-# vLLM's PyPI wheel is built against CUDA 12 so the cublas13 build pulls a
-# cu130-flavoured wheel from vLLM's per-tag index at
-# https://wheels.vllm.ai/<TAG>/cu130/. That URL segment is itself version-locked
-# (no /latest/ alias upstream), so bumping vLLM means rewriting both the URL
-# segment and the version constraint atomically. bump_deps.sh handles git-sha
-# vars in Makefiles; this script handles the two-value rewrite specific to the
-# vLLM requirements file.
-set -xe
-REPO=$1   # vllm-project/vllm
-FILE=$2   # backend/python/vllm/requirements-cublas13-after.txt
-VAR=$3    # VLLM_VERSION (used for output file names so the workflow can read them)
-
-if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
-    echo "usage: $0 <repo> <requirements-file> <var-name>" >&2
-    exit 1
-fi
-
-# /releases/latest returns the most recent non-prerelease tag.
-LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
-    "https://api.github.com/repos/$REPO/releases/latest" \
-    | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
-
-# Strip leading 'v' (vLLM tags are 'v0.20.0', the URL/version use '0.20.0').
-NEW_VERSION="${LATEST_TAG#v}"
-
-set +e
-CURRENT_VERSION=$(grep -oE '^vllm==[0-9]+\.[0-9]+\.[0-9]+' "$FILE" | head -1 | cut -d= -f3)
-set -e
-
-# sed both lines unconditionally — peter-evans/create-pull-request opens no PR
-# when the working tree is clean, so a no-op rewrite is safe.
-sed -i "$FILE" \
-    -e "s|wheels\.vllm\.ai/[^/]*/cu130|wheels.vllm.ai/$NEW_VERSION/cu130|g" \
-    -e "s|^vllm==.*|vllm==$NEW_VERSION|"
-
-if [ -z "$CURRENT_VERSION" ]; then
-    echo "Could not find vllm==X.Y.Z in $FILE."
-    exit 0
-fi
-
-echo "Changes: https://github.com/$REPO/compare/v${CURRENT_VERSION}...${LATEST_TAG}" >> "${VAR}_message.txt"
-echo "${NEW_VERSION}" >> "${VAR}_commit.txt"

.github/gallery-agent/agent.go

@@ -0,0 +1,445 @@
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"regexp"
+	"slices"
+	"strings"
+
+	"github.com/ghodss/yaml"
+	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
+	"github.com/mudler/cogito"
+	"github.com/mudler/cogito/clients"
+	"github.com/mudler/cogito/structures"
+	"github.com/sashabaranov/go-openai/jsonschema"
+)
+
+var (
+	openAIModel      = os.Getenv("OPENAI_MODEL")
+	openAIKey        = os.Getenv("OPENAI_KEY")
+	openAIBaseURL    = os.Getenv("OPENAI_BASE_URL")
+	galleryIndexPath = os.Getenv("GALLERY_INDEX_PATH")
+	//defaultclient
+	llm = clients.NewOpenAILLM(openAIModel, openAIKey, openAIBaseURL)
+)
+
+// cleanTextContent removes trailing spaces, tabs, and normalizes line endings
+// to prevent YAML linting issues like trailing spaces and multiple empty lines
+func cleanTextContent(text string) string {
+	lines := strings.Split(text, "\n")
+	var cleanedLines []string
+	var prevEmpty bool
+	for _, line := range lines {
+		// Remove all trailing whitespace (spaces, tabs, etc.)
+		trimmed := strings.TrimRight(line, " \t\r")
+		// Avoid multiple consecutive empty lines
+		if trimmed == "" {
+			if !prevEmpty {
+				cleanedLines = append(cleanedLines, "")
+			}
+			prevEmpty = true
+		} else {
+			cleanedLines = append(cleanedLines, trimmed)
+			prevEmpty = false
+		}
+	}
+	// Remove trailing empty lines from the result
+	result := strings.Join(cleanedLines, "\n")
+	return stripThinkingTags(strings.TrimRight(result, "\n"))
+}
+
+type galleryModel struct {
+	Name string   `yaml:"name"`
+	Urls []string `yaml:"urls"`
+}
+
+// isModelExisting checks if a specific model ID exists in the gallery using text search
+func isModelExisting(modelID string) (bool, error) {
+	indexPath := getGalleryIndexPath()
+	content, err := os.ReadFile(indexPath)
+	if err != nil {
+		return false, fmt.Errorf("failed to read %s: %w", indexPath, err)
+	}
+
+	var galleryModels []galleryModel
+
+	err = yaml.Unmarshal(content, &galleryModels)
+	if err != nil {
+		return false, fmt.Errorf("failed to unmarshal %s: %w", indexPath, err)
+	}
+
+	for _, galleryModel := range galleryModels {
+		if slices.Contains(galleryModel.Urls, modelID) {
+			return true, nil
+		}
+	}
+
+	return false, nil
+}
+
+// filterExistingModels removes models that already exist in the gallery
+func filterExistingModels(models []ProcessedModel) ([]ProcessedModel, error) {
+	var filteredModels []ProcessedModel
+	for _, model := range models {
+		exists, err := isModelExisting(model.ModelID)
+		if err != nil {
+			fmt.Printf("Error checking if model %s exists: %v, skipping\n", model.ModelID, err)
+			continue
+		}
+
+		if !exists {
+			filteredModels = append(filteredModels, model)
+		} else {
+			fmt.Printf("Skipping existing model: %s\n", model.ModelID)
+		}
+	}
+
+	fmt.Printf("Filtered out %d existing models, %d new models remaining\n",
+		len(models)-len(filteredModels), len(filteredModels))
+
+	return filteredModels, nil
+}
+
+// getGalleryIndexPath returns the gallery index file path, with a default fallback
+func getGalleryIndexPath() string {
+	if galleryIndexPath != "" {
+		return galleryIndexPath
+	}
+	return "gallery/index.yaml"
+}
+
+func stripThinkingTags(content string) string {
+	// Remove content between <thinking> and </thinking> (including multi-line)
+	content = regexp.MustCompile(`(?s)<thinking>.*?</thinking>`).ReplaceAllString(content, "")
+	// Remove content between <think> and </think> (including multi-line)
+	content = regexp.MustCompile(`(?s)<think>.*?</think>`).ReplaceAllString(content, "")
+	// Clean up any extra whitespace
+	content = strings.TrimSpace(content)
+	return content
+}
+
+func getRealReadme(ctx context.Context, repository string) (string, error) {
+	// Create a conversation fragment
+	fragment := cogito.NewEmptyFragment().
+		AddMessage("user",
+			`Your task is to get a clear description of a large language model from huggingface by using the provided tool. I will share with you a repository that might be quantized, and as such probably not by the original model author. We need to get the real  description of the model, and not the one that might be quantized. You will have to call the tool to get the readme more than once by figuring out from the quantized readme which is the base model readme. This is the repository: `+repository)
+
+	// Execute with tools
+	result, err := cogito.ExecuteTools(llm, fragment,
+		cogito.WithIterations(3),
+		cogito.WithMaxAttempts(3),
+		cogito.WithTools(&HFReadmeTool{client: hfapi.NewClient()}))
+	if err != nil {
+		return "", err
+	}
+
+	result = result.AddMessage("user", "Describe the model in a clear and concise way that can be shared in a model gallery.")
+
+	// Get a response
+	_, err = llm.Ask(ctx, result)
+	if err != nil {
+		return "", err
+	}
+
+	content := result.LastMessage().Content
+	return cleanTextContent(content), nil
+}
+
+func selectMostInterestingModels(ctx context.Context, searchResult *SearchResult) ([]ProcessedModel, error) {
+
+	if len(searchResult.Models) == 1 {
+		return searchResult.Models, nil
+	}
+
+	// Create a conversation fragment
+	fragment := cogito.NewEmptyFragment().
+		AddMessage("user",
+			`Your task is to analyze a list of AI models and select the most interesting ones for a model gallery. You will be given detailed information about multiple models including their metadata, file information, and README content.
+
+Consider the following criteria when selecting models:
+1. Model popularity (download count)
+2. Model recency (last modified date)
+3. Model completeness (has preferred model file, README, etc.)
+4. Model uniqueness (not duplicates or very similar models)
+5. Model quality (based on README content and description)
+6. Model utility (practical applications)
+
+You should select models that would be most valuable for users browsing a model gallery. Prioritize models that are:
+- Well-documented with clear READMEs
+- Recently updated
+- Popular (high download count)
+- Have the preferred quantization format available
+- Offer unique capabilities or are from reputable authors
+
+Return your analysis and selection reasoning.`)
+
+	// Add the search results as context
+	modelsInfo := fmt.Sprintf("Found %d models matching '%s' with quantization preference '%s':\n\n",
+		searchResult.TotalModelsFound, searchResult.SearchTerm, searchResult.Quantization)
+
+	for i, model := range searchResult.Models {
+		modelsInfo += fmt.Sprintf("Model %d:\n", i+1)
+		modelsInfo += fmt.Sprintf("  ID: %s\n", model.ModelID)
+		modelsInfo += fmt.Sprintf("  Author: %s\n", model.Author)
+		modelsInfo += fmt.Sprintf("  Downloads: %d\n", model.Downloads)
+		modelsInfo += fmt.Sprintf("  Last Modified: %s\n", model.LastModified)
+		modelsInfo += fmt.Sprintf("  Files: %d files\n", len(model.Files))
+
+		if model.PreferredModelFile != nil {
+			modelsInfo += fmt.Sprintf("  Preferred Model File: %s (%d bytes)\n",
+				model.PreferredModelFile.Path, model.PreferredModelFile.Size)
+		} else {
+			modelsInfo += "  No preferred model file found\n"
+		}
+
+		if model.ReadmeContent != "" {
+			modelsInfo += fmt.Sprintf("  README: %s\n", model.ReadmeContent)
+		}
+
+		if model.ProcessingError != "" {
+			modelsInfo += fmt.Sprintf("  Processing Error: %s\n", model.ProcessingError)
+		}
+
+		modelsInfo += "\n"
+	}
+
+	fragment = fragment.AddMessage("user", modelsInfo)
+
+	fragment = fragment.AddMessage("user", "Based on your analysis, select the top 5 most interesting models and provide a brief explanation for each selection. Also, create a filtered SearchResult with only the selected models. Return just a list of repositories IDs, you will later be asked to output it as a JSON array with the json tool.")
+
+	// Get a response
+	newFragment, err := llm.Ask(ctx, fragment)
+	if err != nil {
+		return nil, err
+	}
+
+	fmt.Println(newFragment.LastMessage().Content)
+	repositories := struct {
+		Repositories []string `json:"repositories"`
+	}{}
+
+	s := structures.Structure{
+		Schema: jsonschema.Definition{
+			Type:                 jsonschema.Object,
+			AdditionalProperties: false,
+			Properties: map[string]jsonschema.Definition{
+				"repositories": {
+					Type:        jsonschema.Array,
+					Items:       &jsonschema.Definition{Type: jsonschema.String},
+					Description: "The trending repositories IDs",
+				},
+			},
+			Required: []string{"repositories"},
+		},
+		Object: &repositories,
+	}
+
+	err = newFragment.ExtractStructure(ctx, llm, s)
+	if err != nil {
+		return nil, err
+	}
+
+	filteredModels := []ProcessedModel{}
+	for _, m := range searchResult.Models {
+		if slices.Contains(repositories.Repositories, m.ModelID) {
+			filteredModels = append(filteredModels, m)
+		}
+	}
+
+	return filteredModels, nil
+}
+
+// ModelMetadata represents extracted metadata from a model
+type ModelMetadata struct {
+	Tags    []string `json:"tags"`
+	License string   `json:"license"`
+}
+
+// extractModelMetadata extracts tags and license from model README and documentation
+func extractModelMetadata(ctx context.Context, model ProcessedModel) ([]string, string, error) {
+	// Create a conversation fragment
+	fragment := cogito.NewEmptyFragment().
+		AddMessage("user",
+			`Your task is to extract metadata from an AI model's README and documentation. You will be provided with:
+1. Model information (ID, author, description)
+2. README content
+
+You need to extract:
+1. **Tags**: An array of relevant tags that describe the model. Use common tags from the gallery such as:
+   - llm, gguf, gpu, cpu, multimodal, image-to-text, text-to-text, text-to-speech, tts
+   - thinking, reasoning, chat, instruction-tuned, code, vision
+   - Model family names (e.g., llama, qwen, mistral, gemma) if applicable
+   - Any other relevant descriptive tags
+   Select 3-8 most relevant tags.
+
+2. **License**: The license identifier (e.g., "apache-2.0", "mit", "llama2", "gpl-3.0", "bsd", "cc-by-4.0").
+   If no license is found, return an empty string.
+
+Return the extracted metadata in a structured format.`)
+
+	// Add model information
+	modelInfo := "Model Information:\n"
+	modelInfo += fmt.Sprintf("  ID: %s\n", model.ModelID)
+	modelInfo += fmt.Sprintf("  Author: %s\n", model.Author)
+	modelInfo += fmt.Sprintf("  Downloads: %d\n", model.Downloads)
+	if model.ReadmeContent != "" {
+		modelInfo += fmt.Sprintf("  README Content:\n%s\n", model.ReadmeContent)
+	} else if model.ReadmeContentPreview != "" {
+		modelInfo += fmt.Sprintf("  README Preview: %s\n", model.ReadmeContentPreview)
+	}
+
+	fragment = fragment.AddMessage("user", modelInfo)
+	fragment = fragment.AddMessage("user", "Extract the tags and license from the model information. Return the metadata as a JSON object with 'tags' (array of strings) and 'license' (string).")
+
+	// Get a response
+	newFragment, err := llm.Ask(ctx, fragment)
+	if err != nil {
+		return nil, "", err
+	}
+
+	// Extract structured metadata
+	metadata := ModelMetadata{}
+
+	s := structures.Structure{
+		Schema: jsonschema.Definition{
+			Type:                 jsonschema.Object,
+			AdditionalProperties: false,
+			Properties: map[string]jsonschema.Definition{
+				"tags": {
+					Type:        jsonschema.Array,
+					Items:       &jsonschema.Definition{Type: jsonschema.String},
+					Description: "Array of relevant tags describing the model",
+				},
+				"license": {
+					Type:        jsonschema.String,
+					Description: "License identifier (e.g., apache-2.0, mit, llama2). Empty string if not found.",
+				},
+			},
+			Required: []string{"tags", "license"},
+		},
+		Object: &metadata,
+	}
+
+	err = newFragment.ExtractStructure(ctx, llm, s)
+	if err != nil {
+		return nil, "", err
+	}
+
+	return metadata.Tags, metadata.License, nil
+}
+
+// extractIconFromReadme scans the README content for image URLs and returns the first suitable icon URL found
+func extractIconFromReadme(readmeContent string) string {
+	if readmeContent == "" {
+		return ""
+	}
+
+	// Regular expressions to match image URLs in various formats (case-insensitive)
+	// Match markdown image syntax: ![alt](url) - case insensitive extensions
+	markdownImageRegex := regexp.MustCompile(`(?i)!\[[^\]]*\]\(([^)]+\.(png|jpg|jpeg|svg|webp|gif))\)`)
+	// Match HTML img tags: <img src="url">
+	htmlImageRegex := regexp.MustCompile(`(?i)<img[^>]+src=["']([^"']+\.(png|jpg|jpeg|svg|webp|gif))["']`)
+	// Match plain URLs ending with image extensions
+	plainImageRegex := regexp.MustCompile(`(?i)https?://[^\s<>"']+\.(png|jpg|jpeg|svg|webp|gif)`)
+
+	// Try markdown format first
+	matches := markdownImageRegex.FindStringSubmatch(readmeContent)
+	if len(matches) > 1 && matches[1] != "" {
+		url := strings.TrimSpace(matches[1])
+		// Prefer HuggingFace CDN URLs or absolute URLs
+		if strings.HasPrefix(strings.ToLower(url), "http") {
+			return url
+		}
+	}
+
+	// Try HTML img tags
+	matches = htmlImageRegex.FindStringSubmatch(readmeContent)
+	if len(matches) > 1 && matches[1] != "" {
+		url := strings.TrimSpace(matches[1])
+		if strings.HasPrefix(strings.ToLower(url), "http") {
+			return url
+		}
+	}
+
+	// Try plain URLs
+	matches = plainImageRegex.FindStringSubmatch(readmeContent)
+	if len(matches) > 0 {
+		url := strings.TrimSpace(matches[0])
+		if strings.HasPrefix(strings.ToLower(url), "http") {
+			return url
+		}
+	}
+
+	return ""
+}
+
+// getHuggingFaceAvatarURL attempts to get the HuggingFace avatar URL for a user
+func getHuggingFaceAvatarURL(author string) string {
+	if author == "" {
+		return ""
+	}
+
+	// Try to fetch user info from HuggingFace API
+	// HuggingFace API endpoint: https://huggingface.co/api/users/{username}
+	baseURL := "https://huggingface.co"
+	userURL := fmt.Sprintf("%s/api/users/%s", baseURL, author)
+
+	req, err := http.NewRequest("GET", userURL, nil)
+	if err != nil {
+		return ""
+	}
+
+	client := &http.Client{}
+	resp, err := client.Do(req)
+	if err != nil {
+		return ""
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		return ""
+	}
+
+	// Parse the response to get avatar URL
+	var userInfo map[string]interface{}
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return ""
+	}
+
+	if err := json.Unmarshal(body, &userInfo); err != nil {
+		return ""
+	}
+
+	// Try to extract avatar URL from response
+	if avatar, ok := userInfo["avatarUrl"].(string); ok && avatar != "" {
+		return avatar
+	}
+	if avatar, ok := userInfo["avatar"].(string); ok && avatar != "" {
+		return avatar
+	}
+
+	return ""
+}
+
+// extractModelIcon extracts icon URL from README or falls back to HuggingFace avatar
+func extractModelIcon(model ProcessedModel) string {
+	// First, try to extract icon from README
+	if icon := extractIconFromReadme(model.ReadmeContent); icon != "" {
+		return icon
+	}
+
+	// Fallback: Try to get HuggingFace user avatar
+	if model.Author != "" {
+		if avatar := getHuggingFaceAvatarURL(model.Author); avatar != "" {
+			return avatar
+		}
+	}
+
+	return ""
+}

.github/gallery-agent/gallery.go

@@ -7,8 +7,8 @@ import (
 	"os"
 	"strings"
 
+	"github.com/ghodss/yaml"
 	"github.com/mudler/LocalAI/core/gallery/importers"
-	"sigs.k8s.io/yaml"
 )
 
 func formatTextContent(text string) string {
@@ -79,20 +79,7 @@ func generateYAMLEntry(model ProcessedModel, quantization string) string {
 	description = cleanTextContent(description)
 	formattedDescription := formatTextContent(description)
 
-	// Strip name and description from config file since they are
-	// already present at the gallery entry level and should not
-	// appear under overrides.
-	configFileContent := modelConfig.ConfigFile
-	var cfgMap map[string]any
-	if err := yaml.Unmarshal([]byte(configFileContent), &cfgMap); err == nil {
-		delete(cfgMap, "name")
-		delete(cfgMap, "description")
-		if cleaned, err := yaml.Marshal(cfgMap); err == nil {
-			configFileContent = string(cleaned)
-		}
-	}
-
-	configFile := formatTextContent(configFileContent)
+	configFile := formatTextContent(modelConfig.ConfigFile)
 
 	filesYAML, _ := yaml.Marshal(modelConfig.Files)
 

.github/gallery-agent/helpers.go

@@ -1,301 +0,0 @@
-package main
-
-import (
-	"encoding/json"
-	"fmt"
-	"io"
-	"net/http"
-	"os"
-	"regexp"
-	"strings"
-
-	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
-	"sigs.k8s.io/yaml"
-)
-
-var galleryIndexPath = os.Getenv("GALLERY_INDEX_PATH")
-
-// getGalleryIndexPath returns the gallery index file path, with a default fallback
-func getGalleryIndexPath() string {
-	if galleryIndexPath != "" {
-		return galleryIndexPath
-	}
-	return "gallery/index.yaml"
-}
-
-type galleryModel struct {
-	Name string   `yaml:"name"`
-	Urls []string `yaml:"urls"`
-}
-
-// loadGalleryURLSet parses gallery/index.yaml once and returns the set of
-// HuggingFace model URLs already present in the gallery.
-func loadGalleryURLSet() (map[string]struct{}, error) {
-	indexPath := getGalleryIndexPath()
-	content, err := os.ReadFile(indexPath)
-	if err != nil {
-		return nil, fmt.Errorf("failed to read %s: %w", indexPath, err)
-	}
-
-	var galleryModels []galleryModel
-	if err := yaml.Unmarshal(content, &galleryModels); err != nil {
-		return nil, fmt.Errorf("failed to unmarshal %s: %w", indexPath, err)
-	}
-
-	set := make(map[string]struct{}, len(galleryModels))
-	for _, gm := range galleryModels {
-		for _, u := range gm.Urls {
-			set[u] = struct{}{}
-		}
-	}
-
-	// Also skip URLs already proposed in open (unmerged) gallery-agent PRs.
-	// The workflow injects these via EXTRA_SKIP_URLS so we don't keep
-	// re-proposing the same model every run while a PR is waiting to merge.
-	for _, line := range strings.FieldsFunc(os.Getenv("EXTRA_SKIP_URLS"), func(r rune) bool {
-		return r == '\n' || r == ',' || r == ' '
-	}) {
-		u := strings.TrimSpace(line)
-		if u != "" {
-			set[u] = struct{}{}
-		}
-	}
-
-	return set, nil
-}
-
-// modelAlreadyInGallery checks whether a HuggingFace model repo is already
-// referenced in the gallery URL set.
-func modelAlreadyInGallery(set map[string]struct{}, modelID string) bool {
-	_, ok := set["https://huggingface.co/"+modelID]
-	return ok
-}
-
-// baseModelFromTags returns the first `base_model:<repo>` value found in the
-// tag list, or "" if none is present. HuggingFace surfaces the base model
-// declared in the model card's YAML frontmatter as such a tag.
-func baseModelFromTags(tags []string) string {
-	for _, t := range tags {
-		if strings.HasPrefix(t, "base_model:") {
-			return strings.TrimPrefix(t, "base_model:")
-		}
-	}
-	return ""
-}
-
-// licenseFromTags returns the `license:<id>` value from the tag list, or "".
-func licenseFromTags(tags []string) string {
-	for _, t := range tags {
-		if strings.HasPrefix(t, "license:") {
-			return strings.TrimPrefix(t, "license:")
-		}
-	}
-	return ""
-}
-
-// curatedTags produces the gallery tag list from HuggingFace's raw tag set.
-// Always includes llm + gguf, then adds whitelisted family / capability
-// markers when they appear in the HF tag list.
-func curatedTags(hfTags []string) []string {
-	whitelist := []string{
-		"gpu", "cpu",
-		"llama", "mistral", "mixtral", "qwen", "qwen2", "qwen3",
-		"gemma", "gemma2", "gemma3", "phi", "phi3", "phi4",
-		"deepseek", "yi", "falcon", "command-r",
-		"vision", "multimodal", "code", "chat",
-		"instruction-tuned", "reasoning", "thinking",
-	}
-	seen := map[string]struct{}{}
-	out := []string{"llm", "gguf"}
-	seen["llm"] = struct{}{}
-	seen["gguf"] = struct{}{}
-
-	hfSet := map[string]struct{}{}
-	for _, t := range hfTags {
-		hfSet[strings.ToLower(t)] = struct{}{}
-	}
-	for _, w := range whitelist {
-		if _, ok := hfSet[w]; ok {
-			if _, dup := seen[w]; !dup {
-				out = append(out, w)
-				seen[w] = struct{}{}
-			}
-		}
-	}
-	return out
-}
-
-// resolveReadme fetches a description-quality README for a (possibly
-// quantized) repo: if a `base_model:` tag is present, fetch the base repo's
-// README; otherwise fall back to the repo's own README.
-func resolveReadme(client *hfapi.Client, modelID string, hfTags []string) (string, error) {
-	if base := baseModelFromTags(hfTags); base != "" && base != modelID {
-		if content, err := client.GetReadmeContent(base, "README.md"); err == nil && strings.TrimSpace(content) != "" {
-			return cleanTextContent(content), nil
-		}
-	}
-	content, err := client.GetReadmeContent(modelID, "README.md")
-	if err != nil {
-		return "", err
-	}
-	return cleanTextContent(content), nil
-}
-
-// extractDescription turns a raw HuggingFace README into a concise plain-text
-// description suitable for embedding in gallery/index.yaml: strips YAML
-// frontmatter, HTML tags/comments, markdown images, link URLs (keeping the
-// link text), markdown tables, and then truncates at a paragraph boundary
-// around ~1200 characters. Raw README should still be used for icon
-// extraction — call this only for the `description:` field.
-func extractDescription(readme string) string {
-	s := readme
-
-	// Strip leading YAML frontmatter: `---\n...\n---\n` at start of file.
-	if strings.HasPrefix(strings.TrimLeft(s, " \t\n"), "---") {
-		trimmed := strings.TrimLeft(s, " \t\n")
-		rest := strings.TrimPrefix(trimmed, "---")
-		if idx := strings.Index(rest, "\n---"); idx >= 0 {
-			after := rest[idx+len("\n---"):]
-			after = strings.TrimPrefix(after, "\n")
-			s = after
-		}
-	}
-
-	// Strip HTML comments and tags.
-	s = regexp.MustCompile(`(?s)<!--.*?-->`).ReplaceAllString(s, "")
-	s = regexp.MustCompile(`(?is)<[^>]+>`).ReplaceAllString(s, "")
-
-	// Strip markdown images entirely.
-	s = regexp.MustCompile(`!\[[^\]]*\]\([^)]*\)`).ReplaceAllString(s, "")
-	// Replace markdown links `[text](url)` with just `text`.
-	s = regexp.MustCompile(`\[([^\]]+)\]\([^)]+\)`).ReplaceAllString(s, "$1")
-
-	// Drop table lines and horizontal rules, and flatten all leading
-	// whitespace: generateYAMLEntry embeds this under a `description: |`
-	// literal block whose indentation is set by the first non-empty line.
-	// If any line has extra leading whitespace (e.g. from an indented
-	// `<p align="center">` block in the original README), YAML will pick
-	// that up as the block's indent and every later line at a smaller
-	// indent blows the block scalar. Stripping leading whitespace here
-	// guarantees uniform 4-space indentation after formatTextContent runs.
-	var kept []string
-	for _, line := range strings.Split(s, "\n") {
-		t := strings.TrimLeft(line, " \t")
-		ts := strings.TrimSpace(t)
-		if strings.HasPrefix(ts, "|") {
-			continue
-		}
-		if strings.HasPrefix(ts, ":--") || strings.HasPrefix(ts, "---") || strings.HasPrefix(ts, "===") {
-			continue
-		}
-		kept = append(kept, t)
-	}
-	s = strings.Join(kept, "\n")
-
-	// Normalise whitespace and drop any leading blank lines so the literal
-	// block in YAML doesn't start with a blank first line (which would
-	// break the indentation detector the same way).
-	s = cleanTextContent(s)
-	s = strings.TrimLeft(s, " \t\n")
-
-	// Truncate at a paragraph boundary around maxLen chars.
-	const maxLen = 1200
-	if len(s) > maxLen {
-		cut := strings.LastIndex(s[:maxLen], "\n\n")
-		if cut < maxLen/3 {
-			cut = maxLen
-		}
-		s = strings.TrimRight(s[:cut], " \t\n") + "\n\n..."
-	}
-
-	return s
-}
-
-// cleanTextContent removes trailing spaces/tabs and collapses multiple empty
-// lines so README content embeds cleanly into YAML without lint noise.
-func cleanTextContent(text string) string {
-	lines := strings.Split(text, "\n")
-	var cleaned []string
-	var prevEmpty bool
-	for _, line := range lines {
-		trimmed := strings.TrimRight(line, " \t\r")
-		if trimmed == "" {
-			if !prevEmpty {
-				cleaned = append(cleaned, "")
-			}
-			prevEmpty = true
-		} else {
-			cleaned = append(cleaned, trimmed)
-			prevEmpty = false
-		}
-	}
-	return strings.TrimRight(strings.Join(cleaned, "\n"), "\n")
-}
-
-// extractIconFromReadme scans README content for an image URL usable as a
-// gallery entry icon.
-func extractIconFromReadme(readmeContent string) string {
-	if readmeContent == "" {
-		return ""
-	}
-
-	markdownImageRegex := regexp.MustCompile(`(?i)!\[[^\]]*\]\(([^)]+\.(png|jpg|jpeg|svg|webp|gif))\)`)
-	htmlImageRegex := regexp.MustCompile(`(?i)<img[^>]+src=["']([^"']+\.(png|jpg|jpeg|svg|webp|gif))["']`)
-	plainImageRegex := regexp.MustCompile(`(?i)https?://[^\s<>"']+\.(png|jpg|jpeg|svg|webp|gif)`)
-
-	if m := markdownImageRegex.FindStringSubmatch(readmeContent); len(m) > 1 && strings.HasPrefix(strings.ToLower(m[1]), "http") {
-		return strings.TrimSpace(m[1])
-	}
-	if m := htmlImageRegex.FindStringSubmatch(readmeContent); len(m) > 1 && strings.HasPrefix(strings.ToLower(m[1]), "http") {
-		return strings.TrimSpace(m[1])
-	}
-	if m := plainImageRegex.FindStringSubmatch(readmeContent); len(m) > 0 && strings.HasPrefix(strings.ToLower(m[0]), "http") {
-		return strings.TrimSpace(m[0])
-	}
-	return ""
-}
-
-// getHuggingFaceAvatarURL returns the HF avatar URL for a user, or "".
-func getHuggingFaceAvatarURL(author string) string {
-	if author == "" {
-		return ""
-	}
-	userURL := fmt.Sprintf("https://huggingface.co/api/users/%s/overview", author)
-	resp, err := http.Get(userURL)
-	if err != nil {
-		return ""
-	}
-	defer resp.Body.Close()
-	if resp.StatusCode != http.StatusOK {
-		return ""
-	}
-	body, err := io.ReadAll(resp.Body)
-	if err != nil {
-		return ""
-	}
-	var info map[string]any
-	if err := json.Unmarshal(body, &info); err != nil {
-		return ""
-	}
-	if v, ok := info["avatarUrl"].(string); ok && v != "" {
-		return v
-	}
-	if v, ok := info["avatar"].(string); ok && v != "" {
-		return v
-	}
-	return ""
-}
-
-// extractModelIcon extracts an icon URL from the README, falling back to the
-// HuggingFace user avatar.
-func extractModelIcon(model ProcessedModel) string {
-	if icon := extractIconFromReadme(model.ReadmeContent); icon != "" {
-		return icon
-	}
-	if model.Author != "" {
-		if avatar := getHuggingFaceAvatarURL(model.Author); avatar != "" {
-			return avatar
-		}
-	}
-	return ""
-}

.github/gallery-agent/main.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"os"
 	"strconv"
+	"strings"
 	"time"
 
 	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
@@ -38,6 +39,16 @@ type ProcessedModel struct {
 	Icon                    string               `json:"icon,omitempty"`
 }
 
+// SearchResult represents the complete result of searching and processing models
+type SearchResult struct {
+	SearchTerm       string           `json:"search_term"`
+	Limit            int              `json:"limit"`
+	Quantization     string           `json:"quantization"`
+	TotalModelsFound int              `json:"total_models_found"`
+	Models           []ProcessedModel `json:"models"`
+	FormattedOutput  string           `json:"formatted_output"`
+}
+
 // AddedModelSummary represents a summary of models added to the gallery
 type AddedModelSummary struct {
 	SearchTerm     string   `json:"search_term"`
@@ -52,16 +63,19 @@ type AddedModelSummary struct {
 func main() {
 	startTime := time.Now()
 
-	// Synthetic mode for local testing
-	if sm := os.Getenv("SYNTHETIC_MODE"); sm == "true" || sm == "1" {
+	// Check for synthetic mode
+	syntheticMode := os.Getenv("SYNTHETIC_MODE")
+	if syntheticMode == "true" || syntheticMode == "1" {
 		fmt.Println("Running in SYNTHETIC MODE - generating random test data")
-		if err := runSyntheticMode(); err != nil {
+		err := runSyntheticMode()
+		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error in synthetic mode: %v\n", err)
 			os.Exit(1)
 		}
 		return
 	}
 
+	// Get configuration from environment variables
 	searchTerm := os.Getenv("SEARCH_TERM")
 	if searchTerm == "" {
 		searchTerm = "GGUF"
@@ -69,7 +83,7 @@ func main() {
 
 	limitStr := os.Getenv("LIMIT")
 	if limitStr == "" {
-		limitStr = "15"
+		limitStr = "5"
 	}
 	limit, err := strconv.Atoi(limitStr)
 	if err != nil {
@@ -78,197 +92,287 @@ func main() {
 	}
 
 	quantization := os.Getenv("QUANTIZATION")
-	if quantization == "" {
-		quantization = "Q4_K_M"
-	}
 
-	maxModelsStr := os.Getenv("MAX_MODELS")
-	if maxModelsStr == "" {
-		maxModelsStr = "1"
+	maxModels := os.Getenv("MAX_MODELS")
+	if maxModels == "" {
+		maxModels = "1"
 	}
-	maxModels, err := strconv.Atoi(maxModelsStr)
+	maxModelsInt, err := strconv.Atoi(maxModels)
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "Error parsing MAX_MODELS: %v\n", err)
 		os.Exit(1)
 	}
 
+	// Print configuration
 	fmt.Printf("Gallery Agent Configuration:\n")
 	fmt.Printf("  Search Term: %s\n", searchTerm)
 	fmt.Printf("  Limit: %d\n", limit)
 	fmt.Printf("  Quantization: %s\n", quantization)
-	fmt.Printf("  Max Models to Add: %d\n", maxModels)
-	fmt.Printf("  Gallery Index Path: %s\n", getGalleryIndexPath())
+	fmt.Printf("  Max Models to Add: %d\n", maxModelsInt)
+	fmt.Printf("  Gallery Index Path: %s\n", os.Getenv("GALLERY_INDEX_PATH"))
 	fmt.Println()
 
-	// Phase 1: load current gallery and query HuggingFace.
-	gallerySet, err := loadGalleryURLSet()
+	result, err := searchAndProcessModels(searchTerm, limit, quantization)
 	if err != nil {
-		fmt.Fprintf(os.Stderr, "Error loading gallery index: %v\n", err)
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
 		os.Exit(1)
 	}
-	fmt.Printf("Loaded %d existing gallery entries\n", len(gallerySet))
 
-	client := hfapi.NewClient()
+	fmt.Println(result.FormattedOutput)
+	var models []ProcessedModel
 
-	fmt.Println("Searching for trending models on HuggingFace...")
-	rawModels, err := client.GetTrending(searchTerm, limit)
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "Error fetching models: %v\n", err)
-		os.Exit(1)
-	}
-	fmt.Printf("Found %d trending models matching %q\n", len(rawModels), searchTerm)
-	totalFound := len(rawModels)
-
-	// Phase 2: drop anything already in the gallery *before* any expensive
-	// per-model work (GetModelDetails, README fetches, icon lookups).
-	fresh := rawModels[:0]
-	for _, m := range rawModels {
-		if modelAlreadyInGallery(gallerySet, m.ModelID) {
-			fmt.Printf("Skipping existing model: %s\n", m.ModelID)
-			continue
+	if len(result.Models) > 1 {
+		fmt.Println("More than one model found (", len(result.Models), "), using AI agent to select the most interesting models")
+		for _, model := range result.Models {
+			fmt.Println("Model: ", model.ModelID)
 		}
-		fresh = append(fresh, m)
+		// Use AI agent to select the most interesting models
+		fmt.Println("Using AI agent to select the most interesting models...")
+		models, err = selectMostInterestingModels(context.Background(), result)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Error in model selection: %v\n", err)
+			// Continue with original result if selection fails
+			models = result.Models
+		}
+	} else if len(result.Models) == 1 {
+		models = result.Models
+		fmt.Println("Only one model found, using it directly")
 	}
-	fmt.Printf("%d candidates after gallery dedup\n", len(fresh))
 
-	// Phase 3: HuggingFace already returned these in trendingScore order —
-	// just cap to MAX_MODELS.
-	if len(fresh) > maxModels {
-		fresh = fresh[:maxModels]
+	fmt.Print(models)
+
+	// Filter out models that already exist in the gallery
+	fmt.Println("Filtering out existing models...")
+	models, err = filterExistingModels(models)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error filtering existing models: %v\n", err)
+		os.Exit(1)
 	}
-	if len(fresh) == 0 {
+
+	// Limit to maxModelsInt after filtering
+	if len(models) > maxModelsInt {
+		models = models[:maxModelsInt]
+	}
+
+	// Track added models for summary
+	var addedModelIDs []string
+	var addedModelURLs []string
+
+	// Generate YAML entries and append to gallery/index.yaml
+	if len(models) > 0 {
+		for _, model := range models {
+			addedModelIDs = append(addedModelIDs, model.ModelID)
+			// Generate Hugging Face URL for the model
+			modelURL := fmt.Sprintf("https://huggingface.co/%s", model.ModelID)
+			addedModelURLs = append(addedModelURLs, modelURL)
+		}
+		fmt.Println("Generating YAML entries for selected models...")
+		err = generateYAMLForModels(context.Background(), models, quantization)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Error generating YAML entries: %v\n", err)
+			os.Exit(1)
+		}
+	} else {
 		fmt.Println("No new models to add to the gallery.")
-		writeSummary(AddedModelSummary{
-			SearchTerm:     searchTerm,
-			TotalFound:     totalFound,
-			ModelsAdded:    0,
-			Quantization:   quantization,
-			ProcessingTime: time.Since(startTime).String(),
-		})
-		return
 	}
 
-	// Phase 4: fetch details and build ProcessedModel entries for survivors.
-	var processed []ProcessedModel
-	quantPrefs := []string{quantization, "Q4_K_M", "Q4_K_S", "Q3_K_M", "Q2_K", "Q8_0"}
-	for _, m := range fresh {
-		fmt.Printf("Processing model: %s (downloads=%d)\n", m.ModelID, m.Downloads)
-
-		pm := ProcessedModel{
-			ModelID:                 m.ModelID,
-			Author:                  m.Author,
-			Downloads:               m.Downloads,
-			LastModified:            m.LastModified,
-			QuantizationPreferences: quantPrefs,
-		}
-
-		details, err := client.GetModelDetails(m.ModelID)
-		if err != nil {
-			fmt.Printf("  Error getting model details: %v (skipping)\n", err)
-			continue
-		}
-
-		preferred := hfapi.FindPreferredModelFile(details.Files, quantPrefs)
-		if preferred == nil {
-			fmt.Printf("  No GGUF file matching %v — skipping\n", quantPrefs)
-			continue
-		}
-
-		pm.Files = make([]ProcessedModelFile, len(details.Files))
-		for j, f := range details.Files {
-			fileType := "other"
-			if f.IsReadme {
-				fileType = "readme"
-			} else if f.Path == preferred.Path {
-				fileType = "model"
-			}
-			pm.Files[j] = ProcessedModelFile{
-				Path:     f.Path,
-				Size:     f.Size,
-				SHA256:   f.SHA256,
-				IsReadme: f.IsReadme,
-				FileType: fileType,
-			}
-			if f.Path == preferred.Path {
-				copyFile := pm.Files[j]
-				pm.PreferredModelFile = &copyFile
-			}
-			if f.IsReadme {
-				copyFile := pm.Files[j]
-				pm.ReadmeFile = &copyFile
-			}
-		}
-
-		// Deterministic README resolution: follow base_model tag if set.
-		// Keep the raw (HTML-bearing) README around while we extract the
-		// icon, then strip it down to a plain-text description for the
-		// `description:` YAML field.
-		readme, err := resolveReadme(client, m.ModelID, m.Tags)
-		if err != nil {
-			fmt.Printf("  Warning: failed to fetch README: %v\n", err)
-		}
-		pm.ReadmeContent = readme
-
-		pm.License = licenseFromTags(m.Tags)
-		pm.Tags = curatedTags(m.Tags)
-		pm.Icon = extractModelIcon(pm)
-
-		if pm.ReadmeContent != "" {
-			pm.ReadmeContent = extractDescription(pm.ReadmeContent)
-			pm.ReadmeContentPreview = truncateString(pm.ReadmeContent, 200)
-		}
-
-		fmt.Printf("  License: %s, Tags: %v, Icon: %s\n", pm.License, pm.Tags, pm.Icon)
-		processed = append(processed, pm)
-	}
-
-	if len(processed) == 0 {
-		fmt.Println("No processable models after detail fetch.")
-		writeSummary(AddedModelSummary{
-			SearchTerm:     searchTerm,
-			TotalFound:     totalFound,
-			ModelsAdded:    0,
-			Quantization:   quantization,
-			ProcessingTime: time.Since(startTime).String(),
-		})
-		return
-	}
-
-	// Phase 5: write YAML entries.
-	var addedIDs, addedURLs []string
-	for _, pm := range processed {
-		addedIDs = append(addedIDs, pm.ModelID)
-		addedURLs = append(addedURLs, "https://huggingface.co/"+pm.ModelID)
-	}
-
-	fmt.Println("Generating YAML entries for selected models...")
-	if err := generateYAMLForModels(context.Background(), processed, quantization); err != nil {
-		fmt.Fprintf(os.Stderr, "Error generating YAML entries: %v\n", err)
-		os.Exit(1)
-	}
-
-	writeSummary(AddedModelSummary{
+	// Create and write summary
+	processingTime := time.Since(startTime).String()
+	summary := AddedModelSummary{
 		SearchTerm:     searchTerm,
-		TotalFound:     totalFound,
-		ModelsAdded:    len(addedIDs),
-		AddedModelIDs:  addedIDs,
-		AddedModelURLs: addedURLs,
+		TotalFound:     result.TotalModelsFound,
+		ModelsAdded:    len(addedModelIDs),
+		AddedModelIDs:  addedModelIDs,
+		AddedModelURLs: addedModelURLs,
 		Quantization:   quantization,
-		ProcessingTime: time.Since(startTime).String(),
-	})
-}
+		ProcessingTime: processingTime,
+	}
 
-func writeSummary(summary AddedModelSummary) {
-	data, err := json.MarshalIndent(summary, "", "  ")
+	// Write summary to file
+	summaryData, err := json.MarshalIndent(summary, "", "  ")
 	if err != nil {
 		fmt.Fprintf(os.Stderr, "Error marshaling summary: %v\n", err)
-		return
+	} else {
+		err = os.WriteFile("gallery-agent-summary.json", summaryData, 0644)
+		if err != nil {
+			fmt.Fprintf(os.Stderr, "Error writing summary file: %v\n", err)
+		} else {
+			fmt.Printf("Summary written to gallery-agent-summary.json\n")
+		}
 	}
-	if err := os.WriteFile("gallery-agent-summary.json", data, 0644); err != nil {
-		fmt.Fprintf(os.Stderr, "Error writing summary file: %v\n", err)
-		return
+}
+
+func searchAndProcessModels(searchTerm string, limit int, quantization string) (*SearchResult, error) {
+	client := hfapi.NewClient()
+	var outputBuilder strings.Builder
+
+	fmt.Println("Searching for models...")
+	// Initialize the result struct
+	result := &SearchResult{
+		SearchTerm:   searchTerm,
+		Limit:        limit,
+		Quantization: quantization,
+		Models:       []ProcessedModel{},
 	}
-	fmt.Println("Summary written to gallery-agent-summary.json")
+
+	models, err := client.GetLatest(searchTerm, limit)
+	if err != nil {
+		return nil, fmt.Errorf("failed to fetch models: %w", err)
+	}
+
+	fmt.Println("Models found:", len(models))
+	result.TotalModelsFound = len(models)
+
+	if len(models) == 0 {
+		outputBuilder.WriteString("No models found.\n")
+		result.FormattedOutput = outputBuilder.String()
+		return result, nil
+	}
+
+	outputBuilder.WriteString(fmt.Sprintf("Found %d models matching '%s':\n\n", len(models), searchTerm))
+
+	// Process each model
+	for i, model := range models {
+		outputBuilder.WriteString(fmt.Sprintf("%d. Processing Model: %s\n", i+1, model.ModelID))
+		outputBuilder.WriteString(fmt.Sprintf("   Author: %s\n", model.Author))
+		outputBuilder.WriteString(fmt.Sprintf("   Downloads: %d\n", model.Downloads))
+		outputBuilder.WriteString(fmt.Sprintf("   Last Modified: %s\n", model.LastModified))
+
+		// Initialize processed model struct
+		processedModel := ProcessedModel{
+			ModelID:                 model.ModelID,
+			Author:                  model.Author,
+			Downloads:               model.Downloads,
+			LastModified:            model.LastModified,
+			QuantizationPreferences: []string{quantization, "Q4_K_M", "Q4_K_S", "Q3_K_M", "Q2_K"},
+		}
+
+		// Get detailed model information
+		details, err := client.GetModelDetails(model.ModelID)
+		if err != nil {
+			errorMsg := fmt.Sprintf("   Error getting model details: %v\n", err)
+			outputBuilder.WriteString(errorMsg)
+			processedModel.ProcessingError = err.Error()
+			result.Models = append(result.Models, processedModel)
+			continue
+		}
+
+		// Define quantization preferences (in order of preference)
+		quantizationPreferences := []string{quantization, "Q4_K_M", "Q4_K_S", "Q3_K_M", "Q2_K"}
+
+		// Find preferred model file
+		preferredModelFile := hfapi.FindPreferredModelFile(details.Files, quantizationPreferences)
+
+		// Process files
+		processedFiles := make([]ProcessedModelFile, len(details.Files))
+		for j, file := range details.Files {
+			fileType := "other"
+			if file.IsReadme {
+				fileType = "readme"
+			} else if preferredModelFile != nil && file.Path == preferredModelFile.Path {
+				fileType = "model"
+			}
+
+			processedFiles[j] = ProcessedModelFile{
+				Path:     file.Path,
+				Size:     file.Size,
+				SHA256:   file.SHA256,
+				IsReadme: file.IsReadme,
+				FileType: fileType,
+			}
+		}
+
+		processedModel.Files = processedFiles
+
+		// Set preferred model file
+		if preferredModelFile != nil {
+			for _, file := range processedFiles {
+				if file.Path == preferredModelFile.Path {
+					processedModel.PreferredModelFile = &file
+					break
+				}
+			}
+		}
+
+		// Print file information
+		outputBuilder.WriteString(fmt.Sprintf("   Files found: %d\n", len(details.Files)))
+
+		if preferredModelFile != nil {
+			outputBuilder.WriteString(fmt.Sprintf("   Preferred Model File: %s (SHA256: %s)\n",
+				preferredModelFile.Path,
+				preferredModelFile.SHA256))
+		} else {
+			outputBuilder.WriteString(fmt.Sprintf("   No model file found with quantization preferences: %v\n", quantizationPreferences))
+		}
+
+		if details.ReadmeFile != nil {
+			outputBuilder.WriteString(fmt.Sprintf("   README File: %s\n", details.ReadmeFile.Path))
+
+			// Find and set readme file
+			for _, file := range processedFiles {
+				if file.IsReadme {
+					processedModel.ReadmeFile = &file
+					break
+				}
+			}
+
+			fmt.Println("Getting real readme for", model.ModelID, "waiting...")
+			// Use agent to get the real readme and prepare the model description
+			readmeContent, err := getRealReadme(context.Background(), model.ModelID)
+			if err == nil {
+				processedModel.ReadmeContent = readmeContent
+				processedModel.ReadmeContentPreview = truncateString(readmeContent, 200)
+				outputBuilder.WriteString(fmt.Sprintf("   README Content Preview: %s\n",
+					processedModel.ReadmeContentPreview))
+			} else {
+				fmt.Printf("   Warning: Failed to get real readme: %v\n", err)
+			}
+			fmt.Println("Real readme got", readmeContent)
+
+			// Extract metadata (tags, license) from README using LLM
+			fmt.Println("Extracting metadata for", model.ModelID, "waiting...")
+			tags, license, err := extractModelMetadata(context.Background(), processedModel)
+			if err == nil {
+				processedModel.Tags = tags
+				processedModel.License = license
+				outputBuilder.WriteString(fmt.Sprintf("   Tags: %v\n", tags))
+				outputBuilder.WriteString(fmt.Sprintf("   License: %s\n", license))
+			} else {
+				fmt.Printf("   Warning: Failed to extract metadata: %v\n", err)
+			}
+
+			// Extract icon from README or use HuggingFace avatar
+			icon := extractModelIcon(processedModel)
+			if icon != "" {
+				processedModel.Icon = icon
+				outputBuilder.WriteString(fmt.Sprintf("   Icon: %s\n", icon))
+			}
+			// Get README content
+			// readmeContent, err := client.GetReadmeContent(model.ModelID, details.ReadmeFile.Path)
+			// if err == nil {
+			// 	processedModel.ReadmeContent = readmeContent
+			// 	processedModel.ReadmeContentPreview = truncateString(readmeContent, 200)
+			// 	outputBuilder.WriteString(fmt.Sprintf("   README Content Preview: %s\n",
+			// 		processedModel.ReadmeContentPreview))
+			// }
+		}
+
+		// Print all files with their checksums
+		outputBuilder.WriteString("   All Files:\n")
+		for _, file := range processedFiles {
+			outputBuilder.WriteString(fmt.Sprintf("     - %s (%s, %d bytes", file.Path, file.FileType, file.Size))
+			if file.SHA256 != "" {
+				outputBuilder.WriteString(fmt.Sprintf(", SHA256: %s", file.SHA256))
+			}
+			outputBuilder.WriteString(")\n")
+		}
+
+		outputBuilder.WriteString("\n")
+		result.Models = append(result.Models, processedModel)
+	}
+
+	result.FormattedOutput = outputBuilder.String()
+	return result, nil
 }
 
 func truncateString(s string, maxLen int) string {
@@ -277,4 +381,3 @@ func truncateString(s string, maxLen int) string {
 	}
 	return s[:maxLen] + "..."
 }
-

.github/gallery-agent/testing.go

@@ -3,7 +3,7 @@ package main
 import (
 	"context"
 	"fmt"
-	"math/rand/v2"
+	"math/rand"
 	"strings"
 	"time"
 )
@@ -13,11 +13,11 @@ func runSyntheticMode() error {
 	generator := NewSyntheticDataGenerator()
 
 	// Generate a random number of synthetic models (1-3)
-	numModels := generator.rand.IntN(3) + 1
+	numModels := generator.rand.Intn(3) + 1
 	fmt.Printf("Generating %d synthetic models for testing...\n", numModels)
 
 	var models []ProcessedModel
-	for range numModels {
+	for i := 0; i < numModels; i++ {
 		model := generator.GenerateProcessedModel()
 		models = append(models, model)
 		fmt.Printf("Generated synthetic model: %s\n", model.ModelID)
@@ -42,14 +42,14 @@ type SyntheticDataGenerator struct {
 // NewSyntheticDataGenerator creates a new synthetic data generator
 func NewSyntheticDataGenerator() *SyntheticDataGenerator {
 	return &SyntheticDataGenerator{
-		rand: rand.New(rand.NewPCG(uint64(time.Now().UnixNano()), 0)),
+		rand: rand.New(rand.NewSource(time.Now().UnixNano())),
 	}
 }
 
 // GenerateProcessedModelFile creates a synthetic ProcessedModelFile
 func (g *SyntheticDataGenerator) GenerateProcessedModelFile() ProcessedModelFile {
 	fileTypes := []string{"model", "readme", "other"}
-	fileType := fileTypes[g.rand.IntN(len(fileTypes))]
+	fileType := fileTypes[g.rand.Intn(len(fileTypes))]
 
 	var path string
 	var isReadme bool
@@ -68,7 +68,7 @@ func (g *SyntheticDataGenerator) GenerateProcessedModelFile() ProcessedModelFile
 
 	return ProcessedModelFile{
 		Path:     path,
-		Size:     int64(g.rand.IntN(1000000000) + 1000000), // 1MB to 1GB
+		Size:     int64(g.rand.Intn(1000000000) + 1000000), // 1MB to 1GB
 		SHA256:   g.randomSHA256(),
 		IsReadme: isReadme,
 		FileType: fileType,
@@ -80,19 +80,19 @@ func (g *SyntheticDataGenerator) GenerateProcessedModel() ProcessedModel {
 	authors := []string{"microsoft", "meta", "google", "openai", "anthropic", "mistralai", "huggingface"}
 	modelNames := []string{"llama", "gpt", "claude", "mistral", "gemma", "phi", "qwen", "codellama"}
 
-	author := authors[g.rand.IntN(len(authors))]
-	modelName := modelNames[g.rand.IntN(len(modelNames))]
+	author := authors[g.rand.Intn(len(authors))]
+	modelName := modelNames[g.rand.Intn(len(modelNames))]
 	modelID := fmt.Sprintf("%s/%s-%s", author, modelName, g.randomString(6))
 
 	// Generate files
-	numFiles := g.rand.IntN(5) + 2 // 2-6 files
+	numFiles := g.rand.Intn(5) + 2 // 2-6 files
 	files := make([]ProcessedModelFile, numFiles)
 
 	// Ensure at least one model file and one readme
 	hasModelFile := false
 	hasReadme := false
 
-	for i := range numFiles {
+	for i := 0; i < numFiles; i++ {
 		files[i] = g.GenerateProcessedModelFile()
 		if files[i].FileType == "model" {
 			hasModelFile = true
@@ -140,27 +140,27 @@ func (g *SyntheticDataGenerator) GenerateProcessedModel() ProcessedModel {
 
 	// Generate sample metadata
 	licenses := []string{"apache-2.0", "mit", "llama2", "gpl-3.0", "bsd", ""}
-	license := licenses[g.rand.IntN(len(licenses))]
+	license := licenses[g.rand.Intn(len(licenses))]
 
 	sampleTags := []string{"llm", "gguf", "gpu", "cpu", "text-to-text", "chat", "instruction-tuned"}
-	numTags := g.rand.IntN(4) + 3 // 3-6 tags
+	numTags := g.rand.Intn(4) + 3 // 3-6 tags
 	tags := make([]string, numTags)
-	for i := range numTags {
-		tags[i] = sampleTags[g.rand.IntN(len(sampleTags))]
+	for i := 0; i < numTags; i++ {
+		tags[i] = sampleTags[g.rand.Intn(len(sampleTags))]
 	}
 	// Remove duplicates
 	tags = g.removeDuplicates(tags)
 
 	// Optionally include icon (50% chance)
 	icon := ""
-	if g.rand.IntN(2) == 0 {
+	if g.rand.Intn(2) == 0 {
 		icon = fmt.Sprintf("https://cdn-avatars.huggingface.co/v1/production/uploads/%s.png", g.randomString(24))
 	}
 
 	return ProcessedModel{
 		ModelID:                 modelID,
 		Author:                  author,
-		Downloads:               g.rand.IntN(1000000) + 1000,
+		Downloads:               g.rand.Intn(1000000) + 1000,
 		LastModified:            g.randomDate(),
 		Files:                   files,
 		PreferredModelFile:      preferredModelFile,
@@ -180,7 +180,7 @@ func (g *SyntheticDataGenerator) randomString(length int) string {
 	const charset = "abcdefghijklmnopqrstuvwxyz0123456789"
 	b := make([]byte, length)
 	for i := range b {
-		b[i] = charset[g.rand.IntN(len(charset))]
+		b[i] = charset[g.rand.Intn(len(charset))]
 	}
 	return string(b)
 }
@@ -189,14 +189,14 @@ func (g *SyntheticDataGenerator) randomSHA256() string {
 	const charset = "0123456789abcdef"
 	b := make([]byte, 64)
 	for i := range b {
-		b[i] = charset[g.rand.IntN(len(charset))]
+		b[i] = charset[g.rand.Intn(len(charset))]
 	}
 	return string(b)
 }
 
 func (g *SyntheticDataGenerator) randomDate() string {
 	now := time.Now()
-	daysAgo := g.rand.IntN(365) // Random date within last year
+	daysAgo := g.rand.Intn(365) // Random date within last year
 	pastDate := now.AddDate(0, 0, -daysAgo)
 	return pastDate.Format("2006-01-02T15:04:05.000Z")
 }
@@ -220,5 +220,5 @@ func (g *SyntheticDataGenerator) generateReadmeContent(modelName, author string)
 		fmt.Sprintf("# %s Language Model\n\nDeveloped by %s, this model represents state-of-the-art performance in natural language understanding and generation.\n\n## Key Features\n\n- Multilingual support\n- Context-aware responses\n- Efficient memory usage\n- Fast inference speed\n\n## Applications\n\n- Chatbots and virtual assistants\n- Content generation\n- Code completion\n- Educational tools", strings.Title(modelName), author),
 	}
 
-	return templates[g.rand.IntN(len(templates))]
+	return templates[g.rand.Intn(len(templates))]
 }

.github/gallery-agent/tools.go

@@ -0,0 +1,46 @@
+package main
+
+import (
+	"fmt"
+
+	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
+	openai "github.com/sashabaranov/go-openai"
+	jsonschema "github.com/sashabaranov/go-openai/jsonschema"
+)
+
+// Get repository README from HF
+type HFReadmeTool struct {
+	client *hfapi.Client
+}
+
+func (s *HFReadmeTool) Execute(args map[string]any) (string, any, error) {
+	q, ok := args["repository"].(string)
+	if !ok {
+		return "", nil, fmt.Errorf("no query")
+	}
+	readme, err := s.client.GetReadmeContent(q, "README.md")
+	if err != nil {
+		return "", nil, err
+	}
+	return readme, nil, nil
+}
+
+func (s *HFReadmeTool) Tool() openai.Tool {
+	return openai.Tool{
+		Type: openai.ToolTypeFunction,
+		Function: &openai.FunctionDefinition{
+			Name:        "hf_readme",
+			Description: "A tool to get the README content of a huggingface repository",
+			Parameters: jsonschema.Definition{
+				Type: jsonschema.Object,
+				Properties: map[string]jsonschema.Definition{
+					"repository": {
+						Type:        jsonschema.String,
+						Description: "The huggingface repository to get the README content of",
+					},
+				},
+				Required: []string{"repository"},
+			},
+		},
+	}
+}

.github/scripts/anchor-digest-in-cache.sh

@@ -1,46 +0,0 @@
-#!/usr/bin/env bash
-# Anchor a backend per-arch digest in quay.io/go-skynet/ci-cache so quay's
-# garbage collector won't reap the manifest before backend_merge.yml runs.
-#
-# Context: backend_build.yml pushes by canonical digest only
-# (push-by-digest=true). Unreferenced manifests on quay can be reaped within
-# ~1-2h, but backend-merge-jobs runs only after the *entire* per-arch build
-# matrix drains (max-parallel: 8 × dozens of entries → ~2h+). Without an
-# anchoring tag, the earliest digests are gone by the time `imagetools create`
-# tries to read them, producing "manifest not found" merge failures.
-#
-# We tag the digest under our internal ci-cache image; quay does not GC tagged
-# manifests. The user-facing manifest list still references the original
-# digest in local-ai-backends. backend_merge.yml deletes the anchor tag after
-# the user-facing manifest is published — see cleanup-keepalive-tags.sh.
-#
-# Required env:
-#   GITHUB_RUN_ID  - current workflow run id (set automatically by GHA)
-#   TAG_SUFFIX     - matrix entry's tag-suffix (e.g. -gpu-nvidia-cuda-12-vllm)
-#   PLATFORM_TAG   - amd64 / arm64 / single (single = singleton matrix entry)
-#   DIGEST         - canonical content digest from build step (sha256:...)
-#
-# Optional env:
-#   ANCHOR_IMAGE   - target image (default: quay.io/go-skynet/ci-cache)
-#   SOURCE_IMAGE   - source image (default: quay.io/go-skynet/local-ai-backends)
-#   GITHUB_STEP_SUMMARY - if set, an anchored-by line is appended to it
-set -euo pipefail
-
-: "${GITHUB_RUN_ID:?}"
-: "${TAG_SUFFIX:?}"
-: "${PLATFORM_TAG:?}"
-: "${DIGEST:?}"
-
-anchor_image="${ANCHOR_IMAGE:-quay.io/go-skynet/ci-cache}"
-source_image="${SOURCE_IMAGE:-quay.io/go-skynet/local-ai-backends}"
-
-tag="keepalive-${GITHUB_RUN_ID}${TAG_SUFFIX}-${PLATFORM_TAG}"
-
-docker buildx imagetools create \
-  -t "${anchor_image}:${tag}" \
-  "${source_image}@${DIGEST}"
-
-echo "anchored ${DIGEST} as ${anchor_image}:${tag}"
-if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then
-  echo "anchored \`${DIGEST}\` as \`${anchor_image}:${tag}\`" >> "${GITHUB_STEP_SUMMARY}"
-fi

.github/scripts/cleanup-keepalive-tags.sh

@@ -1,49 +0,0 @@
-#!/usr/bin/env bash
-# Best-effort cleanup of the keepalive anchor tags written by
-# anchor-digest-in-cache.sh. Called from backend_merge.yml after the
-# user-facing manifest list has been published.
-#
-# Quay's docker registry v2 doesn't allow tag deletes — only digest deletes.
-# The proper delete is the quay REST API, which requires an OAuth-scoped
-# token. We try QUAY_TOKEN as a bearer token: if the secret is an OAuth app
-# token (typical for service accounts) the delete succeeds; otherwise this
-# is a soft no-op and the tag persists until manually pruned.
-#
-# Cleanup failure MUST NOT fail the merge — the merge has already produced
-# the user-facing manifest list at this point and the keepalive tags are
-# pure overhead. We always exit 0.
-#
-# Required env:
-#   GITHUB_RUN_ID  - current workflow run id (set automatically by GHA)
-#   TAG_SUFFIX     - matrix entry's tag-suffix (e.g. -gpu-nvidia-cuda-12-vllm)
-#   QUAY_TOKEN     - bearer token for quay's REST API
-#
-# Optional env:
-#   QUAY_REPO      - target repo (default: go-skynet/ci-cache)
-#   PLATFORM_TAGS  - space-separated list of platform-tag values to try
-#                    (default: "amd64 arm64 single")
-#                    We don't know which platform-tag(s) exist for this
-#                    tag-suffix without an extra API call, so we just try
-#                    all three and ignore 404s for the ones that don't.
-set -uo pipefail
-
-: "${GITHUB_RUN_ID:?}"
-: "${TAG_SUFFIX:?}"
-: "${QUAY_TOKEN:?}"
-
-quay_repo="${QUAY_REPO:-go-skynet/ci-cache}"
-platform_tags="${PLATFORM_TAGS:-amd64 arm64 single}"
-
-for plat in $platform_tags; do
-  tag="keepalive-${GITHUB_RUN_ID}${TAG_SUFFIX}-${plat}"
-  url="https://quay.io/api/v1/repository/${quay_repo}/tag/${tag}"
-  http=$(curl -sS -o /dev/null -w '%{http_code}' \
-    -X DELETE -H "Authorization: Bearer ${QUAY_TOKEN}" "$url" || echo "000")
-  case "$http" in
-    204|200) echo "deleted $tag" ;;
-    404)     echo "not present: $tag" ;;
-    401|403) echo "auth not OAuth-scoped (http $http) for $tag - skipping; orphan tag will persist" ;;
-    *)       echo "unexpected http $http deleting $tag - skipping" ;;
-  esac
-done
-exit 0

.github/workflows/backend_build.yml

@@ -24,17 +24,6 @@ on:
         description: 'Platforms'
         default: ''
         type: string
-      platform-tag:
-        description: |
-          Short tag identifying the platform leg, e.g. "amd64" or "arm64".
-          Used to scope the per-arch registry cache and the digest artifact name.
-          Required for split-and-merge multi-arch builds; pass "amd64" for
-          single-arch amd64 builds too. Optional (default '') during the
-          migration to per-arch matrix expansion; will be flipped to
-          required: true in Phase 6 once all callers pass an explicit value.
-        required: false
-        default: ''
-        type: string
       tag-latest:
         description: 'Tag latest'
         default: ''
@@ -69,20 +58,6 @@ on:
         required: false
         default: '2204'
         type: string
-      amdgpu-targets:
-        description: 'AMD GPU targets for ROCm/HIP builds'
-        required: false
-        default: ''
-        type: string
-      builder-base-image:
-        description: |
-          Pre-built builder base image (e.g. quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64).
-          When set, the variant Dockerfile uses its `builder-prebuilt` stage which FROMs this
-          image directly instead of running its own gRPC stage + apt installs. Empty for
-          backends whose Dockerfile doesn't support a prebuilt base.
-        required: false
-        default: ''
-        type: string
     secrets:
       dockerUsername:
         required: false
@@ -100,22 +75,76 @@ jobs:
         quay_username: ${{ secrets.quayUsername }}
     steps:
 
+
+      - name: Free Disk Space (Ubuntu)
+        if: inputs.runs-on == 'ubuntu-latest'
+        uses: jlumbroso/free-disk-space@main
+        with:
+          # this might remove tools that are actually needed,
+          # if set to "true" but frees about 6 GB
+          tool-cache: true
+          # all of these default to true, but feel free to set to
+          # "false" if necessary for your workflow
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          docker-images: true
+          swap-storage: true
+
+      - name: Force Install GIT latest
+        run: |
+          sudo apt-get update \
+          && sudo apt-get install -y software-properties-common \
+          && sudo apt-get update \
+          && sudo add-apt-repository -y ppa:git-core/ppa \
+          && sudo apt-get update \
+          && sudo apt-get install -y git
+
       - name: Checkout
         uses: actions/checkout@v6
-        with:
-          submodules: true
 
-      - name: Configure apt mirror on runner
-        id: apt_mirror
-        uses: ./.github/actions/configure-apt-mirror
-
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
-        with:
-          mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
-
-      - name: Set up build disk
-        uses: ./.github/actions/setup-build-disk
+      - name: Release space from worker
+        if: inputs.runs-on == 'ubuntu-latest'
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get remove -y microsoft-edge-stable || true
+          sudo apt-get remove -y firefox || true
+          sudo apt-get remove -y powershell || true
+          sudo apt-get remove -y r-base-core || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf "/usr/local/share/boost" || true
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
+          df -h
 
       - name: Docker meta
         id: meta
@@ -172,17 +201,7 @@ jobs:
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
-      # Weekly cache-buster for the per-backend `make` step. Most Python
-      # backends list unpinned deps (torch, transformers, vllm, ...), so a
-      # warm cache freezes upstream versions indefinitely. Rolling this
-      # weekly forces a re-resolve of the install layer at most once per
-      # week, picking up newer wheels without a full cold rebuild.
-      - name: Compute deps refresh key
-        id: deps_refresh
-        run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
-
-      - name: Build and push by digest
-        id: build
+      - name: Build and push
         uses: docker/build-push-action@v7
         if: github.event_name != 'pull_request'
         with:
@@ -195,66 +214,15 @@ jobs:
             BASE_IMAGE=${{ inputs.base-image }}
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
-            AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
-            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
-            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
-            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
-          outputs: |
-            type=image,name=quay.io/go-skynet/local-ai-backends,push-by-digest=true,name-canonical=true,push=true
-            type=image,name=localai/localai-backends,push-by-digest=true,name-canonical=true,push=true
-          # Disable provenance: with mode=max (the default for push:true)
-          # buildx bundles a per-registry attestation manifest into each
-          # registry's manifest list, which makes the resulting list digest
-          # diverge across registries. steps.build.outputs.digest then
-          # only matches one of them, and the merge job's
-          # `imagetools create <reg>@sha256:<digest>` lookup fails on the
-          # other. Disabling provenance keeps the digest content-only and
-          # identical across both registries — required for digest-based
-          # cross-registry merge.
-          provenance: false
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
 
-      - name: Export digest
-        if: github.event_name != 'pull_request'
-        run: |
-          mkdir -p /tmp/digests
-          digest="${{ steps.build.outputs.digest }}"
-          touch "/tmp/digests/${digest#sha256:}"
-
-      # See .github/scripts/anchor-digest-in-cache.sh for why this is needed
-      # and how it interacts with backend_merge.yml's cleanup step.
-      - name: Anchor digest in ci-cache so quay GC won't reap before merge
-        if: github.event_name != 'pull_request'
-        env:
-          TAG_SUFFIX: ${{ inputs.tag-suffix }}
-          PLATFORM_TAG: ${{ inputs.platform-tag || 'single' }}
-          DIGEST: ${{ steps.build.outputs.digest }}
-        run: .github/scripts/anchor-digest-in-cache.sh
-
-      # Artifact name uses a `--` separator between tag-suffix and platform-tag
-      # to avoid prefix collisions during the merge job's pattern-based download.
-      # Tag-suffixes are not prefix-disjoint (e.g. -gpu-nvidia-cuda-12-vllm is a
-      # prefix of -gpu-nvidia-cuda-12-vllm-omni); a single `-` separator plus the
-      # merge-side `digests<tag-suffix>-*` glob would let one merge over-match
-      # the other backend's artifacts. The `-single` placeholder for empty
-      # platform-tag (single-arch entries) keeps the artifact name non-trailing.
-      - name: Upload digest artifact
-        if: github.event_name != 'pull_request'
-        uses: actions/upload-artifact@v7
-        with:
-          name: digests${{ inputs.tag-suffix }}--${{ inputs.platform-tag || 'single' }}
-          path: /tmp/digests/*
-          if-no-files-found: error
-          retention-days: 1
-
-      - name: Build (PR)
+      - name: Build and push (PR)
         uses: docker/build-push-action@v7
         if: github.event_name == 'pull_request'
         with:
@@ -267,15 +235,9 @@ jobs:
             BASE_IMAGE=${{ inputs.base-image }}
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
-            AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
-            DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
-            BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
-            BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
           push: ${{ env.quay_username != '' }}
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/backend_build_darwin.yml

@@ -48,13 +48,6 @@ jobs:
     strategy:
       matrix:
         go-version: ['${{ inputs.go-version }}']
-    env:
-      # Keep the brew Cellar stable across cache restores. Without these,
-      # `brew install` would auto-update brew itself and re-link formulas,
-      # mutating the very paths the cache just restored.
-      HOMEBREW_NO_AUTO_UPDATE: '1'
-      HOMEBREW_NO_INSTALL_CLEANUP: '1'
-      HOMEBREW_NO_ANALYTICS: '1'
     steps:
       - name: Clone
         uses: actions/checkout@v6
@@ -65,190 +58,21 @@ jobs:
         uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go-version }}
-          # Caches ~/go/pkg/mod and ~/Library/Caches/go-build keyed on go.sum.
-          # Shared across every darwin matrix entry — first job in a run warms
-          # it, the rest hit warm.
-          cache: true
+          cache: false
 
       # You can test your matrix by printing the current Go version
       - name: Display Go version
         run: go version
 
-      # ---- Homebrew cache ----
-      # macOS runners have no Docker daemon, so the BuildKit registry cache used
-      # for Linux backend images (see .agents/ci-caching.md) doesn't apply here.
-      # We cache the brew downloads + Cellar entries for the formulas we install
-      # below. Read on every run, write only on master/tag pushes — same policy
-      # as the Linux registry cache.
-      - name: Restore Homebrew cache
-        id: brew-cache
-        uses: actions/cache/restore@v4
-        with:
-          path: |
-            ~/Library/Caches/Homebrew/downloads
-            /opt/homebrew/Cellar/protobuf
-            /opt/homebrew/Cellar/grpc
-            /opt/homebrew/Cellar/protoc-gen-go
-            /opt/homebrew/Cellar/protoc-gen-go-grpc
-            /opt/homebrew/Cellar/libomp
-            /opt/homebrew/Cellar/llvm
-            /opt/homebrew/Cellar/ccache
-            /opt/homebrew/Cellar/blake3
-            /opt/homebrew/Cellar/fmt
-            /opt/homebrew/Cellar/hiredis
-            /opt/homebrew/Cellar/xxhash
-            /opt/homebrew/Cellar/zstd
-          key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
-
       - name: Dependencies
         run: |
-          # ccache is always installed (used by the llama-cpp variant build) so
-          # the brew cache content stays stable across every backend in the
-          # matrix — they all share one cache key.
-          # blake3, fmt, hiredis, xxhash, zstd are ccache's runtime dylib deps.
-          # Without explicitly installing them, a brew cache-hit run restores
-          # ccache's Cellar dir but skips installing those transitive deps,
-          # and ccache fails at runtime with `dyld: Library not loaded`.
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd
-          # Force-reinstall ccache so brew re-validates its full runtime-dep
-          # closure on every run. This is the durable fix: when the upstream
-          # ccache formula gains a new transitive dep (as it has multiple times
-          # already), we don't have to chase missing dylibs one at a time.
-          # The downloads cache makes the reinstall fast (~5s on a hit).
-          brew reinstall ccache
-          # Same pattern for grpc: its CMake config (used by the llama-cpp
-          # `grpc-server` target) does find_package(absl). The cache restores
-          # /opt/homebrew/Cellar/grpc so brew above no-ops the install, but
-          # abseil isn't in our Cellar cache list and never gets installed
-          # alongside, leaving grpc's CMake unable to resolve it. Reinstalling
-          # grpc re-validates and pulls abseil in, mirroring the ccache fix.
-          brew reinstall grpc
-          # The brew cache restores the Cellar dirs but NOT the bin symlinks
-          # at /opt/homebrew/bin/*. brew install above sees the Cellar present
-          # and decides "already installed" without re-linking, so on a cache-
-          # hit run the formulas aren't on PATH. Force-link them; --overwrite
-          # tolerates pre-existing symlinks from earlier installs.
-          brew link --overwrite protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd 2>/dev/null || true
-
-      - name: Save Homebrew cache
-        if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v4
-        with:
-          path: |
-            ~/Library/Caches/Homebrew/downloads
-            /opt/homebrew/Cellar/protobuf
-            /opt/homebrew/Cellar/grpc
-            /opt/homebrew/Cellar/protoc-gen-go
-            /opt/homebrew/Cellar/protoc-gen-go-grpc
-            /opt/homebrew/Cellar/libomp
-            /opt/homebrew/Cellar/llvm
-            /opt/homebrew/Cellar/ccache
-            /opt/homebrew/Cellar/blake3
-            /opt/homebrew/Cellar/fmt
-            /opt/homebrew/Cellar/hiredis
-            /opt/homebrew/Cellar/xxhash
-            /opt/homebrew/Cellar/zstd
-          key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
-
-      # ---- ccache for llama.cpp CMake builds ----
-      # Three CMake variants (fallback, grpc, rpc-server) compile the same
-      # llama.cpp source tree with overlapping flags — ccache dedupes object
-      # files across them. Key on the pinned LLAMA_VERSION so a pin bump
-      # invalidates cleanly; restore-keys fall back to the latest entry for the
-      # same pin so unchanged TUs stay warm even when the cache is fresh.
-      - name: Compute llama.cpp version
-        if: inputs.backend == 'llama-cpp'
-        id: llama-version
-        run: |
-          version=$(grep '^LLAMA_VERSION' backend/cpp/llama-cpp/Makefile | head -1 | cut -d= -f2 | cut -d'?' -f1 | tr -d ' ')
-          echo "version=${version}" >> "$GITHUB_OUTPUT"
-
-      - name: Restore ccache
-        if: inputs.backend == 'llama-cpp'
-        id: ccache-cache
-        uses: actions/cache/restore@v4
-        with:
-          path: ~/Library/Caches/ccache
-          key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
-          restore-keys: |
-            ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-
-
-      - name: Configure ccache
-        if: inputs.backend == 'llama-cpp'
-        run: |
-          mkdir -p "$HOME/Library/Caches/ccache"
-          ccache -M 2G
-          ccache -z
-          # llama-cpp-darwin.sh reads CMAKE_ARGS / CCACHE_DIR from env.
-          {
-            echo "CMAKE_ARGS=${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache"
-            echo "CCACHE_DIR=$HOME/Library/Caches/ccache"
-          } >> "$GITHUB_ENV"
-
-      # ---- Python wheel cache (uv + pip) ----
-      # Mirrors the Linux DEPS_REFRESH cadence (see .agents/ci-caching.md): the
-      # ISO-week segment of the cache key forces at most one cold rebuild per
-      # backend per week, automatically picking up newer wheels for unpinned
-      # deps (torch, mlx, diffusers, …). Restore-keys fall back to the most
-      # recent build of the same backend so off-week PRs still hit warm.
-      - name: Compute weekly cache bucket
-        if: inputs.lang == 'python'
-        id: weekly
-        run: echo "bucket=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
-
-      - name: Restore Python wheel cache
-        if: inputs.lang == 'python'
-        id: pyenv-cache
-        uses: actions/cache/restore@v4
-        with:
-          path: |
-            ~/Library/Caches/pip
-            ~/Library/Caches/uv
-          key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
-          restore-keys: |
-            pyenv-darwin-${{ inputs.backend }}-
-
-      # llama-cpp on Darwin uses a bespoke build script (scripts/build/llama-cpp-darwin.sh)
-      # that compiles three CMake variants from backend/cpp/llama-cpp and bundles dylibs
-      # via otool — it doesn't fit the build-darwin-go-backend / build-darwin-python-backend
-      # mold. Drive it via its dedicated `backends/llama-cpp-darwin` make target instead.
-      - name: Build ${{ inputs.backend }}-darwin (llama-cpp)
-        if: inputs.backend == 'llama-cpp'
-        run: |
-          make protogen-go
-          make backends/llama-cpp-darwin
-
-      - name: Build ds4 backend (Darwin Metal)
-        if: inputs.backend == 'ds4'
-        run: |
-          make backends/ds4-darwin
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm
 
       - name: Build ${{ inputs.backend }}-darwin
-        if: inputs.backend != 'llama-cpp' && inputs.backend != 'ds4'
         run: |
           make protogen-go
           BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
 
-      - name: ccache stats
-        if: inputs.backend == 'llama-cpp'
-        run: ccache -s
-
-      - name: Save ccache
-        if: inputs.backend == 'llama-cpp' && github.event_name != 'pull_request'
-        uses: actions/cache/save@v4
-        with:
-          path: ~/Library/Caches/ccache
-          key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
-
-      - name: Save Python wheel cache
-        if: inputs.lang == 'python' && github.event_name != 'pull_request' && steps.pyenv-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v4
-        with:
-          path: |
-            ~/Library/Caches/pip
-            ~/Library/Caches/uv
-          key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
-
       - name: Upload ${{ inputs.backend }}.tar
         uses: actions/upload-artifact@v7
         with:

.github/workflows/backend_merge.yml

@@ -1,174 +0,0 @@
----
-name: 'merge backend manifest list (reusable)'
-
-# Reusable workflow that joins per-arch digest artifacts (uploaded by
-# backend_build.yml when called with platform-tag) into a single tagged
-# multi-arch manifest list. Called once per backend by backend.yml after
-# both per-arch build jobs succeed.
-
-on:
-  workflow_call:
-    inputs:
-      tag-latest:
-        description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
-        required: false
-        type: string
-        default: ''
-      tag-suffix:
-        description: 'Backend tag suffix (e.g. -cpu-faster-whisper). Used to compute the artifact pattern and the final tag suffix.'
-        required: true
-        type: string
-    secrets:
-      dockerUsername:
-        required: false
-      dockerPassword:
-        required: false
-      quayUsername:
-        required: true
-      quayPassword:
-        required: true
-
-jobs:
-  merge:
-    runs-on: ubuntu-latest
-    env:
-      quay_username: ${{ secrets.quayUsername }}
-    steps:
-      # Sparse checkout: the merge job needs `.github/scripts/` (for the
-      # keepalive cleanup script) but none of the source tree.
-      - name: Checkout (.github/scripts only)
-        uses: actions/checkout@v6
-        with:
-          sparse-checkout: |
-            .github/scripts
-          sparse-checkout-cone-mode: false
-
-      # `--` separator anchors the glob so we don't over-match sibling
-      # backends whose tag-suffix happens to be a prefix of ours
-      # (e.g. -cpu-vllm vs -cpu-vllm-omni). Must stay in sync with the
-      # upload-artifact name in backend_build.yml.
-      - name: Download digests
-        uses: actions/download-artifact@v8
-        with:
-          pattern: digests${{ inputs.tag-suffix }}--*
-          merge-multiple: true
-          path: /tmp/digests
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@master
-
-      - name: Login to DockerHub
-        if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
-        with:
-          username: ${{ secrets.dockerUsername }}
-          password: ${{ secrets.dockerPassword }}
-
-      - name: Login to Quay.io
-        if: ${{ env.quay_username != '' }}
-        uses: docker/login-action@v4
-        with:
-          registry: quay.io
-          username: ${{ secrets.quayUsername }}
-          password: ${{ secrets.quayPassword }}
-
-      - name: Docker meta
-        id: meta
-        if: github.event_name != 'pull_request'
-        uses: docker/metadata-action@v6
-        with:
-          images: |
-            quay.io/go-skynet/local-ai-backends
-            localai/localai-backends
-          tags: |
-            type=ref,event=branch
-            type=semver,pattern={{raw}}
-            type=sha
-          flavor: |
-            latest=${{ inputs.tag-latest }}
-            suffix=${{ inputs.tag-suffix }},onlatest=true
-
-      # Source from ci-cache, not local-ai-backends.
-      #
-      # The build job pushes per-arch manifests to local-ai-backends with
-      # push-by-digest=true (no tag), then anchors a tagged copy into
-      # ci-cache so the manifest can be retrieved hours later when this
-      # merge runs. Quay's manifest GC, however, is per-repository: the
-      # anchor tag in ci-cache protects the manifest there, but the same
-      # digest in local-ai-backends has no tag in *that* repo and gets
-      # reaped independently. Sourcing local-ai-backends@<digest> here
-      # then fails with "manifest not found" — exactly the regression
-      # we hit on v4.2.2 (19/37 multiarch merges failed).
-      #
-      # ci-cache@<digest> resolves because we anchored it there. buildx
-      # imagetools create copies the manifest into local-ai-backends
-      # (cross-repo within the same registry, blobs already cross-mounted
-      # from the original push so no transfer needed) and publishes the
-      # manifest list with the user-facing tags. The resulting manifest
-      # list is fully self-contained in local-ai-backends — child digests
-      # only, no embedded references to ci-cache.
-      - name: Create manifest list and push (quay)
-        if: github.event_name != 'pull_request'
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '
-            .tags
-            | map(select(startswith("quay.io/")))
-            | map("-t " + .)
-            | join(" ")
-          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No quay.io tags from docker/metadata-action; skipping quay merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'quay.io/go-skynet/ci-cache@sha256:%s ' *)
-          fi
-
-      - name: Create manifest list and push (dockerhub)
-        if: github.event_name != 'pull_request'
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '
-            .tags
-            | map(select(startswith("localai/")))
-            | map("-t " + .)
-            | join(" ")
-          ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'localai/localai-backends@sha256:%s ' *)
-          fi
-
-      - name: Inspect manifest
-        if: github.event_name != 'pull_request'
-        run: |
-          set -euo pipefail
-          first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
-            docker buildx imagetools inspect "$first_tag"
-          fi
-
-      # See .github/scripts/cleanup-keepalive-tags.sh for why this is
-      # best-effort and what the failure modes are.
-      - name: Cleanup keepalive tags in ci-cache
-        if: github.event_name != 'pull_request' && success()
-        env:
-          TAG_SUFFIX: ${{ inputs.tag-suffix }}
-          QUAY_TOKEN: ${{ secrets.quayPassword }}
-        run: .github/scripts/cleanup-keepalive-tags.sh
-
-      - name: Job summary
-        if: github.event_name != 'pull_request'
-        run: |
-          set -euo pipefail
-          echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
-          jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
-          echo >> "$GITHUB_STEP_SUMMARY"
-          echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
-          ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"

.github/workflows/backend_pr.yml

@@ -4,23 +4,17 @@ on:
   pull_request:
 
 concurrency:
-  group: ci-backends-pr-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-backends-pr-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   generate-matrix:
     runs-on: ubuntu-latest
     outputs:
-      matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
-      matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
-      matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
-      merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
-      merge-matrix-singlearch: ${{ steps.set-matrix.outputs['merge-matrix-singlearch'] }}
-      has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
-      has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
-      has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
-      has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
-      has-merges-singlearch: ${{ steps.set-matrix.outputs['has-merges-singlearch'] }}
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+      matrix-darwin: ${{ steps.set-matrix.outputs.matrix-darwin }}
+      has-backends: ${{ steps.set-matrix.outputs.has-backends }}
+      has-backends-darwin: ${{ steps.set-matrix.outputs.has-backends-darwin }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -33,9 +27,7 @@ jobs:
           bun add js-yaml
           bun add @octokit/core
 
-      # filters the matrix in backend.yml; splits into single-arch and
-      # multi-arch groups so backend-merge-jobs can `needs:` only the latter
-      # (matches backend.yml's structure).
+      # filters the matrix in backend.yml
       - name: Filter matrix for changed backends
         id: set-matrix
         env:
@@ -43,10 +35,10 @@ jobs:
           GITHUB_EVENT_PATH: ${{ github.event_path }}
         run: bun run scripts/changed-backends.js
 
-  backend-jobs-multiarch:
+  backend-jobs:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build.yml
-    if: needs.generate-matrix.outputs['has-backends-multiarch'] == 'true'
+    if: needs.generate-matrix.outputs.has-backends == 'true'
     with:
       tag-latest: ${{ matrix.tag-latest }}
       tag-suffix: ${{ matrix.tag-suffix }}
@@ -54,83 +46,19 @@ jobs:
       cuda-major-version: ${{ matrix.cuda-major-version }}
       cuda-minor-version: ${{ matrix.cuda-minor-version }}
       platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
       runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
       base-image: ${{ matrix.base-image }}
       backend: ${{ matrix.backend }}
       dockerfile: ${{ matrix.dockerfile }}
       skip-drivers: ${{ matrix.skip-drivers }}
       context: ${{ matrix.context }}
       ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
     secrets:
       quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
     strategy:
       fail-fast: true
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
-  backend-jobs-singlearch:
-    needs: generate-matrix
-    uses: ./.github/workflows/backend_build.yml
-    if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: true
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
-  backend-merge-jobs-multiarch:
-    needs: [generate-matrix, backend-jobs-multiarch]
-    # backend_merge.yml's push-side steps are all gated on
-    # github.event_name != 'pull_request', so on a PR the merge job would
-    # do nothing. Skip it entirely to avoid spinning up an empty runner.
-    # !cancelled() lets the merge run even when a few build legs fail —
-    # see the matching note in backend.yml.
-    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-multiarch'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
-
-  backend-merge-jobs-singlearch:
-    needs: [generate-matrix, backend-jobs-singlearch]
-    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }}
   backend-jobs-darwin:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build_darwin.yml
@@ -138,7 +66,7 @@ jobs:
     with:
       backend: ${{ matrix.backend }}
       build-type: ${{ matrix.build-type }}
-      go-version: "1.25.x"
+      go-version: "1.24.x"
       tag-suffix: ${{ matrix.tag-suffix }}
       lang: ${{ matrix.lang || 'python' }}
       use-pip: ${{ matrix.backend == 'diffusers' }}

.github/workflows/base-images.yml

@@ -1,161 +0,0 @@
----
-name: 'build base-grpc images'
-
-# Builds + pushes pre-compiled builder base images that downstream
-# llama-cpp / ik-llama-cpp / turboquant variant Dockerfiles will FROM
-# (PR 2). Each base contains apt deps + protoc + cmake + gRPC at
-# /opt/grpc + (conditionally) CUDA / ROCm / Vulkan toolchains.
-#
-# Triggers:
-#   - schedule (Saturdays 05:00 UTC) - picks up Ubuntu/CUDA/ROCm
-#     security updates and re-runs ahead of the backend.yml weekly
-#     cron (Sundays 06:00 UTC).
-#   - workflow_dispatch - manual one-off rebuild.
-#   - push to master that touches Dockerfile.base-grpc-builder or
-#     this workflow itself - keeps bases in sync with their inputs.
-#
-# Bootstrap (one-time after this PR merges):
-#   gh workflow run base-images.yml --ref master
-# Wait ~30 min for all 9 matrix variants to push to
-# quay.io/go-skynet/ci-cache:base-grpc-* before merging PR 2.
-
-on:
-  schedule:
-    - cron: '0 5 * * 6'
-  workflow_dispatch:
-  push:
-    branches: [master]
-    paths:
-      - 'backend/Dockerfile.base-grpc-builder'
-      - '.github/workflows/base-images.yml'
-      # The install logic and apt-mirror helper are bind-mounted into
-      # Dockerfile.base-grpc-builder at build time — changes to either
-      # affect the produced base images and must trigger a rebuild.
-      - '.docker/install-base-deps.sh'
-      - '.docker/apt-mirror.sh'
-
-concurrency:
-  group: ci-base-images-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  build:
-    if: github.repository == 'mudler/LocalAI'
-    runs-on: ${{ matrix.runs-on }}
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - tag: 'base-grpc-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:24.04'
-            build-type: ''
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'ubuntu:24.04'
-            build-type: ''
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-cuda-12-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:24.04'
-            build-type: 'cublas'
-            cuda-major-version: '12'
-            cuda-minor-version: '8'
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-cuda-13-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:22.04'
-            build-type: 'cublas'
-            cuda-major-version: '13'
-            cuda-minor-version: '0'
-            ubuntu-version: '2204'
-          - tag: 'base-grpc-cuda-13-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'ubuntu:24.04'
-            build-type: 'cublas'
-            cuda-major-version: '13'
-            cuda-minor-version: '0'
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-rocm-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'rocm/dev-ubuntu-24.04:7.2.1'
-            build-type: 'hipblas'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-vulkan-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'ubuntu:24.04'
-            build-type: 'vulkan'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-vulkan-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'ubuntu:24.04'
-            build-type: 'vulkan'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          - tag: 'base-grpc-intel-amd64'
-            runs-on: 'ubuntu-latest'
-            base-image: 'intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04'
-            build-type: 'sycl'
-            cuda-major-version: ''
-            cuda-minor-version: ''
-            ubuntu-version: '2404'
-          # Legacy JetPack r36.4.0 base for older Jetson devices (CUDA 12).
-          # Distinct from base-grpc-cuda-13-arm64 (Ubuntu 24.04 + CUDA 13 sbsa)
-          # which targets newer Jetsons. Some matrix entries
-          # (-nvidia-l4t-arm64-llama-cpp / -turboquant) still build against
-          # the JetPack image, so we need a matching base.
-          - tag: 'base-grpc-l4t-cuda-12-arm64'
-            runs-on: 'ubuntu-24.04-arm'
-            base-image: 'nvcr.io/nvidia/l4t-jetpack:r36.4.0'
-            build-type: 'l4t'
-            cuda-major-version: '12'
-            cuda-minor-version: '0'
-            ubuntu-version: '2204'
-            # JetPack r36.4.0 already ships CUDA preinstalled at /usr/local/cuda;
-            # apt-installing cuda-nvcc-12-0 from the public repos fails because
-            # those packages aren't published for the JetPack apt feed. Match
-            # the original l4t matrix entry which set skip-drivers: 'true'.
-            skip-drivers: 'true'
-    steps:
-      - uses: actions/checkout@v6
-        with:
-          submodules: false
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
-      - name: Set up build disk
-        uses: ./.github/actions/setup-build-disk
-      - uses: docker/setup-qemu-action@master
-        with:
-          platforms: all
-      - uses: docker/setup-buildx-action@master
-      - uses: docker/login-action@v4
-        with:
-          registry: quay.io
-          username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-          password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-      - uses: docker/build-push-action@v7
-        with:
-          context: .
-          file: ./backend/Dockerfile.base-grpc-builder
-          build-args: |
-            BASE_IMAGE=${{ matrix.base-image }}
-            BUILD_TYPE=${{ matrix.build-type }}
-            CUDA_MAJOR_VERSION=${{ matrix.cuda-major-version }}
-            CUDA_MINOR_VERSION=${{ matrix.cuda-minor-version }}
-            UBUNTU_VERSION=${{ matrix.ubuntu-version }}
-            SKIP_DRIVERS=${{ matrix.skip-drivers || 'false' }}
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }},mode=max,ignore-error=true
-          provenance: false
-          tags: quay.io/go-skynet/ci-cache:${{ matrix.tag }}
-          push: true

.github/workflows/build-test.yaml

@@ -50,8 +50,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:

.github/workflows/bump-inference-defaults.yml

@@ -1,48 +0,0 @@
-name: Bump inference defaults
-
-on:
-  schedule:
-    # Run daily at 06:00 UTC
-    - cron: '0 6 * * *'
-  workflow_dispatch: # Allow manual trigger
-
-permissions:
-  contents: write
-  pull-requests: write
-
-jobs:
-  bump:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v6
-
-      - uses: actions/setup-go@v5
-        with:
-          go-version-file: go.mod
-
-      - name: Re-fetch inference defaults
-        run: make generate-force
-
-      - name: Check for changes
-        id: diff
-        run: |
-          if git diff --quiet core/config/inference_defaults.json; then
-            echo "changed=false" >> "$GITHUB_OUTPUT"
-          else
-            echo "changed=true" >> "$GITHUB_OUTPUT"
-          fi
-
-      - name: Create Pull Request
-        if: steps.diff.outputs.changed == 'true'
-        uses: peter-evans/create-pull-request@v8
-        with:
-          commit-message: "chore: bump inference defaults from unsloth"
-          title: "chore: bump inference defaults from unsloth"
-          body: |
-            Auto-generated update of `core/config/inference_defaults.json` from
-            [unsloth's inference_defaults.json](https://github.com/unslothai/unsloth/blob/main/studio/backend/assets/configs/inference_defaults.json).
-
-            This PR was created automatically by the `bump-inference-defaults` workflow.
-          branch: chore/bump-inference-defaults
-          delete-branch: true
-          labels: automated

.github/workflows/bump_deps.yaml

@@ -14,18 +14,6 @@ jobs:
             variable: "LLAMA_VERSION"
             branch: "master"
             file: "backend/cpp/llama-cpp/Makefile"
-          - repository: "ikawrakow/ik_llama.cpp"
-            variable: "IK_LLAMA_VERSION"
-            branch: "main"
-            file: "backend/cpp/ik-llama-cpp/Makefile"
-          - repository: "TheTom/llama-cpp-turboquant"
-            variable: "TURBOQUANT_VERSION"
-            branch: "feature/turboquant-kv-cache"
-            file: "backend/cpp/turboquant/Makefile"
-          - repository: "antirez/ds4"
-            variable: "DS4_VERSION"
-            branch: "main"
-            file: "backend/cpp/ds4/Makefile"
           - repository: "ggml-org/whisper.cpp"
             variable: "WHISPER_CPP_VERSION"
             branch: "master"
@@ -46,18 +34,6 @@ jobs:
             variable: "ACESTEP_CPP_VERSION"
             branch: "master"
             file: "backend/go/acestep-cpp/Makefile"
-          - repository: "PABannier/sam3.cpp"
-            variable: "SAM3_VERSION"
-            branch: "main"
-            file: "backend/go/sam3-cpp/Makefile"
-          - repository: "predict-woo/qwen3-tts.cpp"
-            variable: "QWEN3TTS_CPP_VERSION"
-            branch: "main"
-            file: "backend/go/qwen3-tts-cpp/Makefile"
-          - repository: "localai-org/vibevoice.cpp"
-            variable: "VIBEVOICE_CPP_VERSION"
-            branch: "master"
-            file: "backend/go/vibevoice-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
@@ -88,37 +64,5 @@ jobs:
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
-  bump-vllm-wheel:
-    # vLLM's cu130 wheel comes from a per-tag index URL (no /latest/ alias),
-    # so the cublas13 requirements file pins both a URL segment and a version
-    # constraint. bump_deps.sh handles git-sha-in-Makefile only — this job
-    # rewrites both values atomically when a new vLLM stable tag ships.
-    if: github.repository == 'mudler/LocalAI'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v6
-      - name: Bump vLLM cu130 wheel pin 🔧
-        id: bump
-        run: |
-          bash .github/bump_vllm_wheel.sh vllm-project/vllm backend/python/vllm/requirements-cublas13-after.txt VLLM_VERSION
-          {
-            echo 'message<<EOF'
-            cat "VLLM_VERSION_message.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          {
-            echo 'commit<<EOF'
-            cat "VLLM_VERSION_commit.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          rm -rfv VLLM_VERSION_message.txt VLLM_VERSION_commit.txt
-      - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
-        with:
-          token: ${{ secrets.UPDATE_BOT_TOKEN }}
-          push-to-fork: ci-forks/LocalAI
-          commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel'
-          title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_VERSION"
-          body: ${{ steps.bump.outputs.message }}
-          signoff: true
+
+

.github/workflows/checksum_checker.yaml

@@ -8,9 +8,15 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
+      - name: Force Install GIT latest
+        run: |
+          sudo apt-get update \
+          && sudo apt-get install -y software-properties-common \
+          && sudo apt-get update \
+          && sudo add-apt-repository -y ppa:git-core/ppa \
+          && sudo apt-get update \
+          && sudo apt-get install -y git
       - uses: actions/checkout@v6
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Install dependencies
         run: |
           sudo apt-get update

.github/workflows/gallery-agent.yaml

@@ -2,7 +2,7 @@ name: Gallery Agent
 on:
 
   schedule:
-    - cron: '0 */12 * * *'  # Run every 4 hours
+    - cron: '0 */3 * * *'  # Run every 4 hours
   workflow_dispatch:
     inputs:
       search_term:
@@ -48,88 +48,21 @@ jobs:
           go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
           go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
           PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Process gallery-agent PR commands
-        env:
-          GH_TOKEN: ${{ secrets.UPDATE_BOT_TOKEN }}
-          REPO: ${{ github.repository }}
-          SEARCH: 'gallery agent in:title'
-        run: |
-          # Walk gallery-agent PRs and act on maintainer comments:
-          #   /gallery-agent blacklist → label `gallery-agent/blacklisted` + close (never repropose)
-          #   /gallery-agent recreate  → close without label (next run may repropose)
-          # Only comments from OWNER / MEMBER / COLLABORATOR are honored so
-          # random users can't drive the bot.
-          #
-          # We scan both open PRs AND recently-closed PRs that don't already
-          # carry the blacklist label. This covers the common flow where a
-          # maintainer writes /gallery-agent blacklist and immediately clicks
-          # Close — without this, the next scheduled run wouldn't see the
-          # command (PR is already closed) and would repropose the model.
-          gh label create gallery-agent/blacklisted \
-            --repo "$REPO" --color ededed \
-            --description "gallery-agent must not repropose this model" 2>/dev/null || true
-
-          prs_open=$(gh pr list --repo "$REPO" --state open --search "$SEARCH" \
-            --json number --jq '.[].number')
-          # Closed PRs from the last 14 days that don't yet have the blacklist label.
-          # Bounded window keeps the scan cheap while covering late-applied commands.
-          since=$(date -u -d '14 days ago' +%Y-%m-%d)
-          prs_closed=$(gh pr list --repo "$REPO" --state closed \
-            --search "$SEARCH closed:>=$since -label:gallery-agent/blacklisted" \
-            --json number --jq '.[].number')
-          prs=$(printf '%s\n%s\n' "$prs_open" "$prs_closed" | sort -u | sed '/^$/d')
-          for pr in $prs; do
-            state=$(gh pr view "$pr" --repo "$REPO" --json state --jq '.state')
-            cmds=$(gh pr view "$pr" --repo "$REPO" --json comments \
-              --jq '.comments[] | select(.authorAssociation=="OWNER" or .authorAssociation=="MEMBER" or .authorAssociation=="COLLABORATOR") | .body')
-            if echo "$cmds" | grep -qE '(^|[[:space:]])/gallery-agent[[:space:]]+blacklist([[:space:]]|$)'; then
-              echo "PR #$pr: blacklist command found (state=$state)"
-              gh pr edit "$pr" --repo "$REPO" --add-label gallery-agent/blacklisted || true
-              if [ "$state" = "OPEN" ]; then
-                gh pr close "$pr" --repo "$REPO" --comment "Blacklisted via \`/gallery-agent blacklist\`. This model will not be reproposed." || true
-              fi
-            elif [ "$state" = "OPEN" ] && echo "$cmds" | grep -qE '(^|[[:space:]])/gallery-agent[[:space:]]+recreate([[:space:]]|$)'; then
-              echo "PR #$pr: recreate command found"
-              gh pr close "$pr" --repo "$REPO" --comment "Closed via \`/gallery-agent recreate\`. The next scheduled run will propose this model again." || true
-            fi
-          done
-
-      - name: Collect skip URLs for the gallery agent
-        id: open_prs
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          REPO: ${{ github.repository }}
-          SEARCH: 'gallery agent in:title'
-        run: |
-          # Skip set =
-          #   URLs from any open gallery-agent PR (avoid duplicate PRs for the same model while one is pending)
-          # + URLs from closed PRs carrying the `gallery-agent/blacklisted` label (hard blacklist)
-          # Plain-closed PRs without the label are ignored — closing a PR is
-          # not by itself a "never propose again" signal; maintainers must
-          # opt in via the /gallery-agent blacklist comment command.
-          urls_open=$(gh pr list --repo "$REPO" --state open --search "$SEARCH" \
-            --json body --jq '[.[].body] | join("\n")' \
-            | grep -oE 'https://huggingface\.co/[^ )]+' || true)
-          urls_blacklist=$(gh pr list --repo "$REPO" --state closed --search "$SEARCH" \
-            --label gallery-agent/blacklisted \
-            --json body --jq '[.[].body] | join("\n")' \
-            | grep -oE 'https://huggingface\.co/[^ )]+' || true)
-          urls=$(printf '%s\n%s\n' "$urls_open" "$urls_blacklist" | sort -u | sed '/^$/d')
-          echo "Skip URLs:"
-          echo "$urls"
-          {
-            echo "urls<<EOF"
-            echo "$urls"
-            echo "EOF"
-          } >> "$GITHUB_OUTPUT"
+      - uses: mudler/localai-github-action@v1.1
+        with:
+          model: 'https://huggingface.co/unsloth/Qwen3.5-2B-GGUF'
 
       - name: Run gallery agent
         env:
+          #OPENAI_MODEL: ${{ secrets.OPENAI_MODEL }}
+          OPENAI_MODE: Qwen3.5-2B-GGUF
+          OPENAI_BASE_URL: "http://localhost:8080"
+          OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
+          #OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
           SEARCH_TERM: ${{ github.event.inputs.search_term || 'GGUF' }}
           LIMIT: ${{ github.event.inputs.limit || '15' }}
           QUANTIZATION: ${{ github.event.inputs.quantization || 'Q4_K_M' }}
           MAX_MODELS: ${{ github.event.inputs.max_models || '1' }}
-          EXTRA_SKIP_URLS: ${{ steps.open_prs.outputs.urls }}
         run: |
           export GALLERY_INDEX_PATH=$PWD/gallery/index.yaml
           go run ./.github/gallery-agent
@@ -191,21 +124,7 @@ jobs:
             
             **Added Models:**
             ${{ steps.read_summary.outputs.added_models || '- No models added' }}
-
-            ### Bot commands
-
-            Maintainers (owner / member / collaborator) can control this PR
-            by leaving a comment with one of:
-
-            - `/gallery-agent recreate` — close this PR; the next scheduled
-              run will propose this model again (useful if the entry needs
-              to be regenerated with fresh metadata).
-            - `/gallery-agent blacklist` — close this PR and permanently
-              prevent the gallery agent from ever reproposing this model.
-
-            Plain "Close" (without a command) is treated as a no-op: the
-            model may be reproposed by a future run.
-
+            
             **Workflow Details:**
             - Triggered by: `${{ github.event_name }}`
             - Run ID: `${{ github.run_id }}`

.github/workflows/generate_grpc_cache.yaml

@@ -0,0 +1,96 @@
+name: 'generate and publish GRPC docker caches'
+
+on:
+  workflow_dispatch:
+
+  schedule:
+    # daily at midnight
+    - cron: '0 0 * * *'
+
+concurrency:
+  group: grpc-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
+
+jobs:
+  generate_caches:
+    if: github.repository == 'mudler/LocalAI'
+    strategy:
+      matrix:
+        include:
+          - grpc-base-image: ubuntu:24.04
+            runs-on: 'ubuntu-latest'
+            platforms: 'linux/amd64,linux/arm64'
+    runs-on: ${{matrix.runs-on}}
+    steps:
+      - name: Release space from worker
+        if: matrix.runs-on == 'ubuntu-latest'
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get remove -y microsoft-edge-stable || true
+          sudo apt-get remove -y firefox || true
+          sudo apt-get remove -y powershell || true
+          sudo apt-get remove -y r-base-core || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf "/usr/local/share/boost" || true
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
+          df -h
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@master
+        with:
+          platforms: all
+
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Cache GRPC
+        uses: docker/build-push-action@v7
+        with:
+          builder: ${{ steps.buildx.outputs.name }}
+          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
+          # This means that even the MAKEFLAGS have to be an EXACT match.
+          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
+          build-args: |
+            GRPC_BASE_IMAGE=${{ matrix.grpc-base-image }}
+            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
+            GRPC_VERSION=v1.65.0
+          context: .
+          file: ./Dockerfile
+          cache-to: type=gha,ignore-error=true
+          cache-from: type=gha
+          target: grpc
+          platforms: ${{ matrix.platforms }}
+          push: false

.github/workflows/generate_intel_image.yaml

@@ -7,8 +7,8 @@ on:
       - master
 
 concurrency:
-  group: intel-cache-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: intel-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   generate_caches:
@@ -16,7 +16,7 @@ jobs:
     strategy:
       matrix:
         include:
-          - base-image: intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04
+          - base-image: intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04
             runs-on: 'arc-runner-set'
             platforms: 'linux/amd64'
     runs-on: ${{matrix.runs-on}}

.github/workflows/gh-pages.yml

@@ -1,75 +0,0 @@
-name: Deploy docs to GitHub Pages
-
-on:
-  push:
-    branches:
-      - master
-    paths:
-      - 'docs/**'
-      - 'gallery/**'
-      - 'images/**'
-      - '.github/ci/modelslist.go'
-      - '.github/workflows/gh-pages.yml'
-  workflow_dispatch:
-
-permissions:
-  contents: read
-  pages: write
-  id-token: write
-
-concurrency:
-  group: pages
-  cancel-in-progress: false
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    env:
-      HUGO_VERSION: "0.146.3"
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v6
-        with:
-          fetch-depth: 0  # needed for enableGitInfo
-          submodules: true
-
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.22'
-          cache: false
-
-      - name: Setup Hugo
-        uses: peaceiris/actions-hugo@v3
-        with:
-          hugo-version: ${{ env.HUGO_VERSION }}
-          extended: true
-
-      - name: Setup Pages
-        id: pages
-        uses: actions/configure-pages@v6
-
-      - name: Generate gallery
-        run: go run ./.github/ci/modelslist.go ./gallery/index.yaml > docs/static/gallery.html
-
-      - name: Build site
-        working-directory: docs
-        run: |
-          mkdir -p layouts/_default
-          hugo --minify --baseURL "${{ steps.pages.outputs.base_url }}/"
-
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@v5
-        with:
-          path: docs/public
-
-  deploy:
-    environment:
-      name: github-pages
-      url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
-    needs: build
-    steps:
-      - name: Deploy to GitHub Pages
-        id: deployment
-        uses: actions/deploy-pages@v5

.github/workflows/image-pr.yml

@@ -5,8 +5,8 @@
     pull_request:
   
   concurrency:
-    group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-    cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
+    cancel-in-progress: true
   
   jobs:
     image-build:
@@ -18,9 +18,9 @@
         cuda-major-version: ${{ matrix.cuda-major-version }}
         cuda-minor-version: ${{ matrix.cuda-minor-version }}
         platforms: ${{ matrix.platforms }}
-        platform-tag: ${{ matrix.platform-tag || '' }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
       secrets:
@@ -59,36 +59,28 @@
               platforms: 'linux/amd64'
               tag-latest: 'false'
               tag-suffix: '-hipblas'
-              base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+              base-image: "rocm/dev-ubuntu-24.04:6.4.4"
+              grpc-base-image: "ubuntu:24.04"
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'sycl'
               platforms: 'linux/amd64'
               tag-latest: 'false'
-              base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
+              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+              grpc-base-image: "ubuntu:24.04"
               tag-suffix: 'sycl'
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
             - build-type: 'vulkan'
-              platforms: 'linux/amd64'
-              platform-tag: 'amd64'
+              platforms: 'linux/amd64,linux/arm64'
               tag-latest: 'false'
               tag-suffix: '-vulkan-core'
               runs-on: 'ubuntu-latest'
               base-image: "ubuntu:24.04"
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'
-            - build-type: 'vulkan'
-              platforms: 'linux/arm64'
-              platform-tag: 'arm64'
-              tag-latest: 'false'
-              tag-suffix: '-vulkan-core'
-              runs-on: 'ubuntu-24.04-arm'
-              base-image: "ubuntu:24.04"
-              makeflags: "--jobs=4 --output-sync=target"
-              ubuntu-version: '2404'
             - build-type: 'cublas'
               cuda-major-version: "13"
               cuda-minor-version: "0"

.github/workflows/image.yml

@@ -9,8 +9,8 @@
         - '*'
   
   concurrency:
-    group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-    cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
+    cancel-in-progress: true
   
   jobs:
     hipblas-jobs:
@@ -25,6 +25,7 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
         ubuntu-codename: ${{ matrix.ubuntu-codename }}
@@ -40,12 +41,13 @@
               platforms: 'linux/amd64'
               tag-latest: 'auto'
               tag-suffix: '-gpu-hipblas'
-              base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+              base-image: "rocm/dev-ubuntu-24.04:6.4.4"
+              grpc-base-image: "ubuntu:24.04"
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-
+  
     core-image-build:
       if: github.repository == 'mudler/LocalAI'
       uses: ./.github/workflows/image_build.yml
@@ -56,9 +58,9 @@
         cuda-major-version: ${{ matrix.cuda-major-version }}
         cuda-minor-version: ${{ matrix.cuda-minor-version }}
         platforms: ${{ matrix.platforms }}
-        platform-tag: ${{ matrix.platform-tag || '' }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         skip-drivers: ${{ matrix.skip-drivers }}
         ubuntu-version: ${{ matrix.ubuntu-version }}
@@ -73,8 +75,7 @@
         matrix:
           include:
             - build-type: ''
-              platforms: 'linux/amd64'
-              platform-tag: 'amd64'
+              platforms: 'linux/amd64,linux/arm64'
               tag-latest: 'auto'
               tag-suffix: ''
               base-image: "ubuntu:24.04"
@@ -83,17 +84,6 @@
               skip-drivers: 'false'
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-            - build-type: ''
-              platforms: 'linux/arm64'
-              platform-tag: 'arm64'
-              tag-latest: 'auto'
-              tag-suffix: ''
-              base-image: "ubuntu:24.04"
-              runs-on: 'ubuntu-24.04-arm'
-              makeflags: "--jobs=4 --output-sync=target"
-              skip-drivers: 'false'
-              ubuntu-version: '2404'
-              ubuntu-codename: 'noble'
             - build-type: 'cublas'
               cuda-major-version: "12"
               cuda-minor-version: "8"
@@ -119,8 +109,7 @@
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
             - build-type: 'vulkan'
-              platforms: 'linux/amd64'
-              platform-tag: 'amd64'
+              platforms: 'linux/amd64,linux/arm64'
               tag-latest: 'auto'
               tag-suffix: '-gpu-vulkan'
               runs-on: 'ubuntu-latest'
@@ -129,141 +118,17 @@
               makeflags: "--jobs=4 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-            - build-type: 'vulkan'
-              platforms: 'linux/arm64'
-              platform-tag: 'arm64'
-              tag-latest: 'auto'
-              tag-suffix: '-gpu-vulkan'
-              runs-on: 'ubuntu-24.04-arm'
-              base-image: "ubuntu:24.04"
-              skip-drivers: 'false'
-              makeflags: "--jobs=4 --output-sync=target"
-              ubuntu-version: '2404'
-              ubuntu-codename: 'noble'
             - build-type: 'intel'
               platforms: 'linux/amd64'
               tag-latest: 'auto'
-              base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
+              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+              grpc-base-image: "ubuntu:24.04"
               tag-suffix: '-gpu-intel'
               runs-on: 'ubuntu-latest'
               makeflags: "--jobs=3 --output-sync=target"
               ubuntu-version: '2404'
               ubuntu-codename: 'noble'
-
-    core-image-merge:
-      # !cancelled(): without it, GHA's default `needs:` cascade skips the
-      # merge whenever any matrix cell of the parent build fails or is
-      # cancelled. Same fix as backend.yml's merge jobs — we still want to
-      # publish the manifest list for tag-suffixes whose legs all succeeded.
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: core-image-build
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: ''
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    gpu-vulkan-image-merge:
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: core-image-build
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-gpu-vulkan'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    # Single-arch server-image merges. Same conceptual fix as the backend
-    # singletons in PR #9781: image_build.yml pushes by canonical digest
-    # only, so without a downstream merge step there's no tag for consumers
-    # (no :latest-gpu-nvidia-cuda-12, no :v<X>-gpu-nvidia-cuda-12, etc.).
-    # Each merge job needs only its parent build matrix and is filtered by
-    # tag-suffix in image_merge.yml's artifact-download pattern.
-    gpu-nvidia-cuda-12-image-merge:
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: core-image-build
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-gpu-nvidia-cuda-12'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    gpu-nvidia-cuda-13-image-merge:
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: core-image-build
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-gpu-nvidia-cuda-13'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    gpu-intel-image-merge:
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: core-image-build
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-gpu-intel'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    gpu-hipblas-image-merge:
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: hipblas-jobs
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-gpu-hipblas'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    nvidia-l4t-arm64-image-merge:
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: gh-runner
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-nvidia-l4t-arm64'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
-    nvidia-l4t-arm64-cuda-13-image-merge:
-      if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
-      needs: gh-runner
-      uses: ./.github/workflows/image_merge.yml
-      with:
-        tag-latest: 'auto'
-        tag-suffix: '-nvidia-l4t-arm64-cuda-13'
-      secrets:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-
+  
     gh-runner:
       if: github.repository == 'mudler/LocalAI'
       uses: ./.github/workflows/image_build.yml
@@ -276,6 +141,7 @@
         platforms: ${{ matrix.platforms }}
         runs-on: ${{ matrix.runs-on }}
         base-image: ${{ matrix.base-image }}
+        grpc-base-image: ${{ matrix.grpc-base-image }}
         makeflags: ${{ matrix.makeflags }}
         skip-drivers: ${{ matrix.skip-drivers }}
         ubuntu-version: ${{ matrix.ubuntu-version }}

.github/workflows/image_build.yml

@@ -8,6 +8,11 @@ on:
         description: 'Base image'
         required: true
         type: string
+      grpc-base-image:
+        description: 'GRPC Base image, must be a compatible image with base-image'
+        required: false
+        default: ''
+        type: string
       build-type:
         description: 'Build type'
         default: ''
@@ -24,15 +29,6 @@ on:
         description: 'Platforms'
         default: ''
         type: string
-      platform-tag:
-        description: |
-          Short tag identifying the platform leg, e.g. "amd64" or "arm64".
-          Used to scope the per-arch registry cache and the digest artifact name.
-          Optional during the migration; will be flipped to required: true once
-          every caller passes an explicit value.
-        required: false
-        default: ''
-        type: string
       tag-latest:
         description: 'Tag latest'
         default: ''
@@ -79,20 +75,73 @@ jobs:
     runs-on: ${{ inputs.runs-on }}
     steps:
 
+      - name: Free Disk Space (Ubuntu)
+        if: inputs.runs-on == 'ubuntu-latest'
+        uses: jlumbroso/free-disk-space@main
+        with:
+          # this might remove tools that are actually needed,
+          # if set to "true" but frees about 6 GB
+          tool-cache: true
+          # all of these default to true, but feel free to set to
+          # "false" if necessary for your workflow
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          docker-images: true
+          swap-storage: true
+      - name: Force Install GIT latest
+        run: |
+          sudo apt-get update \
+          && sudo apt-get install -y software-properties-common \
+          && sudo apt-get update \
+          && sudo add-apt-repository -y ppa:git-core/ppa \
+          && sudo apt-get update \
+          && sudo apt-get install -y git
       - name: Checkout
         uses: actions/checkout@v6
 
-      - name: Configure apt mirror on runner
-        id: apt_mirror
-        uses: ./.github/actions/configure-apt-mirror
-
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
-        with:
-          mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
-
-      - name: Set up build disk
-        uses: ./.github/actions/setup-build-disk
+      - name: Release space from worker
+        if: inputs.runs-on == 'ubuntu-latest'
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get remove -y microsoft-edge-stable || true
+          sudo apt-get remove -y firefox || true
+          sudo apt-get remove -y powershell || true
+          sudo apt-get remove -y r-base-core || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf "/usr/local/share/boost" || true
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
+          df -h
 
       - name: Docker meta
         id: meta
@@ -147,89 +196,59 @@ jobs:
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
-      - name: Build and push by digest
-        id: build
+      - name: Build and push
         uses: docker/build-push-action@v7
         if: github.event_name != 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
+          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
+          # This means that even the MAKEFLAGS have to be an EXACT match.
+          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
+          # This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
           build-args: |
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
+            GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
+            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
+            GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
           context: .
           file: ./Dockerfile
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
-          cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
-          outputs: |
-            type=image,name=quay.io/go-skynet/local-ai,push-by-digest=true,name-canonical=true,push=true
-            type=image,name=localai/localai,push-by-digest=true,name-canonical=true,push=true
-          # See backend_build.yml for the rationale — provenance=mode=max
-          # diverges the manifest-list digest per registry, breaking the
-          # downstream imagetools create lookup.
-          provenance: false
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
-
-      - name: Export digest
-        if: github.event_name != 'pull_request'
-        run: |
-          mkdir -p /tmp/digests
-          digest="${{ steps.build.outputs.digest }}"
-          touch "/tmp/digests/${digest#sha256:}"
-
-      # See .github/scripts/anchor-digest-in-cache.sh for why this is needed
-      # and how it interacts with image_merge.yml's cleanup step. Mirrors the
-      # same anchor in backend_build.yml — quay's per-repo manifest GC reaps
-      # untagged manifests in local-ai before the merge runs.
-      - name: Anchor digest in ci-cache so quay GC won't reap before merge
-        if: github.event_name != 'pull_request'
-        env:
-          TAG_SUFFIX: ${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}
-          PLATFORM_TAG: ${{ inputs.platform-tag || 'single' }}
-          DIGEST: ${{ steps.build.outputs.digest }}
-          SOURCE_IMAGE: quay.io/go-skynet/local-ai
-        run: .github/scripts/anchor-digest-in-cache.sh
-
-      - name: Upload digest artifact
-        if: github.event_name != 'pull_request'
-        uses: actions/upload-artifact@v7
-        with:
-          # `--` separator + 'single' placeholder for empty platform-tag —
-          # same pattern as backend_build.yml. Prevents prefix collisions
-          # in the merge-side glob (e.g. -nvidia-l4t-arm64 is a prefix of
-          # -nvidia-l4t-arm64-cuda-13).
-          name: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}--${{ inputs.platform-tag || 'single' }}
-          path: /tmp/digests/*
-          if-no-files-found: error
-          retention-days: 1
 ### Start testing image
       - name: Build and push
         uses: docker/build-push-action@v7
         if: github.event_name == 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
+          # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
+          # This means that even the MAKEFLAGS have to be an EXACT match.
+          # If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
+          # This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
           build-args: |
             BUILD_TYPE=${{ inputs.build-type }}
             CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
+            GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
+            GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
+            GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
-            APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
-            APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
           context: .
           file: ./Dockerfile
-          cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
+          cache-from: type=gha
           platforms: ${{ inputs.platforms }}
           #push: true
           tags: ${{ steps.meta_pull_request.outputs.tags }}

.github/workflows/image_merge.yml

@@ -1,145 +0,0 @@
----
-name: 'merge LocalAI image manifest list (reusable)'
-
-# Reusable workflow that joins per-arch digest artifacts (uploaded by
-# image_build.yml when called with platform-tag) into a single tagged
-# multi-arch manifest list.
-
-on:
-  workflow_call:
-    inputs:
-      tag-latest:
-        description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
-        required: false
-        type: string
-        default: ''
-      tag-suffix:
-        description: 'Image tag suffix (empty for core image). Used in artifact pattern with a -core placeholder for empty.'
-        required: true
-        type: string
-    secrets:
-      dockerUsername:
-        required: false
-      dockerPassword:
-        required: false
-      quayUsername:
-        required: true
-      quayPassword:
-        required: true
-
-jobs:
-  merge:
-    runs-on: ubuntu-latest
-    env:
-      quay_username: ${{ secrets.quayUsername }}
-    steps:
-      # Sparse checkout: needed for .github/scripts/ (the keepalive cleanup
-      # script). Skips the rest of the source tree.
-      - name: Checkout (.github/scripts only)
-        uses: actions/checkout@v6
-        with:
-          sparse-checkout: |
-            .github/scripts
-          sparse-checkout-cone-mode: false
-
-      - name: Download digests
-        uses: actions/download-artifact@v8
-        with:
-          # `--` separator anchors the glob so we don't over-match sibling
-          # tag-suffixes (e.g. -nvidia-l4t-arm64 vs -nvidia-l4t-arm64-cuda-13).
-          # Must stay in sync with image_build.yml's upload-artifact name.
-          pattern: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}--*
-          merge-multiple: true
-          path: /tmp/digests
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@master
-
-      - name: Login to DockerHub
-        if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
-        with:
-          username: ${{ secrets.dockerUsername }}
-          password: ${{ secrets.dockerPassword }}
-
-      - name: Login to Quay.io
-        uses: docker/login-action@v4
-        with:
-          registry: quay.io
-          username: ${{ secrets.quayUsername }}
-          password: ${{ secrets.quayPassword }}
-
-      - name: Docker meta
-        id: meta
-        uses: docker/metadata-action@v6
-        with:
-          images: |
-            quay.io/go-skynet/local-ai
-            localai/localai
-          tags: |
-            type=ref,event=branch
-            type=semver,pattern={{raw}}
-            type=sha
-          flavor: |
-            latest=${{ inputs.tag-latest }}
-            suffix=${{ inputs.tag-suffix }},onlatest=true
-
-      # Source from ci-cache, not local-ai. See backend_merge.yml for the
-      # detailed rationale — quay's manifest GC is per-repository, so the
-      # untagged digest in local-ai gets reaped while the same content lives
-      # tagged under ci-cache (anchored by image_build.yml). buildx imagetools
-      # create copies the manifest into local-ai (blobs already cross-mounted)
-      # and publishes the manifest list with user-facing tags. End state in
-      # local-ai is self-contained; no embedded reference to ci-cache.
-      - name: Create manifest list and push (quay)
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '.tags | map(select(startswith("quay.io/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No quay.io tags from docker/metadata-action; skipping quay merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'quay.io/go-skynet/ci-cache@sha256:%s ' *)
-          fi
-
-      - name: Create manifest list and push (dockerhub)
-        if: github.event_name != 'pull_request'
-        working-directory: /tmp/digests
-        run: |
-          set -euo pipefail
-          tags=$(jq -cr '.tags | map(select(startswith("localai/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -z "$tags" ]; then
-            echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
-          else
-            # shellcheck disable=SC2086
-            docker buildx imagetools create $tags \
-              $(printf 'localai/localai@sha256:%s ' *)
-          fi
-
-      - name: Inspect manifest
-        run: |
-          set -euo pipefail
-          first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
-          if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
-            docker buildx imagetools inspect "$first_tag"
-          fi
-
-      # See .github/scripts/cleanup-keepalive-tags.sh for the best-effort
-      # semantics — fails soft when the registry credential isn't OAuth-scoped.
-      - name: Cleanup keepalive tags in ci-cache
-        if: github.event_name != 'pull_request' && success()
-        env:
-          TAG_SUFFIX: ${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}
-          QUAY_TOKEN: ${{ secrets.quayPassword }}
-        run: .github/scripts/cleanup-keepalive-tags.sh
-
-      - name: Job summary
-        run: |
-          set -euo pipefail
-          echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
-          jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
-          echo >> "$GITHUB_STEP_SUMMARY"
-          echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
-          ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"

.github/workflows/lint.yml

@@ -1,48 +0,0 @@
----
-name: 'lint'
-
-on:
-  pull_request:
-    paths-ignore:
-      - 'docs/**'
-      - 'examples/**'
-      - 'README.md'
-      - '**/*.md'
-  push:
-    branches:
-      - master
-
-concurrency:
-  group: ci-lint-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  golangci-lint:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v6
-        with:
-          # Full history so golangci-lint's new-from-merge-base can reach
-          # origin/master and compute the diff against it.
-          fetch-depth: 0
-      - uses: actions/setup-go@v5
-        with:
-          go-version: '1.26.x'
-          cache: false
-      - name: install golangci-lint
-        run: |
-          curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh \
-            | sh -s -- -b "$(go env GOPATH)/bin" v2.11.4
-      - name: generate grpc proto sources
-        # pkg/grpc/proto/*.go is generated, not checked in. Several packages
-        # import it, so without this step typecheck fails project-wide.
-        run: make protogen-go
-      - name: stub react-ui dist for go:embed
-        # core/http/app.go has //go:embed react-ui/dist/*; the glob needs at
-        # least one non-hidden entry to satisfy typecheck. We don't run
-        # `make react-ui` here because lint doesn't need the real bundle.
-        run: |
-          mkdir -p core/http/react-ui/dist
-          touch core/http/react-ui/dist/index.html
-      - name: lint
-        run: make lint

.github/workflows/release.yaml

@@ -39,7 +39,7 @@ jobs:
         run: |
           make build-launcher-darwin
       - name: Upload DMG to Release
-        uses: softprops/action-gh-release@v3
+        uses: softprops/action-gh-release@v2
         with:
           files: ./dist/LocalAI.dmg
   launcher-build-linux:
@@ -49,8 +49,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:
@@ -61,6 +59,6 @@ jobs:
           sudo apt-get install golang gcc libgl1-mesa-dev xorg-dev libxkbcommon-dev
           make build-launcher-linux
       - name: Upload Linux launcher artifacts
-        uses: softprops/action-gh-release@v3
+        uses: softprops/action-gh-release@v2
         with:
           files: ./local-ai-launcher-linux.tar.xz

.github/workflows/test-extra.yml

@@ -10,55 +10,10 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-extra-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-extra-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
-  detect-changes:
-    runs-on: ubuntu-latest
-    outputs:
-      run-all: ${{ steps.detect.outputs.run-all }}
-      transformers: ${{ steps.detect.outputs.transformers }}
-      rerankers: ${{ steps.detect.outputs.rerankers }}
-      diffusers: ${{ steps.detect.outputs.diffusers }}
-      coqui: ${{ steps.detect.outputs.coqui }}
-      moonshine: ${{ steps.detect.outputs.moonshine }}
-      pocket-tts: ${{ steps.detect.outputs.pocket-tts }}
-      qwen-tts: ${{ steps.detect.outputs.qwen-tts }}
-      qwen-asr: ${{ steps.detect.outputs.qwen-asr }}
-      nemo: ${{ steps.detect.outputs.nemo }}
-      voxcpm: ${{ steps.detect.outputs.voxcpm }}
-      liquid-audio: ${{ steps.detect.outputs.liquid-audio }}
-      llama-cpp-quantization: ${{ steps.detect.outputs.llama-cpp-quantization }}
-      llama-cpp: ${{ steps.detect.outputs.llama-cpp }}
-      ik-llama-cpp: ${{ steps.detect.outputs.ik-llama-cpp }}
-      turboquant: ${{ steps.detect.outputs.turboquant }}
-      vllm: ${{ steps.detect.outputs.vllm }}
-      sglang: ${{ steps.detect.outputs.sglang }}
-      acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
-      qwen3-tts-cpp: ${{ steps.detect.outputs.qwen3-tts-cpp }}
-      vibevoice-cpp: ${{ steps.detect.outputs.vibevoice-cpp }}
-      localvqe: ${{ steps.detect.outputs.localvqe }}
-      voxtral: ${{ steps.detect.outputs.voxtral }}
-      kokoros: ${{ steps.detect.outputs.kokoros }}
-      insightface: ${{ steps.detect.outputs.insightface }}
-      speaker-recognition: ${{ steps.detect.outputs.speaker-recognition }}
-      sherpa-onnx: ${{ steps.detect.outputs.sherpa-onnx }}
-      whisper: ${{ steps.detect.outputs.whisper }}
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v6
-      - name: Setup Bun
-        uses: oven-sh/setup-bun@v2
-      - name: Install dependencies
-        run: bun add js-yaml @octokit/core
-      - name: Detect changed backends
-        id: detect
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GITHUB_EVENT_PATH: ${{ github.event_path }}
-        run: bun run scripts/changed-backends.js
-
   # Requires CUDA
   # tests-chatterbox-tts:
   #   runs-on: ubuntu-latest
@@ -82,8 +37,6 @@ jobs:
   #          make --jobs=5 --output-sync=target -C backend/python/chatterbox
   #          make --jobs=5 --output-sync=target -C backend/python/chatterbox test
   tests-transformers:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.transformers == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -105,8 +58,6 @@ jobs:
            make --jobs=5 --output-sync=target -C backend/python/transformers
            make --jobs=5 --output-sync=target -C backend/python/transformers test
   tests-rerankers:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.rerankers == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -129,8 +80,6 @@ jobs:
            make --jobs=5 --output-sync=target -C backend/python/rerankers test
 
   tests-diffusers:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.diffusers == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -280,8 +229,6 @@ jobs:
   #          make --jobs=5 --output-sync=target -C backend/python/vllm test
 
   tests-coqui:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.coqui == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -301,8 +248,6 @@ jobs:
           make --jobs=5 --output-sync=target -C backend/python/coqui
           make --jobs=5 --output-sync=target -C backend/python/coqui test
   tests-moonshine:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.moonshine == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -322,8 +267,6 @@ jobs:
           make --jobs=5 --output-sync=target -C backend/python/moonshine
           make --jobs=5 --output-sync=target -C backend/python/moonshine test
   tests-pocket-tts:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.pocket-tts == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -343,8 +286,6 @@ jobs:
           make --jobs=5 --output-sync=target -C backend/python/pocket-tts
           make --jobs=5 --output-sync=target -C backend/python/pocket-tts test
   tests-qwen-tts:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.qwen-tts == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -386,8 +327,6 @@ jobs:
   #         make --jobs=5 --output-sync=target -C backend/python/fish-speech
   #         make --jobs=5 --output-sync=target -C backend/python/fish-speech test
   tests-qwen-asr:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.qwen-asr == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -407,8 +346,6 @@ jobs:
           make --jobs=5 --output-sync=target -C backend/python/qwen-asr
           make --jobs=5 --output-sync=target -C backend/python/qwen-asr test
   tests-nemo:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.nemo == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -428,8 +365,6 @@ jobs:
           make --jobs=5 --output-sync=target -C backend/python/nemo
           make --jobs=5 --output-sync=target -C backend/python/nemo test
   tests-voxcpm:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.voxcpm == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -448,340 +383,7 @@ jobs:
         run: |
           make --jobs=5 --output-sync=target -C backend/python/voxcpm
           make --jobs=5 --output-sync=target -C backend/python/voxcpm test
-  # liquid-audio: LFM2.5-Audio any-to-any backend. The CI smoke test
-  # exercises Health() and LoadModel(mode:finetune) — fine-tune mode
-  # short-circuits before pulling weights (backend.py:192), so no
-  # HuggingFace download or GPU is needed. The full-inference path is
-  # gated on LIQUID_AUDIO_MODEL_ID, which we don't set here.
-  tests-liquid-audio:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.liquid-audio == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential ffmpeg
-          sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Test liquid-audio
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/liquid-audio
-          make --jobs=5 --output-sync=target -C backend/python/liquid-audio test
-  tests-llama-cpp-quantization:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.llama-cpp-quantization == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl git python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Build llama-quantize from llama.cpp
-        run: |
-          git clone --depth 1 https://github.com/ggml-org/llama.cpp.git /tmp/llama.cpp
-          cmake -B /tmp/llama.cpp/build -S /tmp/llama.cpp -DGGML_NATIVE=OFF
-          cmake --build /tmp/llama.cpp/build --target llama-quantize -j$(nproc)
-          sudo cp /tmp/llama.cpp/build/bin/llama-quantize /usr/local/bin/
-      - name: Install backend
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/llama-cpp-quantization
-      - name: Test llama-cpp-quantization
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/llama-cpp-quantization test
-  tests-llama-cpp-grpc:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.llama-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build llama-cpp backend image and run gRPC e2e tests
-        run: |
-          make test-extra-backend-llama-cpp
-  tests-llama-cpp-grpc-transcription:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.llama-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build llama-cpp backend image and run audio transcription gRPC e2e tests
-        run: |
-          make test-extra-backend-llama-cpp-transcription
-  # PR-acceptance smoke gate: always runs on every PR (no detect-changes gate, no
-  # paths filter). Pulls the pre-built master CPU llama-cpp image from quay
-  # instead of building from source, so the cost is a docker pull (~30s) plus the
-  # short Qwen3-0.6B model download. Exercises the full gRPC surface — health,
-  # load, predict, stream — plus the logprobs/logit_bias specs that moved out of
-  # core/http/app_test.go. Anything heavier or per-backend is gated to the
-  # detect-changes path-filter above.
-  tests-llama-cpp-smoke:
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Pull pre-built llama-cpp backend image
-        run: docker pull quay.io/go-skynet/local-ai-backends:master-cpu-llama-cpp
-      - name: Run e2e-backends smoke
-        env:
-          BACKEND_IMAGE: quay.io/go-skynet/local-ai-backends:master-cpu-llama-cpp
-          BACKEND_TEST_CAPS: health,load,predict,stream,logprobs,logit_bias
-        run: |
-          make test-extra-backend
-  # Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked LLM.
-  # Builds the sherpa-onnx Docker image, extracts the rootfs so the e2e suite
-  # can discover the backend binary + shared libs, downloads the three model
-  # bundles (silero-vad, omnilingual-asr, vits-ljs) and drives the realtime
-  # websocket spec end-to-end.
-  tests-sherpa-onnx-realtime:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Setup Node.js
-        uses: actions/setup-node@v6
-        with:
-          node-version: '22'
-      - name: Build sherpa-onnx backend image and run realtime e2e tests
-        run: |
-          make test-extra-e2e-realtime-sherpa
-  # Streaming ASR via the sherpa-onnx online recognizer (zipformer
-  # transducer). Exercises both AudioTranscription (buffered) and
-  # AudioTranscriptionStream (real-time deltas) on the e2e-backends
-  # harness.
-  tests-sherpa-onnx-grpc-transcription:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build sherpa-onnx backend image and run streaming ASR gRPC e2e tests
-        run: |
-          make test-extra-backend-sherpa-onnx-transcription
-  # End-to-end transcription via the e2e-backends gRPC harness against
-  # the whisper.cpp backend. Drives AudioTranscription (offline) and
-  # AudioTranscriptionStream (real, segment-callback-driven deltas) on
-  # ggml-base.en + the JFK 11s clip.
-  tests-whisper-grpc-transcription:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.whisper == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build whisper backend image and run transcription gRPC e2e tests
-        run: |
-          make test-extra-backend-whisper-transcription
-  # VITS TTS via the sherpa-onnx backend. Drives both TTS (file write) and
-  # TTSStream (PCM chunks) on the e2e-backends harness.
-  tests-sherpa-onnx-grpc-tts:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.sherpa-onnx == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build sherpa-onnx backend image and run TTS gRPC e2e tests
-        run: |
-          make test-extra-backend-sherpa-onnx-tts
-  tests-ik-llama-cpp-grpc:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.ik-llama-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build ik-llama-cpp backend image and run gRPC e2e tests
-        run: |
-          make test-extra-backend-ik-llama-cpp
-  tests-turboquant-grpc:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.turboquant == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      # Exercises the turboquant (llama.cpp fork) backend with KV-cache
-      # quantization enabled. The convenience target sets
-      # BACKEND_TEST_CACHE_TYPE_K / _V=q8_0, which are plumbed into the
-      # ModelOptions.CacheTypeKey/Value gRPC fields. LoadModel-success +
-      # backend stdout/stderr (captured by the Ginkgo suite) prove the
-      # cache-type config path reaches the fork's KV-cache init.
-      - name: Build turboquant backend image and run gRPC e2e tests
-        run: |
-          make test-extra-backend-turboquant
-  # tests-vllm-grpc is currently disabled in CI.
-  #
-  # The prebuilt vllm CPU wheel is compiled with AVX-512 VNNI/BF16
-  # instructions, and neither ubuntu-latest nor the bigger-runner pool
-  # offers a stable CPU baseline that supports them — runners come
-  # back with different hardware between runs and SIGILL on import of
-  # vllm.model_executor.models.registry. Compiling vllm from source
-  # via FROM_SOURCE=true works on any CPU but takes 30-50 minutes per
-  # run, which is too slow for a smoke test.
-  #
-  # The test itself (tests/e2e-backends + make test-extra-backend-vllm)
-  # is fully working and validated locally on a host with the right
-  # SIMD baseline. Run it manually with:
-  #
-  #   make test-extra-backend-vllm
-  #
-  # Re-enable this job once we have a self-hosted runner label with
-  # guaranteed AVX-512 VNNI/BF16 support, or once the vllm project
-  # publishes a CPU wheel with a wider baseline.
-  #
-  # tests-vllm-grpc:
-  #   needs: detect-changes
-  #   if: needs.detect-changes.outputs.vllm == 'true' || needs.detect-changes.outputs.run-all == 'true'
-  #   runs-on: bigger-runner
-  #   timeout-minutes: 90
-  #   steps:
-  #     - name: Clone
-  #       uses: actions/checkout@v6
-  #       with:
-  #         submodules: true
-  #     - name: Dependencies
-  #       run: |
-  #         sudo apt-get update
-  #         sudo apt-get install -y --no-install-recommends \
-  #             make build-essential curl unzip ca-certificates git tar
-  #     - name: Setup Go
-  #       uses: actions/setup-go@v5
-  #       with:
-  #         go-version: '1.25.4'
-  #     - name: Free disk space
-  #       run: |
-  #         sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-  #         df -h
-  #     - name: Build vllm (cpu) backend image and run gRPC e2e tests
-  #       run: |
-  #         make test-extra-backend-vllm
-  # tests-sglang-grpc is currently disabled in CI for the same reason as
-  # tests-vllm-grpc: sglang's CPU kernel (sgl-kernel) uses __m512 AVX-512
-  # intrinsics unconditionally in shm.cpp, so the from-source build
-  # requires `-march=sapphirerapids` (already set in install.sh) and the
-  # resulting binary SIGILLs at import on CPUs without AVX-512 VNNI/BF16.
-  # The ubuntu-latest runner pool does not guarantee that ISA baseline.
-  #
-  # The test itself (tests/e2e-backends + make test-extra-backend-sglang)
-  # is fully working and validated locally on a host with the right
-  # SIMD baseline. Run it manually with:
-  #
-  #   make test-extra-backend-sglang
-  #
-  # Re-enable this job once we have a self-hosted runner label with
-  # guaranteed AVX-512 VNNI/BF16 support.
-  #
-  # tests-sglang-grpc:
-  #   needs: detect-changes
-  #   if: needs.detect-changes.outputs.sglang == 'true' || needs.detect-changes.outputs.run-all == 'true'
-  #   runs-on: bigger-runner
-  #   timeout-minutes: 90
-  #   steps:
-  #     - name: Clone
-  #       uses: actions/checkout@v6
-  #       with:
-  #         submodules: true
-  #     - name: Dependencies
-  #       run: |
-  #         sudo apt-get update
-  #         sudo apt-get install -y --no-install-recommends \
-  #             make build-essential curl unzip ca-certificates git tar
-  #     - name: Setup Go
-  #       uses: actions/setup-go@v5
-  #       with:
-  #         go-version: '1.25.4'
-  #     - name: Free disk space
-  #       run: |
-  #         sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-  #         df -h
-  #     - name: Build sglang (cpu) backend image and run gRPC e2e tests
-  #       run: |
-  #         make test-extra-backend-sglang
   tests-acestep-cpp:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.acestep-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -811,152 +413,7 @@ jobs:
       - name: Test acestep-cpp
         run: |
           make --jobs=5 --output-sync=target -C backend/go/acestep-cpp test
-  tests-qwen3-tts-cpp:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.qwen3-tts-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
-      - name: Setup Go
-        uses: actions/setup-go@v5
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Build qwen3-tts-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/qwen3-tts-cpp
-      - name: Test qwen3-tts-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/qwen3-tts-cpp test
-  # Per-backend smoke for vibevoice-cpp: builds the .so + Go binary and
-  # runs `make -C backend/go/vibevoice-cpp test`. test.sh auto-downloads
-  # the published mudler/vibevoice.cpp-models bundle (TTS Q8_0 + ASR Q4_K
-  # + tokenizer + voice) and runs the closed-loop TTS → ASR Go test.
-  tests-vibevoice-cpp:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
-      - name: Setup Go
-        uses: actions/setup-go@v5
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Build vibevoice-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/vibevoice-cpp
-      - name: Test vibevoice-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/vibevoice-cpp test
-  # End-to-end TTS via the e2e-backends gRPC harness. Builds the
-  # vibevoice-cpp Docker image and drives Backend/TTS against it with a
-  # real LocalAI gRPC client.
-  tests-vibevoice-cpp-grpc-tts:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build vibevoice-cpp backend image and run TTS gRPC e2e tests
-        run: |
-          make test-extra-backend-vibevoice-cpp-tts
-  # End-to-end transcription via the e2e-backends gRPC harness. The
-  # vibevoice ASR is a 7B-param model (Q4_K weights ~10 GB on disk)
-  # and the JFK 30 s decode is too heavy for a free 4-core
-  # ubuntu-latest pool runner - two CI attempts got SIGTERM'd during
-  # LoadModel, before the test could even progress. Use the
-  # self-hosted 'bigger-runner' label (same one the GPU image builds
-  # in backend.yml use) and the documented dotnet/ghc/android cache
-  # purge to clear ~10-20 GB of headroom for the model + Docker
-  # image + working dir.
-  tests-vibevoice-cpp-grpc-transcription:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: bigger-runner
-    timeout-minutes: 150
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y --no-install-recommends \
-              make build-essential curl unzip ca-certificates git tar
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Free disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-          df -h
-      - name: Build vibevoice-cpp backend image and run ASR gRPC e2e tests
-        run: |
-          make test-extra-backend-vibevoice-cpp-transcription
-  # End-to-end audio transform via the e2e-backends gRPC harness. The
-  # LocalVQE GGUF is small (~5 MB) and the model is real-time on CPU, so
-  # the default ubuntu-latest pool is plenty.
-  tests-localvqe-grpc-transform:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.localvqe == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 60
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.25.4'
-      - name: Build localvqe backend image and run audio_transform gRPC e2e tests
-        run: |
-          make test-extra-backend-localvqe-transform
   tests-voxtral:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.voxtral == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
@@ -987,77 +444,3 @@ jobs:
       - name: Test voxtral
         run: |
           make --jobs=5 --output-sync=target -C backend/go/voxtral test
-  tests-kokoros:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.kokoros == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake pkg-config protobuf-compiler clang libclang-dev
-          sudo apt-get install -y espeak-ng libespeak-ng-dev libsonic-dev libpcaudio-dev libopus-dev libssl-dev
-          curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
-          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
-      - name: Build kokoros
-        run: |
-          make -C backend/rust/kokoros kokoros-grpc
-      - name: Test kokoros
-        run: |
-          make -C backend/rust/kokoros test
-  tests-insightface-grpc:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.insightface == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y --no-install-recommends \
-              make build-essential curl unzip ca-certificates git tar
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.26.0'
-      - name: Free disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-          df -h
-      - name: Build insightface backend image and run both model configurations
-        run: |
-          make test-extra-backend-insightface-all
-  tests-speaker-recognition-grpc:
-    needs: detect-changes
-    if: needs.detect-changes.outputs.speaker-recognition == 'true' || needs.detect-changes.outputs.run-all == 'true'
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y --no-install-recommends \
-              make build-essential curl ca-certificates git tar
-      - name: Setup Go
-        uses: actions/setup-go@v5
-        with:
-          go-version: '1.26.0'
-      - name: Free disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
-          df -h
-      - name: Build speaker-recognition backend image and run the ECAPA-TDNN configuration
-        run: |
-          make test-extra-backend-speaker-recognition-all

.github/workflows/test.yml

@@ -9,23 +9,70 @@ on:
     tags:
       - '*'
 
+env:
+  GRPC_VERSION: v1.65.0
+
 concurrency:
-  group: ci-tests-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   tests-linux:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        go-version: ['1.26.x']
+        go-version: ['1.25.x']
     steps:
+      - name: Free Disk Space (Ubuntu)
+        uses: jlumbroso/free-disk-space@main
+        with:
+          # this might remove tools that are actually needed,
+          # if set to "true" but frees about 6 GB
+          tool-cache: true
+          # all of these default to true, but feel free to set to
+          # "false" if necessary for your workflow
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          docker-images: true
+          swap-storage: true
+      - name: Release space from worker
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          df -h
       - name: Clone
         uses: actions/checkout@v6
         with:
           submodules: true
-      - name: Free disk space
-        uses: ./.github/actions/free-disk-space
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:
@@ -53,9 +100,73 @@ jobs:
           node-version: '22'
       - name: Build React UI
         run: make react-ui
+      - name: Build backends
+        run: |
+          make backends/transformers
+          mkdir external && mv backends/transformers external/transformers
+          make backends/llama-cpp backends/local-store backends/silero-vad backends/piper backends/whisper backends/stablediffusion-ggml
       - name: Test
         run: |
-          PATH="$PATH:/root/go/bin" make --jobs 5 --output-sync=target test
+          TRANSFORMER_BACKEND=$PWD/external/transformers/run.sh PATH="$PATH:/root/go/bin" GO_TAGS="tts" make --jobs 5 --output-sync=target test
+      - name: Setup tmate session if tests fail
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3.23
+        with:
+          detached: true
+          connect-timeout-seconds: 180
+          limit-access-to-actor: true
+
+  tests-e2e-container:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Release space from worker
+        run: |
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          df -h
+          echo
+          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
+          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
+          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
+          sudo rm -rf /usr/local/lib/android
+          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
+          sudo rm -rf /usr/share/dotnet
+          sudo apt-get remove -y '^mono-.*' || true
+          sudo apt-get remove -y '^ghc-.*' || true
+          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
+          sudo apt-get remove -y 'php.*' || true
+          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
+          sudo apt-get remove -y '^google-.*' || true
+          sudo apt-get remove -y azure-cli || true
+          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
+          sudo apt-get remove -y '^gfortran-.*' || true
+          sudo apt-get autoremove -y
+          sudo apt-get clean
+          echo
+          echo "Listing top largest packages"
+          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
+          head -n 30 <<< "${pkgs}"
+          echo
+          sudo rm -rfv build || true
+          df -h
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Dependencies
+        run: |
+          # Install protoc
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Test
+        run: |
+            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
       - name: Setup tmate session if tests fail
         if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
@@ -68,7 +179,7 @@ jobs:
     runs-on: macos-latest
     strategy:
       matrix:
-        go-version: ['1.26.x']
+        go-version: ['1.25.x']
     steps:
       - name: Clone
         uses: actions/checkout@v6
@@ -84,7 +195,7 @@ jobs:
         run: go version
       - name: Dependencies
         run: |
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus ffmpeg
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus
           pip install --user --no-cache-dir grpcio-tools grpcio
       - name: Setup Node.js
         uses: actions/setup-node@v6
@@ -92,6 +203,10 @@ jobs:
           node-version: '22'
       - name: Build React UI
         run: make react-ui
+      - name: Build llama-cpp-darwin
+        run: |
+          make protogen-go
+          make backends/llama-cpp-darwin
       - name: Test
         run: |
           export C_INCLUDE_PATH=/usr/local/include

.github/workflows/tests-aio.yml

@@ -1,86 +0,0 @@
----
-name: 'tests-aio'
-
-# Runs the all-in-one (AIO) Docker image with real backends + real models.
-# Heavy: builds llama-cpp/whisper/piper/silero-vad/stablediffusion-ggml/local-store
-# and exercises end-to-end inference inside the container. Moved out of test.yml
-# (which used to run on every PR) so PR CI no longer pays this cost.
-#
-# Triggers:
-#   - schedule (nightly @ 04:00 UTC) — catches packaging/image regressions within 24h
-#   - workflow_dispatch — manual run on-demand
-#   - push to master/tags — sanity check after merge / before release
-
-on:
-  schedule:
-    - cron: '0 4 * * *'
-  workflow_dispatch:
-  push:
-    branches:
-      - master
-    tags:
-      - '*'
-
-concurrency:
-  group: ci-tests-aio-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  tests-aio:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Release space from worker
-        run: |
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          df -h
-          echo
-          sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
-          sudo apt-get remove --auto-remove android-sdk-platform-tools || true
-          sudo apt-get purge --auto-remove android-sdk-platform-tools || true
-          sudo rm -rf /usr/local/lib/android
-          sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
-          sudo rm -rf /usr/share/dotnet
-          sudo apt-get remove -y '^mono-.*' || true
-          sudo apt-get remove -y '^ghc-.*' || true
-          sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
-          sudo apt-get remove -y 'php.*' || true
-          sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
-          sudo apt-get remove -y '^google-.*' || true
-          sudo apt-get remove -y azure-cli || true
-          sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
-          sudo apt-get remove -y '^gfortran-.*' || true
-          sudo apt-get autoremove -y
-          sudo apt-get clean
-          echo
-          echo "Listing top largest packages"
-          pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
-          head -n 30 <<< "${pkgs}"
-          echo
-          sudo rm -rfv build || true
-          df -h
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Test
-        run: |
-            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
-      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
-        uses: mxschmitt/action-tmate@v3.23
-        with:
-          detached: true
-          connect-timeout-seconds: 180
-          limit-access-to-actor: true

.github/workflows/tests-e2e.yml

@@ -10,8 +10,8 @@ on:
       - '*'
 
 concurrency:
-  group: ci-tests-e2e-backend-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-e2e-backend-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   tests-e2e-backend:
@@ -24,8 +24,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/tests-ui-e2e.yml

@@ -12,8 +12,8 @@ on:
       - master
 
 concurrency:
-  group: ci-tests-ui-e2e-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: ci-tests-ui-e2e-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  cancel-in-progress: true
 
 jobs:
   tests-ui-e2e:
@@ -26,8 +26,6 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/update_swagger.yaml

@@ -11,8 +11,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
       - uses: actions/setup-go@v5
         with:
           go-version: 'stable'

.gitmodules

@@ -1,6 +1,3 @@
 [submodule "docs/themes/hugo-theme-relearn"]
 	path = docs/themes/hugo-theme-relearn
 	url = https://github.com/McShelby/hugo-theme-relearn.git
-[submodule "backend/rust/kokoros/sources/Kokoros"]
-	path = backend/rust/kokoros/sources/Kokoros
-	url = https://github.com/lucasjinreal/Kokoros

.golangci.yml

@@ -1,53 +0,0 @@
-version: "2"
-
-# Only issues introduced relative to master are reported. Pre-existing issues
-# in the codebase do not fail the lint job; they're treated as a baseline that
-# can be cleaned up incrementally. New code (added lines on a branch) is held
-# to the full linter set. Locally, `make lint-all` overrides this and reports
-# every issue.
-issues:
-  # origin/master because in shallow CI checkouts only the remote-tracking
-  # branch exists; a bare 'master' ref isn't reachable locally.
-  new-from-merge-base: origin/master
-
-linters:
-  default: standard
-  # staticcheck is noisy on this codebase (mostly QF style suggestions like
-  # "could use tagged switch" or "unnecessary fmt.Sprintf"). Re-enable
-  # selectively if a high-signal subset is identified.
-  disable:
-    - staticcheck
-  enable:
-    - forbidigo
-  settings:
-    forbidigo:
-      forbid:
-        - pattern: '^t\.Errorf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(...) instead of t.Errorf. See .agents/coding-style.md.'
-        - pattern: '^t\.Error$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(...) instead of t.Error. See .agents/coding-style.md.'
-        - pattern: '^t\.Fatalf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(Succeed()) / Fail(...) instead of t.Fatalf. See .agents/coding-style.md.'
-        - pattern: '^t\.Fatal$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Expect(...).To(Succeed()) / Fail(...) instead of t.Fatal. See .agents/coding-style.md.'
-        - pattern: '^t\.Run$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Describe/Context/It instead of t.Run. See .agents/coding-style.md.'
-        - pattern: '^t\.Skip$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Skip(...) instead of t.Skip. See .agents/coding-style.md.'
-        - pattern: '^t\.Skipf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Skip(...) instead of t.Skipf. See .agents/coding-style.md.'
-        - pattern: '^t\.SkipNow$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Skip(...) instead of t.SkipNow. See .agents/coding-style.md.'
-        - pattern: '^t\.Logf$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use GinkgoWriter / fmt.Fprintf(GinkgoWriter, ...) instead of t.Logf. See .agents/coding-style.md.'
-        - pattern: '^t\.Log$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use GinkgoWriter / fmt.Fprintln(GinkgoWriter, ...) instead of t.Log. See .agents/coding-style.md.'
-        - pattern: '^t\.Fail$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Fail(...) instead of t.Fail. See .agents/coding-style.md.'
-        - pattern: '^t\.FailNow$'
-          msg: 'LocalAI tests must use Ginkgo/Gomega; use Fail(...) instead of t.FailNow. See .agents/coding-style.md.'
-  exclusions:
-    paths:
-      # Upstream whisper.cpp source tree fetched by the whisper backend Makefile.
-      - 'backend/go/whisper/sources'
-      - 'docs/'

AGENTS.md

@@ -1,36 +1,18 @@
 # LocalAI Agent Instructions
 
-This file is the entry point for AI coding assistants (Claude Code, Cursor, Copilot, Codex, Aider, etc.) working on LocalAI. It is an index to detailed topic guides in the `.agents/` directory. Read the relevant file(s) for the task at hand — you don't need to load all of them.
-
-Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development workflow.
-
-## Policy for AI-Assisted Contributions
-
-LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:
-
-- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
-- **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
-- **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
-- **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.
+This file is an index to detailed topic guides in the `.agents/` directory. Read the relevant file(s) for the task at hand — you don't need to load all of them.
 
 ## Topics
 
 | File | When to read |
 |------|-------------|
-| [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md) | Policy for AI-assisted contributions — licensing, DCO, attribution |
 | [.agents/building-and-testing.md](.agents/building-and-testing.md) | Building the project, running tests, Docker builds for specific platforms |
-| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache, per-arch keys), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, prebuilt `base-grpc-*` images for llama.cpp variants, per-arch native + manifest-merge pattern, `setup-build-disk` `/mnt` relocation, path filter on master push, manual eviction |
-| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist, including importer integration (the `/import-model` dropdown is server-driven from `GET /backends/known`) |
+| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist |
 | [.agents/coding-style.md](.agents/coding-style.md) | Code style, editorconfig, logging, documentation conventions |
 | [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
-| [.agents/vllm-backend.md](.agents/vllm-backend.md) | Working on the vLLM / vLLM-omni backends — native parsers, ChatDelta, CPU build, libnuma packaging, backend hooks |
-| [.agents/sglang-backend.md](.agents/sglang-backend.md) | Working on the SGLang backend — `engine_args` validation against ServerArgs, speculative-decoding (EAGLE/EAGLE3/DFLASH/MTP) recipes, parser handling |
-| [.agents/ds4-backend.md](.agents/ds4-backend.md) | Working on the ds4 backend - DSML state machine, thinking modes, KV cache, Metal+CUDA matrix |
 | [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
 | [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
 | [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |
-| [.agents/adding-gallery-models.md](.agents/adding-gallery-models.md) | Adding GGUF models from HuggingFace to the model gallery |
-| [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) | LocalAI Assistant chat modality — adding admin tools to the in-process MCP server, editing skill prompts, keeping REST + MCP + skills in sync |
 
 ## Quick Reference
 
@@ -38,7 +20,5 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 - **Go style**: Prefer `any` over `interface{}`
 - **Comments**: Explain *why*, not *what*
 - **Docs**: Update `docs/content/` when adding features or changing config
-- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
-- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
 - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
 - **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI

CONTRIBUTING.md

@@ -13,7 +13,6 @@ Thank you for your interest in contributing to LocalAI! We appreciate your time
   - [Development Workflow](#development-workflow)
   - [Creating a Pull Request (PR)](#creating-a-pull-request-pr)
 - [Coding Guidelines](#coding-guidelines)
-- [AI Coding Assistants](#ai-coding-assistants)
 - [Testing](#testing)
 - [Documentation](#documentation)
 - [Community and Communication](#community-and-communication)
@@ -186,7 +185,7 @@ Before jumping into a PR for a massive feature or big change, it is preferred to
 
 This project uses an [`.editorconfig`](.editorconfig) file to define formatting standards (indentation, line endings, charset, etc.). Please configure your editor to respect it.
 
-For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`CLAUDE.md`](CLAUDE.md) symlink) for agent-specific guidelines including build instructions and backend architecture details. Contributions produced with AI assistance must follow the rules in the [AI Coding Assistants](#ai-coding-assistants) section below.
+For AI-assisted development, see [`CLAUDE.md`](CLAUDE.md) for agent-specific guidelines including build instructions and backend architecture details.
 
 ### General Principles
 
@@ -212,26 +211,6 @@ For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`C
 - Reviewers will check for correctness, test coverage, adherence to these guidelines, and clarity of intent.
 - Be responsive to review feedback and keep discussions constructive.
 
-## AI Coding Assistants
-
-LocalAI follows the **same guidelines as the Linux kernel project** for AI-assisted contributions: <https://docs.kernel.org/process/coding-assistants.html>.
-
-The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:
-
-- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
-- **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
-- **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:
-
-  ```
-  Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
-  ```
-
-  Example: `Assisted-by: Claude:claude-opus-4-7 golangci-lint`
-
-  Basic development tools (git, go, make, editors) should not be listed.
-- **The human submitter is responsible** for reviewing, testing, and fully understanding every line of AI-generated code — including verifying that any referenced APIs, flags, or file paths actually exist in the tree.
-- Contributions must remain compatible with LocalAI's **MIT License**.
-
 ## Testing
 
 All new features and bug fixes should include test coverage. The project uses [Ginkgo](https://onsi.github.io/ginkgo/) as its test framework.

Dockerfile

@@ -1,20 +1,13 @@
 ARG BASE_IMAGE=ubuntu:24.04
+ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 ARG INTEL_BASE_IMAGE=${BASE_IMAGE}
 ARG UBUNTU_CODENAME=noble
-# Optional alternate Ubuntu apt mirror(s). Empty = use upstream.
-# See .docker/apt-mirror.sh for accepted values.
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS requirements
 
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 ENV DEBIAN_FRONTEND=noninteractive
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates curl wget espeak-ng libgomp1 \
         ffmpeg libopenblas0 libopenblas-dev libopus0 sox && \
@@ -156,7 +149,6 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
-            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \
@@ -184,7 +176,7 @@ ENV PATH=/opt/rocm/bin:${PATH}
 # The requirements-core target is common to all images.  It should not be placed in requirements-core unless every single build will use it.
 FROM requirements-drivers AS build-requirements
 
-ARG GO_VERSION=1.26.0
+ARG GO_VERSION=1.25.4
 ARG CMAKE_VERSION=3.31.10
 ARG CMAKE_FROM_SOURCE=false
 ARG TARGETARCH
@@ -248,14 +240,10 @@ WORKDIR /build
 # This is a temporary workaround until Intel fixes their repository
 FROM ${INTEL_BASE_IMAGE} AS intel
 ARG UBUNTU_CODENAME=noble
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 RUN wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
 gpg --yes --dearmor --output /usr/share/keyrings/intel-graphics.gpg
 RUN echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu ${UBUNTU_CODENAME}/lts/2350 unified" > /etc/apt/sources.list.d/intel-graphics.list
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         intel-oneapi-runtime-libs && \
     apt-get clean && \
@@ -305,7 +293,7 @@ EOT
 ###################################
 
 # Build React UI
-FROM node:26-slim AS react-ui-builder
+FROM node:25-slim AS react-ui-builder
 WORKDIR /app
 COPY core/http/react-ui/package*.json ./
 RUN npm install
@@ -331,6 +319,7 @@ COPY ./.git ./.git
 # Some of the Go backends use libs from the main src, we could further optimize the caching by building the CPP backends before here
 COPY ./pkg/grpc ./pkg/grpc
 COPY ./pkg/utils ./pkg/utils
+COPY ./pkg/langchain ./pkg/langchain
 
 RUN ls -l ./
 RUN make protogen-go

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -10,13 +10,6 @@ LAUNCHER_BINARY_NAME=local-ai-launcher
 UBUNTU_VERSION?=2404
 UBUNTU_CODENAME?=noble
 
-# Optional Ubuntu apt mirror overrides forwarded to docker builds.
-# Empty = use upstream archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com.
-# Set e.g. APT_MIRROR=http://azure.archive.ubuntu.com to route apt traffic
-# during outages of the default Ubuntu pool.
-APT_MIRROR?=
-APT_PORTS_MIRROR?=
-
 GORELEASER?=
 
 export BUILD_TYPE?=
@@ -72,7 +65,7 @@ endif
 TEST_PATHS?=./api/... ./pkg/... ./core/...
 
 
-.PHONY: all test build vendor lint lint-all
+.PHONY: all test build vendor
 
 all: help
 
@@ -92,7 +85,6 @@ clean: ## Remove build related file
 clean-tests:
 	rm -rf test-models
 	rm -rf test-dir
-	rm -f tests/e2e/mock-backend/mock-backend
 
 ## Install Go tools
 install-go-tools:
@@ -115,7 +107,7 @@ core/http/react-ui/dist: react-ui
 
 ## Build:
 
-build: protogen-go generate install-go-tools core/http/react-ui/dist ## Build the project
+build: protogen-go install-go-tools core/http/react-ui/dist ## Build the project
 	$(info ${GREEN}I local-ai build info:${RESET})
 	$(info ${GREEN}I BUILD_TYPE: ${YELLOW}$(BUILD_TYPE)${RESET})
 	$(info ${GREEN}I GO_TAGS: ${YELLOW}$(GO_TAGS)${RESET})
@@ -151,56 +143,33 @@ osx-signed: build
 run: ## run local-ai
 	CGO_LDFLAGS="$(CGO_LDFLAGS)" $(GOCMD) run ./
 
-prepare-test: protogen-go build-mock-backend
+test-models/testmodel.ggml:
+	mkdir -p test-models
+	mkdir -p test-dir
+	wget -q https://huggingface.co/mradermacher/gpt2-alpaca-gpt4-GGUF/resolve/main/gpt2-alpaca-gpt4.Q4_K_M.gguf -O test-models/testmodel.ggml
+	wget -q https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin -O test-models/whisper-en
+	wget -q https://huggingface.co/mudler/all-MiniLM-L6-v2/resolve/main/ggml-model-q4_0.bin -O test-models/bert
+	wget -q https://cdn.openai.com/whisper/draft-20220913a/micro-machines.wav -O test-dir/audio.wav
+	cp tests/models_fixtures/* test-models
+
+prepare-test: protogen-go
+	cp tests/models_fixtures/* test-models
 
 ########################################################
 ## Tests
 ########################################################
 
 ## Test targets
-## After the test-suite reorg (see plans/test-reorg) the default `make test`
-## no longer downloads multi-GB GGUF/whisper fixtures or builds llama-cpp /
-## transformers / piper / whisper / stablediffusion-ggml. core/http/app_test.go
-## now drives the mock-backend binary built by build-mock-backend; real-backend
-## inference moved into tests/e2e-backends/ (per-backend, path-filtered) and
-## tests/e2e-aio/ (nightly).
-test: prepare-test
+test: test-models/testmodel.ggml protogen-go
 	@echo 'Running tests'
 	export GO_TAGS="debug"
+	$(MAKE) prepare-test
 	OPUS_SHIM_LIBRARY=$(abspath ./pkg/opus/shim/libopusshim.so) \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) --fail-fast -v -r $(TEST_PATHS)
-
-########################################################
-## Lint
-########################################################
-## Runs golangci-lint with config from .golangci.yml. Includes the standard
-## linter set plus forbidigo, which enforces the Ginkgo/Gomega-only test
-## convention documented in .agents/coding-style.md.
-##
-## LINT_EXCLUDE_DIRS_RE matches directories whose Go packages can't typecheck
-## without C/C++ headers we don't install in the lint runner (cgo wrappers
-## around llama.cpp, piper/spdlog, silero-vad/onnxruntime, and Fyne/OpenGL for
-## the launcher). Their compile-time correctness is enforced by their own
-## build pipelines. Keep this as a deny list — `go list ./...` discovers
-## everything else automatically, so new packages are scanned by default.
-LINT_EXCLUDE_DIRS_RE=/(backend/go/(piper|silero-vad|llm)|cmd/launcher)(/|$$)
-
-lint:
-	@command -v golangci-lint >/dev/null 2>&1 || { \
-		echo 'golangci-lint not installed. Install: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest'; \
-		exit 1; \
-	}
-	golangci-lint run $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
-
-## Like `lint` but reports every issue, including the pre-existing baseline
-## that `lint` ignores via .golangci.yml's new-from-merge-base. Use this to
-## see what's available to clean up.
-lint-all:
-	@command -v golangci-lint >/dev/null 2>&1 || { \
-		echo 'golangci-lint not installed. Install: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest'; \
-		exit 1; \
-	}
-	golangci-lint run --new=false --new-from-merge-base= --new-from-rev= $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
+	HUGGINGFACE_GRPC=$(abspath ./)/backend/python/transformers/run.sh TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="!llama-gguf"  --flake-attempts $(TEST_FLAKES) --fail-fast -v -r $(TEST_PATHS)
+	$(MAKE) test-llama-gguf
+	$(MAKE) test-tts
+	$(MAKE) test-stablediffusion
 
 ########################################################
 ## E2E AIO tests (uses standard image with pre-configured models)
@@ -216,8 +185,6 @@ docker-build-e2e:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		-t local-ai:tests -f Dockerfile .
 
@@ -232,20 +199,6 @@ run-e2e-aio: protogen-go
 	@echo 'Running e2e AIO tests'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e-aio
 
-# vLLM multi-node DP smoke (CPU). Builds local-ai:tests and the
-# cpu-vllm backend from the current working tree, then drives a
-# head + headless follower via testcontainers-go and asserts a chat
-# completion. BuildKit caches both images, so re-runs only rebuild
-# what changed. The test lives under tests/e2e/distributed and is
-# selected by the VLLMMultinode label so it doesn't run alongside
-# the other distributed-suite tests by default.
-test-e2e-vllm-multinode: docker-build-e2e extract-backend-vllm protogen-go
-	@echo 'Running e2e vLLM multi-node DP test'
-	LOCALAI_IMAGE=local-ai \
-	LOCALAI_IMAGE_TAG=tests \
-	LOCALAI_VLLM_BACKEND_DIR=$(abspath ./local-backends/vllm) \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='VLLMMultinode' -v -r ./tests/e2e/distributed
-
 ########################################################
 ## E2E tests
 ########################################################
@@ -259,8 +212,6 @@ prepare-e2e:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
 		-t localai-tests .
@@ -285,12 +236,20 @@ teardown-e2e:
 ## Integration and unit tests
 ########################################################
 
-## Storage / vector-store integration. Requires the local-store backend to
-## be available — we build it on demand and pass its location via
-## BACKENDS_PATH (the model loader looks there for the gRPC binary).
-test-stores: backends/local-store
-	BACKENDS_PATH=$(abspath ./)/backends \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r tests/integration
+test-llama-gguf: prepare-test
+	TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="llama-gguf" --flake-attempts $(TEST_FLAKES) -v -r $(TEST_PATHS)
+
+test-tts: prepare-test
+	TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="tts" --flake-attempts $(TEST_FLAKES) -v -r $(TEST_PATHS)
+
+test-stablediffusion: prepare-test
+	TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="stablediffusion" --flake-attempts $(TEST_FLAKES) -v -r $(TEST_PATHS)
+
+test-stores:
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="stores" --flake-attempts $(TEST_FLAKES) -v -r tests/integration
 
 test-opus:
 	@echo 'Running opus backend tests'
@@ -302,8 +261,6 @@ test-opus-docker:
 	docker build --target builder \
 	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),) \
 	  --build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
-	  --build-arg APT_MIRROR=$(APT_MIRROR) \
-	  --build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 	  --build-arg BACKEND=opus \
 	  -t localai-opus-test -f backend/Dockerfile.golang .
 	docker run --rm localai-opus-test \
@@ -313,13 +270,23 @@ test-realtime: build-mock-backend
 	@echo 'Running realtime e2e tests (mock backend)'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime && !real-models" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
 
-# Container-based real-model realtime testing. Build env vars / pipeline
-# definition kept here so test-realtime-models-docker can drive a fully wired
-# pipeline (VAD + STT + LLM + TTS) from inside a containerised runner.
+# Real-model realtime tests. Set REALTIME_TEST_MODEL to use your own pipeline,
+# or leave unset to auto-build one from the component env vars below.
 REALTIME_VAD?=silero-vad-ggml
 REALTIME_STT?=whisper-1
 REALTIME_LLM?=qwen3-0.6b
 REALTIME_TTS?=tts-1
+REALTIME_BACKENDS_PATH?=$(abspath ./)/backends
+
+test-realtime-models: build-mock-backend
+	@echo 'Running realtime e2e tests (real models)'
+	REALTIME_TEST_MODEL=$${REALTIME_TEST_MODEL:-realtime-test-pipeline} \
+	REALTIME_VAD=$(REALTIME_VAD) \
+	REALTIME_STT=$(REALTIME_STT) \
+	REALTIME_LLM=$(REALTIME_LLM) \
+	REALTIME_TTS=$(REALTIME_TTS) \
+	REALTIME_BACKENDS_PATH=$(REALTIME_BACKENDS_PATH) \
+	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
 
 # --- Container-based real-model testing ---
 
@@ -333,7 +300,7 @@ local-backends:
 
 extract-backend-%: docker-build-% local-backends
 	@echo "Extracting backend $*..."
-	@CID=$$(docker create --entrypoint=/run.sh local-ai-backend:$*) && \
+	@CID=$$(docker create local-ai-backend:$*) && \
 	  rm -rf local-backends/$* && mkdir -p local-backends/$* && \
 	  docker cp $$CID:/ - | tar -xf - -C local-backends/$* && \
 	  docker rm $$CID > /dev/null
@@ -345,8 +312,6 @@ test-realtime-models-docker: build-mock-backend
 	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),cublas) \
 	  --build-arg CUDA_MAJOR_VERSION=$(or $(CUDA_MAJOR_VERSION),13) \
 	  --build-arg CUDA_MINOR_VERSION=$(or $(CUDA_MINOR_VERSION),0) \
-	  --build-arg APT_MIRROR=$(APT_MIRROR) \
-	  --build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 	  -t localai-test-runner .
 	docker run --rm \
 	  $(REALTIME_DOCKER_FLAGS) \
@@ -430,25 +395,9 @@ protoc:
 .PHONY: protogen-go
 protogen-go: protoc install-go-tools
 	mkdir -p pkg/grpc/proto
-	# install-go-tools writes protoc-gen-go and protoc-gen-go-grpc into
-	# $(shell go env GOPATH)/bin, which isn't on every dev's PATH. protoc
-	# resolves its code-gen plugins via PATH, so without this prefix the
-	# generate step fails with "protoc-gen-go: program not found". Prepend
-	# GOPATH/bin so the freshly-installed plugins win without requiring a
-	# shell-profile change.
-	PATH="$$(go env GOPATH)/bin:$$PATH" ./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
+	./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
     backend/backend.proto
 
-core/config/inference_defaults.json: ## Fetch inference defaults from unsloth (only if missing)
-	$(GOCMD) generate ./core/config/...
-
-.PHONY: generate
-generate: core/config/inference_defaults.json ## Ensure inference defaults exist
-
-.PHONY: generate-force
-generate-force: ## Re-fetch inference defaults from unsloth (always)
-	$(GOCMD) generate ./core/config/...
-
 .PHONY: protogen-go-clean
 protogen-go-clean:
 	$(RM) pkg/grpc/proto/backend.pb.go pkg/grpc/proto/backend_grpc.pb.go
@@ -461,9 +410,7 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/chatterbox
 	$(MAKE) -C backend/python/vllm
 	$(MAKE) -C backend/python/vllm-omni
-	$(MAKE) -C backend/python/sglang
 	$(MAKE) -C backend/python/vibevoice
-	$(MAKE) -C backend/python/liquid-audio
 	$(MAKE) -C backend/python/moonshine
 	$(MAKE) -C backend/python/pocket-tts
 	$(MAKE) -C backend/python/qwen-tts
@@ -472,14 +419,9 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/qwen-asr
 	$(MAKE) -C backend/python/nemo
 	$(MAKE) -C backend/python/voxcpm
-	$(MAKE) -C backend/python/faster-whisper
 	$(MAKE) -C backend/python/whisperx
 	$(MAKE) -C backend/python/ace-step
 	$(MAKE) -C backend/python/trl
-	$(MAKE) -C backend/python/tinygrad
-	$(MAKE) -C backend/python/insightface
-	$(MAKE) -C backend/python/speaker-recognition
-	$(MAKE) -C backend/rust/kokoros kokoros-grpc
 
 test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/transformers test
@@ -489,7 +431,6 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/vllm test
 	$(MAKE) -C backend/python/vllm-omni test
 	$(MAKE) -C backend/python/vibevoice test
-	$(MAKE) -C backend/python/liquid-audio test
 	$(MAKE) -C backend/python/moonshine test
 	$(MAKE) -C backend/python/pocket-tts test
 	$(MAKE) -C backend/python/qwen-tts test
@@ -498,455 +439,9 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/qwen-asr test
 	$(MAKE) -C backend/python/nemo test
 	$(MAKE) -C backend/python/voxcpm test
-	$(MAKE) -C backend/python/faster-whisper test
 	$(MAKE) -C backend/python/whisperx test
 	$(MAKE) -C backend/python/ace-step test
 	$(MAKE) -C backend/python/trl test
-	$(MAKE) -C backend/python/tinygrad test
-	$(MAKE) -C backend/python/insightface test
-	$(MAKE) -C backend/python/speaker-recognition test
-	$(MAKE) -C backend/rust/kokoros test
-
-##
-## End-to-end gRPC tests that exercise a built backend container image.
-##
-## The test suite in tests/e2e-backends is backend-agnostic. You drive it via env
-## vars (see tests/e2e-backends/backend_test.go for the full list) and the
-## capability-driven harness picks which gRPC RPCs to exercise:
-##
-##   BACKEND_IMAGE            Required. Docker image to test, e.g. local-ai-backend:llama-cpp.
-##   BACKEND_TEST_MODEL_URL   URL of a model file to download and load.
-##   BACKEND_TEST_MODEL_FILE  Path to an already-downloaded model (skips download).
-##   BACKEND_TEST_MODEL_NAME  HuggingFace repo id (e.g. Qwen/Qwen2.5-0.5B-Instruct).
-##                            Use this instead of MODEL_URL for backends that
-##                            resolve HF model ids natively (vllm, vllm-omni).
-##   BACKEND_TEST_CAPS        Comma-separated capabilities, default "health,load,predict,stream".
-##                            Adds "tools" to exercise ChatDelta tool call extraction.
-##   BACKEND_TEST_PROMPT      Override the prompt used in predict/stream specs.
-##   BACKEND_TEST_OPTIONS     Comma-separated Options[] entries forwarded to LoadModel,
-##                            e.g. "tool_parser:hermes,reasoning_parser:qwen3".
-##
-## Direct usage (image already built, no docker-build-* dependency):
-##
-##   make test-extra-backend BACKEND_IMAGE=local-ai-backend:llama-cpp \
-##       BACKEND_TEST_MODEL_URL=https://.../model.gguf
-##
-## Convenience wrappers below build a specific backend image first, then run the
-## suite against it.
-##
-BACKEND_TEST_MODEL_URL?=https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q8_0.gguf
-
-## Generic target — runs the suite against whatever BACKEND_IMAGE points at.
-## Depends on protogen-go so pkg/grpc/proto is generated before `go test`.
-test-extra-backend: protogen-go
-	@test -n "$$BACKEND_IMAGE" || { echo "BACKEND_IMAGE must be set" >&2; exit 1; }
-	BACKEND_IMAGE="$$BACKEND_IMAGE" \
-	BACKEND_TEST_MODEL_URL="$${BACKEND_TEST_MODEL_URL:-$(BACKEND_TEST_MODEL_URL)}" \
-	BACKEND_TEST_MODEL_FILE="$$BACKEND_TEST_MODEL_FILE" \
-	BACKEND_TEST_MODEL_NAME="$$BACKEND_TEST_MODEL_NAME" \
-	BACKEND_TEST_MMPROJ_URL="$$BACKEND_TEST_MMPROJ_URL" \
-	BACKEND_TEST_MMPROJ_FILE="$$BACKEND_TEST_MMPROJ_FILE" \
-	BACKEND_TEST_AUDIO_URL="$$BACKEND_TEST_AUDIO_URL" \
-	BACKEND_TEST_AUDIO_FILE="$$BACKEND_TEST_AUDIO_FILE" \
-	BACKEND_TEST_CAPS="$$BACKEND_TEST_CAPS" \
-	BACKEND_TEST_PROMPT="$$BACKEND_TEST_PROMPT" \
-	BACKEND_TEST_OPTIONS="$$BACKEND_TEST_OPTIONS" \
-	BACKEND_TEST_TOOL_PROMPT="$$BACKEND_TEST_TOOL_PROMPT" \
-	BACKEND_TEST_TOOL_NAME="$$BACKEND_TEST_TOOL_NAME" \
-	BACKEND_TEST_CACHE_TYPE_K="$$BACKEND_TEST_CACHE_TYPE_K" \
-	BACKEND_TEST_CACHE_TYPE_V="$$BACKEND_TEST_CACHE_TYPE_V" \
-	BACKEND_TEST_FACE_IMAGE_1_URL="$$BACKEND_TEST_FACE_IMAGE_1_URL" \
-	BACKEND_TEST_FACE_IMAGE_1_FILE="$$BACKEND_TEST_FACE_IMAGE_1_FILE" \
-	BACKEND_TEST_FACE_IMAGE_2_URL="$$BACKEND_TEST_FACE_IMAGE_2_URL" \
-	BACKEND_TEST_FACE_IMAGE_2_FILE="$$BACKEND_TEST_FACE_IMAGE_2_FILE" \
-	BACKEND_TEST_FACE_IMAGE_3_URL="$$BACKEND_TEST_FACE_IMAGE_3_URL" \
-	BACKEND_TEST_FACE_IMAGE_3_FILE="$$BACKEND_TEST_FACE_IMAGE_3_FILE" \
-	BACKEND_TEST_VERIFY_DISTANCE_CEILING="$$BACKEND_TEST_VERIFY_DISTANCE_CEILING" \
-	go test -v -timeout 30m ./tests/e2e-backends/...
-
-## Convenience wrappers: build the image, then exercise it.
-test-extra-backend-llama-cpp: docker-build-llama-cpp
-	BACKEND_IMAGE=local-ai-backend:llama-cpp \
-	BACKEND_TEST_CAPS=health,load,predict,stream,logprobs,logit_bias \
-	$(MAKE) test-extra-backend
-
-test-extra-backend-ik-llama-cpp: docker-build-ik-llama-cpp
-	BACKEND_IMAGE=local-ai-backend:ik-llama-cpp $(MAKE) test-extra-backend
-
-## turboquant: exercises the llama.cpp-fork backend with the fork's
-## *TurboQuant-specific* KV-cache types (turbo3 for both K and V). turbo3
-## is what makes this backend distinct from stock llama-cpp — picking q8_0
-## here would only test the standard llama.cpp code path that the upstream
-## llama-cpp backend already covers. The fork auto-enables flash_attention
-## when turbo3/turbo4 are active, so we don't need to set it explicitly.
-test-extra-backend-turboquant: docker-build-turboquant
-	BACKEND_IMAGE=local-ai-backend:turboquant \
-	BACKEND_TEST_CACHE_TYPE_K=q8_0 \
-	BACKEND_TEST_CACHE_TYPE_V=turbo3 \
-	$(MAKE) test-extra-backend
-
-## Audio transcription wrapper for the llama-cpp backend.
-## Drives the new AudioTranscription / AudioTranscriptionStream RPCs against
-## ggml-org/Qwen3-ASR-0.6B-GGUF (a small ASR model that requires its mmproj
-## audio encoder companion). The audio fixture is a short public-domain
-## "jfk.wav" clip ggml-org bundles with whisper.cpp's CI assets.
-test-extra-backend-llama-cpp-transcription: docker-build-llama-cpp
-	BACKEND_IMAGE=local-ai-backend:llama-cpp \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main/Qwen3-ASR-0.6B-Q8_0.gguf \
-	BACKEND_TEST_MMPROJ_URL=https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	BACKEND_TEST_CTX_SIZE=2048 \
-	$(MAKE) test-extra-backend
-
-## vllm is resolved from a HuggingFace model id (no file download) and
-## exercises Predict + streaming + tool-call extraction via the hermes parser.
-## Requires a host CPU with the SIMD instructions the prebuilt vllm CPU
-## wheel was compiled against (AVX-512 VNNI/BF16); older CPUs will SIGILL
-## on import — on CI this means using the bigger-runner label.
-test-extra-backend-vllm: docker-build-vllm
-	BACKEND_IMAGE=local-ai-backend:vllm \
-	BACKEND_TEST_MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct \
-	BACKEND_TEST_CAPS=health,load,predict,stream,tools \
-	BACKEND_TEST_OPTIONS=tool_parser:hermes \
-	$(MAKE) test-extra-backend
-
-## vllm multi-node data-parallel smoke test. Runs LocalAI head + a
-## `local-ai p2p-worker vllm` follower in docker compose against
-## Qwen2.5-0.5B with data_parallel_size=2. Requires 2 NVIDIA GPUs and
-## nvidia-container-runtime on the host — vLLM v1's DP coordinator is
-## not viable on CPU so this cannot run in CI without GPU.
-test-extra-backend-vllm-multinode:
-	./tests/e2e/vllm-multinode/smoke.sh
-
-## tinygrad mirrors the vllm target (same model, same caps, same parser) so
-## the two backends are directly comparable. The LLM path covers Predict,
-## streaming and native tool-call extraction. Companion targets below cover
-## embeddings, Stable Diffusion and Whisper — run them individually or via
-## the `test-extra-backend-tinygrad-all` aggregate.
-test-extra-backend-tinygrad: docker-build-tinygrad
-	BACKEND_IMAGE=local-ai-backend:tinygrad \
-	BACKEND_TEST_MODEL_NAME=Qwen/Qwen3-0.6B \
-	BACKEND_TEST_CAPS=health,load,predict,stream,tools \
-	BACKEND_TEST_OPTIONS=tool_parser:hermes \
-	$(MAKE) test-extra-backend
-
-## tinygrad — embeddings via LLM last-hidden-state pooling. Reuses the same
-## Qwen3-0.6B as the chat target so we don't need a separate BERT vendor;
-## the Embedding RPC mean-pools and L2-normalizes the last-layer hidden
-## state.
-test-extra-backend-tinygrad-embeddings: docker-build-tinygrad
-	BACKEND_IMAGE=local-ai-backend:tinygrad \
-	BACKEND_TEST_MODEL_NAME=Qwen/Qwen3-0.6B \
-	BACKEND_TEST_CAPS=health,load,embeddings \
-	$(MAKE) test-extra-backend
-
-## tinygrad — Stable Diffusion 1.5. The original CompVis/runwayml repos have
-## been gated, so we use the community-maintained mirror at
-## stable-diffusion-v1-5/stable-diffusion-v1-5 with the EMA-only pruned
-## checkpoint (~4.3GB). Step count is kept low (4) so a CPU-only run finishes
-## in a few minutes; bump BACKEND_TEST_IMAGE_STEPS for higher quality.
-test-extra-backend-tinygrad-sd: docker-build-tinygrad
-	BACKEND_IMAGE=local-ai-backend:tinygrad \
-	BACKEND_TEST_MODEL_NAME=stable-diffusion-v1-5/stable-diffusion-v1-5 \
-	BACKEND_TEST_CAPS=health,load,image \
-	$(MAKE) test-extra-backend
-
-## tinygrad — Whisper. Loads OpenAI's tiny.en checkpoint (smallest at ~75MB)
-## from the original azure CDN through tinygrad's `fetch` helper, and
-## transcribes the canonical jfk.wav fixture from whisper.cpp's CI samples.
-## Exercises both AudioTranscription and AudioTranscriptionStream.
-test-extra-backend-tinygrad-whisper: docker-build-tinygrad
-	BACKEND_IMAGE=local-ai-backend:tinygrad \
-	BACKEND_TEST_MODEL_NAME=openai/whisper-tiny.en \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	$(MAKE) test-extra-backend
-
-test-extra-backend-tinygrad-all: \
-	test-extra-backend-tinygrad \
-	test-extra-backend-tinygrad-embeddings \
-	test-extra-backend-tinygrad-sd \
-	test-extra-backend-tinygrad-whisper
-
-## insightface — face recognition.
-##
-## Face fixtures default to the sample images shipped in the
-## deepinsight/insightface repository (MIT-licensed). For offline/local
-## runs override with BACKEND_TEST_FACE_IMAGE_{1,2,3}_FILE pointing at
-## local paths.
-FACE_IMAGE_1_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
-FACE_IMAGE_2_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
-FACE_IMAGE_3_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/mask_white.jpg
-## Known spoof fixture used by the face_antispoof e2e cap. This is
-## upstream's own `image_F2.jpg` (Silent-Face repo, via yakhyo mirror)
-## — verified to classify as is_real=false with score < 0.05 on the
-## MiniFASNetV2 + MiniFASNetV1SE ensemble.
-FACE_SPOOF_IMAGE_URL ?= https://github.com/yakhyo/face-anti-spoofing/raw/main/assets/image_F2.jpg
-
-## Host-side cache for the OpenCV Zoo face ONNX files used by the
-## opencv e2e target. The backend image no longer bakes model weights —
-## gallery installs bring them via `files:` — but the e2e suite drives
-## LoadModel over gRPC directly without going through the gallery. We
-## pre-download the ONNX files to a stable host path and pass absolute
-## paths in BACKEND_TEST_OPTIONS; `make` skips the downloads when the
-## SHA-256 already matches.
-INSIGHTFACE_OPENCV_DIR := /tmp/localai-insightface-opencv-cache
-INSIGHTFACE_OPENCV_YUNET_URL := https://github.com/opencv/opencv_zoo/raw/main/models/face_detection_yunet/face_detection_yunet_2023mar.onnx
-INSIGHTFACE_OPENCV_SFACE_URL := https://github.com/opencv/opencv_zoo/raw/main/models/face_recognition_sface/face_recognition_sface_2021dec.onnx
-INSIGHTFACE_OPENCV_YUNET_SHA := 8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4
-INSIGHTFACE_OPENCV_SFACE_SHA := 0ba9fbfa01b5270c96627c4ef784da859931e02f04419c829e83484087c34e79
-
-## buffalo_sc (insightface) — pack zip + SHA-256 mirrors the gallery
-## entry so the e2e target matches exactly what `local-ai models install
-## insightface-buffalo-sc` would have fetched. Smallest insightface pack
-## (~16MB) — keeps CI fast while still covering the insightface engine
-## code path end-to-end.
-INSIGHTFACE_BUFFALO_SC_DIR := /tmp/localai-insightface-buffalo-sc-cache
-INSIGHTFACE_BUFFALO_SC_URL := https://github.com/deepinsight/insightface/releases/download/v0.7/buffalo_sc.zip
-INSIGHTFACE_BUFFALO_SC_SHA := 57d31b56b6ffa911c8a73cfc1707c73cab76efe7f13b675a05223bf42de47c72
-
-## Silent-Face antispoofing (MiniFASNetV2 + MiniFASNetV1SE) — shared
-## between the buffalo_sc and opencv e2e targets. Both ONNX files are
-## ~1.7MB, Apache 2.0. URLs + SHAs mirror the gallery entries.
-INSIGHTFACE_ANTISPOOF_DIR := /tmp/localai-insightface-antispoof-cache
-INSIGHTFACE_ANTISPOOF_V2_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV2.onnx
-INSIGHTFACE_ANTISPOOF_V2_SHA := b32929adc2d9c34b9486f8c4c7bc97c1b69bc0ea9befefc380e4faae4e463907
-INSIGHTFACE_ANTISPOOF_V1SE_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV1SE.onnx
-INSIGHTFACE_ANTISPOOF_V1SE_SHA := ebab7f90c7833fbccd46d3a555410e78d969db5438e169b6524be444862b3676
-
-.PHONY: insightface-opencv-models
-insightface-opencv-models:
-	@mkdir -p $(INSIGHTFACE_OPENCV_DIR)
-	@if [ "$$(sha256sum $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_OPENCV_YUNET_SHA)" ]; then \
-		echo "Fetching YuNet..."; \
-		curl -fsSL -o $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx $(INSIGHTFACE_OPENCV_YUNET_URL); \
-		echo "$(INSIGHTFACE_OPENCV_YUNET_SHA)  $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx" | sha256sum -c; \
-	fi
-	@if [ "$$(sha256sum $(INSIGHTFACE_OPENCV_DIR)/sface.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_OPENCV_SFACE_SHA)" ]; then \
-		echo "Fetching SFace..."; \
-		curl -fsSL -o $(INSIGHTFACE_OPENCV_DIR)/sface.onnx $(INSIGHTFACE_OPENCV_SFACE_URL); \
-		echo "$(INSIGHTFACE_OPENCV_SFACE_SHA)  $(INSIGHTFACE_OPENCV_DIR)/sface.onnx" | sha256sum -c; \
-	fi
-
-.PHONY: insightface-antispoof-models
-insightface-antispoof-models:
-	@mkdir -p $(INSIGHTFACE_ANTISPOOF_DIR)
-	@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V2_SHA)" ]; then \
-		echo "Fetching MiniFASNetV2..."; \
-		curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx $(INSIGHTFACE_ANTISPOOF_V2_URL); \
-		echo "$(INSIGHTFACE_ANTISPOOF_V2_SHA)  $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx" | sha256sum -c; \
-	fi
-	@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA)" ]; then \
-		echo "Fetching MiniFASNetV1SE..."; \
-		curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx $(INSIGHTFACE_ANTISPOOF_V1SE_URL); \
-		echo "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA)  $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx" | sha256sum -c; \
-	fi
-
-.PHONY: insightface-buffalo-sc-models
-insightface-buffalo-sc-models:
-	@mkdir -p $(INSIGHTFACE_BUFFALO_SC_DIR)
-	@if [ "$$(sha256sum $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_BUFFALO_SC_SHA)" ]; then \
-		echo "Fetching buffalo_sc..."; \
-		curl -fsSL -o $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip $(INSIGHTFACE_BUFFALO_SC_URL); \
-		echo "$(INSIGHTFACE_BUFFALO_SC_SHA)  $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip" | sha256sum -c; \
-		rm -f $(INSIGHTFACE_BUFFALO_SC_DIR)/*.onnx; \
-	fi
-	@if [ ! -f "$(INSIGHTFACE_BUFFALO_SC_DIR)/det_500m.onnx" ]; then \
-		echo "Extracting buffalo_sc..."; \
-		unzip -o -q $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip -d $(INSIGHTFACE_BUFFALO_SC_DIR); \
-	fi
-
-## buffalo_sc — smallest insightface pack (SCRFD-500MF detector + MBF
-## recognizer, ~16MB). Exercises the insightface engine code path
-## (model_zoo-backed inference) without the ~326MB buffalo_l download.
-## No age/gender/landmark heads — face_analyze is dropped from caps.
-## The pack is pre-fetched on the host and passed as `root:<dir>` since
-## the e2e suite drives LoadModel directly without going through
-## LocalAI's gallery flow (which is what would normally populate
-## ModelPath and in turn the engine's `_model_dir` option).
-test-extra-backend-insightface-buffalo-sc: docker-build-insightface insightface-buffalo-sc-models insightface-antispoof-models
-	BACKEND_IMAGE=local-ai-backend:insightface \
-	BACKEND_TEST_MODEL_NAME=insightface-buffalo-sc \
-	BACKEND_TEST_OPTIONS=engine:insightface,model_pack:buffalo_sc,root:$(INSIGHTFACE_BUFFALO_SC_DIR),antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
-	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
-	BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
-	BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
-	BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
-	BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
-	BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
-	$(MAKE) test-extra-backend
-
-## OpenCV Zoo YuNet + SFace — Apache 2.0, commercial-safe. face_analyze
-## cap is dropped (SFace has no demographic head). The ONNX files are
-## pre-fetched on the host via the insightface-opencv-models target and
-## passed as absolute paths, since the e2e suite drives LoadModel
-## directly without going through LocalAI's gallery flow.
-test-extra-backend-insightface-opencv: docker-build-insightface insightface-opencv-models insightface-antispoof-models
-	BACKEND_IMAGE=local-ai-backend:insightface \
-	BACKEND_TEST_MODEL_NAME=insightface-opencv \
-	BACKEND_TEST_OPTIONS=engine:onnx_direct,detector_onnx:$(INSIGHTFACE_OPENCV_DIR)/yunet.onnx,recognizer_onnx:$(INSIGHTFACE_OPENCV_DIR)/sface.onnx,antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
-	BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
-	BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
-	BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
-	BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
-	BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
-	BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
-	$(MAKE) test-extra-backend
-
-## Aggregate — runs both face-recognition model configurations so CI
-## catches regressions across engines together.
-test-extra-backend-insightface-all: \
-	test-extra-backend-insightface-buffalo-sc \
-	test-extra-backend-insightface-opencv
-
-## speaker-recognition — voice (speaker) biometrics.
-##
-## Audio fixtures default to the speechbrain test samples served
-## straight from their GitHub repo — public, no auth needed, and they
-## ship as 16kHz mono WAV/FLAC which is exactly what the engine wants.
-## example{1,2,5} are three different speakers; the suite treats
-## example1 as the "same-image twin" probe (verify(clip, clip) must
-## return distance≈0) and the other two as cross-speaker ceilings.
-## Override with BACKEND_TEST_VOICE_AUDIO_{1,2,3}_FILE for offline runs.
-VOICE_AUDIO_1_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example1.wav
-VOICE_AUDIO_2_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example2.flac
-VOICE_AUDIO_3_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example5.wav
-
-## ECAPA-TDNN via SpeechBrain — default CI configuration. Auto-downloads
-## the checkpoint from HuggingFace on first LoadModel (bundled in the
-## backend image pip install). 192-d embeddings, cosine-distance based.
-## The e2e suite drives LoadModel directly so we don't rely on LocalAI's
-## gallery flow here.
-test-extra-backend-speaker-recognition-ecapa: docker-build-speaker-recognition
-	BACKEND_IMAGE=local-ai-backend:speaker-recognition \
-	BACKEND_TEST_MODEL_NAME=speechbrain/spkrec-ecapa-voxceleb \
-	BACKEND_TEST_OPTIONS=engine:speechbrain,source:speechbrain/spkrec-ecapa-voxceleb \
-	BACKEND_TEST_CAPS=health,load,voice_embed,voice_verify \
-	BACKEND_TEST_VOICE_AUDIO_1_URL=$(VOICE_AUDIO_1_URL) \
-	BACKEND_TEST_VOICE_AUDIO_2_URL=$(VOICE_AUDIO_2_URL) \
-	BACKEND_TEST_VOICE_AUDIO_3_URL=$(VOICE_AUDIO_3_URL) \
-	BACKEND_TEST_VOICE_VERIFY_DISTANCE_CEILING=0.4 \
-	$(MAKE) test-extra-backend
-
-## Aggregate — today there's only one voice config; the target exists
-## so the CI workflow matches the insightface-all naming convention and
-## can grow to include WeSpeaker / 3D-Speaker later.
-test-extra-backend-speaker-recognition-all: \
-	test-extra-backend-speaker-recognition-ecapa
-
-## Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked
-## LLM. Extracts the sherpa-onnx Docker image rootfs, downloads the three
-## gallery-referenced model bundles (silero-vad, omnilingual-asr, vits-ljs),
-## writes the corresponding model config YAMLs, and runs the realtime
-## websocket spec in tests/e2e with REALTIME_* env vars wiring the sherpa
-## slots into the pipeline. The LLM slot stays on the in-repo mock-backend
-## registered unconditionally by tests/e2e/e2e_suite_test.go. See
-## tests/e2e/run-realtime-sherpa.sh for the full orchestration.
-test-extra-e2e-realtime-sherpa: build-mock-backend docker-build-sherpa-onnx protogen-go react-ui
-	bash tests/e2e/run-realtime-sherpa.sh
-
-## Streaming ASR via the sherpa-onnx online recognizer. Uses the streaming
-## zipformer English model (encoder/decoder/joiner int8 + tokens) from the
-## sherpa-onnx gallery entry. Drives both AudioTranscription and
-## AudioTranscriptionStream via the e2e-backends gRPC harness; streaming
-## emits real partial deltas during decode. Each file is renamed on download
-## to the shape sherpa-onnx's online loader expects (encoder.int8.onnx etc.).
-test-extra-backend-sherpa-onnx-transcription: docker-build-sherpa-onnx
-	BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#encoder.int8.onnx' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/decoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#decoder.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/joiner-epoch-99-avg-1-chunk-16-left-128.int8.onnx#joiner.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/tokens.txt' \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	BACKEND_TEST_OPTIONS=subtype=online \
-	$(MAKE) test-extra-backend
-
-## VITS TTS via the sherpa-onnx backend. Pulls the individual files from
-## HuggingFace (the vits-ljs release tarball lives on the k2-fsa github
-## but is also mirrored as discrete files on HF). Exercises both
-## TTS (write-to-file) and TTSStream (PCM chunks + WAV header) via the
-## e2e-backends gRPC harness.
-test-extra-backend-sherpa-onnx-tts: docker-build-sherpa-onnx
-	BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/vits-ljs.onnx#vits-ljs.onnx' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/tokens.txt|https://huggingface.co/csukuangfj/vits-ljs/resolve/main/lexicon.txt' \
-	BACKEND_TEST_CAPS=health,load,tts \
-	$(MAKE) test-extra-backend
-
-## VibeVoice TTS via the vibevoice-cpp backend. ModelFile is the
-## realtime gguf; the supplementary tokenizer + voice prompt land
-## alongside it under the harness's models dir and are wired through
-## via the standard Options[] convention (tokenizer=, voice=).
-test-extra-backend-vibevoice-cpp-tts: docker-build-vibevoice-cpp
-	BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-realtime-0.5B-q8_0.gguf#vibevoice-realtime-0.5B-q8_0.gguf' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf|https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/voice-en-Carter_man.gguf#voice-en-Carter_man.gguf' \
-	BACKEND_TEST_OPTIONS=tokenizer:tokenizer.gguf,voice:voice-en-Carter_man.gguf \
-	BACKEND_TEST_CAPS=health,load,tts \
-	$(MAKE) test-extra-backend
-
-## VibeVoice ASR (long-form, with diarization). type=asr tells the
-## backend's Load() to slot ModelFile into the asr_model role; the
-## tokenizer is supplied via Options[]. Uses the Q4_K quant (~10 GB)
-## rather than Q8_0 (~14 GB) so the bundle fits inside ubuntu-latest's
-## post-image disk budget.
-test-extra-backend-vibevoice-cpp-transcription: docker-build-vibevoice-cpp
-	BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-asr-q4_k.gguf#vibevoice-asr-q4_k.gguf' \
-	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf' \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_OPTIONS=type:asr,tokenizer:tokenizer.gguf \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	$(MAKE) test-extra-backend
-
-## Audio transcription wrapper for the whisper.cpp backend.
-## Drives the AudioTranscription / AudioTranscriptionStream RPCs against
-## ggml-base.en (~145 MB) using the JFK 11s clip. The streaming spec
-## asserts len(deltas) >= 1 and concat(deltas) == final.Text - whisper-
-## specific multi-segment assertions live in backend/go/whisper/gowhisper_test.go.
-test-extra-backend-whisper-transcription: docker-build-whisper
-	BACKEND_IMAGE=local-ai-backend:whisper \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	$(MAKE) test-extra-backend
-
-## LocalVQE audio transform (joint AEC + noise suppression + dereverb).
-## Exercises the audio_transform capability end-to-end: batch transform
-## of a real WAV fixture and bidi streaming of synthetic silent frames.
-test-extra-backend-localvqe-transform: docker-build-localvqe
-	BACKEND_IMAGE=local-ai-backend:localvqe \
-	BACKEND_TEST_MODEL_URL='https://huggingface.co/LocalAI-io/LocalVQE/resolve/main/localvqe-v1-1.3M-f32.gguf#localvqe-v1-1.3M-f32.gguf' \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,audio_transform \
-	$(MAKE) test-extra-backend
-
-## sglang mirrors the vllm setup: HuggingFace model id, same tiny Qwen,
-## tool-call extraction via sglang's native qwen parser. CPU builds use
-## sglang's upstream pyproject_cpu.toml recipe (see backend/python/sglang/install.sh).
-test-extra-backend-sglang: docker-build-sglang
-	BACKEND_IMAGE=local-ai-backend:sglang \
-	BACKEND_TEST_MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct \
-	BACKEND_TEST_CAPS=health,load,predict,stream,tools \
-	BACKEND_TEST_OPTIONS=tool_parser:qwen \
-	$(MAKE) test-extra-backend
-
-
-## mlx is Apple-Silicon-first — the MLX backend auto-detects the right tool
-## parser from the chat template, so no tool_parser: option is needed (it
-## would be ignored at runtime). Run this on macOS / arm64 with Metal; the
-## Linux/CPU mlx variant is untested in CI.
-test-extra-backend-mlx: docker-build-mlx
-	BACKEND_IMAGE=local-ai-backend:mlx \
-	BACKEND_TEST_MODEL_NAME=mlx-community/Qwen2.5-0.5B-Instruct-4bit \
-	BACKEND_TEST_CAPS=health,load,predict,stream,tools \
-	$(MAKE) test-extra-backend
-
-test-extra-backend-mlx-vlm: docker-build-mlx-vlm
-	BACKEND_IMAGE=local-ai-backend:mlx-vlm \
-	BACKEND_TEST_MODEL_NAME=mlx-community/Qwen2.5-0.5B-Instruct-4bit \
-	BACKEND_TEST_CAPS=health,load,predict,stream,tools \
-	$(MAKE) test-extra-backend
 
 DOCKER_IMAGE?=local-ai
 IMAGE_TYPE?=core
@@ -963,8 +458,6 @@ docker:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE) .
 
 docker-cuda12:
@@ -978,13 +471,11 @@ docker-cuda12:
 		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE)-cuda-12 .
 
 docker-image-intel:
 	docker build \
-		--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04 \
+		--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04 \
 		--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
@@ -993,8 +484,6 @@ docker-image-intel:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE) .
 
 ########################################################
@@ -1011,10 +500,6 @@ backends/llama-cpp-darwin: build
 	bash ./scripts/build/llama-cpp-darwin.sh
 	./local-ai backends install "ocifile://$(abspath ./backend-images/llama-cpp.tar)"
 
-backends/ds4-darwin: build
-	bash ./scripts/build/ds4-darwin.sh
-	./local-ai backends install "ocifile://$(abspath ./backend-images/ds4.tar)"
-
 build-darwin-python-backend: build
 	bash ./scripts/build/python-darwin.sh
 
@@ -1051,15 +536,6 @@ backend-images:
 # Backend metadata: BACKEND_NAME | DOCKERFILE_TYPE | BUILD_CONTEXT | PROGRESS_FLAG | NEEDS_BACKEND_ARG
 # llama-cpp is special - uses llama-cpp Dockerfile and doesn't need BACKEND arg
 BACKEND_LLAMA_CPP = llama-cpp|llama-cpp|.|false|false
-# ik-llama-cpp is a fork of llama.cpp with superior CPU performance
-BACKEND_IK_LLAMA_CPP = ik-llama-cpp|ik-llama-cpp|.|false|false
-# turboquant is a llama.cpp fork with TurboQuant KV-cache quantization.
-# Reuses backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
-BACKEND_TURBOQUANT = turboquant|turboquant|.|false|false
-# ds4 is antirez/ds4, a DeepSeek V4 Flash-specific inference engine.
-# Single-model; hardware-only validation lives at tests/e2e-backends/
-# (BACKEND_BINARY mode); see docs/superpowers/plans/2026-05-11-ds4-backend.md.
-BACKEND_DS4 = ds4|ds4|.|false|false
 
 # Golang backends
 BACKEND_PIPER = piper|golang|.|false|true
@@ -1070,11 +546,7 @@ BACKEND_STABLEDIFFUSION_GGML = stablediffusion-ggml|golang|.|--progress=plain|tr
 BACKEND_WHISPER = whisper|golang|.|false|true
 BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
-BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
-BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
-BACKEND_LOCALVQE = localvqe|golang|.|false|true
 BACKEND_OPUS = opus|golang|.|false|true
-BACKEND_SHERPA_ONNX = sherpa-onnx|golang|.|false|true
 
 # Python backends with root context
 BACKEND_RERANKERS = rerankers|python|.|false|true
@@ -1083,18 +555,14 @@ BACKEND_OUTETTS = outetts|python|.|false|true
 BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
 BACKEND_COQUI = coqui|python|.|false|true
 BACKEND_RFDETR = rfdetr|python|.|false|true
-BACKEND_INSIGHTFACE = insightface|python|.|false|true
-BACKEND_SPEAKER_RECOGNITION = speaker-recognition|python|.|false|true
 BACKEND_KITTEN_TTS = kitten-tts|python|.|false|true
 BACKEND_NEUTTS = neutts|python|.|false|true
 BACKEND_KOKORO = kokoro|python|.|false|true
 BACKEND_VLLM = vllm|python|.|false|true
 BACKEND_VLLM_OMNI = vllm-omni|python|.|false|true
-BACKEND_SGLANG = sglang|python|.|false|true
 BACKEND_DIFFUSERS = diffusers|python|.|--progress=plain|true
 BACKEND_CHATTERBOX = chatterbox|python|.|false|true
 BACKEND_VIBEVOICE = vibevoice|python|.|--progress=plain|true
-BACKEND_LIQUID_AUDIO = liquid-audio|python|.|--progress=plain|true
 BACKEND_MOONSHINE = moonshine|python|.|false|true
 BACKEND_POCKET_TTS = pocket-tts|python|.|false|true
 BACKEND_QWEN_TTS = qwen-tts|python|.|false|true
@@ -1105,18 +573,8 @@ BACKEND_NEMO = nemo|python|.|false|true
 BACKEND_VOXCPM = voxcpm|python|.|false|true
 BACKEND_WHISPERX = whisperx|python|.|false|true
 BACKEND_ACE_STEP = ace-step|python|.|false|true
-BACKEND_MLX = mlx|python|.|false|true
-BACKEND_MLX_VLM = mlx-vlm|python|.|false|true
 BACKEND_MLX_DISTRIBUTED = mlx-distributed|python|./|false|true
 BACKEND_TRL = trl|python|.|false|true
-BACKEND_LLAMA_CPP_QUANTIZATION = llama-cpp-quantization|python|.|false|true
-BACKEND_TINYGRAD = tinygrad|python|.|false|true
-
-# Rust backends
-BACKEND_KOKOROS = kokoros|rust|.|false|true
-
-# C++ backends (Go wrapper with purego)
-BACKEND_SAM3_CPP = sam3-cpp|golang|.|false|true
 
 # Helper function to build docker image for a backend
 # Usage: $(call docker-build-backend,BACKEND_NAME,DOCKERFILE_TYPE,BUILD_CONTEXT,PROGRESS_FLAG,NEEDS_BACKEND_ARG)
@@ -1128,10 +586,6 @@ define docker-build-backend
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg APT_MIRROR=$(APT_MIRROR) \
-		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
-		$(if $(FROM_SOURCE),--build-arg FROM_SOURCE=$(FROM_SOURCE)) \
-		$(if $(AMDGPU_TARGETS),--build-arg AMDGPU_TARGETS=$(AMDGPU_TARGETS)) \
 		$(if $(filter true,$(5)),--build-arg BACKEND=$(1)) \
 		-t local-ai-backend:$(1) -f backend/Dockerfile.$(2) $(3)
 endef
@@ -1144,9 +598,6 @@ endef
 
 # Generate all docker-build targets
 $(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_IK_LLAMA_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_TURBOQUANT)))
-$(eval $(call generate-docker-build-target,$(BACKEND_DS4)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PIPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCAL_STORE)))
 $(eval $(call generate-docker-build-target,$(BACKEND_HUGGINGFACE)))
@@ -1161,18 +612,14 @@ $(eval $(call generate-docker-build-target,$(BACKEND_OUTETTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
-$(eval $(call generate-docker-build-target,$(BACKEND_INSIGHTFACE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_SPEAKER_RECOGNITION)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KITTEN_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VLLM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VLLM_OMNI)))
-$(eval $(call generate-docker-build-target,$(BACKEND_SGLANG)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DIFFUSERS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_CHATTERBOX)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_LIQUID_AUDIO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MOONSHINE)))
 $(eval $(call generate-docker-build-target,$(BACKEND_POCKET_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN_TTS)))
@@ -1184,24 +631,14 @@ $(eval $(call generate-docker-build-target,$(BACKEND_VOXCPM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_MLX)))
-$(eval $(call generate-docker-build-target,$(BACKEND_MLX_VLM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MLX_DISTRIBUTED)))
 $(eval $(call generate-docker-build-target,$(BACKEND_TRL)))
-$(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP_QUANTIZATION)))
-$(eval $(call generate-docker-build-target,$(BACKEND_TINYGRAD)))
-$(eval $(call generate-docker-build-target,$(BACKEND_KOKOROS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_SAM3_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX)))
 
 # Pattern rule for docker-save targets
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx
+docker-build-backends: docker-build-llama-cpp docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl
 
 ########################################################
 ### Mock Backend for E2E Tests

README.md

@@ -5,17 +5,35 @@
 </h1>
 
 <p align="center">
+<a href="https://github.com/go-skynet/LocalAI/fork" target="blank">
+<img src="https://img.shields.io/github/forks/go-skynet/LocalAI?style=for-the-badge" alt="LocalAI forks"/>
+</a>
 <a href="https://github.com/go-skynet/LocalAI/stargazers" target="blank">
 <img src="https://img.shields.io/github/stars/go-skynet/LocalAI?style=for-the-badge" alt="LocalAI stars"/>
 </a>
+<a href="https://github.com/go-skynet/LocalAI/pulls" target="blank">
+<img src="https://img.shields.io/github/issues-pr/go-skynet/LocalAI?style=for-the-badge" alt="LocalAI pull-requests"/>
+</a>
 <a href='https://github.com/go-skynet/LocalAI/releases'>
 <img src='https://img.shields.io/github/release/go-skynet/LocalAI?&label=Latest&style=for-the-badge'>
 </a>
+</p>
+
+<p align="center">
 <a href="LICENSE" target="blank">
 <img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge" alt="LocalAI License"/>
 </a>
 </p>
 
+<p align="center">
+<a href="https://hub.docker.com/r/localai/localai" target="blank">
+<img src="https://img.shields.io/badge/dockerhub-images-important.svg?logo=Docker" alt="LocalAI Docker hub"/>
+</a>
+<a href="https://quay.io/repository/go-skynet/local-ai?tab=tags&tag=latest" target="blank">
+<img src="https://img.shields.io/badge/quay.io-images-important.svg?" alt="LocalAI Quay.io"/>
+</a>
+</p>
+
 <p align="center">
 <a href="https://twitter.com/LocalAI_API" target="blank">
 <img src="https://img.shields.io/badge/X-%23000000.svg?style=for-the-badge&logo=X&logoColor=white&label=LocalAI_API" alt="Follow LocalAI_API"/>
@@ -29,186 +47,310 @@
 <a href="https://trendshift.io/repositories/5539" target="_blank"><img src="https://trendshift.io/api/badge/repositories/5539" alt="mudler%2FLocalAI | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </p>
 
-**LocalAI** is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.
+> :bulb: Get help - [❓FAQ](https://localai.io/faq/) [💭Discussions](https://github.com/go-skynet/LocalAI/discussions) [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) [:book: Documentation website](https://localai.io/)
+>
+> [💻 Quickstart](https://localai.io/basics/getting_started/) [🖼️ Models](https://models.localai.io/) [🚀 Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap) [🛫 Examples](https://github.com/mudler/LocalAI-examples) Try on 
+[![Telegram](https://img.shields.io/badge/Telegram-2CA5E0?style=for-the-badge&logo=telegram&logoColor=white)](https://t.me/localaiofficial_bot)
 
-- **Drop-in API compatibility** — OpenAI, Anthropic, ElevenLabs APIs
-- **36+ backends** — llama.cpp, vLLM, transformers, whisper, diffusers, MLX...
-- **Any hardware** — NVIDIA, AMD, Intel, Apple Silicon, Vulkan, or CPU-only
-- **Multi-user ready** — API key auth, user quotas, role-based access
-- **Built-in AI agents** — autonomous agents with tool use, RAG, MCP, and skills
-- **Privacy-first** — your data never leaves your infrastructure
+[![tests](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml)[![Build and Release](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml)[![build container images](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml)[![Bump dependencies](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml)[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/localai)](https://artifacthub.io/packages/search?repo=localai)
 
-Created by [Ettore Di Giacinto](https://github.com/mudler) and maintained by the [LocalAI team](#team).
+<p align="center">
+<a href="https://github.com/mudler/LocalAI-examples" target="blank">
+<img src="https://img.shields.io/badge/📦_Examples_Repository-Browse_Ready--to--Run_Examples-blue?style=for-the-badge" alt="LocalAI Examples Repository"/>
+</a>
+</p>
 
-> [:book: Documentation](https://localai.io/) | [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) | [💻 Quickstart](https://localai.io/basics/getting_started/) | [🖼️ Models](https://models.localai.io/) | [❓FAQ](https://localai.io/faq/)
+**LocalAI** is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI (Elevenlabs, Anthropic... ) API specifications for local AI inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families. Does not require GPU. It is created and maintained by [Ettore Di Giacinto](https://github.com/mudler).
 
-## Guided tour
+## Screenshots / Video
+
+### Chat, Model gallery
 
 https://github.com/user-attachments/assets/08cbb692-57da-48f7-963d-2e7b43883c18
 
-<details>
-
-<summary>
-Click to see more!
-</summary>
-
-#### User and auth
-
-https://github.com/user-attachments/assets/228fa9ad-81a3-4d43-bfb9-31557e14a36c
-
-#### Agents
+### Agents
 
 https://github.com/user-attachments/assets/6270b331-e21d-4087-a540-6290006b381a
 
-#### Usage metrics per user
+### Youtube video
 
-https://github.com/user-attachments/assets/cbb03379-23b4-4e3d-bd26-d152f057007f
+<h1 align="center">
+  <br>
+  <a href="https://www.youtube.com/watch?v=PDqYhB9nNHA" target="_blank"> <img width="300" src="https://img.youtube.com/vi/PDqYhB9nNHA/0.jpg"> </a><br>
+<br>
+</h1>
 
-#### Fine-tuning and Quantization
+## 💻 Quickstart
 
-https://github.com/user-attachments/assets/5ba4ace9-d3df-4795-b7d4-b0b404ea71ee
-
-#### WebRTC
-
-https://github.com/user-attachments/assets/ed88e34c-fed3-4b83-8a67-4716a9feeb7b
-
-</details>
-
-## Quickstart
-
-### macOS
+### macOS Download:
 
 <a href="https://github.com/mudler/LocalAI/releases/latest/download/LocalAI.dmg">
   <img src="https://img.shields.io/badge/Download-macOS-blue?style=for-the-badge&logo=apple&logoColor=white" alt="Download LocalAI for macOS"/>
 </a>
 
-> **Note:** The DMG is not signed by Apple. After installing, run: `sudo xattr -d com.apple.quarantine /Applications/LocalAI.app`. See [#6268](https://github.com/mudler/LocalAI/issues/6268) for details.
+> Note: the DMGs are not signed by Apple as quarantined. See https://github.com/mudler/LocalAI/issues/6268 for a workaround, fix is tracked here: https://github.com/mudler/LocalAI/issues/6244
+> Install the DMG and paste this code into terminal: `sudo xattr -d com.apple.quarantine /Applications/LocalAI.app`
 
 ### Containers (Docker, podman, ...)
 
-> Already ran LocalAI before? Use `docker start -i local-ai` to restart an existing container.
+> **💡 Docker Run vs Docker Start**
+> 
+> - `docker run` creates and starts a new container. If a container with the same name already exists, this command will fail.
+> - `docker start` starts an existing container that was previously created with `docker run`.
+> 
+> If you've already run LocalAI before and want to start it again, use: `docker start -i local-ai`
 
-#### CPU only:
+#### CPU only image:
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
 ```
 
-#### NVIDIA GPU:
+#### NVIDIA GPU Images:
 
 ```bash
-# CUDA 13
+# CUDA 13.0
 docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13
 
-# CUDA 12
+# CUDA 12.0
 docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12
 
-# NVIDIA Jetson ARM64 (CUDA 12, for AGX Orin and similar)
+# NVIDIA Jetson (L4T) ARM64
+# CUDA 12 (for Nvidia AGX Orin and similar platforms)
 docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64
 
-# NVIDIA Jetson ARM64 (CUDA 13, for DGX Spark)
+# CUDA 13 (for Nvidia DGX Spark)
 docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13
 ```
 
-#### AMD GPU (ROCm):
+#### AMD GPU Images (ROCm):
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
 ```
 
-#### Intel GPU (oneAPI):
+#### Intel GPU Images (oneAPI):
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intel
 ```
 
-#### Vulkan GPU:
+#### Vulkan GPU Images:
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan
 ```
 
-### Loading models
+To load models:
 
 ```bash
-# From the model gallery (see available models with `local-ai models list` or at https://models.localai.io)
+# From the model gallery (see available models with `local-ai models list`, in the WebUI from the model tab, or visiting https://models.localai.io)
 local-ai run llama-3.2-1b-instruct:q4_k_m
-# From Huggingface
+# Start LocalAI with the phi-2 model directly from huggingface
 local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
-# From the Ollama OCI registry
+# Install and run a model from the Ollama OCI registry
 local-ai run ollama://gemma:2b
-# From a YAML config
+# Run a model from a configuration file
 local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
-# From a standard OCI registry (e.g., Docker Hub)
+# Install and run a model from a standard OCI registry (e.g., Docker Hub)
 local-ai run oci://localai/phi-2:latest
 ```
 
-> **Automatic Backend Detection**: LocalAI automatically detects your GPU capabilities and downloads the appropriate backend. For advanced options, see [GPU Acceleration](https://localai.io/features/gpu-acceleration/).
+> ⚡ **Automatic Backend Detection**: When you install models from the gallery or YAML files, LocalAI automatically detects your system's GPU capabilities (NVIDIA, AMD, Intel) and downloads the appropriate backend. For advanced configuration options, see [GPU Acceleration](https://localai.io/features/gpu-acceleration/#automatic-backend-detection).
 
-For more details, see the [Getting Started guide](https://localai.io/basics/getting_started/).
+For more information, see [💻 Getting started](https://localai.io/basics/getting_started/index.html), if you are interested in our roadmap items and future enhancements, you can see the [Issues labeled as Roadmap here](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
 
-## Latest News
+## 📰 Latest project news
+- March 2026: [Agent management](https://github.com/mudler/LocalAI/pull/8820), [New React UI](https://github.com/mudler/LocalAI/pull/8772), [WebRTC](https://github.com/mudler/LocalAI/pull/8790),[MLX-distributed via P2P and RDMA](https://github.com/mudler/LocalAI/pull/8801), [MCP Apps, MCP Client-side](https://github.com/mudler/LocalAI/pull/8947)
+- February 2026: [Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245), [ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396)
+- January 2026: **LocalAI 3.10.0** - Major release with Anthropic API support, Open Responses API for stateful agents, video & image generation suite (LTX-2), unified GPU backends, tool streaming & XML parsing, system-aware backend gallery, crash fixes for AVX-only CPUs and AMD VRAM reporting, request tracing, and new backends: **Moonshine** (ultra-fast transcription), **Pocket-TTS** (lightweight TTS). Vulkan arm64 builds now available. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0).
+- December 2025: [Dynamic Memory Resource reclaimer](https://github.com/mudler/LocalAI/pull/7583), [Automatic fitting of models to multiple GPUS(llama.cpp)](https://github.com/mudler/LocalAI/pull/7584), [Added Vibevoice backend](https://github.com/mudler/LocalAI/pull/7494)
+- November 2025: Major improvements to the UX. Among these: [Import models via URL](https://github.com/mudler/LocalAI/pull/7245) and [Multiple chats and history](https://github.com/mudler/LocalAI/pull/7325)
+- October 2025: 🔌 [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/) support added for agentic capabilities with external tools
+- September 2025: New Launcher application for MacOS and Linux, extended support to many backends for Mac and Nvidia L4T devices. Models: Added MLX-Audio, WAN 2.2. WebUI improvements and Python-based backends now ships portable python environments.
+- August 2025: MLX, MLX-VLM, Diffusers and llama.cpp are now supported on Mac M1/M2/M3+ chips ( with `development` suffix in the gallery ): https://github.com/mudler/LocalAI/pull/6049 https://github.com/mudler/LocalAI/pull/6119 https://github.com/mudler/LocalAI/pull/6121 https://github.com/mudler/LocalAI/pull/6060
+- July/August 2025: 🔍 [Object Detection](https://localai.io/features/object-detection/) added to the API featuring [rf-detr](https://github.com/roboflow/rf-detr)
+- July 2025: All backends migrated outside of the main binary. LocalAI is now more lightweight, small, and automatically downloads the required backend to run the model. [Read the release notes](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)
+- June 2025: [Backend management](https://github.com/mudler/LocalAI/pull/5607) has been added. Attention: extras images are going to be deprecated from the next release! Read [the backend management PR](https://github.com/mudler/LocalAI/pull/5607).
+- May 2025: [Audio input](https://github.com/mudler/LocalAI/pull/5466) and [Reranking](https://github.com/mudler/LocalAI/pull/5396) in llama.cpp backend, [Realtime API](https://github.com/mudler/LocalAI/pull/5392),  Support to Gemma, SmollVLM, and more multimodal models (available in the gallery).
+- May 2025: Important: image name changes [See release](https://github.com/mudler/LocalAI/releases/tag/v2.29.0)
+- Apr 2025: Rebrand, WebUI enhancements
+- Apr 2025: [LocalAGI](https://github.com/mudler/LocalAGI) and [LocalRecall](https://github.com/mudler/LocalRecall) join the LocalAI family stack.
+- Apr 2025: WebUI overhaul
+- Feb 2025: Backend cleanup, Breaking changes, new backends (kokoro, OutelTTS, faster-whisper), Nvidia L4T images
+- Jan 2025: LocalAI model release: https://huggingface.co/mudler/LocalAI-functioncall-phi-4-v0.3, SANA support in diffusers: https://github.com/mudler/LocalAI/pull/4603
+- Dec 2024: stablediffusion.cpp backend (ggml) added ( https://github.com/mudler/LocalAI/pull/4289 )
+- Nov 2024: Bark.cpp backend added ( https://github.com/mudler/LocalAI/pull/4287 )
+- Nov 2024: Voice activity detection models (**VAD**) added to the API: https://github.com/mudler/LocalAI/pull/4204
+- Oct 2024: examples moved to [LocalAI-examples](https://github.com/mudler/LocalAI-examples)
+- Aug 2024:  🆕 FLUX-1, [P2P Explorer](https://explorer.localai.io)
+- July 2024: 🔥🔥 🆕 P2P Dashboard, LocalAI Federated mode and AI Swarms: https://github.com/mudler/LocalAI/pull/2723. P2P Global community pools: https://github.com/mudler/LocalAI/issues/3113
+- May 2024: 🔥🔥 Decentralized P2P llama.cpp:  https://github.com/mudler/LocalAI/pull/2343 (peer2peer llama.cpp!) 👉 Docs  https://localai.io/features/distribute/
+- May 2024: 🔥🔥 Distributed inferencing: https://github.com/mudler/LocalAI/pull/2324
+- April 2024: Reranker API: https://github.com/mudler/LocalAI/pull/2121
 
-- **April 2026**: [Voice recognition](https://github.com/mudler/LocalAI/pull/9500), [Face recognition, identification & liveness detection](https://github.com/mudler/LocalAI/pull/9480), [Ollama API compatibility](https://github.com/mudler/LocalAI/pull/9284), [Video generation in stable-diffusion.ggml](https://github.com/mudler/LocalAI/pull/9420), [Backend versioning with auto-upgrade](https://github.com/mudler/LocalAI/pull/9315), [Pin models & load-on-demand toggle](https://github.com/mudler/LocalAI/pull/9309), [Universal model importer](https://github.com/mudler/LocalAI/pull/9466), new backends: [sglang](https://github.com/mudler/LocalAI/pull/9359), [ik-llama-cpp](https://github.com/mudler/LocalAI/pull/9326), [TurboQuant](https://github.com/mudler/LocalAI/pull/9355), [sam.cpp](https://github.com/mudler/LocalAI/pull/9288), [Kokoros](https://github.com/mudler/LocalAI/pull/9212), [qwen3tts.cpp](https://github.com/mudler/LocalAI/pull/9316), [tinygrad multimodal](https://github.com/mudler/LocalAI/pull/9364)
-- **March 2026**: [Agent management](https://github.com/mudler/LocalAI/pull/8820), [New React UI](https://github.com/mudler/LocalAI/pull/8772), [WebRTC](https://github.com/mudler/LocalAI/pull/8790), [MLX-distributed via P2P and RDMA](https://github.com/mudler/LocalAI/pull/8801), [MCP Apps, MCP Client-side](https://github.com/mudler/LocalAI/pull/8947)
-- **February 2026**: [Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245), [ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396)
-- **January 2026**: **LocalAI 3.10.0** — Anthropic API support, Open Responses API, video & image generation (LTX-2), unified GPU backends, tool streaming, Moonshine, Pocket-TTS. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0)
-- **December 2025**: [Dynamic Memory Resource reclaimer](https://github.com/mudler/LocalAI/pull/7583), [Automatic multi-GPU model fitting (llama.cpp)](https://github.com/mudler/LocalAI/pull/7584), [Vibevoice backend](https://github.com/mudler/LocalAI/pull/7494)
-- **November 2025**: [Import models via URL](https://github.com/mudler/LocalAI/pull/7245), [Multiple chats and history](https://github.com/mudler/LocalAI/pull/7325)
-- **October 2025**: [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/) support for agentic capabilities
-- **September 2025**: New Launcher for macOS and Linux, extended backend support for Mac and Nvidia L4T, MLX-Audio, WAN 2.2
-- **August 2025**: MLX, MLX-VLM, Diffusers, llama.cpp now supported on Apple Silicon
-- **July 2025**: All backends migrated outside the main binary — [lightweight, modular architecture](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)
+Roadmap items: [List of issues](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
 
-For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [News page](https://localai.io/basics/news/).
+## 🚀 [Features](https://localai.io/features/)
 
-## Features
+- 🧩 [Backend Gallery](https://localai.io/backends/): Install/remove backends on the fly, powered by OCI images — fully customizable and API-driven.
+- 📖 [Text generation with GPTs](https://localai.io/features/text-generation/) (`llama.cpp`, `transformers`, `vllm` ... [:book: and more](https://localai.io/model-compatibility/index.html#model-compatibility-table))
+- 🗣 [Text to Audio](https://localai.io/features/text-to-audio/)
+- 🔈 [Audio to Text](https://localai.io/features/audio-to-text/)
+- 🎨 [Image generation](https://localai.io/features/image-generation)
+- 🔥 [OpenAI-alike tools API](https://localai.io/features/openai-functions/) 
+- ⚡ [Realtime API](https://localai.io/features/openai-realtime/) (Speech-to-speech) 
+- 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/)
+- ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/)
+- 🖼️ [Download Models directly from Huggingface ](https://localai.io/models/)
+- 🥽 [Vision API](https://localai.io/features/gpt-vision/)
+- 🔍 [Object Detection](https://localai.io/features/object-detection/)
+- 📈 [Reranker API](https://localai.io/features/reranker/)
+- 🆕🖧 [P2P Inferencing](https://localai.io/features/distribute/)
+- 🆕🔌 [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/) - Agentic capabilities with external tools and [LocalAGI's Agentic capabilities](https://github.com/mudler/LocalAGI)
+- 🆕🤖 [Built-in Agents](https://localai.io/features/agents/) - Autonomous AI agents with tool use, knowledge base (RAG), skills, SSE streaming, import/export, and [Agent Hub](https://agenthub.localai.io) — powered by [LocalAGI](https://github.com/mudler/LocalAGI)
+- 🔊 Voice activity detection (Silero-VAD support)
+- 🌍 Integrated WebUI!
 
-- [Text generation](https://localai.io/features/text-generation/) (`llama.cpp`, `transformers`, `vllm` ... [and more](https://localai.io/model-compatibility/))
-- [Text to Audio](https://localai.io/features/text-to-audio/)
-- [Audio to Text](https://localai.io/features/audio-to-text/)
-- [Image generation](https://localai.io/features/image-generation)
-- [OpenAI-compatible tools API](https://localai.io/features/openai-functions/)
-- [Realtime API](https://localai.io/features/openai-realtime/) (Speech-to-speech)
-- [Embeddings generation](https://localai.io/features/embeddings/)
-- [Constrained grammars](https://localai.io/features/constrained_grammars/)
-- [Download models from Huggingface](https://localai.io/models/)
-- [Vision API](https://localai.io/features/gpt-vision/)
-- [Object Detection](https://localai.io/features/object-detection/)
-- [Reranker API](https://localai.io/features/reranker/)
-- [P2P Inferencing](https://localai.io/features/distribute/)
-- [Distributed Mode](https://localai.io/features/distributed-mode/) — Horizontal scaling with PostgreSQL + NATS
-- [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/)
-- [Built-in Agents](https://localai.io/features/agents/) — Autonomous AI agents with tool use, RAG, skills, SSE streaming, and [Agent Hub](https://agenthub.localai.io)
-- [Backend Gallery](https://localai.io/backends/) — Install/remove backends on the fly via OCI images
-- Voice Activity Detection (Silero-VAD)
-- Integrated WebUI
+## 🧩 Supported Backends & Acceleration
 
-## Supported Backends & Acceleration
+LocalAI supports a comprehensive range of AI backends with multiple acceleration options:
 
-LocalAI supports **36+ backends** including llama.cpp, vLLM, transformers, whisper.cpp, diffusers, MLX, MLX-VLM, and many more. Hardware acceleration is available for **NVIDIA** (CUDA 12/13), **AMD** (ROCm), **Intel** (oneAPI/SYCL), **Apple Silicon** (Metal), **Vulkan**, and **NVIDIA Jetson** (L4T). All backends can be installed on-the-fly from the [Backend Gallery](https://localai.io/backends/).
+### Text Generation & Language Models
+| Backend | Description | Acceleration Support |
+|---------|-------------|---------------------|
+| **llama.cpp** | LLM inference in C/C++ | CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, CPU |
+| **vLLM** | Fast LLM inference with PagedAttention | CUDA 12/13, ROCm, Intel |
+| **transformers** | HuggingFace transformers framework | CUDA 12/13, ROCm, Intel, CPU |
+| **MLX** | Apple Silicon LLM inference | Metal (M1/M2/M3+) |
+| **MLX-VLM** | Apple Silicon Vision-Language Models | Metal (M1/M2/M3+) |
+| **vLLM Omni** | Multimodal vLLM with vision and audio | CUDA 12/13, ROCm, Intel |
 
-See the full [Backend & Model Compatibility Table](https://localai.io/model-compatibility/) and [GPU Acceleration guide](https://localai.io/features/gpu-acceleration/).
+### Audio & Speech Processing
+| Backend | Description | Acceleration Support |
+|---------|-------------|---------------------|
+| **whisper.cpp** | OpenAI Whisper in C/C++ | CUDA 12/13, ROCm, Intel SYCL, Vulkan, CPU |
+| **faster-whisper** | Fast Whisper with CTranslate2 | CUDA 12/13, ROCm, Intel, CPU |
+| **moonshine** | Ultra-fast transcription engine for low-end devices | CUDA 12/13, Metal, CPU |
+| **coqui** | Advanced TTS with 1100+ languages | CUDA 12/13, ROCm, Intel, CPU |
+| **kokoro** | Lightweight TTS model | CUDA 12/13, ROCm, Intel, CPU |
+| **chatterbox** | Production-grade TTS | CUDA 12/13, CPU |
+| **piper** | Fast neural TTS system | CPU |
+| **kitten-tts** | Kitten TTS models | CPU |
+| **silero-vad** | Voice Activity Detection | CPU |
+| **neutts** | Text-to-speech with voice cloning | CUDA 12/13, ROCm, CPU |
+| **vibevoice** | Real-time TTS with voice cloning | CUDA 12/13, ROCm, Intel, CPU |
+| **pocket-tts** | Lightweight CPU-based TTS | CUDA 12/13, ROCm, Intel, CPU |
+| **qwen-tts** | High-quality TTS with custom voice, voice design, and voice cloning | CUDA 12/13, ROCm, Intel, CPU |
+| **nemo** | NVIDIA NeMo framework for speech models | CUDA 12/13, ROCm, Intel, CPU |
+| **outetts** | OuteTTS with voice cloning | CUDA 12/13, CPU |
+| **faster-qwen3-tts** | Faster Qwen3 TTS | CUDA 12/13, ROCm, Intel, CPU |
+| **qwen-asr** | Qwen ASR speech recognition | CUDA 12/13, ROCm, Intel, CPU |
+| **voxcpm** | VoxCPM speech understanding | CUDA 12/13, Metal, CPU |
+| **whisperx** | Enhanced Whisper transcription | CUDA 12/13, ROCm, Intel, CPU |
+| **ace-step** | Music generation from text descriptions, lyrics, or audio samples | CUDA 12/13, ROCm, Intel, Metal, CPU |
 
-## Resources
+### Image & Video Generation
+| Backend | Description | Acceleration Support |
+|---------|-------------|---------------------|
+| **stablediffusion.cpp** | Stable Diffusion in C/C++ | CUDA 12/13, Intel SYCL, Vulkan, CPU |
+| **diffusers** | HuggingFace diffusion models | CUDA 12/13, ROCm, Intel, Metal, CPU |
 
-- [Documentation](https://localai.io/)
-- [LLM fine-tuning guide](https://localai.io/docs/advanced/fine-tuning/)
-- [Build from source](https://localai.io/basics/build/)
-- [Kubernetes installation](https://localai.io/basics/getting_started/#run-localai-in-kubernetes)
-- [Integrations & community projects](https://localai.io/docs/integrations/)
-- [Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)
-- [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
-- [Examples](https://github.com/mudler/LocalAI-examples)
+### Specialized AI Tasks
+| Backend | Description | Acceleration Support |
+|---------|-------------|---------------------|
+| **rfdetr** | Real-time object detection | CUDA 12/13, Intel, CPU |
+| **rerankers** | Document reranking API | CUDA 12/13, ROCm, Intel, CPU |
+| **local-store** | Vector database | CPU |
+| **huggingface** | HuggingFace API integration | API-based |
 
-## Team
+### Hardware Acceleration Matrix
 
-LocalAI is maintained by a small team of humans, together with the wider community of contributors.
+| Acceleration Type | Supported Backends | Hardware Support |
+|-------------------|-------------------|------------------|
+| **NVIDIA CUDA 12** | All CUDA-compatible backends | Nvidia hardware |
+| **NVIDIA CUDA 13** | All CUDA-compatible backends | Nvidia hardware |
+| **AMD ROCm** | llama.cpp, whisper, vllm, transformers, diffusers, rerankers, coqui, kokoro, neutts, vibevoice, pocket-tts, qwen-tts, ace-step | AMD Graphics |
+| **Intel oneAPI** | llama.cpp, whisper, stablediffusion, vllm, transformers, diffusers, rfdetr, rerankers, coqui, kokoro, vibevoice, pocket-tts, qwen-tts, ace-step | Intel Arc, Intel iGPUs |
+| **Apple Metal** | llama.cpp, whisper, diffusers, MLX, MLX-VLM, moonshine, ace-step | Apple M1/M2/M3+ |
+| **Vulkan** | llama.cpp, whisper, stablediffusion | Cross-platform GPUs |
+| **NVIDIA Jetson (CUDA 12)** | llama.cpp, whisper, stablediffusion, diffusers, rfdetr, ace-step | ARM64 embedded AI (AGX Orin, etc.) |
+| **NVIDIA Jetson (CUDA 13)** | llama.cpp, whisper, stablediffusion, diffusers, rfdetr | ARM64 embedded AI (DGX Spark) |
+| **CPU Optimized** | All backends | AVX/AVX2/AVX512, quantization support |
 
-- **[Ettore Di Giacinto](https://github.com/mudler)** — original author and project lead
-- **[Richard Palethorpe](https://github.com/richiejp)** — maintainer
+### 🔗 Community and integrations
 
-A huge thank you to everyone who contributes code, reviews PRs, files issues, and helps users in [Discord](https://discord.gg/uJAeKSAGDy) — LocalAI is a community-driven project and wouldn't exist without you. See the full [contributors list](https://github.com/mudler/LocalAI/graphs/contributors).
+Build and deploy custom containers:
+- https://github.com/sozercan/aikit
+
+WebUIs:
+- https://github.com/Jirubizu/localai-admin
+- https://github.com/go-skynet/LocalAI-frontend
+- QA-Pilot(An interactive chat project that leverages LocalAI LLMs for rapid understanding and navigation of GitHub code repository) https://github.com/reid41/QA-Pilot
+
+Agentic Libraries:
+- https://github.com/mudler/cogito
+
+MCPs:
+- https://github.com/mudler/MCPs
+
+OS Assistant:
+
+- https://github.com/mudler/Keygeist - Keygeist is an AI-powered keyboard operator that listens for key combinations and responds with AI-generated text typed directly into your Linux box.
+
+Model galleries
+- https://github.com/go-skynet/model-gallery
+
+Voice:
+- https://github.com/richiejp/VoxInput
+
+Other:
+- Helm chart https://github.com/go-skynet/helm-charts
+- VSCode extension https://github.com/badgooooor/localai-vscode-plugin
+- Langchain: https://python.langchain.com/docs/integrations/providers/localai/
+- Terminal utility https://github.com/djcopley/ShellOracle
+- Local Smart assistant https://github.com/mudler/LocalAGI
+- Home Assistant https://github.com/drndos/hass-openai-custom-conversation / https://github.com/valentinfrlch/ha-llmvision / https://github.com/loryanstrant/HA-LocalAI-Monitor
+- Discord bot https://github.com/mudler/LocalAGI/tree/main/examples/discord
+- Slack bot https://github.com/mudler/LocalAGI/tree/main/examples/slack
+- Shell-Pilot(Interact with LLM using LocalAI models via pure shell scripts on your Linux or MacOS system) https://github.com/reid41/shell-pilot
+- Telegram bot https://github.com/mudler/LocalAI/tree/master/examples/telegram-bot
+- Another Telegram Bot https://github.com/JackBekket/Hellper
+- Auto-documentation https://github.com/JackBekket/Reflexia
+- Github bot which answer on issues, with code and documentation as context https://github.com/JackBekket/GitHelper
+- Github Actions: https://github.com/marketplace/actions/start-localai
+- Examples: https://github.com/mudler/LocalAI/tree/master/examples/
+  
+
+### 🔗 Resources
+
+- [LLM finetuning guide](https://localai.io/docs/advanced/fine-tuning/)
+- [How to build locally](https://localai.io/basics/build/index.html)
+- [How to install in Kubernetes](https://localai.io/basics/getting_started/index.html#run-localai-in-kubernetes)
+- [Projects integrating LocalAI](https://localai.io/docs/integrations/)
+- [How tos section](https://io.midori-ai.xyz/howtos/) (curated by our community)
+
+## :book: 🎥 [Media, Blogs, Social](https://localai.io/basics/news/#media-blogs-social)
+
+- 🆕 [LocalAI Autonomous Dev Team Blog Post](https://mudler.pm/posts/2026/02/28/a-call-to-open-source-maintainers-stop-babysitting-ai-how-i-built-a-100-local-autonomous-dev-team-to-maintain-localai-and-why-you-should-too/)
+
+- [Run Visual studio code with LocalAI (SUSE)](https://www.suse.com/c/running-ai-locally/)
+- 🆕 [Run LocalAI on Jetson Nano Devkit](https://mudler.pm/posts/local-ai-jetson-nano-devkit/)
+- [Run LocalAI on AWS EKS with Pulumi](https://www.pulumi.com/blog/low-code-llm-apps-with-local-ai-flowise-and-pulumi/)
+- [Run LocalAI on AWS](https://staleks.hashnode.dev/installing-localai-on-aws-ec2-instance)
+- [Create a slackbot for teams and OSS projects that answer to documentation](https://mudler.pm/posts/smart-slackbot-for-teams/)
+- [LocalAI meets k8sgpt](https://www.youtube.com/watch?v=PKrDNuJ_dfE)
+- [Question Answering on Documents locally with LangChain, LocalAI, Chroma, and GPT4All](https://mudler.pm/posts/localai-question-answering/)
+- [Tutorial to use k8sgpt with LocalAI](https://medium.com/@tyler_97636/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65)
+
+
+## 🤖 Autonomous Development Team
+
+LocalAI is now helped being maintained (for small tasks!) by a full team of autonomous AI agents led by an AI Scrum Master! This experiment demonstrates how open source projects can leverage AI agents for sustainable, long-term maintenance.
+
+- **📊 Live Reports**: [Automatically generated reports](http://reports.localai.io)
+- **📋 Project Board**: [Agent task tracking](https://github.com/users/mudler/projects/6)
+- **📝 Blog Post**: [Learn about the autonomous dev team experiment](https://mudler.pm/posts/2026/02/28/a-call-to-open-source-maintainers-stop-babysitting-ai-how-i-built-a-100-local-autonomous-dev-team-to-maintain-localai-and-why-you-should-too/)
 
 ## Citation
 
@@ -224,7 +366,7 @@ If you utilize this repository, data in a downstream project, please consider ci
   howpublished = {\url{https://github.com/go-skynet/LocalAI}},
 ```
 
-## Sponsors
+## ❤️ Sponsors
 
 > Do you find LocalAI useful?
 
@@ -243,19 +385,19 @@ A huge thank you to our generous sponsors who support this project covering CI e
 
 ### Individual sponsors
 
-A special thanks to individual sponsors, a full list is on [GitHub](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler). Special shout out to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!
+A special thanks to individual sponsors that contributed to the project, a full list is in [Github](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler), a special shout out goes to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!
 
-## Star history
+## 🌟 Star history
 
 [![LocalAI Star history Chart](https://api.star-history.com/svg?repos=go-skynet/LocalAI&type=Date)](https://star-history.com/#go-skynet/LocalAI&Date)
 
-## License
+## 📖 License
 
-LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/) and maintained by the [LocalAI team](#team).
+LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/).
 
 MIT - Author Ettore Di Giacinto <mudler@localai.io>
 
-## Acknowledgements
+## 🙇 Acknowledgements
 
 LocalAI couldn't have been built without the help of great software already available from the community. Thank you!
 
@@ -268,9 +410,9 @@ LocalAI couldn't have been built without the help of great software already avai
 - https://github.com/rhasspy/piper
 - [exo](https://github.com/exo-explore/exo) for the MLX distributed auto-parallel sharding implementation
 
-## Contributors
+## 🤗 Contributors
 
-This is a community project, a special thanks to our contributors!
+This is a community project, a special thanks to our contributors! 🤗
 <a href="https://github.com/go-skynet/LocalAI/graphs/contributors">
   <img src="https://contrib.rocks/image?repo=go-skynet/LocalAI" />
 </a>

backend/Dockerfile.base-grpc-builder

@@ -1,98 +0,0 @@
-# syntax=docker/dockerfile:1.7
-#
-# Pre-built builder base image for LocalAI's C++ backends.
-#
-# This Dockerfile is the source of truth for the
-# `quay.io/go-skynet/ci-cache:base-grpc-*` images that
-# `.github/workflows/base-images.yml` builds and pushes. The output of a
-# build is a fully-prepped builder layer containing:
-#
-#   - apt build deps (build-essential, ccache, git, make, pkg-config,
-#     libcurl4-openssl-dev, libssl-dev, curl, unzip, wget, ca-certificates)
-#   - cmake (apt or, when CMAKE_FROM_SOURCE=true, compiled from
-#     ${CMAKE_VERSION})
-#   - protoc v27.1 at /usr/local/bin/protoc
-#   - gRPC ${GRPC_VERSION} compiled and installed at /opt/grpc
-#   - Conditional CUDA toolkit (BUILD_TYPE=cublas|l4t, SKIP_DRIVERS=false)
-#     including the cuda-13 + arm64 cudss/nvpl special case
-#   - Conditional ROCm/HIP build deps (BUILD_TYPE=hipblas)
-#   - Conditional Vulkan SDK 1.4.335.0 (BUILD_TYPE=vulkan)
-#
-# Variants built by the workflow (matrix in base-images.yml):
-#
-#   base-grpc-amd64                 ubuntu:24.04, CPU-only
-#   base-grpc-arm64                 ubuntu:24.04, CPU-only
-#   base-grpc-cuda-12-amd64         ubuntu:24.04 + CUDA 12.8
-#   base-grpc-cuda-13-amd64         ubuntu:22.04 + CUDA 13.0
-#   base-grpc-cuda-13-arm64         ubuntu:24.04 + CUDA 13.0 (sbsa)
-#   base-grpc-l4t-cuda-12-arm64     ubuntu:22.04 + CUDA 12.x (legacy JetPack)
-#   base-grpc-rocm-amd64            rocm/dev-ubuntu-24.04:7.2.1 + hipblas
-#   base-grpc-vulkan-amd64          ubuntu:24.04 + Vulkan SDK 1.4.335
-#   base-grpc-vulkan-arm64          ubuntu:24.04 + Vulkan SDK ARM 1.4.335
-#   base-grpc-intel-amd64           intel/oneapi-basekit:2025.3.2 (sycl)
-#
-# This is a SINGLE-stage Dockerfile by design: the final image IS the
-# builder base. The intermediate gRPC compile happens inside this same
-# stage so consumer Dockerfiles in PR 2 can simply
-# `FROM quay.io/go-skynet/ci-cache:base-grpc-<variant>` without needing a
-# COPY --from=grpc step. /opt/grpc is the canonical install prefix and
-# downstream builds will add it to CMAKE_PREFIX_PATH (or copy to
-# /usr/local) the same way Dockerfile.llama-cpp does today.
-#
-# Install logic lives in .docker/install-base-deps.sh, which is also
-# bind-mounted by the variant Dockerfiles' builder-fromsource stage.
-# This guarantees bit-equivalence between the prebuilt CI base and the
-# from-source local-dev path — both invoke the same script with the
-# same env inputs.
-
-ARG BASE_IMAGE=ubuntu:24.04
-
-FROM ${BASE_IMAGE}
-
-ARG BASE_IMAGE=ubuntu:24.04
-ARG BUILD_TYPE=""
-ARG CUDA_MAJOR_VERSION=""
-ARG CUDA_MINOR_VERSION=""
-ARG CMAKE_FROM_SOURCE=false
-# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain
-# detection / arch table issues.
-ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG SKIP_DRIVERS=false
-ARG TARGETARCH
-ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
-ARG AMDGPU_TARGETS=""
-
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    DEBIAN_FRONTEND=noninteractive
-
-# CUDA on PATH (no-op when CUDA isn't installed)
-ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
-ENV PATH=/opt/rocm/bin:${PATH}
-
-WORKDIR /build
-
-# Single RUN that delegates to .docker/install-base-deps.sh — the same
-# script the variant Dockerfiles' builder-fromsource stage runs.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
-
-WORKDIR /

backend/Dockerfile.ds4

@@ -1,41 +0,0 @@
-ARG BASE_IMAGE=ubuntu:24.04
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
-
-# BASE_IMAGE is either ubuntu:24.04 (for cpu builds) or nvidia/cuda:13.0.0-devel-ubuntu24.04
-# (for cublas builds). Both ship apt + Ubuntu Noble packages; the nvidia/cuda base
-# additionally provides /usr/local/cuda. Darwin (Metal) builds bypass this Dockerfile
-# entirely via scripts/build/ds4-darwin.sh.
-FROM ${BASE_IMAGE} AS builder
-ARG BUILD_TYPE
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    DEBIAN_FRONTEND=noninteractive \
-    PATH=/usr/local/cuda/bin:${PATH}
-
-WORKDIR /build
-
-# Install build-time deps via plain apt - install-base-deps.sh's full pipeline
-# (CUDA keyring + from-source gRPC) is unnecessary here:
-#   - CUDA: when BASE_IMAGE=nvidia/cuda:*, /usr/local/cuda is already populated;
-#     for the cpu build we don't need CUDA at all.
-#   - gRPC/Protobuf: system apt packages are sufficient; ds4's wrapper only links
-#     against them, it doesn't ship the gRPC source tree.
-#   - nlohmann-json: dsml_renderer's only third-party dep.
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
-        git cmake build-essential pkg-config ca-certificates \
-        libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc \
-        nlohmann-json3-dev && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
-
-COPY . /LocalAI
-
-RUN --mount=type=cache,target=/root/.ccache,id=ds4-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package
-
-FROM scratch
-COPY --from=builder /LocalAI/backend/cpp/ds4/package/. ./

backend/Dockerfile.golang

@@ -1,6 +1,4 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -16,20 +14,8 @@ ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
-ARG AMDGPU_TARGETS
-ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 
-# gcc-14 is the default on noble (ubuntu:24.04) but absent from jammy
-# (the L4T jetpack r36.4.0 base). LocalVQE specifically needs it; the
-# other Go backends compile fine with the default gcc shipped via
-# build-essential. So: try gcc-14 from the configured repos, fall back
-# gracefully when it's not available so jammy-based builds don't fail
-# at the apt step.
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         git ccache \
@@ -37,12 +23,6 @@ RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mi
         make cmake wget libopenblas-dev \
         curl unzip \
         libssl-dev && \
-    if apt-cache show gcc-14 >/dev/null 2>&1 && apt-cache show g++-14 >/dev/null 2>&1; then \
-        apt-get install -y --no-install-recommends gcc-14 g++-14 && \
-        update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100 \
-            --slave /usr/bin/g++ g++ /usr/bin/g++-14 \
-            --slave /usr/bin/gcov gcov /usr/bin/gcov-14; \
-    fi && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -167,7 +147,6 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
-            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \

backend/Dockerfile.ik-llama-cpp

@@ -1,151 +0,0 @@
-ARG BASE_IMAGE=ubuntu:24.04
-# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
-# when no prebuilt base is supplied. The builder-prebuilt stage is only
-# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
-# content here is harmless — BuildKit prunes the unreferenced builder.
-ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
-# BUILDER_TARGET selects which builder stage the final scratch image copies
-# package output from. Declared at global scope (before any FROM) so it's
-# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
-# `make backends/ik-llama-cpp` on the from-source path.
-ARG BUILDER_TARGET=builder-fromsource
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
-
-
-# ============================================================================
-# Stage: builder-fromsource — self-contained build path.
-# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
-# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
-# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
-# default; local `make backends/ik-llama-cpp`).
-#
-# The install script is the same one that backend/Dockerfile.base-grpc-builder
-# runs, so the result is bit-equivalent to the prebuilt-base path
-# (builder-prebuilt below).
-# ============================================================================
-FROM ${BASE_IMAGE} AS builder-fromsource
-ARG BUILD_TYPE
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
-ARG CMAKE_FROM_SOURCE=false
-# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
-ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG SKIP_DRIVERS=false
-ARG TARGETARCH
-ARG TARGETVARIANT
-ARG GO_VERSION=1.25.4
-ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-ARG AMDGPU_TARGETS=""
-ARG BACKEND=rerankers
-# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ARG CMAKE_ARGS
-
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
-    CMAKE_ARGS=${CMAKE_ARGS} \
-    DEBIAN_FRONTEND=noninteractive
-
-# CUDA on PATH (no-op when CUDA isn't installed)
-ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
-ENV PATH=/opt/rocm/bin:${PATH}
-
-WORKDIR /build
-
-# Install everything via the shared script — the same one that
-# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
-# this from-source path are bit-equivalent.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
-
-# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
-# CMake's find_package finds it at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
-# for the rationale. Distinct mount id so ik-llama-cpp's cache doesn't
-# overlap with llama-cpp's — ik_llama.cpp is a different fork with
-# different source.
-#
-# The compile body is shared with builder-prebuilt via .docker/ik-llama-cpp-compile.sh.
-RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-
-# Copy libraries using a script to handle architecture differences
-RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
-
-
-# ============================================================================
-# Stage: builder-prebuilt — uses the pre-built base from
-# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
-# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
-# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
-# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
-# builder-base-image).
-# ============================================================================
-FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
-
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
-# /usr/local. Mirror what the from-source path does so the compile step
-# can find gRPC at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
-
-
-# ============================================================================
-# Final stage — copies package output from one of the two builders.
-# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
-#
-# BuildKit doesn't support variable expansion in `COPY --from=` directly,
-# so we resolve the ARG by aliasing the chosen builder to a fixed stage
-# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
-# BUILDER_TARGET itself is declared as a global ARG at the top of this
-# file (required for use in FROM), so we just re-import it into this
-# stage's scope before the FROM directive.
-# ============================================================================
-FROM ${BUILDER_TARGET} AS builder
-
-FROM scratch
-
-
-# Copy all available binaries (the build process only creates the appropriate ones for the target architecture)
-COPY --from=builder /LocalAI/backend/cpp/ik-llama-cpp/package/. ./

backend/Dockerfile.llama-cpp

@@ -1,155 +1,284 @@
 ARG BASE_IMAGE=ubuntu:24.04
-# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
-# when no prebuilt base is supplied. The builder-prebuilt stage is only
-# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
-# content here is harmless — BuildKit prunes the unreferenced builder.
-ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
-# BUILDER_TARGET selects which builder stage the final scratch image copies
-# package output from. Declared at global scope (before any FROM) so it's
-# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
-# `make backends/llama-cpp` on the from-source path.
-ARG BUILDER_TARGET=builder-fromsource
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
+ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 
 
-# ============================================================================
-# Stage: builder-fromsource — self-contained build path.
-# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
-# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
-# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
-# default; local `make backends/llama-cpp`).
-#
-# The install script is the same one that backend/Dockerfile.base-grpc-builder
-# runs, so the result is bit-equivalent to the prebuilt-base path
-# (builder-prebuilt below).
-# ============================================================================
-FROM ${BASE_IMAGE} AS builder-fromsource
-ARG BUILD_TYPE
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
+# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
+# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
+FROM ${GRPC_BASE_IMAGE} AS grpc
+
+# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+
+ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
+
+WORKDIR /build
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        ca-certificates \
+        build-essential curl libssl-dev \
+        git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
+# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
+# and running make install in the target container
+RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
+    mkdir -p /build/grpc/cmake/build && \
+    cd /build/grpc/cmake/build && \
+    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
+    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
+    make && \
+    make install && \
+    rm -rf /build
+
+FROM ${BASE_IMAGE} AS builder
+ARG CMAKE_FROM_SOURCE=false
+ARG CMAKE_VERSION=3.31.10
+# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+ARG BACKEND=rerankers
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
 ARG SKIP_DRIVERS=false
+ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
+ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
+ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-ARG AMDGPU_TARGETS
-# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ARG CMAKE_ARGS
 
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
-    CMAKE_ARGS=${CMAKE_ARGS} \
-    DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ccache git \
+        ca-certificates \
+        make \
+        pkg-config libcurl4-openssl-dev \
+        curl unzip \
+        libssl-dev wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
 
-# CUDA on PATH (no-op when CUDA isn't installed)
+# Cuda
 ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
+
+# HipBLAS requirements
 ENV PATH=/opt/rocm/bin:${PATH}
 
-WORKDIR /build
 
-# Install everything via the shared script — the same one that
-# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
-# this from-source path are bit-equivalent.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
+# Vulkan requirements
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils wget gpg-agent && \
+        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
+            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            mkdir -p /opt/vulkan-sdk && \
+            mv 1.4.335.0 /opt/vulkan-sdk/ && \
+            cd /opt/vulkan-sdk/1.4.335.0 && \
+            ./vulkansdk --no-deps --maxjobs \
+                vulkan-loader \
+                vulkan-validationlayers \
+                vulkan-extensionlayer \
+                vulkan-tools \
+                shaderc && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
+            rm -rf /opt/vulkan-sdk
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            mkdir vulkan && cd vulkan && \
+            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
+            tar -xvf vulkan-sdk.tar.xz && \
+            rm vulkan-sdk.tar.xz && \
+            cd 1.4.335.0 && \
+            cp -rfv aarch64/bin/* /usr/bin/ && \
+            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
+            cp -rfv aarch64/include/* /usr/include/ && \
+            cp -rfv aarch64/share/* /usr/share/ && \
+            cd ../.. && \
+            rm -rf vulkan
+        fi
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# CuBLAS requirements
+RUN <<EOT bash
+    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
+            else
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
+            fi
+        fi
+        dpkg -i cuda-keyring_1.1-1_all.deb && \
+        rm -f cuda-keyring_1.1-1_all.deb && \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
+            apt-get install -y --no-install-recommends \
+            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        fi
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+
+# https://github.com/NVIDIA/Isaac-GR00T/issues/343
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
+        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
+        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get install -y nvpl
+    fi
+EOT
+
+# If we are building with clblas support, we need the libraries for the builds
+RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libclblast-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* \
+    ; fi
+
+RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            hipblas-dev \
+            rocblas-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* && \
+        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
+        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
+        ldconfig \
+    ; fi
+
+RUN echo "TARGETARCH: $TARGETARCH"
+
+# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
+# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
+# here so that we can generate the grpc code for the stablediffusion build
+RUN <<EOT bash
+    if [ "amd64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+    if [ "arm64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+EOT
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+COPY --from=grpc /opt/grpc /usr/local
 
-# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
-# CMake's find_package finds it at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
 
 COPY . /LocalAI
 
-# BuildKit cache mount for ccache. Persists compiler outputs across builds
-# via the registry cache (cache-to: type=registry,mode=max in CI). On a
-# LLAMA_VERSION bump most TUs are byte-identical to the previous version's
-# preprocessed source — ccache returns the previous .o file and skips the
-# real compile. Same for LocalAI source changes that don't touch llama.cpp.
-# CMAKE_*_COMPILER_LAUNCHER threads ccache through CMake to wrap gcc/g++/nvcc.
-# sharing=locked serializes concurrent writes if multiple matrix variants
-# share the same cache mount id.
-#
-# The compile body is shared with builder-prebuilt via .docker/llama-cpp-compile.sh.
-RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
+RUN <<'EOT' bash
+set -euxo pipefail
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
+fi
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  cd /LocalAI/backend/cpp/llama-cpp
+  make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
+else
+  cd /LocalAI/backend/cpp/llama-cpp
+  make llama-cpp-avx
+  make llama-cpp-avx2
+  make llama-cpp-avx512
+  make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
+fi
+EOT
 
 
 # Copy libraries using a script to handle architecture differences
 RUN make -BC /LocalAI/backend/cpp/llama-cpp package
 
 
-# ============================================================================
-# Stage: builder-prebuilt — uses the pre-built base from
-# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
-# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
-# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
-# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
-# builder-base-image).
-# ============================================================================
-FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
-
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-ARG AMDGPU_TARGETS
-ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
-# /usr/local. The variant Dockerfile's from-source path does that too;
-# mirror it here so the compile step can find gRPC at the canonical
-# prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-RUN make -BC /LocalAI/backend/cpp/llama-cpp package
-
-
-# ============================================================================
-# Final stage — copies package output from one of the two builders.
-# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
-#
-# BuildKit doesn't support variable expansion in `COPY --from=` directly,
-# so we resolve the ARG by aliasing the chosen builder to a fixed stage
-# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
-# BUILDER_TARGET itself is declared as a global ARG at the top of this
-# file (required for use in FROM), so we just re-import it into this
-# stage's scope before the FROM directive.
-# ============================================================================
-FROM ${BUILDER_TARGET} AS builder
-
 FROM scratch
 
 

backend/Dockerfile.python

@@ -1,6 +1,4 @@
 ARG BASE_IMAGE=ubuntu:24.04
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -15,12 +13,8 @@ ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
 
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
+RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         ccache \
@@ -35,7 +29,6 @@ RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mi
         curl python3-pip \
         python-is-python3 \
         python3-dev llvm \
-        libnuma1 libgomp1 \
         python3-venv make cmake && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
@@ -168,7 +161,6 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y --no-install-recommends \
             hipblas-dev \
-            hipblaslt-dev \
             rocblas-dev && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \
@@ -203,19 +195,6 @@ COPY backend/backend.proto /${BACKEND}/backend.proto
 COPY backend/python/common/ /${BACKEND}/common
 COPY scripts/build/package-gpu-libs.sh /package-gpu-libs.sh
 
-# Optional per-backend source build toggle (e.g. vllm on CPU can set
-# FROM_SOURCE=true to compile against the build host SIMD instead of
-# pulling a prebuilt wheel). Default empty — most backends ignore it.
-ARG FROM_SOURCE=""
-ENV FROM_SOURCE=${FROM_SOURCE}
-
-# Cache-buster for the per-backend `make` step. Most Python backends list
-# unpinned deps (torch, transformers, vllm, ...), so a warm registry cache
-# would otherwise freeze upstream versions indefinitely. CI passes a value
-# that rolls weekly so the install layer is rebuilt at most once per week
-# and picks up newer wheels from PyPI / nightly indexes.
-ARG DEPS_REFRESH=initial
-
 RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
 
 # Package GPU libraries into the backend's lib directory
@@ -230,4 +209,4 @@ RUN if [ -f "/${BACKEND}/package.sh" ]; then \
 
 FROM scratch
 ARG BACKEND=rerankers
-COPY --from=builder /${BACKEND}/ /
+COPY --from=builder /${BACKEND}/ /

backend/Dockerfile.rust

@@ -1,45 +0,0 @@
-ARG BASE_IMAGE=ubuntu:24.04
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
-
-FROM ${BASE_IMAGE} AS builder
-ARG BACKEND=kokoros
-ENV DEBIAN_FRONTEND=noninteractive
-ARG TARGETARCH
-ARG TARGETVARIANT
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-
-RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
-    apt-get update && \
-    apt-get install -y --no-install-recommends \
-        build-essential \
-        git ccache \
-        ca-certificates \
-        make cmake wget \
-        curl unzip \
-        clang \
-        pkg-config \
-        libssl-dev \
-        espeak-ng libespeak-ng-dev \
-        libsonic-dev libpcaudio-dev \
-        libopus-dev \
-        protobuf-compiler && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
-
-# Install Rust
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
-ENV PATH="/root/.cargo/bin:${PATH}"
-
-COPY . /LocalAI
-
-RUN git config --global --add safe.directory /LocalAI
-
-RUN make -C /LocalAI/backend/rust/${BACKEND} build
-
-FROM scratch
-ARG BACKEND=kokoros
-
-COPY --from=builder /LocalAI/backend/rust/${BACKEND}/package/. ./

backend/Dockerfile.turboquant

@@ -1,160 +0,0 @@
-ARG BASE_IMAGE=ubuntu:24.04
-# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
-# when no prebuilt base is supplied. The builder-prebuilt stage is only
-# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
-# content here is harmless — BuildKit prunes the unreferenced builder.
-ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
-# BUILDER_TARGET selects which builder stage the final scratch image copies
-# package output from. Declared at global scope (before any FROM) so it's
-# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
-# `make backends/turboquant` on the from-source path.
-ARG BUILDER_TARGET=builder-fromsource
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
-
-
-# ============================================================================
-# Stage: builder-fromsource — self-contained build path.
-# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
-# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
-# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
-# default; local `make backends/turboquant`).
-#
-# The install script is the same one that backend/Dockerfile.base-grpc-builder
-# runs, so the result is bit-equivalent to the prebuilt-base path
-# (builder-prebuilt below).
-# ============================================================================
-FROM ${BASE_IMAGE} AS builder-fromsource
-ARG BUILD_TYPE
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
-ARG CMAKE_FROM_SOURCE=false
-# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
-ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG SKIP_DRIVERS=false
-ARG TARGETARCH
-ARG TARGETVARIANT
-ARG GO_VERSION=1.25.4
-ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-ARG AMDGPU_TARGETS=""
-ARG BACKEND=rerankers
-# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ARG CMAKE_ARGS
-
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
-    CMAKE_ARGS=${CMAKE_ARGS} \
-    DEBIAN_FRONTEND=noninteractive
-
-# CUDA on PATH (no-op when CUDA isn't installed)
-ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
-ENV PATH=/opt/rocm/bin:${PATH}
-
-WORKDIR /build
-
-# Install everything via the shared script — the same one that
-# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
-# this from-source path are bit-equivalent.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
-
-# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
-# CMake's find_package finds it at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
-# for rationale. turboquant is a llama.cpp fork that reuses
-# backend/cpp/llama-cpp source via a thin wrapper Makefile, so MOST TUs
-# are content-identical to the upstream llama-cpp build. Sharing a cache
-# id with llama-cpp could give cross-fork hits — but for now keep them
-# separate so a regression in one doesn't poison the other. Revisit
-# sharing after measuring the actual hit rate.
-#
-# The compile body is shared with builder-prebuilt via .docker/turboquant-compile.sh.
-RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-
-# Copy libraries using a script to handle architecture differences
-RUN make -BC /LocalAI/backend/cpp/turboquant package
-
-
-# ============================================================================
-# Stage: builder-prebuilt — uses the pre-built base from
-# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
-# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
-# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
-# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
-# builder-base-image).
-# ============================================================================
-FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
-
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-# AMDGPU_TARGETS must be forwarded into the env here too — backend/cpp/llama-cpp/Makefile
-# (which the turboquant Makefile reuses via a sibling build dir) errors out when the var
-# is empty on a hipblas build, and the prebuilt path is what CI exercises most of the
-# time. The builder-fromsource stage above already does this; mirror it here.
-ARG AMDGPU_TARGETS
-ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
-# /usr/local. Mirror what the from-source path does so the compile step
-# can find gRPC at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-RUN make -BC /LocalAI/backend/cpp/turboquant package
-
-
-# ============================================================================
-# Final stage — copies package output from one of the two builders.
-# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
-#
-# BuildKit doesn't support variable expansion in `COPY --from=` directly,
-# so we resolve the ARG by aliasing the chosen builder to a fixed stage
-# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
-# BUILDER_TARGET itself is declared as a global ARG at the top of this
-# file (required for use in FROM), so we just re-import it into this
-# stage's scope before the FROM directive.
-# ============================================================================
-FROM ${BUILDER_TARGET} AS builder
-
-FROM scratch
-
-
-# Copy all available binaries (the build process only creates the appropriate ones for the target architecture)
-COPY --from=builder /LocalAI/backend/cpp/turboquant/package/. ./

backend/backend.proto

@@ -17,18 +17,12 @@ service Backend {
   rpc GenerateImage(GenerateImageRequest) returns (Result) {}
   rpc GenerateVideo(GenerateVideoRequest) returns (Result) {}
   rpc AudioTranscription(TranscriptRequest) returns (TranscriptResult) {}
-  rpc AudioTranscriptionStream(TranscriptRequest) returns (stream TranscriptStreamResponse) {}
   rpc TTS(TTSRequest) returns (Result) {}
   rpc TTSStream(TTSRequest) returns (stream Reply) {}
   rpc SoundGeneration(SoundGenerationRequest) returns (Result) {}
   rpc TokenizeString(PredictOptions) returns (TokenizationResponse) {}
   rpc Status(HealthMessage) returns (StatusResponse) {}
   rpc Detect(DetectOptions) returns (DetectResponse) {}
-  rpc FaceVerify(FaceVerifyRequest) returns (FaceVerifyResponse) {}
-  rpc FaceAnalyze(FaceAnalyzeRequest) returns (FaceAnalyzeResponse) {}
-  rpc VoiceVerify(VoiceVerifyRequest) returns (VoiceVerifyResponse) {}
-  rpc VoiceAnalyze(VoiceAnalyzeRequest) returns (VoiceAnalyzeResponse) {}
-  rpc VoiceEmbed(VoiceEmbedRequest) returns (VoiceEmbedResponse) {}
 
   rpc StoresSet(StoresSetOptions) returns (Result) {}
   rpc StoresDelete(StoresDeleteOptions) returns (Result) {}
@@ -41,19 +35,9 @@ service Backend {
 
   rpc VAD(VADRequest) returns (VADResponse) {}
 
-  rpc Diarize(DiarizeRequest) returns (DiarizeResponse) {}
-
   rpc AudioEncode(AudioEncodeRequest) returns (AudioEncodeResult) {}
   rpc AudioDecode(AudioDecodeRequest) returns (AudioDecodeResult) {}
 
-  rpc AudioTransform(AudioTransformRequest) returns (AudioTransformResult) {}
-  rpc AudioTransformStream(stream AudioTransformFrameRequest) returns (stream AudioTransformFrameResponse) {}
-  // AudioToAudioStream is the bidirectional any-to-any S2S RPC. Backends
-  // that load a speech-to-speech model consume input audio frames and emit
-  // interleaved audio + transcript + tool-call deltas as typed events.
-  // Backends without S2S support return UNIMPLEMENTED.
-  rpc AudioToAudioStream(stream AudioToAudioRequest) returns (stream AudioToAudioResponse) {}
-
   rpc ModelMetadata(ModelOptions) returns (ModelMetadataResponse) {}
 
   // Fine-tuning RPCs
@@ -62,12 +46,6 @@ service Backend {
   rpc StopFineTune(FineTuneStopRequest) returns (Result) {}
   rpc ListCheckpoints(ListCheckpointsRequest) returns (ListCheckpointsResponse) {}
   rpc ExportModel(ExportModelRequest) returns (Result) {}
-
-  // Quantization RPCs
-  rpc StartQuantization(QuantizationRequest) returns (QuantizationJobResult) {}
-  rpc QuantizationProgress(QuantizationProgressRequest) returns (stream QuantizationProgressUpdate) {}
-  rpc StopQuantization(QuantizationStopRequest) returns (Result) {}
-
 }
 
 // Define the empty request
@@ -195,7 +173,6 @@ message PredictOptions {
   int32 Logprobs = 50;  // Number of top logprobs to return (maps to OpenAI logprobs parameter)
   int32 TopLogprobs = 51;  // Number of top logprobs to return per token (maps to OpenAI top_logprobs parameter)
   map<string, string> Metadata = 52;  // Generic per-request metadata (e.g., enable_thinking)
-  float MinP = 53;  // Minimum probability sampling threshold (0.0 = disabled)
 }
 
 // ToolCallDelta represents an incremental tool call update from the C++ parser.
@@ -320,11 +297,6 @@ message ModelOptions {
   bool Reranking = 71;
 
   repeated string Overrides = 72;
-
-  // EngineArgs carries a JSON-encoded map of backend-native engine arguments
-  // applied verbatim to the backend's engine constructor (e.g. vLLM AsyncEngineArgs).
-  // Unknown keys produce an error at LoadModel time.
-  string EngineArgs = 73;
 }
 
 message Result {
@@ -343,27 +315,11 @@ message TranscriptRequest {
   bool translate = 5;
   bool diarize = 6;
   string prompt = 7;
-  float temperature = 8;
-  repeated string timestamp_granularities = 9;
-  bool stream = 10;
 }
 
 message TranscriptResult {
   repeated TranscriptSegment segments = 1;
   string text = 2;
-  string language = 3;
-  float duration = 4;
-}
-
-message TranscriptStreamResponse {
-  string delta = 1;
-  TranscriptResult final_result = 2;
-}
-
-message TranscriptWord {
-  int64 start = 1;
-  int64 end = 2;
-  string text = 3;
 }
 
 message TranscriptSegment {
@@ -373,7 +329,6 @@ message TranscriptSegment {
   string text = 4;
   repeated int32 tokens = 5;
   string speaker = 6;
-  repeated TranscriptWord words = 7;
 }
 
 message GenerateImageRequest {
@@ -430,43 +385,6 @@ message VADResponse {
   repeated VADSegment segments = 1;
 }
 
-// --- Speaker diarization messages ---
-//
-// Pure speaker diarization: "who spoke when". Returns time-stamped segments
-// labelled with cluster IDs (the same string for the same speaker across
-// segments). Some backends (e.g. vibevoice.cpp) produce diarization as a
-// by-product of ASR and may also fill in `text` per segment; backends with a
-// dedicated diarization pipeline (e.g. sherpa-onnx pyannote) leave `text`
-// empty and emit only the segmentation.
-
-message DiarizeRequest {
-  string dst = 1;                      // path to audio file (HTTP layer materialises uploads to a temp file)
-  uint32 threads = 2;
-  string language = 3;                 // optional; only meaningful for transcription-bundling backends
-  int32  num_speakers = 4;             // exact speaker count if known (>0 forces); 0 = auto
-  int32  min_speakers = 5;             // hint when auto-detecting; 0 = unset
-  int32  max_speakers = 6;             // hint when auto-detecting; 0 = unset
-  float  clustering_threshold = 7;     // distance threshold when num_speakers unknown; 0 = backend default
-  float  min_duration_on = 8;          // discard segments shorter than this (seconds); 0 = backend default
-  float  min_duration_off = 9;         // merge gaps shorter than this (seconds); 0 = backend default
-  bool   include_text = 10;            // when the backend can emit per-segment transcript for free, ask it to populate `text`
-}
-
-message DiarizeSegment {
-  int32  id = 1;
-  float  start = 2;                    // seconds
-  float  end = 3;                      // seconds
-  string speaker = 4;                  // backend-emitted speaker label (e.g. "0", "SPEAKER_00")
-  string text = 5;                     // optional per-segment transcript (empty unless include_text and supported)
-}
-
-message DiarizeResponse {
-  repeated DiarizeSegment segments = 1;
-  int32  num_speakers = 2;             // count of distinct speaker labels in `segments`
-  float  duration = 3;                 // total audio duration in seconds (0 if unknown)
-  string language = 4;                 // optional, when the backend bundles transcription
-}
-
 message SoundGenerationRequest {
   string text = 1;
   string model = 2;
@@ -519,10 +437,6 @@ message Message {
 
 message DetectOptions {
   string src = 1;
-  string prompt = 2;           // Text prompt (for SAM 3 PCS mode)
-  repeated float points = 3;   // Point coordinates as [x1, y1, label1, x2, y2, label2, ...] (label: 1=pos, 0=neg)
-  repeated float boxes = 4;    // Box coordinates as [x1, y1, x2, y2, ...]
-  float threshold = 5;         // Detection confidence threshold
 }
 
 message Detection {
@@ -532,119 +446,12 @@ message Detection {
   float height = 4;
   float confidence = 5;
   string class_name = 6;
-  bytes mask = 7;              // PNG-encoded binary segmentation mask
 }
 
 message DetectResponse {
   repeated Detection Detections = 1;
 }
 
-// --- Face recognition messages ---
-
-message FacialArea {
-  float x = 1;
-  float y = 2;
-  float w = 3;
-  float h = 4;
-}
-
-message FaceVerifyRequest {
-  string img1 = 1;              // base64-encoded image
-  string img2 = 2;              // base64-encoded image
-  float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
-  bool   anti_spoofing = 4;     // run MiniFASNet liveness on each image; failed liveness forces verified=false
-}
-
-message FaceVerifyResponse {
-  bool       verified = 1;
-  float      distance = 2;      // 1 - cosine_similarity
-  float      threshold = 3;
-  float      confidence = 4;    // 0-100
-  string     model = 5;         // e.g. "buffalo_l"
-  FacialArea img1_area = 6;
-  FacialArea img2_area = 7;
-  float      processing_time_ms = 8;
-  bool       img1_is_real = 9;          // anti-spoofing result when enabled
-  float      img1_antispoof_score = 10;
-  bool       img2_is_real = 11;
-  float      img2_antispoof_score = 12;
-}
-
-message FaceAnalyzeRequest {
-  string          img = 1;          // base64-encoded image
-  repeated string actions = 2;      // subset of ["age","gender","emotion","race"]; empty = all-supported
-  bool            anti_spoofing = 3;
-}
-
-message FaceAnalysis {
-  FacialArea         region = 1;
-  float              face_confidence = 2;
-  float              age = 3;
-  string             dominant_gender = 4;   // "Man" | "Woman"
-  map<string, float> gender = 5;
-  string             dominant_emotion = 6;  // reserved; empty in MVP
-  map<string, float> emotion = 7;
-  string             dominant_race = 8;     // not populated
-  map<string, float> race = 9;
-  bool               is_real = 10;          // anti-spoofing result when enabled
-  float              antispoof_score = 11;
-}
-
-message FaceAnalyzeResponse {
-  repeated FaceAnalysis faces = 1;
-}
-
-// --- Voice (speaker) recognition messages ---
-//
-// Analogous to the Face* messages above, but for speaker biometrics.
-// Audio fields accept a filesystem path (same convention as
-// TranscriptRequest.dst). The HTTP layer materialises base64 / URL /
-// data-URI inputs to a temp file before calling the gRPC backend.
-
-message VoiceVerifyRequest {
-  string audio1 = 1;            // path to first audio clip
-  string audio2 = 2;            // path to second audio clip
-  float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
-  bool   anti_spoofing = 4;     // reserved for future AASIST bolt-on
-}
-
-message VoiceVerifyResponse {
-  bool   verified = 1;
-  float  distance = 2;          // 1 - cosine_similarity
-  float  threshold = 3;
-  float  confidence = 4;        // 0-100
-  string model = 5;             // e.g. "speechbrain/spkrec-ecapa-voxceleb"
-  float  processing_time_ms = 6;
-}
-
-message VoiceAnalyzeRequest {
-  string          audio = 1;        // path to audio clip
-  repeated string actions = 2;      // subset of ["age","gender","emotion"]; empty = all-supported
-}
-
-message VoiceAnalysis {
-  float              start = 1;          // segment start time in seconds (0 if single-utterance)
-  float              end = 2;            // segment end time in seconds
-  float              age = 3;
-  string             dominant_gender = 4;
-  map<string, float> gender = 5;
-  string             dominant_emotion = 6;
-  map<string, float> emotion = 7;
-}
-
-message VoiceAnalyzeResponse {
-  repeated VoiceAnalysis segments = 1;
-}
-
-message VoiceEmbedRequest {
-  string audio = 1;              // path to audio clip
-}
-
-message VoiceEmbedResponse {
-  repeated float embedding = 1;
-  string         model = 2;
-}
-
 message ToolFormatMarkers {
   string format_type = 1;           // "json_native", "tag_with_json", "tag_with_tagged"
 
@@ -672,7 +479,7 @@ message ToolFormatMarkers {
   string id_field = 16;            // e.g., "id"
   bool fun_name_is_key = 17;
   bool tools_array_wrapped = 18;
-  reserved 19;
+  bool uses_python_dicts = 19;
 
   // Reasoning markers
   string reasoning_start = 20;     // e.g., "<think>"
@@ -723,148 +530,10 @@ message AudioDecodeResult {
   int32 samples_per_frame = 3;
 }
 
-// Generic audio transform: an audio-in, audio-out operation, optionally
-// conditioned on a second reference signal. Concrete transforms include
-// AEC + noise suppression + dereverberation (LocalVQE), voice conversion
-// (reference = target speaker), pitch shifting, etc.
-message AudioTransformRequest {
-  string audio_path = 1;             // required, primary input file path
-  string reference_path = 2;         // optional auxiliary; empty => zero-fill
-  string dst = 3;                    // required, output file path
-  map<string, string> params = 4;    // backend-specific tuning
-}
-
-message AudioTransformResult {
-  string dst = 1;
-  int32  sample_rate = 2;
-  int32  samples = 3;
-  bool   reference_provided = 4;
-}
-
-// Bidirectional streaming audio transform. The first message MUST carry a
-// Config; subsequent messages carry Frames. A second Config mid-stream
-// resets streaming state before the next frame.
-message AudioTransformFrameRequest {
-  oneof payload {
-    AudioTransformStreamConfig config = 1;
-    AudioTransformFrame        frame  = 2;
-  }
-}
-
-message AudioTransformStreamConfig {
-  enum SampleFormat {
-    F32_LE = 0;
-    S16_LE = 1;
-  }
-  SampleFormat sample_format = 1;
-  int32 sample_rate = 2;             // 0 => backend default
-  int32 frame_samples = 3;           // 0 => backend default
-  map<string, string> params = 4;
-  bool reset = 5;                    // reset streaming state before next frame
-}
-
-message AudioTransformFrame {
-  bytes audio_pcm = 1;               // frame_samples samples in stream's format
-  bytes reference_pcm = 2;           // empty => zero-fill (silent reference)
-}
-
-message AudioTransformFrameResponse {
-  bytes pcm = 1;
-  int64 frame_index = 2;
-}
-
-// === AudioToAudioStream messages =========================================
-//
-// Bidirectional stream between the LocalAI core and an any-to-any audio
-// model. The client opens the stream with a Config payload, then alternates
-// Frame (input audio) and Control (turn boundaries, function-call results,
-// session updates) payloads. The server streams back typed events: audio
-// frames carry PCM in `pcm`; transcript / tool-call deltas carry JSON in
-// `meta`; the stream ends with a `response.done` (success) or `error` event.
-
-message AudioToAudioRequest {
-  oneof payload {
-    AudioToAudioConfig  config  = 1;
-    AudioToAudioFrame   frame   = 2;
-    AudioToAudioControl control = 3;
-  }
-}
-
-message AudioToAudioConfig {
-  // PCM format for client→server audio. 0 => backend default
-  // (16 kHz for the LFM2-Audio Conformer encoder).
-  int32 input_sample_rate = 1;
-  // Preferred server→client audio rate. 0 => backend default
-  // (24 kHz for the LFM2-Audio vocoder).
-  int32 output_sample_rate = 2;
-  // Optional system prompt override. Empty => backend chooses based on
-  // mode (e.g. "Respond with interleaved text and audio.").
-  string system_prompt = 3;
-  // Optional baked-voice id. Models that only ship a fixed set of
-  // voices (e.g. LFM2-Audio: us_male/us_female/uk_male/uk_female) match
-  // this against their voice table; an empty string keeps the default.
-  string voice = 4;
-  // JSON-encoded array of tool definitions in OpenAI Chat Completions
-  // format. Empty => no tools.
-  string tools = 5;
-  // Free-form sampling / decoding parameters (temperature, top_k,
-  // max_new_tokens, audio_top_k, etc).
-  map<string, string> params = 6;
-  // True => reset any session-scoped state before processing further
-  // frames on this stream. The first Config implicitly resets.
-  bool reset = 7;
-}
-
-message AudioToAudioFrame {
-  // Raw PCM s16le mono at config.input_sample_rate. Empty pcm + end_of_input
-  // is a valid "user finished speaking" marker without trailing audio.
-  bytes pcm = 1;
-  // Marks the last frame of a user turn. The backend may begin emitting
-  // a response immediately after seeing this.
-  bool end_of_input = 2;
-}
-
-message AudioToAudioControl {
-  // Free-form control event names. Initial set:
-  //   "input_audio_buffer.commit"     — user finished speaking
-  //   "response.cancel"               — abort in-flight generation
-  //   "conversation.item.create"      — inject a non-audio item (e.g.
-  //                                     function_call_output as JSON in
-  //                                     `payload`)
-  //   "session.update"                — re-configure mid-stream
-  string event = 1;
-  // Event-specific JSON payload.
-  bytes payload = 2;
-}
-
-message AudioToAudioResponse {
-  // Event identifies what this frame carries. Mirrors the OpenAI Realtime
-  // API server-event names where applicable. Initial set:
-  //   "response.audio.delta"
-  //   "response.audio_transcript.delta"
-  //   "response.function_call_arguments.delta"
-  //   "response.function_call_arguments.done"
-  //   "response.done"
-  //   "error"
-  string event = 1;
-  // Populated when event = response.audio.delta.
-  bytes pcm = 2;
-  // Populated alongside pcm to identify its rate. 0 => same as the
-  // session's negotiated output_sample_rate.
-  int32 sample_rate = 3;
-  // JSON payload for non-PCM events (transcript chunk, tool args, error
-  // body).
-  bytes meta = 4;
-  // Monotonic per-stream counter, useful for client reordering and
-  // debugging.
-  int64 sequence = 5;
-}
-
 message ModelMetadataResponse {
   bool supports_thinking = 1;
   string rendered_template = 2;  // The rendered chat template with enable_thinking=true (empty if not applicable)
   ToolFormatMarkers tool_format = 3;  // Auto-detected tool format markers from differential template analysis
-  string media_marker = 4;  // Marker the backend expects in the prompt for each multimodal input (images/audio/video). Empty when the backend does not use a marker.
 }
 
 // Fine-tuning messages
@@ -968,37 +637,3 @@ message ExportModelRequest {
   string model = 5;                   // base model name (for merge operations)
   map<string, string> extra_options = 6;
 }
-
-// Quantization messages
-
-message QuantizationRequest {
-  string model = 1;                      // HF model name or local path
-  string quantization_type = 2;          // q4_k_m, q5_k_m, q8_0, f16, etc.
-  string output_dir = 3;                 // where to write output files
-  string job_id = 4;                     // client-assigned job ID
-  map<string, string> extra_options = 5; // hf_token, custom flags, etc.
-}
-
-message QuantizationJobResult {
-  string job_id = 1;
-  bool success = 2;
-  string message = 3;
-}
-
-message QuantizationProgressRequest {
-  string job_id = 1;
-}
-
-message QuantizationProgressUpdate {
-  string job_id = 1;
-  float progress_percent = 2;
-  string status = 3;                     // queued, downloading, converting, quantizing, completed, failed, stopped
-  string message = 4;
-  string output_file = 5;               // set when completed — path to the output GGUF file
-  map<string, float> extra_metrics = 6;  // e.g. file_size_mb, compression_ratio
-}
-
-message QuantizationStopRequest {
-  string job_id = 1;
-}
-

backend/cpp/ds4/.gitignore

@@ -1,9 +0,0 @@
-ds4/
-build/
-package/
-grpc-server
-*.o
-backend.pb.cc
-backend.pb.h
-backend.grpc.pb.cc
-backend.grpc.pb.h

backend/cpp/ds4/CMakeLists.txt

@@ -1,101 +0,0 @@
-cmake_minimum_required(VERSION 3.15)
-project(ds4-grpc-server LANGUAGES CXX C)
-
-set(CMAKE_CXX_STANDARD 17)
-set(CMAKE_CXX_STANDARD_REQUIRED ON)
-set(TARGET grpc-server)
-
-option(DS4_NATIVE "Compile with -march=native / -mcpu=native" ON)
-set(DS4_GPU "cpu" CACHE STRING "GPU backend: cpu, cuda, or metal")
-set(DS4_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ds4" CACHE PATH "Path to cloned ds4 source")
-
-find_package(Threads REQUIRED)
-find_package(Protobuf CONFIG QUIET)
-if(NOT Protobuf_FOUND)
-    find_package(Protobuf REQUIRED)
-endif()
-find_package(gRPC CONFIG QUIET)
-if(NOT gRPC_FOUND)
-    # Ubuntu's apt-installed grpc++ does not ship a CMake config - fall back.
-    find_library(GRPCPP_LIB grpc++ REQUIRED)
-    find_library(GRPCPP_REFLECTION_LIB grpc++_reflection REQUIRED)
-    add_library(gRPC::grpc++ INTERFACE IMPORTED)
-    set_target_properties(gRPC::grpc++ PROPERTIES INTERFACE_LINK_LIBRARIES "${GRPCPP_LIB}")
-    add_library(gRPC::grpc++_reflection INTERFACE IMPORTED)
-    set_target_properties(gRPC::grpc++_reflection PROPERTIES INTERFACE_LINK_LIBRARIES "${GRPCPP_REFLECTION_LIB}")
-endif()
-
-find_program(_PROTOC NAMES protoc REQUIRED)
-find_program(_GRPC_CPP_PLUGIN NAMES grpc_cpp_plugin REQUIRED)
-
-get_filename_component(HW_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/../../backend.proto" ABSOLUTE)
-get_filename_component(HW_PROTO_PATH "${HW_PROTO}" PATH)
-
-set(HW_PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.cc")
-set(HW_PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.h")
-set(HW_GRPC_SRCS  "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.cc")
-set(HW_GRPC_HDRS  "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.h")
-
-add_custom_command(
-    OUTPUT "${HW_PROTO_SRCS}" "${HW_PROTO_HDRS}" "${HW_GRPC_SRCS}" "${HW_GRPC_HDRS}"
-    COMMAND ${_PROTOC}
-    ARGS --grpc_out "${CMAKE_CURRENT_BINARY_DIR}"
-         --cpp_out  "${CMAKE_CURRENT_BINARY_DIR}"
-         -I "${HW_PROTO_PATH}"
-         --plugin=protoc-gen-grpc="${_GRPC_CPP_PLUGIN}"
-         "${HW_PROTO}"
-    DEPENDS "${HW_PROTO}")
-
-add_library(hw_grpc_proto STATIC
-    ${HW_GRPC_SRCS} ${HW_GRPC_HDRS}
-    ${HW_PROTO_SRCS} ${HW_PROTO_HDRS})
-target_include_directories(hw_grpc_proto PUBLIC ${CMAKE_CURRENT_BINARY_DIR})
-
-set(DS4_OBJS "${DS4_DIR}/ds4.o")
-if(DS4_GPU STREQUAL "cuda")
-    list(APPEND DS4_OBJS "${DS4_DIR}/ds4_cuda.o")
-elseif(DS4_GPU STREQUAL "metal")
-    list(APPEND DS4_OBJS "${DS4_DIR}/ds4_metal.o")
-elseif(DS4_GPU STREQUAL "cpu")
-    set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
-endif()
-
-add_executable(${TARGET}
-    grpc-server.cpp
-    dsml_parser.cpp
-    dsml_renderer.cpp
-    kv_cache.cpp)
-
-target_include_directories(${TARGET} PRIVATE ${DS4_DIR})
-
-foreach(obj ${DS4_OBJS})
-    target_sources(${TARGET} PRIVATE ${obj})
-    set_source_files_properties(${obj} PROPERTIES EXTERNAL_OBJECT TRUE GENERATED TRUE)
-endforeach()
-
-target_link_libraries(${TARGET} PRIVATE
-    hw_grpc_proto
-    gRPC::grpc++
-    gRPC::grpc++_reflection
-    protobuf::libprotobuf
-    Threads::Threads
-    m)
-
-if(DS4_GPU STREQUAL "cuda")
-    find_package(CUDAToolkit REQUIRED)
-    target_link_libraries(${TARGET} PRIVATE CUDA::cudart CUDA::cublas)
-elseif(DS4_GPU STREQUAL "metal")
-    find_library(FOUNDATION_LIB Foundation REQUIRED)
-    find_library(METAL_LIB Metal REQUIRED)
-    target_link_libraries(${TARGET} PRIVATE ${FOUNDATION_LIB} ${METAL_LIB})
-elseif(DS4_GPU STREQUAL "cpu")
-    target_compile_definitions(${TARGET} PRIVATE DS4_NO_GPU)
-endif()
-
-if(DS4_NATIVE)
-    if(APPLE)
-        target_compile_options(${TARGET} PRIVATE -mcpu=native)
-    else()
-        target_compile_options(${TARGET} PRIVATE -march=native)
-    endif()
-endif()

backend/cpp/ds4/Makefile

@@ -1,78 +0,0 @@
-# ds4 backend Makefile.
-#
-# Upstream pin lives below as DS4_VERSION?=950e8e6474a1c9fabe04e669d607606a7ef8824f
-# (.github/bump_deps.sh) can find and update it - matches the
-# llama-cpp / ik-llama-cpp / turboquant convention.
-
-DS4_VERSION?=950e8e6474a1c9fabe04e669d607606a7ef8824f
-DS4_REPO?=https://github.com/antirez/ds4
-
-CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
-BUILD_DIR := build
-
-BUILD_TYPE ?=
-NATIVE ?= false
-JOBS ?= $(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
-
-UNAME_S := $(shell uname -s)
-
-CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
-
-ifeq ($(BUILD_TYPE),cublas)
-    CMAKE_ARGS += -DDS4_GPU=cuda
-    DS4_OBJ_TARGET := ds4.o ds4_cuda.o
-else ifeq ($(UNAME_S),Darwin)
-    CMAKE_ARGS += -DDS4_GPU=metal
-    DS4_OBJ_TARGET := ds4.o ds4_metal.o
-else
-    # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
-    CMAKE_ARGS += -DDS4_GPU=cpu
-    DS4_OBJ_TARGET := ds4_cpu.o
-endif
-
-ifneq ($(NATIVE),true)
-    CMAKE_ARGS += -DDS4_NATIVE=OFF
-endif
-
-.PHONY: grpc-server package clean purge test all
-all: grpc-server
-
-# Clone the upstream ds4 source at the pinned commit. Directory acts as the
-# target so make only re-clones when missing. After a DS4_VERSION bump,
-# run 'make purge && make' to refetch (or rely on CI's clean build).
-ds4:
-	mkdir -p ds4
-	cd ds4 && \
-	git init -q && \
-	git remote add origin $(DS4_REPO) && \
-	git fetch --depth 1 origin $(DS4_VERSION) && \
-	git checkout FETCH_HEAD
-
-# Build ds4's engine object files via its own Makefile, which already encodes
-# the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
-ds4/ds4.o: ds4
-ifeq ($(BUILD_TYPE),cublas)
-	+$(MAKE) -C ds4 ds4.o ds4_cuda.o
-else ifeq ($(UNAME_S),Darwin)
-	+$(MAKE) -C ds4 ds4.o ds4_metal.o
-else
-	+$(MAKE) -C ds4 ds4_cpu.o
-endif
-
-grpc-server: ds4/ds4.o
-	mkdir -p $(BUILD_DIR)
-	cd $(BUILD_DIR) && cmake $(CMAKE_ARGS) $(CURRENT_MAKEFILE_DIR) && cmake --build . --config Release -j $(JOBS)
-	cp $(BUILD_DIR)/grpc-server grpc-server
-
-package: grpc-server
-	bash package.sh
-
-test:
-	@echo "ds4 backend: e2e coverage at tests/e2e-backends/ (BACKEND_BINARY mode)"
-
-clean:
-	rm -rf $(BUILD_DIR) grpc-server package
-	if [ -d ds4 ]; then $(MAKE) -C ds4 clean; fi
-
-purge: clean
-	rm -rf ds4

backend/cpp/ds4/dsml_parser.cpp

@@ -1,359 +0,0 @@
-#include "dsml_parser.h"
-
-#include <algorithm>
-#include <cstdio>
-#include <cstring>
-#include <chrono>
-#include <random>
-#include <string>
-#include <vector>
-
-namespace ds4cpp {
-
-namespace {
-
-constexpr const char *kThinkOpen      = "<think>";
-constexpr const char *kThinkClose     = "</think>";
-constexpr const char *kToolsOpen      = "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>";   // <｜DSML｜tool_calls>
-constexpr const char *kToolsClose     = "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>"; // </｜DSML｜tool_calls>
-constexpr const char *kInvokeOpenPfx  = "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\""; // <｜DSML｜invoke name="
-constexpr const char *kInvokeClose    = "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>";       // </｜DSML｜invoke>
-constexpr const char *kParamOpenPfx   = "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter name=\""; // <｜DSML｜parameter name="
-constexpr const char *kParamClose     = "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>";       // </｜DSML｜parameter>
-
-// All structural markers the parser might encounter - used to detect "buf
-// might be a partial marker, don't drain yet" conditions.
-const std::vector<std::string> &all_markers() {
-    static const std::vector<std::string> v = {
-        kThinkOpen, kThinkClose,
-        kToolsOpen, kToolsClose,
-        kInvokeOpenPfx, kInvokeClose,
-        kParamOpenPfx, kParamClose,
-    };
-    return v;
-}
-
-// Returns true if `buf` could be a *prefix* of any marker (i.e., we should
-// wait for more text before draining as plain content). The marker-prefix
-// loop handles fixed markers exactly. For markers with variable-length
-// internal data (kInvokeOpenPfx, kParamOpenPfx have an open quote, then the
-// tool/param name, then a closing quote and `>`), we also wait while buf
-// starts with `<` and has not yet seen a `>`: the leading `<` could be the
-// start of one of those open markers, or a literal that we can confirm only
-// once we know what follows. Anything after the first `>` arrives is either
-// consumed by TryConsumeMarker or emitted as a literal `<` by the caller.
-bool looks_like_prefix(const std::string &buf) {
-    for (const auto &m : all_markers()) {
-        if (m.size() > buf.size() && m.compare(0, buf.size(), buf) == 0) return true;
-    }
-    if (!buf.empty() && buf[0] == '<' && buf.find('>') == std::string::npos) {
-        return true;
-    }
-    return false;
-}
-
-bool consume_literal(std::string &buf, const std::string &lit) {
-    if (buf.compare(0, lit.size(), lit) == 0) {
-        buf.erase(0, lit.size());
-        return true;
-    }
-    return false;
-}
-
-// Find the next '<' in buf starting at offset; returns std::string::npos if none.
-size_t next_tag(const std::string &buf, size_t off = 0) {
-    return buf.find('<', off);
-}
-
-std::string json_escape(const std::string &in) {
-    std::string out;
-    out.reserve(in.size() + 2);
-    for (char c : in) {
-        switch (c) {
-            case '"':  out += "\\\""; break;
-            case '\\': out += "\\\\"; break;
-            case '\b': out += "\\b"; break;
-            case '\f': out += "\\f"; break;
-            case '\n': out += "\\n"; break;
-            case '\r': out += "\\r"; break;
-            case '\t': out += "\\t"; break;
-            default:
-                if (static_cast<unsigned char>(c) < 0x20) {
-                    char tmp[8];
-                    std::snprintf(tmp, sizeof(tmp), "\\u%04x", c);
-                    out += tmp;
-                } else {
-                    out += c;
-                }
-        }
-    }
-    return out;
-}
-
-} // namespace
-
-DsmlParser::DsmlParser() = default;
-
-bool DsmlParser::IsInDsmlStructural() const {
-    switch (state_) {
-        case State::TOOL_CALLS:
-        case State::INVOKE:
-            return true;
-        case State::PARAM_VALUE:  // payload bytes; user sampling applies
-        case State::TEXT:
-        case State::THINK:
-            return false;
-    }
-    return false;
-}
-
-void DsmlParser::EmitArgsChunk(const std::string &chunk, std::vector<ParserEvent> &out) {
-    if (chunk.empty()) return;
-    ParserEvent e;
-    e.type = ParserEvent::TOOL_ARGS;
-    e.text = chunk;
-    e.index = tool_index_;
-    out.push_back(std::move(e));
-}
-
-void DsmlParser::FinishCurrentToolCall(std::vector<ParserEvent> &out) {
-    if (tool_index_ < 0) return;
-    // Close the JSON object that was opened on the first parameter.
-    if (args_emitted_open_brace_) {
-        EmitArgsChunk("}", out);
-    } else {
-        EmitArgsChunk("{}", out);
-    }
-    ParserEvent e;
-    e.type = ParserEvent::TOOL_END;
-    e.index = tool_index_;
-    out.push_back(std::move(e));
-    current_tool_name_.clear();
-    args_emitted_open_brace_ = false;
-    args_param_count_ = 0;
-}
-
-bool DsmlParser::TryConsumeMarker(std::vector<ParserEvent> &out) {
-    switch (state_) {
-    case State::TEXT: {
-        if (consume_literal(buf_, kThinkOpen))   { state_ = State::THINK;       return true; }
-        if (consume_literal(buf_, kToolsOpen))   { state_ = State::TOOL_CALLS;  return true; }
-        return false;
-    }
-    case State::THINK: {
-        if (consume_literal(buf_, kThinkClose))  { state_ = State::TEXT;        return true; }
-        return false;
-    }
-    case State::TOOL_CALLS: {
-        if (consume_literal(buf_, kToolsClose))  { state_ = State::TEXT;        return true; }
-        // <｜DSML｜invoke name="X">
-        if (buf_.compare(0, std::strlen(kInvokeOpenPfx), kInvokeOpenPfx) == 0) {
-            size_t close_q = buf_.find('"', std::strlen(kInvokeOpenPfx));
-            if (close_q == std::string::npos) return false; // need more bytes
-            size_t close_gt = buf_.find('>', close_q);
-            if (close_gt == std::string::npos) return false;
-            current_tool_name_ = buf_.substr(std::strlen(kInvokeOpenPfx),
-                                             close_q - std::strlen(kInvokeOpenPfx));
-            tool_index_++;
-            buf_.erase(0, close_gt + 1);
-            ParserEvent e;
-            e.type = ParserEvent::TOOL_START;
-            e.tool_name = current_tool_name_;
-            e.tool_id   = RandomToolId();
-            e.index     = tool_index_;
-            out.push_back(std::move(e));
-            args_emitted_open_brace_ = false;
-            args_param_count_ = 0;
-            state_ = State::INVOKE;
-            return true;
-        }
-        return false;
-    }
-    case State::INVOKE: {
-        if (consume_literal(buf_, kInvokeClose)) {
-            FinishCurrentToolCall(out);
-            state_ = State::TOOL_CALLS;
-            return true;
-        }
-        // <｜DSML｜parameter name="K" string="true|false">
-        if (buf_.compare(0, std::strlen(kParamOpenPfx), kParamOpenPfx) == 0) {
-            size_t close_q = buf_.find('"', std::strlen(kParamOpenPfx));
-            if (close_q == std::string::npos) return false;
-            size_t string_attr = buf_.find("string=\"", close_q);
-            if (string_attr == std::string::npos) return false;
-            size_t string_q = buf_.find('"', string_attr + 8);
-            if (string_q == std::string::npos) return false;
-            size_t close_gt = buf_.find('>', string_q);
-            if (close_gt == std::string::npos) return false;
-            param_name_ = buf_.substr(std::strlen(kParamOpenPfx),
-                                      close_q - std::strlen(kParamOpenPfx));
-            std::string string_val = buf_.substr(string_attr + 8,
-                                                 string_q - (string_attr + 8));
-            param_is_string_ = (string_val == "true");
-            param_value_.clear();
-            buf_.erase(0, close_gt + 1);
-            // Emit args JSON opener / separator.
-            std::string opener;
-            if (!args_emitted_open_brace_) { opener = "{"; args_emitted_open_brace_ = true; }
-            else                            { opener = ","; }
-            opener += "\"" + json_escape(param_name_) + "\":";
-            if (param_is_string_) opener += "\"";
-            EmitArgsChunk(opener, out);
-            args_param_count_++;
-            state_ = State::PARAM_VALUE;
-            return true;
-        }
-        return false;
-    }
-    case State::PARAM_VALUE: {
-        if (consume_literal(buf_, kParamClose)) {
-            if (param_is_string_) EmitArgsChunk("\"", out);
-            state_ = State::INVOKE;
-            return true;
-        }
-        return false;
-    }
-    }
-    return false;
-}
-
-void DsmlParser::DrainPlain(std::vector<ParserEvent> &out) {
-    // Drain everything up to the next '<' that *might* start a marker.
-    // Anything before the next '<' is safe to emit; the '<...' tail stays buffered.
-    while (!buf_.empty()) {
-        size_t lt = next_tag(buf_, 0);
-        if (lt == std::string::npos) {
-            // No tag at all - emit (or accumulate) the whole buffer.
-            ParserEvent e;
-            if (state_ == State::PARAM_VALUE) {
-                std::string esc = param_is_string_ ? json_escape(buf_) : buf_;
-                EmitArgsChunk(esc, out);
-            } else if (state_ == State::THINK) {
-                e.type = ParserEvent::REASONING;
-                e.text = buf_;
-                out.push_back(std::move(e));
-            } else if (state_ == State::TEXT) {
-                e.type = ParserEvent::CONTENT;
-                e.text = buf_;
-                out.push_back(std::move(e));
-            }
-            // Inside INVOKE / TOOL_CALLS with no marker, raw bytes are
-            // structural whitespace - discard.
-            buf_.clear();
-            return;
-        }
-        if (lt > 0) {
-            std::string chunk = buf_.substr(0, lt);
-            buf_.erase(0, lt);
-            ParserEvent e;
-            if (state_ == State::PARAM_VALUE) {
-                std::string esc = param_is_string_ ? json_escape(chunk) : chunk;
-                EmitArgsChunk(esc, out);
-            } else if (state_ == State::THINK) {
-                e.type = ParserEvent::REASONING;
-                e.text = chunk;
-                out.push_back(std::move(e));
-            } else if (state_ == State::TEXT) {
-                e.type = ParserEvent::CONTENT;
-                e.text = chunk;
-                out.push_back(std::move(e));
-            }
-        }
-        // buf_[0] == '<' - try consuming a marker. If we consumed one, loop again.
-        if (!TryConsumeMarker(out)) {
-            // Could be a partial marker - wait for more bytes.
-            if (looks_like_prefix(buf_)) return;
-            // Otherwise this '<' is a literal - emit one char and continue.
-            std::string one(1, buf_[0]);
-            buf_.erase(0, 1);
-            ParserEvent e;
-            if (state_ == State::PARAM_VALUE) {
-                std::string esc = param_is_string_ ? json_escape(one) : one;
-                EmitArgsChunk(esc, out);
-            } else if (state_ == State::THINK) {
-                e.type = ParserEvent::REASONING;
-                e.text = one;
-                out.push_back(std::move(e));
-            } else if (state_ == State::TEXT) {
-                e.type = ParserEvent::CONTENT;
-                e.text = one;
-                out.push_back(std::move(e));
-            }
-        }
-    }
-}
-
-void DsmlParser::Feed(const std::string &chunk, std::vector<ParserEvent> &out) {
-    buf_ += chunk;
-    DrainPlain(out);
-}
-
-void DsmlParser::Flush(std::vector<ParserEvent> &out) {
-    // At flush time we no longer wait for marker completion - drain everything
-    // (the trailing bytes won't grow). Mirror DrainPlain's state-aware
-    // classification: PARAM_VALUE bytes become TOOL_ARGS, THINK bytes become
-    // REASONING, TEXT bytes become CONTENT, and INVOKE/TOOL_CALLS bytes are
-    // structural whitespace (discarded).
-    auto emit_plain = [&](const std::string &chunk) {
-        if (chunk.empty()) return;
-        if (state_ == State::PARAM_VALUE) {
-            std::string esc = param_is_string_ ? json_escape(chunk) : chunk;
-            EmitArgsChunk(esc, out);
-            return;
-        }
-        if (state_ == State::THINK) {
-            ParserEvent e;
-            e.type = ParserEvent::REASONING;
-            e.text = chunk;
-            out.push_back(std::move(e));
-            return;
-        }
-        if (state_ == State::TEXT) {
-            ParserEvent e;
-            e.type = ParserEvent::CONTENT;
-            e.text = chunk;
-            out.push_back(std::move(e));
-            return;
-        }
-        // INVOKE / TOOL_CALLS: structural whitespace, discard.
-    };
-    while (!buf_.empty()) {
-        size_t lt = next_tag(buf_, 0);
-        if (lt == std::string::npos) {
-            emit_plain(buf_);
-            buf_.clear();
-            return;
-        }
-        if (lt > 0) {
-            std::string chunk = buf_.substr(0, lt);
-            buf_.erase(0, lt);
-            emit_plain(chunk);
-        }
-        if (!TryConsumeMarker(out)) {
-            // Definitely a literal '<' now (no chance of more bytes arriving).
-            std::string one(1, buf_[0]);
-            buf_.erase(0, 1);
-            emit_plain(one);
-        }
-    }
-    // If we ended mid-tool-call (model truncated), close it cleanly.
-    if (state_ == State::INVOKE || state_ == State::PARAM_VALUE) {
-        if (state_ == State::PARAM_VALUE && param_is_string_) EmitArgsChunk("\"", out);
-        FinishCurrentToolCall(out);
-        state_ = State::TEXT;
-    }
-}
-
-std::string RandomToolId() {
-    static thread_local std::mt19937_64 rng{
-        static_cast<uint64_t>(std::chrono::system_clock::now().time_since_epoch().count())};
-    const char *alphabet =
-        "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
-    std::string out = "call_";
-    for (int i = 0; i < 16; ++i) {
-        out += alphabet[rng() % 62];
-    }
-    return out;
-}
-
-} // namespace ds4cpp

backend/cpp/ds4/dsml_parser.h

@@ -1,77 +0,0 @@
-#pragma once
-#include <functional>
-#include <string>
-#include <vector>
-
-namespace ds4cpp {
-
-struct ParserEvent {
-    enum Type { CONTENT, REASONING, TOOL_START, TOOL_ARGS, TOOL_END };
-    Type type;
-    std::string text;        // CONTENT, REASONING, TOOL_ARGS
-    std::string tool_name;   // TOOL_START
-    std::string tool_id;     // TOOL_START (caller-assigned)
-    int index = 0;           // TOOL_START / TOOL_ARGS / TOOL_END
-};
-
-// Streaming parser. Stateless across instances; one per Predict call.
-class DsmlParser {
-public:
-    DsmlParser();
-
-    // Feed a chunk of raw model-emitted text. Appends classified events to
-    // `out`. May buffer the tail of `chunk` internally if it looks like a
-    // marker prefix.
-    void Feed(const std::string &chunk, std::vector<ParserEvent> &out);
-
-    // Flush any remaining buffered text as CONTENT (called at generation end).
-    void Flush(std::vector<ParserEvent> &out);
-
-    // True when the parser is inside a DSML structural position - that is,
-    // tags/markers between tool-call boundaries where the model is expected
-    // to emit protocol bytes verbatim. Mirrors ds4_server.c's "force
-    // temperature=0 unless dsml_decode_state_uses_payload_sampling" rule:
-    //
-    //   TEXT / THINK                  -> false (user sampling applies)
-    //   PARAM_VALUE                   -> false (payload uses user sampling)
-    //   TOOL_CALLS / INVOKE           -> true  (structural; force greedy)
-    //
-    // Callers should use this BEFORE the next sample() call to pick the
-    // effective temperature; the parser's state reflects what's already
-    // been consumed, so it predicts the next token's classification.
-    bool IsInDsmlStructural() const;
-
-private:
-    enum class State { TEXT, THINK, TOOL_CALLS, INVOKE, PARAM_VALUE };
-    State state_ = State::TEXT;
-    std::string buf_;
-    std::string current_tool_name_;
-    int tool_index_ = -1;
-    // While parsing a parameter value:
-    std::string param_name_;
-    bool param_is_string_ = true;
-    std::string param_value_;
-    // Incrementally-built arguments JSON for the active tool call.
-    std::string args_json_so_far_;
-    bool args_emitted_open_brace_ = false;
-    int args_param_count_ = 0;
-
-    // Try to consume one structural marker starting at buf_[0]. Returns true
-    // and advances state if a complete marker was consumed; false if the
-    // buffer is ambiguous (could be a marker prefix).
-    bool TryConsumeMarker(std::vector<ParserEvent> &out);
-
-    // Drain plain text from buf_ as far as we're sure it's not a marker prefix.
-    // Emits CONTENT or REASONING depending on current state.
-    void DrainPlain(std::vector<ParserEvent> &out);
-
-    // Emit the next chunk of arguments JSON to the consumer.
-    void EmitArgsChunk(const std::string &chunk, std::vector<ParserEvent> &out);
-    void FinishCurrentToolCall(std::vector<ParserEvent> &out);
-};
-
-// Generate a random tool call ID (e.g. "call_AbCdEf"). Used by the gRPC layer
-// when assigning IDs to streamed tool calls.
-std::string RandomToolId();
-
-} // namespace ds4cpp

backend/cpp/ds4/dsml_renderer.cpp

@@ -1,140 +0,0 @@
-#include "dsml_renderer.h"
-
-// We accept either nlohmann::json (if available) or fall back to a tiny
-// hand-rolled parser. The LocalAI tree already has nlohmann/json bundled
-// in vendor paths; we use the apt-installed nlohmann-json3-dev (installed
-// in Task 11 step 1) when present, otherwise the bundled copy.
-#if __has_include(<nlohmann/json.hpp>)
-#include <nlohmann/json.hpp>
-using json = nlohmann::json;
-#else
-#error "nlohmann/json.hpp not found; install nlohmann-json3-dev"
-#endif
-
-#include <sstream>
-
-namespace ds4cpp {
-
-namespace {
-
-void render_param(std::ostringstream &os, const std::string &name,
-                  const json &value) {
-    bool is_string = value.is_string();
-    os << "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter name=\"" << name
-       << "\" string=\"" << (is_string ? "true" : "false") << "\">";
-    if (is_string) {
-        os << value.get<std::string>();
-    } else {
-        os << value.dump();
-    }
-    os << "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>\n";
-}
-
-} // namespace
-
-std::string RenderAssistantToolCalls(const std::string &tool_calls_json) {
-    if (tool_calls_json.empty()) return "";
-    json arr;
-    try {
-        arr = json::parse(tool_calls_json);
-    } catch (const std::exception &) {
-        return "";
-    }
-    if (!arr.is_array() || arr.empty()) return "";
-
-    std::ostringstream os;
-    os << "\n\n<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\n";
-    for (const auto &call : arr) {
-        // OpenAI shape: { id, type, function: { name, arguments (JSON string) } }
-        // Anthropic shape comes through normalized by LocalAI.
-        std::string name;
-        std::string args_str;
-        if (call.contains("function")) {
-            const auto &fn = call["function"];
-            if (fn.contains("name") && fn["name"].is_string())
-                name = fn["name"].get<std::string>();
-            if (fn.contains("arguments") && fn["arguments"].is_string())
-                args_str = fn["arguments"].get<std::string>();
-        }
-        os << "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\"" << name << "\">\n";
-        if (!args_str.empty()) {
-            json args;
-            try {
-                args = json::parse(args_str);
-            } catch (...) {
-                args = json{};
-            }
-            if (args.is_object()) {
-                for (auto it = args.begin(); it != args.end(); ++it) {
-                    render_param(os, it.key(), it.value());
-                }
-            }
-        }
-        os << "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>\n";
-    }
-    os << "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>";
-    return os.str();
-}
-
-std::string RenderToolResult(const std::string &tool_call_id, const std::string &content) {
-    std::ostringstream os;
-    // ds4_server.c wraps tool results in a "tool_result" DSML tag carrying
-    // the tool_call_id. Match that shape.
-    os << "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_result id=\"" << tool_call_id << "\">"
-       << content
-       << "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_result>";
-    return os.str();
-}
-
-std::string RenderToolsManifest(const std::string &tools_json) {
-    if (tools_json.empty()) return "";
-    json arr;
-    try {
-        arr = json::parse(tools_json);
-    } catch (const std::exception &) {
-        return "";
-    }
-    if (!arr.is_array() || arr.empty()) return "";
-
-    // Extract each OpenAI tool's `function` object, dump as compact JSON, one
-    // per line. Mirrors openai_function_schema_from_tool() in ds4_server.c.
-    std::ostringstream schemas;
-    for (const auto &tool : arr) {
-        if (tool.contains("function") && tool["function"].is_object()) {
-            schemas << tool["function"].dump() << "\n";
-        } else if (tool.is_object()) {
-            // Anthropic / direct-schema form: pass through.
-            schemas << tool.dump() << "\n";
-        }
-    }
-    if (schemas.tellp() == std::streampos(0)) return "";
-
-    // Verbatim text from ds4_server.c append_tools_prompt_text. Do NOT
-    // paraphrase - the model was trained on these exact bytes.
-    std::ostringstream os;
-    os << "## Tools\n\n"
-          "You have access to a set of tools to help answer the user question. "
-          "You can invoke tools by writing a \"<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\" block like the following:\n\n"
-          "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\n"
-          "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\"$TOOL_NAME\">\n"
-          "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter name=\"$PARAMETER_NAME\" string=\"true|false\">$PARAMETER_VALUE</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>\n"
-          "...\n"
-          "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>\n"
-          "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\"$TOOL_NAME2\">\n"
-          "...\n"
-          "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>\n"
-          "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\n\n"
-          "String parameters should be specified as raw text and set `string=\"true\"`. "
-          "Preserve characters such as `>`, `&`, and `&&` exactly; never replace normal string characters with XML or HTML entity escapes. "
-          "Only if a string value itself contains the exact closing parameter tag `</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>`, write that tag as `&lt;/\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>` inside the value. "
-          "For all other types (numbers, booleans, arrays, objects), pass the value in JSON format and set `string=\"false\"`.\n\n"
-          "If thinking_mode is enabled (triggered by <think>), you MUST output your complete reasoning inside <think>...</think> BEFORE any tool calls or final response.\n\n"
-          "Otherwise, output directly after </think> with tool calls or final response.\n\n"
-          "### Available Tool Schemas\n\n"
-       << schemas.str()
-       << "\nYou MUST strictly follow the above defined tool name and parameter schemas to invoke tool calls. "
-          "Use the exact parameter names from the schemas.";
-    return os.str();
-}
-
-} // namespace ds4cpp

backend/cpp/ds4/dsml_renderer.h

@@ -1,27 +0,0 @@
-#pragma once
-#include <string>
-
-namespace ds4cpp {
-
-// Render an assistant message's tool_calls JSON array into the DSML block
-// that ds4 expects in its prompt. `tool_calls_json` is the value of
-// proto.Message.tool_calls (OpenAI shape: array of {id, type, function:{name, arguments}}).
-// Returns the DSML text to append after the assistant's content.
-std::string RenderAssistantToolCalls(const std::string &tool_calls_json);
-
-// Render a role="tool" message into the DSML "tool result" block. ds4's
-// prompt template expects tool results inside a specific tag; we wrap the
-// `content` with that tag and include the `tool_call_id` so the model can
-// correlate.
-std::string RenderToolResult(const std::string &tool_call_id, const std::string &content);
-
-// Render the "## Tools" manifest that ds4 expects in the SYSTEM prompt when
-// tools are available. Without this preamble the model has no idea tools
-// exist and will not emit DSML tool calls. Mirrors append_tools_prompt_text()
-// in ds4_server.c (~line 1646): a fixed preamble + "### Available Tool
-// Schemas" section + one JSON schema per line (extracted from each OpenAI
-// tool's .function object) + a fixed closing instruction. Returns empty
-// when tools_json is empty / unparseable.
-std::string RenderToolsManifest(const std::string &tools_json);
-
-} // namespace ds4cpp

backend/cpp/ds4/grpc-server.cpp

@@ -1,696 +0,0 @@
-// ds4 LocalAI gRPC backend.
-//
-// Wraps antirez/ds4's `ds4_engine_*` / `ds4_session_*` public API
-// (see ds4/ds4.h) over LocalAI's backend.proto. Tool calls, thinking
-// mode, and disk KV cache are wired in follow-up commits; this commit
-// is just the bind/listen/Health/Free skeleton.
-
-#include "backend.pb.h"
-#include "backend.grpc.pb.h"
-
-#include "dsml_parser.h"   // populated in Task 12
-#include "dsml_renderer.h" // populated in Task 16
-#include "kv_cache.h"      // populated in Task 17
-
-extern "C" {
-#include "ds4.h"
-}
-
-#include <grpcpp/grpcpp.h>
-#include <grpcpp/server.h>
-#include <grpcpp/server_builder.h>
-#include <grpcpp/ext/proto_server_reflection_plugin.h>
-
-#include <atomic>
-#include <chrono>
-#include <csignal>
-#include <cstring>
-#include <iostream>
-#include <memory>
-#include <mutex>
-#include <string>
-#include <thread>
-#include <vector>
-
-using grpc::Server;
-using grpc::ServerBuilder;
-using grpc::ServerContext;
-using grpc::ServerWriter;
-// NOTE: do NOT alias `grpc::Status` as `Status` - the Status RPC method below
-// would shadow the type, breaking the other RPC method declarations that use
-// it as a return type. Use GStatus instead.
-using GStatus = ::grpc::Status;
-using grpc::StatusCode;
-
-namespace {
-
-// Global state - ds4 is single-engine-per-process by design.
-std::mutex g_engine_mu;
-ds4_engine *g_engine = nullptr;
-ds4_session *g_session = nullptr;
-int g_ctx_size = 32768;
-std::string g_kv_cache_dir; // empty disables disk cache
-
-std::atomic<Server *> g_server{nullptr};
-
-// Parse a "key:value" option string. Returns empty when no colon.
-static std::pair<std::string, std::string> split_option(const std::string &opt) {
-    auto colon = opt.find(':');
-    if (colon == std::string::npos) return {opt, ""};
-    return {opt.substr(0, colon), opt.substr(colon + 1)};
-}
-
-static void append_token_text(ds4_engine *engine, int token, std::string &out) {
-    size_t len = 0;
-    const char *text = ds4_token_text(engine, token, &len);
-    if (text && len > 0) out.append(text, len);
-}
-
-struct CollectCtx {
-    ds4_engine *engine;
-    std::string raw_buf;  // exact raw bytes for Reply.message
-    ds4cpp::DsmlParser parser;
-    backend::Reply *reply;
-    int tokens;
-
-    // Per-tool aggregation: accumulate ChatDelta tool_calls so we emit one
-    // delta with all calls, mirroring how vllm's non-streaming path returns.
-    struct Pending {
-        std::string id;
-        std::string name;
-        std::string args;
-    };
-    std::vector<Pending> pending;
-
-    std::string content_buf;
-    std::string reasoning_buf;
-};
-
-static void apply_events(CollectCtx *c, const std::vector<ds4cpp::ParserEvent> &events) {
-    for (const auto &e : events) {
-        switch (e.type) {
-        case ds4cpp::ParserEvent::CONTENT:
-            c->content_buf += e.text;
-            break;
-        case ds4cpp::ParserEvent::REASONING:
-            c->reasoning_buf += e.text;
-            break;
-        case ds4cpp::ParserEvent::TOOL_START:
-            if ((int)c->pending.size() <= e.index)
-                c->pending.resize(e.index + 1);
-            c->pending[e.index].id = e.tool_id;
-            c->pending[e.index].name = e.tool_name;
-            break;
-        case ds4cpp::ParserEvent::TOOL_ARGS:
-            if ((int)c->pending.size() > e.index)
-                c->pending[e.index].args += e.text;
-            break;
-        case ds4cpp::ParserEvent::TOOL_END:
-            // No-op for non-streaming: the final delta is emitted at the end.
-            break;
-        }
-    }
-}
-
-static void collect_emit(void *ud, int token) {
-    auto *c = static_cast<CollectCtx *>(ud);
-    if (token == ds4_token_eos(c->engine)) return;
-    size_t len = 0;
-    const char *text = ds4_token_text(c->engine, token, &len);
-    if (!text || len == 0) return;
-    std::string chunk(text, len);
-    c->raw_buf += chunk;
-    std::vector<ds4cpp::ParserEvent> events;
-    c->parser.Feed(chunk, events);
-    apply_events(c, events);
-    c->tokens++;
-}
-static void collect_done(void *) {}
-
-struct StreamCtx {
-    ds4_engine *engine;
-    ServerWriter<backend::Reply> *writer;
-    ds4cpp::DsmlParser parser;
-    int tokens;
-    bool aborted;
-    // Track which tool indices we've seen TOOL_START for, so subsequent
-    // ARGS deltas can elide the redundant id/name fields.
-    std::vector<bool> tool_started;
-};
-
-static void stream_emit(void *ud, int token) {
-    auto *s = static_cast<StreamCtx *>(ud);
-    if (s->aborted) return;
-    if (token == ds4_token_eos(s->engine)) return;
-    size_t len = 0;
-    const char *text = ds4_token_text(s->engine, token, &len);
-    if (!text || len == 0) return;
-    std::string chunk(text, len);
-    std::vector<ds4cpp::ParserEvent> events;
-    s->parser.Feed(chunk, events);
-    if (events.empty()) { s->tokens++; return; }
-
-    backend::Reply reply;
-    auto *delta = reply.add_chat_deltas();
-    bool any_field = false;
-    for (const auto &e : events) {
-        switch (e.type) {
-        case ds4cpp::ParserEvent::CONTENT:
-            delta->set_content(delta->content() + e.text);
-            any_field = true;
-            break;
-        case ds4cpp::ParserEvent::REASONING:
-            delta->set_reasoning_content(delta->reasoning_content() + e.text);
-            any_field = true;
-            break;
-        case ds4cpp::ParserEvent::TOOL_START: {
-            if ((int)s->tool_started.size() <= e.index)
-                s->tool_started.resize(e.index + 1, false);
-            s->tool_started[e.index] = true;
-            auto *tc = delta->add_tool_calls();
-            tc->set_index(e.index);
-            tc->set_id(e.tool_id);
-            tc->set_name(e.tool_name);
-            any_field = true;
-            break;
-        }
-        case ds4cpp::ParserEvent::TOOL_ARGS: {
-            auto *tc = delta->add_tool_calls();
-            tc->set_index(e.index);
-            tc->set_arguments(e.text);
-            any_field = true;
-            break;
-        }
-        case ds4cpp::ParserEvent::TOOL_END:
-            // No marker delta needed - the Go side closes the tool call on
-            // the final aggregator pass.
-            break;
-        }
-    }
-    reply.set_message(chunk);
-    reply.set_tokens(1);
-    if (any_field) {
-        if (!s->writer->Write(reply)) s->aborted = true;
-    }
-    s->tokens++;
-}
-static void stream_done(void *) {}
-
-// Per-thread RNG seed for ds4_session_sample. Initialized lazily from
-// system_clock; ds4 owns the random walk after that.
-static uint64_t *get_rng() {
-    static thread_local uint64_t seed = 0;
-    if (seed == 0) {
-        seed = static_cast<uint64_t>(
-            std::chrono::system_clock::now().time_since_epoch().count());
-        if (seed == 0) seed = 1;
-    }
-    return &seed;
-}
-
-struct SampleParams {
-    float temperature;
-    int top_k;
-    float top_p;
-    float min_p;
-};
-
-// Compute the effective sampling parameters for the next token, mirroring
-// ds4_server.c:7102-7115:
-//   - thinking mode enabled -> override (T=1, top_k=0, top_p=1, min_p=0)
-//   - inside DSML structural position (tool-call markers) -> force T=0
-//   - otherwise -> the request's user-supplied sampling settings
-// The parser argument carries state from tokens emitted so far; its
-// IsInDsmlStructural() predicts the next token's classification.
-static SampleParams compute_sample_params(const backend::PredictOptions *request,
-                                          const ds4cpp::DsmlParser &parser,
-                                          bool think_enabled);
-
-static ds4_think_mode parse_think_mode(const backend::PredictOptions *request) {
-    // Per the vllm backend convention, "enable_thinking" gates thinking on/off,
-    // and "reasoning_effort" picks the strength when on.
-    const auto &md = request->metadata();
-    auto et = md.find("enable_thinking");
-    bool enabled = true; // default ON per ds4-server
-    if (et != md.end()) enabled = (et->second == "true" || et->second == "1");
-    if (!enabled) return DS4_THINK_NONE;
-    auto re = md.find("reasoning_effort");
-    if (re != md.end() && (re->second == "max" || re->second == "xhigh"))
-        return DS4_THINK_MAX;
-    return DS4_THINK_HIGH;
-}
-
-static SampleParams compute_sample_params(const backend::PredictOptions *request,
-                                          const ds4cpp::DsmlParser &parser,
-                                          bool think_enabled) {
-    SampleParams p = {
-        request->temperature(),
-        request->topk(),
-        request->topp(),
-        request->minp(),
-    };
-    if (think_enabled) {
-        // Match ds4-server: thinking mode wants creativity in the reasoning
-        // pass and the trailing content, so the entire generation overrides
-        // sampling unless DSML structural bytes take over below.
-        p.temperature = 1.0f;
-        p.top_k = 0;
-        p.top_p = 1.0f;
-        p.min_p = 0.0f;
-    }
-    if (parser.IsInDsmlStructural()) {
-        // Tool-call structural bytes (tags, markers, headers) must parse
-        // cleanly. Force greedy regardless of user/thinking settings.
-        p.temperature = 0.0f;
-    }
-    return p;
-}
-
-// Build the rendered text for cache keying. We feed the same text the model
-// will see; that lets the cache survive small client-side reformatting of
-// chat history (the cache is keyed on bytes, not tokens).
-static std::string render_prompt_text(const backend::PredictOptions *request) {
-    // Two-mode: either the raw prompt or the chat-template path. We mirror
-    // build_prompt's branching but accumulate text (not tokens) so we can
-    // SHA1 it for the cache key. ds4_session caches a tokens-indexed
-    // checkpoint, but the disk format keys on bytes per ds4-server's design.
-    if (!request->usetokenizertemplate() || request->messages_size() == 0) {
-        return request->prompt();
-    }
-    std::string out;
-    const std::string sys_role = "system";
-    for (const auto &m : request->messages()) {
-        if (m.role() == sys_role) { out += "[sys] " + m.content() + "\n"; break; }
-    }
-    for (const auto &m : request->messages()) {
-        if (m.role() == sys_role) continue;
-        out += "[" + m.role() + "] " + m.content() + "\n";
-    }
-    return out;
-}
-
-ds4cpp::KvCache g_kv_cache;
-
-// Try to recover prefill state for `rendered`. Returns the matched prefix length.
-static size_t maybe_load_cache(const std::string &rendered) {
-    if (!g_kv_cache.enabled() || !g_session) return 0;
-    return g_kv_cache.LoadLongestPrefix(g_session, rendered, g_ctx_size);
-}
-
-static void maybe_save_cache(const std::string &rendered) {
-    if (g_kv_cache.enabled() && g_session) {
-        g_kv_cache.Save(g_session, rendered, g_ctx_size);
-    }
-}
-
-static void build_prompt(ds4_engine *engine, const backend::PredictOptions *request,
-                         ds4_tokens *out) {
-    if (!request->usetokenizertemplate() || request->messages_size() == 0) {
-        ds4_tokenize_text(engine, request->prompt().c_str(), out);
-        return;
-    }
-    // Chat-template path: render via ds4's helpers.
-    ds4_chat_begin(engine, out);
-
-    ds4_think_mode think = parse_think_mode(request);
-
-    // ds4_encode_chat_prompt is convenient when there is exactly one
-    // system+user pair, but for arbitrary turn lists we use the granular
-    // append helpers. Pull the first system message (if any), then append
-    // every other message in order.
-    const std::string sys_role = "system";
-    std::string system_text;
-    for (const auto &m : request->messages()) {
-        if (m.role() == sys_role) { system_text = m.content(); break; }
-    }
-    // Inject the tools manifest into the system prompt when tools are present.
-    // ds4 was trained to emit DSML tool calls ONLY when this preamble is in
-    // the system message - without it, the model has no idea tools exist and
-    // the e2e tool-call test will fail. The renderer lives in dsml_renderer
-    // and is a verbatim port of ds4_server.c's append_tools_prompt_text.
-    std::string tools_manifest;
-    if (!request->tools().empty()) {
-        tools_manifest = ds4cpp::RenderToolsManifest(request->tools());
-    }
-    if (!system_text.empty() || !tools_manifest.empty()) {
-        std::string combined = system_text;
-        if (!tools_manifest.empty()) {
-            if (!combined.empty()) combined += "\n\n";
-            combined += tools_manifest;
-        }
-        ds4_chat_append_message(engine, out, "system", combined.c_str());
-    }
-    for (const auto &m : request->messages()) {
-        if (m.role() == sys_role) continue;
-        if (m.role() == "assistant" && !m.tool_calls().empty()) {
-            std::string combined = m.content();
-            combined += ds4cpp::RenderAssistantToolCalls(m.tool_calls());
-            ds4_chat_append_message(engine, out, "assistant", combined.c_str());
-        } else if (m.role() == "tool") {
-            std::string body = ds4cpp::RenderToolResult(m.tool_call_id(), m.content());
-            ds4_chat_append_message(engine, out, "user", body.c_str());
-        } else {
-            ds4_chat_append_message(engine, out, m.role().c_str(), m.content().c_str());
-        }
-    }
-    ds4_chat_append_assistant_prefix(engine, out, think);
-}
-
-class DS4Backend final : public backend::Backend::Service {
-public:
-    GStatus Health(ServerContext *, const backend::HealthMessage *,
-                  backend::Reply *reply) override {
-        reply->set_message(std::string("OK"));
-        return GStatus::OK;
-    }
-
-    GStatus Free(ServerContext *, const backend::HealthMessage *,
-                backend::Result *result) override {
-        std::lock_guard<std::mutex> lock(g_engine_mu);
-        if (g_session) { ds4_session_free(g_session); g_session = nullptr; }
-        if (g_engine)  { ds4_engine_close(g_engine);  g_engine  = nullptr; }
-        result->set_success(true);
-        return GStatus::OK;
-    }
-
-    GStatus LoadModel(ServerContext *, const backend::ModelOptions *request,
-                     backend::Result *result) override {
-        std::lock_guard<std::mutex> lock(g_engine_mu);
-
-        if (g_engine) {
-            if (g_session) { ds4_session_free(g_session); g_session = nullptr; }
-            ds4_engine_close(g_engine);
-            g_engine = nullptr;
-        }
-
-        std::string model_path = request->modelfile();
-        if (model_path.empty()) model_path = request->model();
-        if (model_path.empty()) {
-            result->set_success(false);
-            result->set_message("ds4: ModelOptions.Model or .ModelFile must be set");
-            return GStatus::OK;
-        }
-
-        std::string mtp_path;
-        int mtp_draft = 0;
-        float mtp_margin = 3.0f;
-        for (const auto &opt : request->options()) {
-            auto [k, v] = split_option(opt);
-            if (k == "mtp_path") mtp_path = v;
-            else if (k == "mtp_draft") mtp_draft = std::stoi(v);
-            else if (k == "mtp_margin") mtp_margin = std::stof(v);
-            else if (k == "kv_cache_dir") g_kv_cache_dir = v;
-        }
-
-        g_kv_cache.SetDir(g_kv_cache_dir);
-
-        ds4_engine_options opt = {};
-        opt.model_path = model_path.c_str();
-        opt.mtp_path = mtp_path.empty() ? nullptr : mtp_path.c_str();
-        opt.n_threads = request->threads() > 0 ? request->threads() : 0;
-        opt.mtp_draft_tokens = mtp_draft;
-        opt.mtp_margin = mtp_margin;
-        opt.directional_steering_file = nullptr;
-        opt.warm_weights = false;
-        opt.quality = false;
-
-#if defined(DS4_NO_GPU)
-        opt.backend = DS4_BACKEND_CPU;
-#elif defined(__APPLE__)
-        opt.backend = DS4_BACKEND_METAL;
-#else
-        opt.backend = DS4_BACKEND_CUDA;
-#endif
-
-        int rc = ds4_engine_open(&g_engine, &opt);
-        if (rc != 0 || !g_engine) {
-            result->set_success(false);
-            result->set_message("ds4_engine_open failed (rc=" + std::to_string(rc) + ")");
-            return GStatus::OK;
-        }
-
-        g_ctx_size = request->contextsize() > 0 ? request->contextsize() : 32768;
-        rc = ds4_session_create(&g_session, g_engine, g_ctx_size);
-        if (rc != 0 || !g_session) {
-            ds4_engine_close(g_engine);
-            g_engine = nullptr;
-            result->set_success(false);
-            result->set_message("ds4_session_create failed (rc=" + std::to_string(rc) + ")");
-            return GStatus::OK;
-        }
-
-        result->set_success(true);
-        result->set_message("loaded " + model_path);
-        return GStatus::OK;
-    }
-
-    GStatus TokenizeString(ServerContext *, const backend::PredictOptions *request,
-                          backend::TokenizationResponse *response) override {
-        std::lock_guard<std::mutex> lock(g_engine_mu);
-        if (!g_engine) return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
-        ds4_tokens out = {};
-        ds4_tokenize_text(g_engine, request->prompt().c_str(), &out);
-        for (int i = 0; i < out.len; ++i) response->add_tokens(out.v[i]);
-        response->set_length(out.len);
-        ds4_tokens_free(&out);
-        return GStatus::OK;
-    }
-
-    GStatus Predict(ServerContext *, const backend::PredictOptions *request,
-                   backend::Reply *reply) override {
-        std::lock_guard<std::mutex> lock(g_engine_mu);
-        if (!g_engine || !g_session) {
-            return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
-        }
-        ds4_tokens prompt = {};
-        build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
-
-        CollectCtx collect = {g_engine, "", {}, reply, 0, {}, "", ""};
-        std::string cache_key = render_prompt_text(request);
-        size_t cache_hit = maybe_load_cache(cache_key);
-        (void)cache_hit; // future: skip prompt prefix if hit covers full prompt
-
-        // Manual generation loop on g_session. When MTP speculative weights
-        // were loaded (LoadModel option 'mtp_path:'), we use the
-        // ds4_session_eval_speculative_argmax path which may accept N>1
-        // tokens per outer iteration. Otherwise per-token argmax + eval.
-        // Either way g_session advances so the disk KV cache picks up a
-        // real checkpoint after the call (see maybe_save_cache below).
-        char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
-        int prompt_len = prompt.len;
-        ds4_tokens_free(&prompt);
-        if (rc == 0) {
-            const int eos = ds4_token_eos(g_engine);
-            const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
-            int produced = 0;
-            while (produced < n_predict) {
-                SampleParams sp = compute_sample_params(request, collect.parser, think_enabled);
-                int first;
-                if (sp.temperature <= 0.0f) {
-                    first = ds4_session_argmax(g_session);
-                } else {
-                    first = ds4_session_sample(g_session,
-                                               sp.temperature, sp.top_k,
-                                               sp.top_p, sp.min_p, get_rng());
-                }
-                if (first == eos) break;
-                // MTP only when sampling is greedy (ds4-server gate).
-                if (draft_max > 0 && sp.temperature <= 0.0f) {
-                    constexpr int kAcceptedMax = 8;
-                    int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
-                    int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
-                        accepted, cap, err, sizeof(err));
-                    if (n < 0) { rc = -1; break; }
-                    bool stop = false;
-                    for (int j = 0; j < n; ++j) {
-                        if (accepted[j] == eos) { stop = true; break; }
-                        collect_emit(&collect, accepted[j]);
-                        if (++produced >= n_predict) { stop = true; break; }
-                    }
-                    if (stop) break;
-                } else {
-                    collect_emit(&collect, first);
-                    if (++produced >= n_predict) break;
-                    rc = ds4_session_eval(g_session, first, err, sizeof(err));
-                    if (rc != 0) break;
-                }
-            }
-            collect_done(&collect);
-        }
-        maybe_save_cache(cache_key);
-
-        // Flush any buffered parser state.
-        std::vector<ds4cpp::ParserEvent> events;
-        collect.parser.Flush(events);
-        apply_events(&collect, events);
-
-        if (rc != 0) {
-            return GStatus(StatusCode::INTERNAL,
-                          std::string("ds4 generation failed: ") + err);
-        }
-
-        // Emit one ChatDelta with content/reasoning/tool_calls.
-        auto *delta = reply->add_chat_deltas();
-        delta->set_content(collect.content_buf);
-        delta->set_reasoning_content(collect.reasoning_buf);
-        for (size_t i = 0; i < collect.pending.size(); ++i) {
-            auto *tc = delta->add_tool_calls();
-            tc->set_index(static_cast<int32_t>(i));
-            tc->set_id(collect.pending[i].id);
-            tc->set_name(collect.pending[i].name);
-            tc->set_arguments(collect.pending[i].args);
-        }
-
-        reply->set_message(collect.raw_buf);
-        reply->set_tokens(collect.tokens);
-        reply->set_prompt_tokens(prompt_len);
-        return GStatus::OK;
-    }
-
-    GStatus PredictStream(ServerContext *, const backend::PredictOptions *request,
-                         ServerWriter<backend::Reply> *writer) override {
-        std::lock_guard<std::mutex> lock(g_engine_mu);
-        if (!g_engine || !g_session) {
-            return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
-        }
-        ds4_tokens prompt = {};
-        build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
-
-        StreamCtx s = {g_engine, writer, {}, 0, false, {}};
-        std::string cache_key = render_prompt_text(request);
-        size_t cache_hit = maybe_load_cache(cache_key);
-        (void)cache_hit;
-
-        // Manual loop on g_session - see Predict() above for the rationale.
-        // MTP speculative path used when ds4_engine_mtp_draft_tokens > 0.
-        char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
-        ds4_tokens_free(&prompt);
-        if (rc == 0) {
-            const int eos = ds4_token_eos(g_engine);
-            const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
-            int produced = 0;
-            while (produced < n_predict && !s.aborted) {
-                SampleParams sp = compute_sample_params(request, s.parser, think_enabled);
-                int first;
-                if (sp.temperature <= 0.0f) {
-                    first = ds4_session_argmax(g_session);
-                } else {
-                    first = ds4_session_sample(g_session,
-                                               sp.temperature, sp.top_k,
-                                               sp.top_p, sp.min_p, get_rng());
-                }
-                if (first == eos) break;
-                if (draft_max > 0 && sp.temperature <= 0.0f) {
-                    constexpr int kAcceptedMax = 8;
-                    int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
-                    int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
-                        accepted, cap, err, sizeof(err));
-                    if (n < 0) { rc = -1; break; }
-                    bool stop = false;
-                    for (int j = 0; j < n; ++j) {
-                        if (accepted[j] == eos) { stop = true; break; }
-                        stream_emit(&s, accepted[j]);
-                        if (s.aborted) { stop = true; break; }
-                        if (++produced >= n_predict) { stop = true; break; }
-                    }
-                    if (stop) break;
-                } else {
-                    stream_emit(&s, first);
-                    if (s.aborted || ++produced >= n_predict) break;
-                    rc = ds4_session_eval(g_session, first, err, sizeof(err));
-                    if (rc != 0) break;
-                }
-            }
-            stream_done(&s);
-        }
-        maybe_save_cache(cache_key);
-
-        // Flush parser state.
-        std::vector<ds4cpp::ParserEvent> events;
-        s.parser.Flush(events);
-        if (!events.empty() && !s.aborted) {
-            backend::Reply reply;
-            auto *delta = reply.add_chat_deltas();
-            for (const auto &e : events) {
-                if (e.type == ds4cpp::ParserEvent::CONTENT) {
-                    delta->set_content(delta->content() + e.text);
-                } else if (e.type == ds4cpp::ParserEvent::REASONING) {
-                    delta->set_reasoning_content(delta->reasoning_content() + e.text);
-                }
-            }
-            s.writer->Write(reply);
-        }
-
-        if (rc != 0 && !s.aborted) {
-            return GStatus(StatusCode::INTERNAL,
-                          std::string("ds4 generation failed: ") + err);
-        }
-        return GStatus::OK;
-    }
-
-    GStatus Status(ServerContext *, const backend::HealthMessage *,
-                  backend::StatusResponse *response) override {
-        std::lock_guard<std::mutex> lock(g_engine_mu);
-        response->set_state(g_engine ? backend::StatusResponse::READY
-                                     : backend::StatusResponse::UNINITIALIZED);
-        return GStatus::OK;
-    }
-};
-
-void RunServer(const std::string &addr) {
-    DS4Backend service;
-    grpc::EnableDefaultHealthCheckService(true);
-    grpc::reflection::InitProtoReflectionServerBuilderPlugin();
-
-    ServerBuilder builder;
-    builder.AddListeningPort(addr, grpc::InsecureServerCredentials());
-    builder.RegisterService(&service);
-    builder.SetMaxReceiveMessageSize(64 * 1024 * 1024);
-    builder.SetMaxSendMessageSize(64 * 1024 * 1024);
-
-    std::unique_ptr<Server> server(builder.BuildAndStart());
-    if (!server) {
-        std::cerr << "ds4 grpc-server: failed to bind " << addr << "\n";
-        std::exit(1);
-    }
-    g_server = server.get();
-    std::cerr << "ds4 grpc-server listening on " << addr << "\n";
-    server->Wait();
-}
-
-void signal_handler(int) {
-    if (auto *srv = g_server.load()) {
-        srv->Shutdown(std::chrono::system_clock::now() +
-                      std::chrono::seconds(3));
-    }
-}
-
-} // namespace
-
-int main(int argc, char *argv[]) {
-    std::string addr = "127.0.0.1:50051";
-    for (int i = 1; i < argc; ++i) {
-        std::string a = argv[i];
-        const std::string addr_flag = "--addr=";
-        if (a.rfind(addr_flag, 0) == 0) addr = a.substr(addr_flag.size());
-        else if (a == "--addr" && i + 1 < argc) addr = argv[++i];
-        else if (a == "--help" || a == "-h") {
-            std::cout << "Usage: grpc-server --addr=HOST:PORT\n";
-            return 0;
-        }
-    }
-    std::signal(SIGINT, signal_handler);
-    std::signal(SIGTERM, signal_handler);
-    RunServer(addr);
-    return 0;
-}

backend/cpp/ds4/kv_cache.cpp

@@ -1,205 +0,0 @@
-#include "kv_cache.h"
-
-#include <cerrno>
-#include <cstdio>
-#include <cstring>
-#include <dirent.h>
-#include <fstream>
-#include <sys/stat.h>
-#include <vector>
-
-namespace ds4cpp {
-
-namespace {
-
-// Minimal SHA1 (public domain reference). 30 lines; used only here.
-struct Sha1 {
-    uint32_t h[5];
-    uint64_t bits;
-    uint8_t block[64];
-    size_t used;
-    Sha1() { h[0]=0x67452301; h[1]=0xEFCDAB89; h[2]=0x98BADCFE; h[3]=0x10325476; h[4]=0xC3D2E1F0; bits=0; used=0; }
-    static uint32_t rol(uint32_t x, int n){ return (x<<n)|(x>>(32-n)); }
-    void transform(const uint8_t *b) {
-        uint32_t w[80];
-        for (int i=0;i<16;i++) w[i] = (uint32_t)b[i*4]<<24 | (uint32_t)b[i*4+1]<<16 | (uint32_t)b[i*4+2]<<8 | b[i*4+3];
-        for (int i=16;i<80;i++) w[i] = rol(w[i-3]^w[i-8]^w[i-14]^w[i-16], 1);
-        uint32_t a=h[0],bb=h[1],c=h[2],d=h[3],e=h[4];
-        for (int i=0;i<80;i++) {
-            uint32_t f,k;
-            if (i<20)      { f=(bb&c)|((~bb)&d); k=0x5A827999; }
-            else if (i<40) { f=bb^c^d;            k=0x6ED9EBA1; }
-            else if (i<60) { f=(bb&c)|(bb&d)|(c&d); k=0x8F1BBCDC; }
-            else           { f=bb^c^d;            k=0xCA62C1D6; }
-            uint32_t t = rol(a,5)+f+e+k+w[i];
-            e=d; d=c; c=rol(bb,30); bb=a; a=t;
-        }
-        h[0]+=a; h[1]+=bb; h[2]+=c; h[3]+=d; h[4]+=e;
-    }
-    void update(const void *p, size_t n) {
-        const uint8_t *bp = (const uint8_t*)p;
-        bits += (uint64_t)n*8;
-        while (n) {
-            size_t take = 64-used;
-            if (take>n) take=n;
-            std::memcpy(block+used, bp, take);
-            used += take; bp += take; n -= take;
-            if (used == 64) { transform(block); used = 0; }
-        }
-    }
-    void final(uint8_t out[20]) {
-        uint8_t pad[64] = {0x80};
-        size_t padlen = (used < 56) ? (56-used) : (120-used);
-        uint64_t lb = bits;
-        uint8_t len[8];
-        for (int i=0;i<8;i++) len[7-i] = (uint8_t)(lb >> (i*8));
-        update(pad, padlen);
-        update(len, 8);
-        for (int i=0;i<5;i++) {
-            out[i*4]   = h[i]>>24;
-            out[i*4+1] = h[i]>>16;
-            out[i*4+2] = h[i]>>8;
-            out[i*4+3] = h[i];
-        }
-    }
-};
-
-std::string mkdir_p(const std::string &d) {
-    if (d.empty()) return d;
-    struct stat st{};
-    if (stat(d.c_str(), &st) == 0) return d;
-    mkdir(d.c_str(), 0755);
-    return d;
-}
-
-bool file_exists(const std::string &p) {
-    struct stat st{};
-    return stat(p.c_str(), &st) == 0;
-}
-
-} // namespace
-
-std::string Sha1Hex(const void *data, size_t len) {
-    Sha1 s;
-    s.update(data, len);
-    uint8_t out[20];
-    s.final(out);
-    char hex[41];
-    for (int i = 0; i < 20; ++i) std::snprintf(hex + i*2, 3, "%02x", out[i]);
-    hex[40] = 0;
-    return std::string(hex);
-}
-
-KvCache::KvCache() = default;
-
-void KvCache::SetDir(const std::string &dir) {
-    dir_ = dir;
-    if (!dir_.empty()) {
-        mkdir_p(dir_);
-        std::fprintf(stderr, "ds4 KvCache: enabled at %s\n", dir_.c_str());
-    } else {
-        std::fprintf(stderr, "ds4 KvCache: disabled (no dir set)\n");
-    }
-}
-
-std::string KvCache::Path(const std::string &rendered_text) const {
-    if (dir_.empty()) return "";
-    return dir_ + "/" + Sha1Hex(rendered_text.data(), rendered_text.size()) + ".kv";
-}
-
-size_t KvCache::LoadLongestPrefix(ds4_session *session,
-                                  const std::string &rendered_text,
-                                  int ctx_size) {
-    if (dir_.empty() || !session) return 0;
-    // Strategy: enumerate all .kv files in dir, read their stored prefix
-    // header, pick the longest one that is also a prefix of rendered_text.
-    DIR *d = opendir(dir_.c_str());
-    if (!d) return 0;
-    struct dirent *de;
-    size_t best_len = 0;
-    std::string best_path;
-    while ((de = readdir(d)) != nullptr) {
-        std::string name = de->d_name;
-        if (name.size() < 4 || name.substr(name.size()-3) != ".kv") continue;
-        std::string path = dir_ + "/" + name;
-        std::ifstream f(path, std::ios::binary);
-        if (!f) continue;
-        char magic[4]; f.read(magic, 4);
-        if (f.gcount() != 4 || std::memcmp(magic, "DS4G", 4) != 0) continue;
-        uint32_t version=0, file_ctx=0, prefix_len=0;
-        f.read((char*)&version, 4); f.read((char*)&file_ctx, 4); f.read((char*)&prefix_len, 4);
-        if (version != 1) continue;
-        if ((int)file_ctx != ctx_size) continue;
-        if (prefix_len > rendered_text.size()) continue;
-        std::vector<char> prefix(prefix_len);
-        f.read(prefix.data(), prefix_len);
-        if (std::memcmp(prefix.data(), rendered_text.data(), prefix_len) != 0) continue;
-        if (prefix_len > best_len) {
-            best_len = prefix_len;
-            best_path = path;
-        }
-    }
-    closedir(d);
-    if (best_len == 0) return 0;
-
-    // Load best_path's payload into session.
-    std::ifstream f(best_path, std::ios::binary);
-    char magic[4]; f.read(magic, 4);
-    uint32_t version, file_ctx, prefix_len;
-    f.read((char*)&version, 4); f.read((char*)&file_ctx, 4); f.read((char*)&prefix_len, 4);
-    f.seekg(prefix_len, std::ios::cur);
-    uint64_t payload_bytes = 0;
-    f.read((char*)&payload_bytes, 8);
-    // ds4_session_load_payload reads from a FILE*; reopen via fopen.
-    FILE *fp = std::fopen(best_path.c_str(), "rb");
-    if (!fp) return 0;
-    // Seek past header + prefix + payload_bytes field.
-    std::fseek(fp, 4 + 4 + 4 + 4 + prefix_len + 8, SEEK_SET);
-    char errbuf[256] = {0};
-    int rc = ds4_session_load_payload(session, fp, payload_bytes, errbuf, sizeof(errbuf));
-    std::fclose(fp);
-    if (rc != 0) return 0;
-    return best_len;
-}
-
-void KvCache::Save(ds4_session *session, const std::string &rendered_text, int ctx_size) {
-    if (dir_.empty()) {
-        std::fprintf(stderr, "ds4 KvCache::Save: skipped (dir empty)\n");
-        return;
-    }
-    if (!session) {
-        std::fprintf(stderr, "ds4 KvCache::Save: skipped (session null)\n");
-        return;
-    }
-    std::string path = Path(rendered_text);
-    uint64_t payload_bytes = ds4_session_payload_bytes(session);
-    std::fprintf(stderr, "ds4 KvCache::Save: path=%s payload_bytes=%llu prefix_len=%zu\n",
-                 path.c_str(), (unsigned long long)payload_bytes, rendered_text.size());
-    FILE *fp = std::fopen(path.c_str(), "wb");
-    if (!fp) {
-        std::fprintf(stderr, "ds4 KvCache::Save: fopen failed: %s\n", std::strerror(errno));
-        return;
-    }
-    char magic[4] = {'D','S','4','G'};
-    uint32_t version = 1;
-    uint32_t ctx = static_cast<uint32_t>(ctx_size);
-    uint32_t prefix_len = static_cast<uint32_t>(rendered_text.size());
-    std::fwrite(magic, 4, 1, fp);
-    std::fwrite(&version, 4, 1, fp);
-    std::fwrite(&ctx, 4, 1, fp);
-    std::fwrite(&prefix_len, 4, 1, fp);
-    std::fwrite(rendered_text.data(), prefix_len, 1, fp);
-    std::fwrite(&payload_bytes, 8, 1, fp);
-    char errbuf[256] = {0};
-    int rc = ds4_session_save_payload(session, fp, errbuf, sizeof(errbuf));
-    std::fclose(fp);
-    if (rc != 0) {
-        std::fprintf(stderr, "ds4 KvCache::Save: ds4_session_save_payload rc=%d err=%s; removing %s\n",
-                     rc, errbuf, path.c_str());
-        std::remove(path.c_str());
-    } else {
-        std::fprintf(stderr, "ds4 KvCache::Save: wrote %s ok\n", path.c_str());
-    }
-}
-
-} // namespace ds4cpp

backend/cpp/ds4/kv_cache.h

@@ -1,44 +0,0 @@
-#pragma once
-#include <string>
-extern "C" {
-#include "ds4.h"
-}
-
-namespace ds4cpp {
-
-// Disk-backed KV cache for ds4 sessions. Keyed by SHA1(rendered prompt prefix).
-// Format (our own, NOT bit-compatible with ds4-server's KVC files - interop
-// is a follow-up plan):
-//
-//   "DS4G" (4 bytes magic) + u32 version=1 + u32 ctx_size +
-//   u32 prefix_text_len + prefix_text + u64 payload_bytes + payload
-class KvCache {
-public:
-    KvCache(); // disabled (dir empty)
-
-    // Set the cache directory. Empty disables.
-    void SetDir(const std::string &dir);
-
-    // Returns the cache file path for a given rendered text prefix.
-    std::string Path(const std::string &rendered_text) const;
-
-    // Look up the longest cached prefix that is also a prefix of
-    // `rendered_text`. Loads it into `session` if found. Returns the
-    // matched prefix length in bytes (0 if no hit).
-    size_t LoadLongestPrefix(ds4_session *session,
-                             const std::string &rendered_text,
-                             int ctx_size);
-
-    // Save the current session, associated with this rendered text prefix.
-    void Save(ds4_session *session, const std::string &rendered_text, int ctx_size);
-
-    bool enabled() const { return !dir_.empty(); }
-
-private:
-    std::string dir_;
-};
-
-// Compute SHA1 of arbitrary bytes; returns 40-char hex.
-std::string Sha1Hex(const void *data, size_t len);
-
-} // namespace ds4cpp

backend/cpp/ds4/package.sh

@@ -1,39 +0,0 @@
-#!/bin/bash
-set -e
-CURDIR=$(dirname "$(realpath "$0")")
-REPO_ROOT="${CURDIR}/../../.."
-
-mkdir -p "$CURDIR/package/lib"
-cp -avf "$CURDIR/grpc-server" "$CURDIR/package/"
-cp -rfv "$CURDIR/run.sh"     "$CURDIR/package/"
-
-UNAME_S=$(uname -s)
-if [ "$UNAME_S" = "Darwin" ]; then
-    # Darwin: bundle dylibs via otool -L (handled by scripts/build/ds4-darwin.sh).
-    echo "package.sh: Darwin handled by ds4-darwin.sh"
-    exit 0
-fi
-
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 "$CURDIR/package/lib/ld.so"
-    LIBDIR=/lib/x86_64-linux-gnu
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    cp -arfLv /lib/ld-linux-aarch64.so.1 "$CURDIR/package/lib/ld.so"
-    LIBDIR=/lib/aarch64-linux-gnu
-else
-    echo "package.sh: unknown architecture" >&2; exit 1
-fi
-
-for lib in libc.so.6 libgcc_s.so.1 libstdc++.so.6 libm.so.6 libgomp.so.1 \
-           libdl.so.2 librt.so.1 libpthread.so.0; do
-    cp -arfLv "$LIBDIR/$lib" "$CURDIR/package/lib/$lib"
-done
-
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "ds4 package contents:"
-ls -lah "$CURDIR/package/" "$CURDIR/package/lib/"

backend/cpp/ds4/run.sh

@@ -1,9 +0,0 @@
-#!/bin/bash
-# Entry point for the ds4 backend image / BACKEND_BINARY mode.
-set -e
-CURDIR=$(dirname "$(realpath "$0")")
-export LD_LIBRARY_PATH="$CURDIR/lib:$LD_LIBRARY_PATH"
-if [ -f "$CURDIR/lib/ld.so" ]; then
-    exec "$CURDIR/lib/ld.so" "$CURDIR/grpc-server" "$@"
-fi
-exec "$CURDIR/grpc-server" "$@"

backend/cpp/ik-llama-cpp/CMakeLists.txt

@@ -1,78 +0,0 @@
-## Clip/LLaVA library for multimodal support — built locally from copied sources
-set(TARGET myclip)
-add_library(${TARGET} clip.cpp clip.h llava.cpp llava.h)
-install(TARGETS ${TARGET} LIBRARY)
-target_include_directories(myclip PUBLIC .)
-target_include_directories(myclip PUBLIC ../..)
-target_include_directories(myclip PUBLIC ../../common)
-target_link_libraries(${TARGET} PRIVATE common ggml llama ${CMAKE_THREAD_LIBS_INIT})
-target_compile_features(${TARGET} PRIVATE cxx_std_11)
-if (NOT MSVC)
-    target_compile_options(${TARGET} PRIVATE -Wno-cast-qual)
-endif()
-
-set(TARGET grpc-server)
-set(CMAKE_CXX_STANDARD 17)
-cmake_minimum_required(VERSION 3.15)
-set(TARGET grpc-server)
-set(_PROTOBUF_LIBPROTOBUF libprotobuf)
-set(_REFLECTION grpc++_reflection)
-
-if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
-    if (CMAKE_HOST_SYSTEM_PROCESSOR MATCHES "arm64")
-        set(HOMEBREW_DEFAULT_PREFIX "/opt/homebrew")
-    else()
-        set(HOMEBREW_DEFAULT_PREFIX "/usr/local")
-    endif()
-    link_directories("${HOMEBREW_DEFAULT_PREFIX}/lib")
-    include_directories("${HOMEBREW_DEFAULT_PREFIX}/include")
-endif()
-
-find_package(absl CONFIG REQUIRED)
-find_package(Protobuf CONFIG REQUIRED)
-find_package(gRPC CONFIG REQUIRED)
-
-find_program(_PROTOBUF_PROTOC protoc)
-set(_GRPC_GRPCPP grpc++)
-find_program(_GRPC_CPP_PLUGIN_EXECUTABLE grpc_cpp_plugin)
-
-include_directories(${CMAKE_CURRENT_BINARY_DIR})
-include_directories(${Protobuf_INCLUDE_DIRS})
-
-message(STATUS "Using protobuf version ${Protobuf_VERSION} | Protobuf_INCLUDE_DIRS: ${Protobuf_INCLUDE_DIRS} | CMAKE_CURRENT_BINARY_DIR: ${CMAKE_CURRENT_BINARY_DIR}")
-
-# Proto file
-get_filename_component(hw_proto "../../../../../../backend/backend.proto" ABSOLUTE)
-get_filename_component(hw_proto_path "${hw_proto}" PATH)
-
-set(hw_proto_srcs "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.cc")
-set(hw_proto_hdrs "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.h")
-set(hw_grpc_srcs "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.cc")
-set(hw_grpc_hdrs "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.h")
-
-add_custom_command(
-      OUTPUT "${hw_proto_srcs}" "${hw_proto_hdrs}" "${hw_grpc_srcs}" "${hw_grpc_hdrs}"
-      COMMAND ${_PROTOBUF_PROTOC}
-      ARGS --grpc_out "${CMAKE_CURRENT_BINARY_DIR}"
-        --cpp_out "${CMAKE_CURRENT_BINARY_DIR}"
-        -I "${hw_proto_path}"
-        --plugin=protoc-gen-grpc="${_GRPC_CPP_PLUGIN_EXECUTABLE}"
-        "${hw_proto}"
-      DEPENDS "${hw_proto}")
-
-add_library(hw_grpc_proto
-  ${hw_grpc_srcs}
-  ${hw_grpc_hdrs}
-  ${hw_proto_srcs}
-  ${hw_proto_hdrs} )
-
-add_executable(${TARGET} grpc-server.cpp json.hpp)
-target_link_libraries(${TARGET} PRIVATE common llama myclip ${CMAKE_THREAD_LIBS_INIT} absl::flags hw_grpc_proto
-  absl::flags_parse
-  gRPC::${_REFLECTION}
-  gRPC::${_GRPC_GRPCPP}
-  protobuf::${_PROTOBUF_LIBPROTOBUF})
-target_compile_features(${TARGET} PRIVATE cxx_std_11)
-if(TARGET BUILD_INFO)
-  add_dependencies(${TARGET} BUILD_INFO)
-endif()

backend/cpp/ik-llama-cpp/Makefile

@@ -1,167 +0,0 @@
-
-IK_LLAMA_VERSION?=5cc0d86c760e9858e4bed4418400bb39dbe025f2
-LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
-
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-ONEAPI_VARS?=/opt/intel/oneapi/setvars.sh
-TARGET?=--target grpc-server
-JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
-ARCH?=$(shell uname -m)
-
-# Disable Shared libs as we are linking on static gRPC and we can't mix shared and static
-CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=OFF
-
-CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
-ifeq ($(NATIVE),false)
-	CMAKE_ARGS+=-DGGML_NATIVE=OFF -DLLAMA_OPENSSL=OFF
-endif
-# If build type is cublas, then we set -DGGML_CUDA=ON to CMAKE_ARGS automatically
-ifeq ($(BUILD_TYPE),cublas)
-	CMAKE_ARGS+=-DGGML_CUDA=ON
-# If build type is openblas then we set -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
-# to CMAKE_ARGS automatically
-else ifeq ($(BUILD_TYPE),openblas)
-	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
-# If build type is clblas (openCL) we set -DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
-else ifeq ($(BUILD_TYPE),clblas)
-	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
-# If it's hipblas we do have also to set CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++
-else ifeq ($(BUILD_TYPE),hipblas)
-	ROCM_HOME ?= /opt/rocm
-	ROCM_PATH ?= /opt/rocm
-	export CXX=$(ROCM_HOME)/llvm/bin/clang++
-	export CC=$(ROCM_HOME)/llvm/bin/clang
-	AMDGPU_TARGETS?=gfx803,gfx900,gfx906,gfx908,gfx90a,gfx942,gfx1010,gfx1030,gfx1032,gfx1100,gfx1101,gfx1102,gfx1200,gfx1201
-	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
-else ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DGGML_VULKAN=1
-else ifeq ($(OS),Darwin)
-	ifeq ($(BUILD_TYPE),)
-		BUILD_TYPE=metal
-	endif
-	ifneq ($(BUILD_TYPE),metal)
-		CMAKE_ARGS+=-DGGML_METAL=OFF
-	else
-		CMAKE_ARGS+=-DGGML_METAL=ON
-		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
-		CMAKE_ARGS+=-DGGML_METAL_USE_BF16=ON
-		CMAKE_ARGS+=-DGGML_OPENMP=OFF
-	endif
-	TARGET+=--target ggml-metal
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f16)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx \
-		-DCMAKE_CXX_FLAGS="-fsycl" \
-		-DGGML_SYCL_F16=ON
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f32)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx \
-		-DCMAKE_CXX_FLAGS="-fsycl"
-endif
-
-INSTALLED_PACKAGES=$(CURDIR)/../grpc/installed_packages
-INSTALLED_LIB_CMAKE=$(INSTALLED_PACKAGES)/lib/cmake
-ADDED_CMAKE_ARGS=-Dabsl_DIR=${INSTALLED_LIB_CMAKE}/absl \
-				 -DProtobuf_DIR=${INSTALLED_LIB_CMAKE}/protobuf \
-				 -Dutf8_range_DIR=${INSTALLED_LIB_CMAKE}/utf8_range \
-				 -DgRPC_DIR=${INSTALLED_LIB_CMAKE}/grpc \
-				 -DCMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES=${INSTALLED_PACKAGES}/include
-build-ik-llama-cpp-grpc-server:
-# Conditionally build grpc for the backend to use if needed
-ifdef BUILD_GRPC_FOR_BACKEND_LLAMA
-	$(MAKE) -C ../../grpc build
-	_PROTOBUF_PROTOC=${INSTALLED_PACKAGES}/bin/proto \
-	_GRPC_CPP_PLUGIN_EXECUTABLE=${INSTALLED_PACKAGES}/bin/grpc_cpp_plugin \
-	PATH="${INSTALLED_PACKAGES}/bin:${PATH}" \
-	CMAKE_ARGS="${CMAKE_ARGS} ${ADDED_CMAKE_ARGS}" \
-	IK_LLAMA_VERSION=$(IK_LLAMA_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../$(VARIANT) grpc-server
-else
-	echo "BUILD_GRPC_FOR_BACKEND_LLAMA is not defined."
-	IK_LLAMA_VERSION=$(IK_LLAMA_VERSION) $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../$(VARIANT) grpc-server
-endif
-
-ik-llama-cpp-avx2: llama.cpp
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx2-build
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx2-build purge
-	$(info ${GREEN}I ik-llama-cpp build info:avx2${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on" $(MAKE) VARIANT="ik-llama-cpp-avx2-build" build-ik-llama-cpp-grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx2-build/grpc-server ik-llama-cpp-avx2
-
-ik-llama-cpp-avx512: llama.cpp
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx512-build
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx512-build purge
-	$(info ${GREEN}I ik-llama-cpp build info:avx512${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on" $(MAKE) VARIANT="ik-llama-cpp-avx512-build" build-ik-llama-cpp-grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx512-build/grpc-server ik-llama-cpp-avx512
-
-ik-llama-cpp-avx: llama.cpp
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx-build
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx-build purge
-	$(info ${GREEN}I ik-llama-cpp build info:avx${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) VARIANT="ik-llama-cpp-avx-build" build-ik-llama-cpp-grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-avx-build/grpc-server ik-llama-cpp-avx
-
-ik-llama-cpp-fallback: llama.cpp
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-fallback-build
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-fallback-build purge
-	$(info ${GREEN}I ik-llama-cpp build info:fallback${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) VARIANT="ik-llama-cpp-fallback-build" build-ik-llama-cpp-grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-fallback-build/grpc-server ik-llama-cpp-fallback
-
-ik-llama-cpp-grpc: llama.cpp
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-grpc-build
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-grpc-build purge
-	$(info ${GREEN}I ik-llama-cpp build info:grpc${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" TARGET="--target grpc-server --target rpc-server" $(MAKE) VARIANT="ik-llama-cpp-grpc-build" build-ik-llama-cpp-grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-grpc-build/grpc-server ik-llama-cpp-grpc
-
-ik-llama-cpp-rpc-server: ik-llama-cpp-grpc
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../ik-llama-cpp-grpc-build/llama.cpp/build/bin/rpc-server ik-llama-cpp-rpc-server
-
-llama.cpp:
-	mkdir -p llama.cpp
-	cd llama.cpp && \
-	git init && \
-	git remote add origin $(LLAMA_REPO)  && \
-	git fetch origin && \
-	git checkout -b build $(IK_LLAMA_VERSION) && \
-	git submodule update --init --recursive --depth 1 --single-branch
-
-llama.cpp/examples/grpc-server: llama.cpp
-	mkdir -p llama.cpp/examples/grpc-server
-	bash prepare.sh
-
-rebuild:
-	bash prepare.sh
-	rm -rf grpc-server
-	$(MAKE) grpc-server
-
-package:
-	bash package.sh
-
-purge:
-	rm -rf llama.cpp/build
-	rm -rf llama.cpp/examples/grpc-server
-	rm -rf grpc-server
-
-clean: purge
-	rm -rf llama.cpp
-
-grpc-server: llama.cpp llama.cpp/examples/grpc-server
-	@echo "Building grpc-server with $(BUILD_TYPE) build type and $(CMAKE_ARGS)"
-ifneq (,$(findstring sycl,$(BUILD_TYPE)))
-	+bash -c "source $(ONEAPI_VARS); \
-	cd llama.cpp && mkdir -p build && cd build && cmake .. $(CMAKE_ARGS) && cmake --build . --config Release -j $(JOBS) $(TARGET)"
-else
-	+cd llama.cpp && mkdir -p build && cd build && cmake .. $(CMAKE_ARGS) && cmake --build . --config Release -j $(JOBS) $(TARGET)
-endif
-	cp llama.cpp/build/bin/grpc-server .

backend/cpp/ik-llama-cpp/package.sh

@@ -1,58 +0,0 @@
-#!/bin/bash
-
-# Script to copy the appropriate libraries based on architecture
-# This script is used in the final stage of the Dockerfile
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-# Create lib directory
-mkdir -p $CURDIR/package/lib
-
-cp -avrf $CURDIR/ik-llama-cpp-* $CURDIR/package/
-cp -rfv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    # x86_64 architecture
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    # ARM64 architecture
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-# The GPU library packaging script will detect BUILD_TYPE and copy appropriate GPU libraries
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/cpp/ik-llama-cpp/patches/0001-fix-missing-cstdint-include.patch

@@ -1,10 +0,0 @@
---- a/ggml/src/iqk/iqk_common.h
-+++ b/ggml/src/iqk/iqk_common.h
-@@ -9,6 +9,7 @@
- #pragma once
-
- #include "iqk_config.h"
-+#include <cstdint>
-
- #if defined IQK_IMPLEMENT
-

backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch

@@ -1,11 +0,0 @@
---- a/examples/llava/clip.cpp
-+++ b/examples/llava/clip.cpp
-@@ -2494,7 +2494,7 @@
-             }
-             new_data = work.data();
-
--            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr);
-+            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr, nullptr);
-         } else {
-             new_type = cur->type;
-             new_data = cur->data;

backend/cpp/ik-llama-cpp/prepare.sh

@@ -1,49 +0,0 @@
-#!/bin/bash
-
-## Patches
-
-## Apply patches from the `patches` directory
-if [ -d "patches" ]; then
-    for patch in $(ls patches); do
-        echo "Applying patch $patch"
-        patch -d llama.cpp/ -p1 < patches/$patch
-    done
-fi
-
-set -e
-
-cp -r CMakeLists.txt llama.cpp/examples/grpc-server/
-cp -r grpc-server.cpp llama.cpp/examples/grpc-server/
-cp -r utils.hpp llama.cpp/examples/grpc-server/
-cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/examples/grpc-server/
-
-## Copy clip/llava files for multimodal support (built as myclip library)
-cp -rfv llama.cpp/examples/llava/clip.h llama.cpp/examples/grpc-server/clip.h
-cp -rfv llama.cpp/examples/llava/clip.cpp llama.cpp/examples/grpc-server/clip.cpp
-cp -rfv llama.cpp/examples/llava/llava.cpp llama.cpp/examples/grpc-server/llava.cpp
-# Prepend llama.h include to llava.h
-echo '#include "llama.h"' > llama.cpp/examples/grpc-server/llava.h
-cat llama.cpp/examples/llava/llava.h >> llama.cpp/examples/grpc-server/llava.h
-# Copy clip-impl.h if it exists
-if [ -f llama.cpp/examples/llava/clip-impl.h ]; then
-    cp -rfv llama.cpp/examples/llava/clip-impl.h llama.cpp/examples/grpc-server/clip-impl.h
-fi
-# Copy stb_image.h
-if [ -f llama.cpp/vendor/stb/stb_image.h ]; then
-    cp -rfv llama.cpp/vendor/stb/stb_image.h llama.cpp/examples/grpc-server/stb_image.h
-elif [ -f llama.cpp/common/stb_image.h ]; then
-    cp -rfv llama.cpp/common/stb_image.h llama.cpp/examples/grpc-server/stb_image.h
-fi
-
-## Fix API compatibility in llava.cpp (llama_n_embd -> llama_model_n_embd)
-if [ -f llama.cpp/examples/grpc-server/llava.cpp ]; then
-    sed -i 's/llama_n_embd(/llama_model_n_embd(/g' llama.cpp/examples/grpc-server/llava.cpp
-fi
-
-set +e
-if grep -q "grpc-server" llama.cpp/examples/CMakeLists.txt; then
-    echo "grpc-server already added"
-else
-    echo "add_subdirectory(grpc-server)" >> llama.cpp/examples/CMakeLists.txt
-fi
-set -e

backend/cpp/ik-llama-cpp/run.sh

@@ -1,40 +0,0 @@
-#!/bin/bash
-set -ex
-
-# Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath $0)")
-
-cd /
-
-echo "CPU info:"
-grep -e "model\sname" /proc/cpuinfo | head -1
-grep -e "flags" /proc/cpuinfo | head -1
-
-# ik_llama.cpp requires AVX2 — default to avx2 binary
-BINARY=ik-llama-cpp-avx2
-
-if [ -e $CURDIR/ik-llama-cpp-fallback ] && ! grep -q -e "\savx2\s" /proc/cpuinfo ; then
-	echo "CPU:    AVX2   NOT found, using fallback"
-	BINARY=ik-llama-cpp-fallback
-fi
-
-# Extend ld library path with the dir where this script is located/lib
-if [ "$(uname)" == "Darwin" ]; then
-	export DYLD_LIBRARY_PATH=$CURDIR/lib:$DYLD_LIBRARY_PATH
-	#export DYLD_FALLBACK_LIBRARY_PATH=$CURDIR/lib:$DYLD_FALLBACK_LIBRARY_PATH
-else
-	export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
-fi
-
-# If there is a lib/ld.so, use it
-if [ -f $CURDIR/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using binary: $BINARY"
-	exec $CURDIR/lib/ld.so $CURDIR/$BINARY "$@"
-fi
-
-echo "Using binary: $BINARY"
-exec $CURDIR/$BINARY "$@"
-
-# We should never reach this point, however just in case we do, run fallback
-exec $CURDIR/ik-llama-cpp-fallback "$@"

backend/cpp/ik-llama-cpp/utils.hpp

@@ -1,483 +0,0 @@
-// https://github.com/ggerganov/llama.cpp/blob/master/examples/server/utils.hpp
-
-#pragma once
-
-#include <string>
-#include <vector>
-#include <set>
-#include <mutex>
-#include <condition_variable>
-#include <unordered_map>
-
-#include "json.hpp"
-
-#include "clip.h"
-
-using json = nlohmann::json;
-
-extern bool server_verbose;
-
-#ifndef SERVER_VERBOSE
-#define SERVER_VERBOSE 1
-#endif
-
-#if SERVER_VERBOSE != 1
-#define LOG_VERBOSE(MSG, ...)
-#else
-#define LOG_VERBOSE(MSG, ...)                                            \
-    do                                                                   \
-    {                                                                    \
-        if (server_verbose)                                              \
-        {                                                                \
-            server_log("VERBOSE", __func__, __LINE__, MSG, __VA_ARGS__); \
-        }                                                                \
-    } while (0)
-#endif
-
-#define LOG_ERROR(  MSG, ...) server_log("ERROR",   __func__, __LINE__, MSG, __VA_ARGS__)
-#define LOG_WARNING(MSG, ...) server_log("WARNING", __func__, __LINE__, MSG, __VA_ARGS__)
-#define LOG_INFO(   MSG, ...) server_log("INFO",    __func__, __LINE__, MSG, __VA_ARGS__)
-
-//
-// parallel
-//
-
-enum server_state {
-    SERVER_STATE_LOADING_MODEL,  // Server is starting up, model not fully loaded yet
-    SERVER_STATE_READY,          // Server is ready and model is loaded
-    SERVER_STATE_ERROR           // An error occurred, load_model failed
-};
-
-enum task_type {
-    TASK_TYPE_COMPLETION,
-    TASK_TYPE_CANCEL,
-    TASK_TYPE_NEXT_RESPONSE
-};
-
-struct task_server {
-    int id = -1; // to be filled by llama_server_queue
-    int target_id;
-    task_type type;
-    json data;
-    bool infill_mode = false;
-    bool embedding_mode = false;
-    int multitask_id = -1;
-};
-
-struct task_result {
-    int id;
-    int multitask_id = -1;
-    bool stop;
-    bool error;
-    json result_json;
-};
-
-struct task_multi {
-    int id;
-    std::set<int> subtasks_remaining{};
-    std::vector<task_result> results{};
-};
-
-// TODO: can become bool if we can't find use of more states
-enum slot_state
-{
-    IDLE,
-    PROCESSING,
-};
-
-enum slot_command
-{
-    NONE,
-    LOAD_PROMPT,
-    RELEASE,
-};
-
-struct slot_params
-{
-    bool stream       = true;
-    bool cache_prompt = false; // remember the prompt to avoid reprocessing all prompt
-
-    uint32_t seed      = -1; // RNG seed
-    int32_t  n_keep    =  0; // number of tokens to keep from initial prompt
-    int32_t  n_predict = -1; // new tokens to predict
-
-    std::vector<std::string> antiprompt;
-
-    json input_prefix;
-    json input_suffix;
-};
-
-struct slot_image
-{
-    int32_t id;
-
-    bool request_encode_image = false;
-    float * image_embedding = nullptr;
-    int32_t image_tokens = 0;
-
-    clip_image_u8 * img_data;
-
-    std::string prefix_prompt; // before of this image
-};
-
-// completion token output with probabilities
-struct completion_token_output
-{
-    struct token_prob
-    {
-        llama_token tok;
-        float prob;
-    };
-
-    std::vector<token_prob> probs;
-    llama_token tok;
-    std::string text_to_send;
-};
-
-static inline void server_log(const char *level, const char *function, int line,
-                       const char *message, const nlohmann::ordered_json &extra)
-{
-    nlohmann::ordered_json log
-    {
-        {"timestamp", time(nullptr)},
-        {"level",     level},
-        {"function",  function},
-        {"line",      line},
-        {"message",   message},
-    };
-
-    if (!extra.empty())
-    {
-        log.merge_patch(extra);
-    }
-
-    const std::string str = log.dump(-1, ' ', false, json::error_handler_t::replace);
-    printf("%.*s\n", (int)str.size(), str.data());
-    fflush(stdout);
-}
-
-//
-// server utils
-//
-
-template <typename T>
-static T json_value(const json &body, const std::string &key, const T &default_value)
-{
-    // Fallback null to default value
-    return body.contains(key) && !body.at(key).is_null()
-        ? body.value(key, default_value)
-        : default_value;
-}
-
-inline std::string format_chatml(std::vector<json> messages)
-{
-    std::ostringstream chatml_msgs;
-
-    for (auto it = messages.begin(); it != messages.end(); ++it) {
-        chatml_msgs << "<|im_start|>"
-                    << json_value(*it, "role",    std::string("user")) << '\n';
-        chatml_msgs << json_value(*it, "content", std::string(""))
-                    << "<|im_end|>\n";
-    }
-
-    chatml_msgs << "<|im_start|>assistant" << '\n';
-
-    return chatml_msgs.str();
-}
-
-//
-// work queue utils
-//
-
-struct llama_server_queue {
-    int id = 0;
-    std::mutex mutex_tasks;
-    // queues
-    std::vector<task_server> queue_tasks;
-    std::vector<task_server> queue_tasks_deferred;
-    std::vector<task_multi> queue_multitasks;
-    std::condition_variable condition_tasks;
-    // callback functions
-    std::function<void(task_server&)> callback_new_task;
-    std::function<void(task_multi&)> callback_finish_multitask;
-    std::function<void(void)> callback_all_task_finished;
-
-    // Add a new task to the end of the queue
-    int post(task_server task) {
-        std::unique_lock<std::mutex> lock(mutex_tasks);
-        if (task.id == -1) {
-            task.id = id++;
-        }
-        queue_tasks.push_back(std::move(task));
-        condition_tasks.notify_one();
-        return task.id;
-    }
-
-    // Add a new task, but defer until one slot is available
-    void defer(task_server task) {
-        std::unique_lock<std::mutex> lock(mutex_tasks);
-        queue_tasks_deferred.push_back(std::move(task));
-    }
-
-    // Get the next id for creating anew task
-    int get_new_id() {
-        std::unique_lock<std::mutex> lock(mutex_tasks);
-        return id++;
-    }
-
-    // Register function to process a new task
-    void on_new_task(std::function<void(task_server&)> callback) {
-        callback_new_task = callback;
-    }
-
-    // Register function to process a multitask
-    void on_finish_multitask(std::function<void(task_multi&)> callback) {
-        callback_finish_multitask = callback;
-    }
-
-    // Register the function to be called when the batch of tasks is finished
-    void on_all_tasks_finished(std::function<void(void)> callback) {
-        callback_all_task_finished = callback;
-    }
-
-    // Call when the state of one slot is changed
-    void notify_slot_changed() {
-        // move deferred tasks back to main loop
-        std::unique_lock<std::mutex> lock(mutex_tasks);
-        for (auto & task : queue_tasks_deferred) {
-            queue_tasks.push_back(std::move(task));
-        }
-        queue_tasks_deferred.clear();
-    }
-
-    // Start the main loop. This call is blocking
-    [[noreturn]]
-    void start_loop() {
-        while (true) {
-            // new task arrived
-            LOG_VERBOSE("have new task", {});
-            {
-                while (true)
-                {
-                    std::unique_lock<std::mutex> lock(mutex_tasks);
-                    if (queue_tasks.empty()) {
-                        lock.unlock();
-                        break;
-                    }
-                    task_server task = queue_tasks.front();
-                    queue_tasks.erase(queue_tasks.begin());
-                    lock.unlock();
-                    LOG_VERBOSE("callback_new_task", {});
-                    callback_new_task(task);
-                }
-                LOG_VERBOSE("callback_all_task_finished", {});
-                // process and update all the multitasks
-                auto queue_iterator = queue_multitasks.begin();
-                while (queue_iterator != queue_multitasks.end())
-                {
-                    if (queue_iterator->subtasks_remaining.empty())
-                    {
-                        // all subtasks done == multitask is done
-                        task_multi current_multitask = *queue_iterator;
-                        callback_finish_multitask(current_multitask);
-                        // remove this multitask
-                        queue_iterator = queue_multitasks.erase(queue_iterator);
-                    }
-                    else
-                    {
-                        ++queue_iterator;
-                    }
-                }
-                // all tasks in the current loop is finished
-                callback_all_task_finished();
-            }
-            LOG_VERBOSE("wait for new task", {});
-            // wait for new task
-            {
-                std::unique_lock<std::mutex> lock(mutex_tasks);
-                if (queue_tasks.empty()) {
-                    condition_tasks.wait(lock, [&]{
-                        return !queue_tasks.empty();
-                    });
-                }
-            }
-        }
-    }
-
-    //
-    // functions to manage multitasks
-    //
-
-    // add a multitask by specifying the id of all subtask (subtask is a task_server)
-    void add_multitask(int multitask_id, std::vector<int>& sub_ids)
-    {
-        std::lock_guard<std::mutex> lock(mutex_tasks);
-        task_multi multi;
-        multi.id = multitask_id;
-        std::copy(sub_ids.begin(), sub_ids.end(), std::inserter(multi.subtasks_remaining, multi.subtasks_remaining.end()));
-        queue_multitasks.push_back(multi);
-    }
-
-    // updatethe remaining subtasks, while appending results to multitask
-    void update_multitask(int multitask_id, int subtask_id, task_result& result)
-    {
-        std::lock_guard<std::mutex> lock(mutex_tasks);
-        for (auto& multitask : queue_multitasks)
-        {
-            if (multitask.id == multitask_id)
-            {
-                multitask.subtasks_remaining.erase(subtask_id);
-                multitask.results.push_back(result);
-            }
-        }
-    }
-};
-
-struct llama_server_response {
-    typedef std::function<void(int, int, task_result&)> callback_multitask_t;
-    callback_multitask_t callback_update_multitask;
-    // for keeping track of all tasks waiting for the result
-    std::set<int> waiting_task_ids;
-    // the main result queue
-    std::vector<task_result> queue_results;
-    std::mutex mutex_results;
-    std::condition_variable condition_results;
-
-    void add_waiting_task_id(int task_id) {
-        std::unique_lock<std::mutex> lock(mutex_results);
-        waiting_task_ids.insert(task_id);
-    }
-
-    void remove_waiting_task_id(int task_id) {
-        std::unique_lock<std::mutex> lock(mutex_results);
-        waiting_task_ids.erase(task_id);
-    }
-
-    // This function blocks the thread until there is a response for this task_id
-    task_result recv(int task_id) {
-        while (true)
-        {
-            std::unique_lock<std::mutex> lock(mutex_results);
-            condition_results.wait(lock, [&]{
-                return !queue_results.empty();
-            });
-            LOG_VERBOSE("condition_results unblock", {});
-
-            for (int i = 0; i < (int) queue_results.size(); i++)
-            {
-                if (queue_results[i].id == task_id)
-                {
-                    assert(queue_results[i].multitask_id == -1);
-                    task_result res = queue_results[i];
-                    queue_results.erase(queue_results.begin() + i);
-                    return res;
-                }
-            }
-        }
-
-        // should never reach here
-    }
-
-    // Register the function to update multitask
-    void on_multitask_update(callback_multitask_t callback) {
-        callback_update_multitask = callback;
-    }
-
-    // Send a new result to a waiting task_id
-    void send(task_result result) {
-        std::unique_lock<std::mutex> lock(mutex_results);
-        LOG_VERBOSE("send new result", {});
-        for (auto& task_id : waiting_task_ids) {
-            // LOG_TEE("waiting task id %i \n", task_id);
-            // for now, tasks that have associated parent multitasks just get erased once multitask picks up the result
-            if (result.multitask_id == task_id)
-            {
-                LOG_VERBOSE("callback_update_multitask", {});
-                callback_update_multitask(task_id, result.id, result);
-                continue;
-            }
-
-            if (result.id == task_id)
-            {
-                LOG_VERBOSE("queue_results.push_back", {});
-                queue_results.push_back(result);
-                condition_results.notify_one();
-                return;
-            }
-        }
-    }
-};
-
-//
-// base64 utils (TODO: move to common in the future)
-//
-
-static const std::string base64_chars =
-             "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
-             "abcdefghijklmnopqrstuvwxyz"
-             "0123456789+/";
-
-static inline bool is_base64(uint8_t c)
-{
-    return (isalnum(c) || (c == '+') || (c == '/'));
-}
-
-static inline std::vector<uint8_t> base64_decode(const std::string & encoded_string)
-{
-    int i = 0;
-    int j = 0;
-    int in_ = 0;
-
-    int in_len = encoded_string.size();
-
-    uint8_t char_array_4[4];
-    uint8_t char_array_3[3];
-
-    std::vector<uint8_t> ret;
-
-    while (in_len-- && (encoded_string[in_] != '=') && is_base64(encoded_string[in_]))
-    {
-        char_array_4[i++] = encoded_string[in_]; in_++;
-        if (i == 4)
-        {
-            for (i = 0; i <4; i++)
-            {
-                char_array_4[i] = base64_chars.find(char_array_4[i]);
-            }
-
-            char_array_3[0] = ((char_array_4[0]      ) << 2) + ((char_array_4[1] & 0x30) >> 4);
-            char_array_3[1] = ((char_array_4[1] & 0xf) << 4) + ((char_array_4[2] & 0x3c) >> 2);
-            char_array_3[2] = ((char_array_4[2] & 0x3) << 6) +   char_array_4[3];
-
-            for (i = 0; (i < 3); i++)
-            {
-                ret.push_back(char_array_3[i]);
-            }
-            i = 0;
-        }
-    }
-
-    if (i)
-    {
-        for (j = i; j <4; j++)
-        {
-            char_array_4[j] = 0;
-        }
-
-        for (j = 0; j <4; j++)
-        {
-            char_array_4[j] = base64_chars.find(char_array_4[j]);
-        }
-
-        char_array_3[0] = ((char_array_4[0]      ) << 2) + ((char_array_4[1] & 0x30) >> 4);
-        char_array_3[1] = ((char_array_4[1] & 0xf) << 4) + ((char_array_4[2] & 0x3c) >> 2);
-        char_array_3[2] = ((char_array_4[2] & 0x3) << 6) +   char_array_4[3];
-
-        for (j = 0; (j < i - 1); j++)
-        {
-            ret.push_back(char_array_3[j]);
-        }
-    }
-
-    return ret;
-}

backend/cpp/llama-cpp/CMakeLists.txt

@@ -62,18 +62,7 @@ add_executable(${TARGET} grpc-server.cpp json.hpp httplib.h)
 target_include_directories(${TARGET} PRIVATE ../llava)
 target_include_directories(${TARGET} PRIVATE ${CMAKE_SOURCE_DIR})
 
-# Upstream llama.cpp renamed the `common` helpers library to `llama-common`.
-# Forks that branched before the rename (e.g. llama-cpp-turboquant) still
-# expose it as `common`. Detect which one is present so the same CMakeLists
-# drives both builds — otherwise an unresolved name silently degrades to a
-# plain `-l` flag and the PUBLIC include dir (where common.h lives) is lost.
-if (TARGET llama-common)
-    set(_LLAMA_COMMON_TARGET llama-common)
-else()
-    set(_LLAMA_COMMON_TARGET common)
-endif()
-
-target_link_libraries(${TARGET} PRIVATE ${_LLAMA_COMMON_TARGET} llama mtmd ${CMAKE_THREAD_LIBS_INIT} absl::flags hw_grpc_proto
+target_link_libraries(${TARGET} PRIVATE common llama mtmd ${CMAKE_THREAD_LIBS_INIT} absl::flags hw_grpc_proto
   absl::flags_parse
   gRPC::${_REFLECTION}
   gRPC::${_GRPC_GRPCPP}

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=1348f67c58f561808136e8a152a9eddec168f221
+LLAMA_VERSION?=5744d7ec430e2f875a393770195fda530560773f
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=
@@ -33,10 +33,7 @@ else ifeq ($(BUILD_TYPE),hipblas)
 	ROCM_PATH ?= /opt/rocm
 	export CXX=$(ROCM_HOME)/llvm/bin/clang++
 	export CC=$(ROCM_HOME)/llvm/bin/clang
-	AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
-ifeq ($(strip $(AMDGPU_TARGETS)),)
-$(error AMDGPU_TARGETS is empty — set it to a comma-separated list of gfx targets e.g. gfx1100,gfx1101)
-endif
+	AMDGPU_TARGETS?=gfx803,gfx900,gfx906,gfx908,gfx90a,gfx942,gfx1010,gfx1030,gfx1032,gfx1100,gfx1101,gfx1102,gfx1200,gfx1201
 	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DGGML_VULKAN=1
@@ -135,7 +132,7 @@ llama.cpp:
 	cd llama.cpp && \
 	git init && \
 	git remote add origin $(LLAMA_REPO)  && \
-	git fetch --all --tags && \
+	git fetch origin && \
 	git checkout -b build $(LLAMA_VERSION) && \
 	git submodule update --init --recursive --depth 1 --single-branch
 

backend/cpp/llama-cpp/grpc-server.cpp

@@ -10,14 +10,6 @@
 #include "server-task.cpp"
 #include "server-queue.cpp"
 #include "server-common.cpp"
-// server-chat.cpp exists only in llama.cpp after the upstream refactor that
-// split OAI/Anthropic/Responses/transcription conversion helpers out of
-// server-common.cpp. When present, server-context.cpp and server-task.cpp
-// above call into it, so we must pull its definitions into this TU or the
-// link fails. __has_include keeps the source compatible with older pins.
-#if __has_include("server-chat.cpp")
-#include "server-chat.cpp"
-#endif
 #include "server-context.cpp"
 
 // LocalAI
@@ -30,15 +22,8 @@
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
 #include <grpcpp/grpcpp.h>
 #include <grpcpp/health_check_service_interface.h>
-#include <grpcpp/security/server_credentials.h>
 #include <regex>
-#include <algorithm>
 #include <atomic>
-#include <cstdlib>
-#include <fstream>
-#include <iterator>
-#include <list>
-#include <map>
 #include <mutex>
 #include <signal.h>
 #include <thread>
@@ -52,64 +37,6 @@ using grpc::Server;
 using grpc::ServerBuilder;
 using grpc::ServerContext;
 using grpc::Status;
-
-// gRPC bearer token auth for distributed mode.
-// Reads LOCALAI_GRPC_AUTH_TOKEN from the environment. When set, rejects
-// requests without a matching "authorization: Bearer <token>" metadata header.
-
-// Cached auth token — empty means auth is disabled.
-static std::string g_grpc_auth_token;
-
-// Minimal constant-time comparison (avoids OpenSSL dependency)
-static int ct_memcmp(const void* a, const void* b, size_t n) {
-    const unsigned char* pa = static_cast<const unsigned char*>(a);
-    const unsigned char* pb = static_cast<const unsigned char*>(b);
-    unsigned char result = 0;
-    for (size_t i = 0; i < n; i++) {
-        result |= pa[i] ^ pb[i];
-    }
-    return result;
-}
-
-// Returns OK when auth is disabled or the token matches.
-static grpc::Status checkAuth(grpc::ServerContext* context) {
-    if (g_grpc_auth_token.empty()) {
-        return grpc::Status::OK;
-    }
-    auto metadata = context->client_metadata();
-    auto it = metadata.find("authorization");
-    if (it != metadata.end()) {
-        std::string expected = "Bearer " + g_grpc_auth_token;
-        std::string got(it->second.data(), it->second.size());
-        if (expected.size() == got.size() &&
-            ct_memcmp(expected.data(), got.data(), expected.size()) == 0) {
-            return grpc::Status::OK;
-        }
-    }
-    return grpc::Status(grpc::StatusCode::UNAUTHENTICATED, "invalid token");
-}
-
-// Minimal base64 encoder. The C++ backend already pulls in base64_decode from
-// llama.cpp's server-common.cpp, but no encoder is exposed — and we need one to
-// hand audio bytes to the existing PredictOptions.audios path (which expects
-// base64-encoded strings, just like images).
-static std::string base64_encode_bytes(const unsigned char* data, size_t len) {
-    static const char tbl[] =
-        "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
-    std::string out;
-    out.reserve(((len + 2) / 3) * 4);
-    for (size_t i = 0; i < len; i += 3) {
-        uint32_t triple = (uint32_t(data[i]) << 16);
-        if (i + 1 < len) triple |= (uint32_t(data[i + 1]) << 8);
-        if (i + 2 < len) triple |= uint32_t(data[i + 2]);
-        out.push_back(tbl[(triple >> 18) & 0x3F]);
-        out.push_back(tbl[(triple >> 12) & 0x3F]);
-        out.push_back(i + 1 < len ? tbl[(triple >> 6) & 0x3F] : '=');
-        out.push_back(i + 2 < len ? tbl[triple & 0x3F]        : '=');
-    }
-    return out;
-}
-
 // END LocalAI
 
 
@@ -209,7 +136,6 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
     data["mirostat_eta"] = predict->mirostateta();
     data["n_keep"] = predict->nkeep();
     data["seed"] = predict->seed();
-    data["min_p"] = predict->minp();
 
 
     std::string grammar_str = predict->grammar();
@@ -318,12 +244,6 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const
     data["ignore_eos"] = predict->ignoreeos();
     data["embeddings"] = predict->embeddings();
 
-    // Speculative decoding per-request overrides
-    // NDraft maps to speculative.n_max (maximum draft tokens per speculation step)
-    if (predict->ndraft() > 0) {
-        data["speculative.n_max"] = predict->ndraft();
-    }
-
     // Add the correlationid to json data
     data["correlation_id"] = predict->correlationid();
 
@@ -442,30 +362,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
     if (!request->mmproj().empty()) {
       params.mmproj.path = request->mmproj();
     }
-
-    // Draft model for speculative decoding
-    if (!request->draftmodel().empty()) {
-        params.speculative.draft.mparams.path = request->draftmodel();
-        // Default to draft type if a draft model is set but no explicit type.
-        // Upstream (post ggml-org/llama.cpp#22838) made the speculative type a
-        // vector; the turboquant fork still uses the legacy scalar. The
-        // LOCALAI_LEGACY_LLAMA_CPP_SPEC macro is injected by
-        // backend/cpp/turboquant/patch-grpc-server.sh for fork builds only.
-        // Upstream renamed COMMON_SPECULATIVE_TYPE_DRAFT -> ..._DRAFT_SIMPLE
-        // in ggml-org/llama.cpp#22964; the fork still uses the old name.
-#ifdef LOCALAI_LEGACY_LLAMA_CPP_SPEC
-        if (params.speculative.type == COMMON_SPECULATIVE_TYPE_NONE) {
-            params.speculative.type = COMMON_SPECULATIVE_TYPE_DRAFT;
-        }
-#else
-        const bool no_spec_type = params.speculative.types.empty() ||
-            (params.speculative.types.size() == 1 && params.speculative.types[0] == COMMON_SPECULATIVE_TYPE_NONE);
-        if (no_spec_type) {
-            params.speculative.types = { COMMON_SPECULATIVE_TYPE_DRAFT_SIMPLE };
-        }
-#endif
-    }
-
     //  params.model_alias ??
     params.model_alias.insert(request->modelfile());
     if (!request->cachetypekey().empty()) {
@@ -659,21 +555,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
                 params.no_op_offload = false;
             }
-        } else if (!strcmp(optname, "split_mode") || !strcmp(optname, "sm")) {
-            // Accepts: none | layer | row | tensor (the latter requires a llama.cpp build
-            // that includes ggml-org/llama.cpp#19378, FlashAttention enabled, and KV-cache
-            // quantization disabled).
-            if (optval != NULL) {
-                if (optval_str == "none") {
-                    params.split_mode = LLAMA_SPLIT_MODE_NONE;
-                } else if (optval_str == "layer") {
-                    params.split_mode = LLAMA_SPLIT_MODE_LAYER;
-                } else if (optval_str == "row") {
-                    params.split_mode = LLAMA_SPLIT_MODE_ROW;
-                } else if (optval_str == "tensor") {
-                    params.split_mode = LLAMA_SPLIT_MODE_TENSOR;
-                }
-            }
         } else if (!strcmp(optname, "kv_unified") || !strcmp(optname, "unified_kv")) {
             if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
                 params.kv_unified = true;
@@ -688,360 +569,7 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                     // If conversion fails, keep default value (8)
                 }
             }
-
-        // --- physical batch size (upstream -ub / --ubatch-size) ---
-        // Note: line ~482 already aliases n_ubatch to n_batch as a default; this
-        // option lets users decouple the two (useful for embeddings/rerank).
-        } else if (!strcmp(optname, "n_ubatch") || !strcmp(optname, "ubatch")) {
-            if (optval != NULL) {
-                try { params.n_ubatch = std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- main-model batch threads (upstream -tb / --threads-batch) ---
-        } else if (!strcmp(optname, "threads_batch") || !strcmp(optname, "n_threads_batch")) {
-            if (optval != NULL) {
-                try {
-                    int n = std::stoi(optval_str);
-                    if (n <= 0) n = (int)std::thread::hardware_concurrency();
-                    params.cpuparams_batch.n_threads = n;
-                } catch (...) {}
-            }
-
-        // --- pooling type for embeddings (upstream --pooling) ---
-        } else if (!strcmp(optname, "pooling_type") || !strcmp(optname, "pooling")) {
-            if (optval != NULL) {
-                if      (optval_str == "none") params.pooling_type = LLAMA_POOLING_TYPE_NONE;
-                else if (optval_str == "mean") params.pooling_type = LLAMA_POOLING_TYPE_MEAN;
-                else if (optval_str == "cls")  params.pooling_type = LLAMA_POOLING_TYPE_CLS;
-                else if (optval_str == "last") params.pooling_type = LLAMA_POOLING_TYPE_LAST;
-                else if (optval_str == "rank") params.pooling_type = LLAMA_POOLING_TYPE_RANK;
-                // unknown values silently leave UNSPECIFIED (auto-detect)
-            }
-
-        // --- llama log verbosity threshold (upstream -lv / --verbosity) ---
-        } else if (!strcmp(optname, "verbosity")) {
-            if (optval != NULL) {
-                try { params.verbosity = std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- O_DIRECT model loading (upstream --direct-io) ---
-        } else if (!strcmp(optname, "direct_io") || !strcmp(optname, "use_direct_io")) {
-            if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
-                params.use_direct_io = true;
-            } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
-                params.use_direct_io = false;
-            }
-
-        // --- embedding normalization (upstream --embd-normalize) ---
-        // -1 none, 0 max-abs, 1 taxicab, 2 L2 (default), >2 p-norm
-        } else if (!strcmp(optname, "embd_normalize") || !strcmp(optname, "embedding_normalize")) {
-            if (optval != NULL) {
-                try { params.embd_normalize = std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- reasoning parser (upstream --reasoning-format) ---
-        // Picks the parser for <think> blocks emitted by reasoning models.
-        // none / auto / deepseek / deepseek-legacy
-        } else if (!strcmp(optname, "reasoning_format")) {
-            if (optval != NULL) {
-                if      (optval_str == "none")             params.reasoning_format = COMMON_REASONING_FORMAT_NONE;
-                else if (optval_str == "auto")             params.reasoning_format = COMMON_REASONING_FORMAT_AUTO;
-                else if (optval_str == "deepseek")         params.reasoning_format = COMMON_REASONING_FORMAT_DEEPSEEK;
-                else if (optval_str == "deepseek-legacy" || optval_str == "deepseek_legacy")
-                                                            params.reasoning_format = COMMON_REASONING_FORMAT_DEEPSEEK_LEGACY;
-                // unknown values silently keep the upstream default (DEEPSEEK)
-            }
-
-        // --- reasoning budget (upstream --reasoning-budget) ---
-        // -1 unlimited, 0 disabled, >0 token budget for thinking blocks.
-        // Distinct from per-request `enable_thinking` (chat_template_kwargs).
-        } else if (!strcmp(optname, "enable_reasoning") || !strcmp(optname, "reasoning_budget")) {
-            if (optval != NULL) {
-                try { params.enable_reasoning = std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- prefill assistant turn (upstream --no-prefill-assistant) ---
-        } else if (!strcmp(optname, "prefill_assistant")) {
-            if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
-                params.prefill_assistant = true;
-            } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
-                params.prefill_assistant = false;
-            }
-
-        // --- mmproj GPU offload (upstream --no-mmproj-offload, inverted) ---
-        } else if (!strcmp(optname, "mmproj_use_gpu") || !strcmp(optname, "mmproj_offload")) {
-            if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
-                params.mmproj_use_gpu = true;
-            } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
-                params.mmproj_use_gpu = false;
-            }
-
-        // --- per-image vision token budget (upstream --image-min/max-tokens) ---
-        } else if (!strcmp(optname, "image_min_tokens")) {
-            if (optval != NULL) {
-                try { params.image_min_tokens = std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "image_max_tokens")) {
-            if (optval != NULL) {
-                try { params.image_max_tokens = std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- main-model tensor buffer overrides (upstream --override-tensor) ---
-        // Format: <tensor regex>=<buffer type>,<tensor regex>=<buffer type>,...
-        // Mirrors the existing `draft_override_tensor` parser below.
-        } else if (!strcmp(optname, "override_tensor") || !strcmp(optname, "tensor_buft_overrides")) {
-            ggml_backend_load_all();
-            std::map<std::string, ggml_backend_buffer_type_t> buft_list;
-            for (size_t i = 0; i < ggml_backend_dev_count(); ++i) {
-                auto * dev = ggml_backend_dev_get(i);
-                auto * buft = ggml_backend_dev_buffer_type(dev);
-                if (buft) {
-                    buft_list[ggml_backend_buft_name(buft)] = buft;
-                }
-            }
-            static std::list<std::string> override_names;
-            std::string cur;
-            auto flush = [&](const std::string & spec) {
-                auto pos = spec.find('=');
-                if (pos == std::string::npos) return;
-                const std::string name = spec.substr(0, pos);
-                const std::string type = spec.substr(pos + 1);
-                auto it = buft_list.find(type);
-                if (it == buft_list.end()) return; // unknown buffer type: ignore
-                override_names.push_back(name);
-                params.tensor_buft_overrides.push_back(
-                    {override_names.back().c_str(), it->second});
-            };
-            for (char c : optval_str) {
-                if (c == ',') { if (!cur.empty()) { flush(cur); cur.clear(); } }
-                else { cur.push_back(c); }
-            }
-            if (!cur.empty()) flush(cur);
-
-        // Speculative decoding options
-        } else if (!strcmp(optname, "spec_type") || !strcmp(optname, "speculative_type")) {
-#ifdef LOCALAI_LEGACY_LLAMA_CPP_SPEC
-            // Fork only knows a single scalar `type`. Take the first comma-
-            // separated value and assign it via the singular helper.
-            std::string first = optval_str;
-            const auto comma = first.find(',');
-            if (comma != std::string::npos) first = first.substr(0, comma);
-            auto type = common_speculative_type_from_name(first);
-            if (type != COMMON_SPECULATIVE_TYPE_COUNT) {
-                params.speculative.type = type;
-            }
-#else
-            // Upstream switched to a vector of types (comma-separated for multi-type
-            // chaining via common_speculative_types_from_names). We keep accepting a
-            // single value here, but also tolerate comma-separated lists.
-            //
-            // ggml-org/llama.cpp#22964 also renamed the registered names from
-            // underscore- to dash-separated form, and replaced the bare
-            // `draft`/`eagle3` aliases with `draft-simple`/`draft-eagle3`. We
-            // normalize each token here so existing model configs keep working.
-            auto normalize_spec_name = [](std::string s) -> std::string {
-                std::replace(s.begin(), s.end(), '_', '-');
-                if (s == "draft")  return "draft-simple";
-                if (s == "eagle3") return "draft-eagle3";
-                return s;
-            };
-            std::vector<std::string> names;
-            std::string item;
-            for (char c : optval_str) {
-                if (c == ',') {
-                    if (!item.empty()) { names.push_back(normalize_spec_name(item)); item.clear(); }
-                } else {
-                    item.push_back(c);
-                }
-            }
-            if (!item.empty()) names.push_back(normalize_spec_name(item));
-            auto parsed = common_speculative_types_from_names(names);
-            if (!parsed.empty()) {
-                params.speculative.types = parsed;
-            }
-#endif
-        } else if (!strcmp(optname, "spec_n_max") || !strcmp(optname, "draft_max")) {
-            if (optval != NULL) {
-                try { params.speculative.draft.n_max = std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_n_min") || !strcmp(optname, "draft_min")) {
-            if (optval != NULL) {
-                try { params.speculative.draft.n_min = std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_p_min") || !strcmp(optname, "draft_p_min")) {
-            if (optval != NULL) {
-                try { params.speculative.draft.p_min = std::stof(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_p_split")) {
-            if (optval != NULL) {
-                try { params.speculative.draft.p_split = std::stof(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_size_n") || !strcmp(optname, "ngram_size_n")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_simple.size_n = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_size_m") || !strcmp(optname, "ngram_size_m")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_simple.size_m = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_min_hits") || !strcmp(optname, "ngram_min_hits")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_simple.min_hits = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "draft_gpu_layers")) {
-            if (optval != NULL) {
-                try { params.speculative.draft.n_gpu_layers = std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "draft_ctx_size")) {
-            // The draft context size is no longer a separate field upstream: the draft
-            // shares the target context size. Accept the option for backward
-            // compatibility but silently ignore it.
-
-// Everything below relies on struct shape introduced in ggml-org/llama.cpp#22838
-// (parallel drafting): `ngram_mod`, `ngram_map_k`, `ngram_map_k4v`,
-// `ngram_cache`, and the `draft.{cache_type_*, cpuparams*, tensor_buft_overrides}`
-// fields. The turboquant fork branched before that, so its build defines
-// LOCALAI_LEGACY_LLAMA_CPP_SPEC via patch-grpc-server.sh and these option
-// keys become unrecognized (silently dropped, like any unknown opt) for it.
-//
-// The `#ifdef LOCALAI_LEGACY_LLAMA_CPP_SPEC` / `#else` split below sits at the
-// closing-brace position of the `draft_ctx_size` branch on purpose: in the
-// legacy build the chain ends here (the brace closes draft_ctx_size), and in
-// the modern build the chain continues with `} else if (...)` instead, so the
-// brace count stays balanced under both branches of the preprocessor.
-#ifdef LOCALAI_LEGACY_LLAMA_CPP_SPEC
         }
-#else
-        // --- ngram_mod family (upstream --spec-ngram-mod-*) ---
-        } else if (!strcmp(optname, "spec_ngram_mod_n_min")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_mod.n_min = std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_mod_n_max")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_mod.n_max = std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_mod_n_match")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_mod.n_match = std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- ngram_map_k family (upstream --spec-ngram-map-k-*) ---
-        } else if (!strcmp(optname, "spec_ngram_map_k_size_n")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_map_k.size_n = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_map_k_size_m")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_map_k.size_m = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_map_k_min_hits")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_map_k.min_hits = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- ngram_map_k4v family (upstream --spec-ngram-map-k4v-*) ---
-        } else if (!strcmp(optname, "spec_ngram_map_k4v_size_n")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_map_k4v.size_n = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_map_k4v_size_m")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_map_k4v.size_m = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-        } else if (!strcmp(optname, "spec_ngram_map_k4v_min_hits")) {
-            if (optval != NULL) {
-                try { params.speculative.ngram_map_k4v.min_hits = (uint16_t)std::stoi(optval_str); } catch (...) {}
-            }
-
-        // --- ngram lookup caches (upstream --lookup-cache-static / -dynamic) ---
-        } else if (!strcmp(optname, "spec_lookup_cache_static") || !strcmp(optname, "lookup_cache_static")) {
-            params.speculative.ngram_cache.lookup_cache_static = optval_str;
-        } else if (!strcmp(optname, "spec_lookup_cache_dynamic") || !strcmp(optname, "lookup_cache_dynamic")) {
-            params.speculative.ngram_cache.lookup_cache_dynamic = optval_str;
-
-        // --- draft model KV cache types (upstream --spec-draft-type-k / -v) ---
-        } else if (!strcmp(optname, "draft_cache_type_k") || !strcmp(optname, "spec_draft_cache_type_k")) {
-            params.speculative.draft.cache_type_k = kv_cache_type_from_str(optval_str);
-        } else if (!strcmp(optname, "draft_cache_type_v") || !strcmp(optname, "spec_draft_cache_type_v")) {
-            params.speculative.draft.cache_type_v = kv_cache_type_from_str(optval_str);
-
-        // --- draft model thread counts (upstream --spec-draft-threads / -batch) ---
-        } else if (!strcmp(optname, "draft_threads") || !strcmp(optname, "spec_draft_threads")) {
-            if (optval != NULL) {
-                try {
-                    int n = std::stoi(optval_str);
-                    if (n <= 0) n = (int)std::thread::hardware_concurrency();
-                    params.speculative.draft.cpuparams.n_threads = n;
-                } catch (...) {}
-            }
-        } else if (!strcmp(optname, "draft_threads_batch") || !strcmp(optname, "spec_draft_threads_batch")) {
-            if (optval != NULL) {
-                try {
-                    int n = std::stoi(optval_str);
-                    if (n <= 0) n = (int)std::thread::hardware_concurrency();
-                    params.speculative.draft.cpuparams_batch.n_threads = n;
-                } catch (...) {}
-            }
-
-        // --- draft model MoE on CPU (upstream --spec-draft-cpu-moe / --spec-draft-n-cpu-moe) ---
-        } else if (!strcmp(optname, "draft_cpu_moe") || !strcmp(optname, "spec_draft_cpu_moe")) {
-            // Bool-style flag: optval may be missing, "true"/"1"/"yes" enables.
-            const bool enable = (optval == NULL) ||
-                optval_str == "true" || optval_str == "1" || optval_str == "yes" ||
-                optval_str == "on" || optval_str == "enabled";
-            if (enable) {
-                params.speculative.draft.tensor_buft_overrides.push_back(llm_ffn_exps_cpu_override());
-            }
-        } else if (!strcmp(optname, "draft_n_cpu_moe") || !strcmp(optname, "spec_draft_n_cpu_moe")) {
-            if (optval != NULL) {
-                try {
-                    int n = std::stoi(optval_str);
-                    if (n < 0) n = 0;
-                    // Keep override-name storage alive for the lifetime of the params struct
-                    // (mirrors upstream arg.cpp behavior with a function-local static).
-                    static std::list<std::string> buft_overrides_draft;
-                    for (int i = 0; i < n; ++i) {
-                        buft_overrides_draft.push_back(llm_ffn_exps_block_regex(i));
-                        params.speculative.draft.tensor_buft_overrides.push_back(
-                            {buft_overrides_draft.back().c_str(), ggml_backend_cpu_buffer_type()});
-                    }
-                } catch (...) {}
-            }
-
-        // --- draft model tensor buffer overrides (upstream --spec-draft-override-tensor) ---
-        } else if (!strcmp(optname, "draft_override_tensor") || !strcmp(optname, "spec_draft_override_tensor")) {
-            // Format: <tensor regex>=<buffer type>,<tensor regex>=<buffer type>,...
-            // We replicate upstream's parse_tensor_buffer_overrides (static in arg.cpp).
-            ggml_backend_load_all();
-            std::map<std::string, ggml_backend_buffer_type_t> buft_list;
-            for (size_t i = 0; i < ggml_backend_dev_count(); ++i) {
-                auto * dev = ggml_backend_dev_get(i);
-                auto * buft = ggml_backend_dev_buffer_type(dev);
-                if (buft) {
-                    buft_list[ggml_backend_buft_name(buft)] = buft;
-                }
-            }
-            static std::list<std::string> draft_override_names;
-            std::string cur;
-            auto flush = [&](const std::string & spec) {
-                auto pos = spec.find('=');
-                if (pos == std::string::npos) return;
-                const std::string name = spec.substr(0, pos);
-                const std::string type = spec.substr(pos + 1);
-                auto it = buft_list.find(type);
-                if (it == buft_list.end()) return; // unknown buffer type: ignore
-                draft_override_names.push_back(name);
-                params.speculative.draft.tensor_buft_overrides.push_back(
-                    {draft_override_names.back().c_str(), it->second});
-            };
-            for (char c : optval_str) {
-                if (c == ',') { if (!cur.empty()) { flush(cur); cur.clear(); } }
-                else { cur.push_back(c); }
-            }
-            if (!cur.empty()) flush(cur);
-        }
-#endif // LOCALAI_LEGACY_LLAMA_CPP_SPEC — closes the `else`/`#ifdef` opened at draft_ctx_size
     }
 
     // Set params.n_parallel from environment variable if not set via options (fallback)
@@ -1185,17 +713,13 @@ private:
 public:
     BackendServiceImpl(server_context& ctx) : ctx_server(ctx) {}
 
-    grpc::Status Health(ServerContext* context, const backend::HealthMessage* /*request*/, backend::Reply* reply) override {
-        auto auth = checkAuth(context);
-        if (!auth.ok()) return auth;
+    grpc::Status Health(ServerContext* /*context*/, const backend::HealthMessage* /*request*/, backend::Reply* reply) override {
         // Implement Health RPC
         reply->set_message("OK");
         return Status::OK;
     }
 
-    grpc::Status LoadModel(ServerContext* context, const backend::ModelOptions* request, backend::Result* result) override {
-        auto auth = checkAuth(context);
-        if (!auth.ok()) return auth;
+    grpc::Status LoadModel(ServerContext* /*context*/, const backend::ModelOptions* request, backend::Result* result) override {
         // Implement LoadModel RPC
         common_params params;
         params_parse(ctx_server, request, params);
@@ -1261,8 +785,8 @@ public:
             if (!params.mmproj.path.empty()) {
                 error_msg += " (with mmproj: " + params.mmproj.path + ")";
             }
-            if (params.speculative.has_dft() && !params.speculative.draft.mparams.path.empty()) {
-                error_msg += " (with draft model: " + params.speculative.draft.mparams.path + ")";
+            if (params.speculative.has_dft() && !params.speculative.mparams_dft.path.empty()) {
+                error_msg += " (with draft model: " + params.speculative.mparams_dft.path + ")";
             }
             
             // Add captured error details if available
@@ -1394,8 +918,6 @@ public:
     }
 
     grpc::Status PredictStream(grpc::ServerContext* context, const backend::PredictOptions* request, grpc::ServerWriter<backend::Reply>* writer) override {
-        auto auth = checkAuth(context);
-        if (!auth.ok()) return auth;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -1683,7 +1205,6 @@ public:
 
                 body_json["messages"] = messages_json;
                 body_json["stream"] = true; // PredictStream is always streaming
-                body_json["stream_options"] = {{"include_usage", true}}; // Ensure token counts in final chunk
 
                 // Check if grammar is provided from Go layer (NoGrammar=false)
                 // If grammar is provided, we must use it and NOT let template generate grammar from tools
@@ -1988,15 +1509,11 @@ public:
                         ctx_server.impl->vocab,
                         params_base,
                         ctx_server.get_meta().slot_n_ctx,
-                        ctx_server.get_meta().logit_bias_eog,
                         data);
                 task.id_slot = json_value(data, "id_slot", -1);
 
-                // OAI-compat: enable autoparser (PEG-based chat parsing) so that
-                // reasoning, tool calls, and content are classified into ChatDeltas.
-                // Without this, the PEG parser never produces diffs and the Go side
-                // cannot detect tool calls or separate reasoning from content.
-                task.params.res_type                 = TASK_RESPONSE_TYPE_OAI_CHAT;
+                // OAI-compat
+                task.params.res_type                 = TASK_RESPONSE_TYPE_NONE;
                 task.params.oaicompat_cmpl_id         = completion_id;
                 // oaicompat_model is already populated by params_from_json_cmpl
 
@@ -2021,47 +1538,19 @@ public:
             return grpc::Status(grpc::StatusCode::INTERNAL, error_json.value("message", "Error occurred"));
         }
 
-        // Lambda to build a Reply from JSON + attach chat deltas from a result.
-        // Handles both native format ({"content": "..."}) and OAI chat format
-        // ({"choices": [{"delta": {"content": "...", "reasoning": "..."}}]}).
+        // Lambda to build a Reply from JSON + attach chat deltas from a result
         auto build_reply_from_json = [](const json & res_json, server_task_result * raw_result) -> backend::Reply {
             backend::Reply reply;
-            std::string completion_text;
-
-            if (res_json.contains("choices")) {
-                // OAI chat format — extract content from choices[0].delta
-                const auto & choices = res_json.at("choices");
-                if (!choices.empty()) {
-                    const auto & delta = choices[0].value("delta", json::object());
-                    if (delta.contains("content") && !delta.at("content").is_null()) {
-                        completion_text = delta.at("content").get<std::string>();
-                    }
-                }
-            } else {
-                // Native llama.cpp format
-                completion_text = res_json.value("content", "");
-            }
-
+            std::string completion_text = res_json.value("content", "");
             reply.set_message(completion_text);
+            reply.set_tokens(res_json.value("tokens_predicted", 0));
+            reply.set_prompt_tokens(res_json.value("tokens_evaluated", 0));
 
-            // Token counts: native format has top-level fields,
-            // OAI format has them in "usage" (final chunk only)
-            if (res_json.contains("usage")) {
-                const auto & usage = res_json.at("usage");
-                reply.set_tokens(usage.value("completion_tokens", 0));
-                reply.set_prompt_tokens(usage.value("prompt_tokens", 0));
-            } else {
-                reply.set_tokens(res_json.value("tokens_predicted", 0));
-                reply.set_prompt_tokens(res_json.value("tokens_evaluated", 0));
-            }
-
-            // Timings: present as top-level "timings" in both formats
             if (res_json.contains("timings")) {
                 reply.set_timing_prompt_processing(res_json.at("timings").value("prompt_ms", 0.0));
                 reply.set_timing_token_generation(res_json.at("timings").value("predicted_ms", 0.0));
             }
 
-            // Logprobs: extract_logprobs_from_json handles both formats
             json logprobs_json = extract_logprobs_from_json(res_json);
             if (!logprobs_json.empty() && !logprobs_json.is_null()) {
                 reply.set_logprobs(logprobs_json.dump());
@@ -2070,12 +1559,6 @@ public:
             return reply;
         };
 
-        // Attach chat deltas from the autoparser to a Reply.
-        // When diffs are available, populate ChatDeltas on the reply.
-        // The raw message is always preserved so the Go side can use it
-        // for reasoning extraction and tool call parsing as a fallback
-        // (important in distributed mode where ChatDeltas may not be
-        // the primary parsing path).
         auto attach_chat_deltas = [](backend::Reply & reply, server_task_result * raw_result) {
             // Try streaming partial result first
             auto* partial = dynamic_cast<server_task_result_cmpl_partial*>(raw_result);
@@ -2090,23 +1573,12 @@ public:
             }
         };
 
-        // Process first result.
-        // When TASK_RESPONSE_TYPE_OAI_CHAT is used, the first token may
-        // produce a JSON array with a role-init element followed by the
-        // actual content element. We must only attach chat deltas to the
-        // content element — attaching to both would duplicate the first
-        // token since oaicompat_msg_diffs is the same for both.
+        // Process first result
         json first_res_json = first_result->to_json();
         if (first_res_json.is_array()) {
             for (const auto & res : first_res_json) {
                 auto reply = build_reply_from_json(res, first_result.get());
-                // Skip chat deltas for role-init elements (have "role" in
-                // delta but no content/reasoning diffs of their own).
-                bool is_role_init = res.contains("choices") && !res["choices"].empty() &&
-                                    res["choices"][0].value("delta", json::object()).contains("role");
-                if (!is_role_init) {
-                    attach_chat_deltas(reply, first_result.get());
-                }
+                attach_chat_deltas(reply, first_result.get());
                 writer->Write(reply);
             }
         } else {
@@ -2130,11 +1602,7 @@ public:
             if (res_json.is_array()) {
                 for (const auto & res : res_json) {
                     auto reply = build_reply_from_json(res, result.get());
-                    bool is_role_init = res.contains("choices") && !res["choices"].empty() &&
-                                        res["choices"][0].value("delta", json::object()).contains("role");
-                    if (!is_role_init) {
-                        attach_chat_deltas(reply, result.get());
-                    }
+                    attach_chat_deltas(reply, result.get());
                     writer->Write(reply);
                 }
             } else {
@@ -2153,8 +1621,6 @@ public:
     }
 
     grpc::Status Predict(ServerContext* context, const backend::PredictOptions* request, backend::Reply* reply) override {
-         auto auth = checkAuth(context);
-         if (!auth.ok()) return auth;
          if (params_base.model.path.empty()) {
              return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
          }
@@ -2772,13 +2238,11 @@ public:
                         ctx_server.impl->vocab,
                         params_base,
                         ctx_server.get_meta().slot_n_ctx,
-                        ctx_server.get_meta().logit_bias_eog,
                         data);
                 task.id_slot = json_value(data, "id_slot", -1);
 
-                // OAI-compat: enable autoparser (PEG-based chat parsing) so that
-                // reasoning, tool calls, and content are classified into ChatDeltas.
-                task.params.res_type                 = TASK_RESPONSE_TYPE_OAI_CHAT;
+                // OAI-compat
+                task.params.res_type                 = TASK_RESPONSE_TYPE_NONE;
                 task.params.oaicompat_cmpl_id         = completion_id;
                 // oaicompat_model is already populated by params_from_json_cmpl
 
@@ -2809,48 +2273,25 @@ public:
                 auto* final_res = dynamic_cast<server_task_result_cmpl_final*>(all_results.results[0].get());
                 GGML_ASSERT(final_res != nullptr);
                 json result_json = all_results.results[0]->to_json();
+                reply->set_message(result_json.value("content", ""));
 
-                // Handle both native format ({"content": "...", "tokens_predicted": N})
-                // and OAI chat format ({"choices": [{"message": {"content": "..."}}],
-                // "usage": {"completion_tokens": N, "prompt_tokens": N}}).
-                std::string completion_text;
-                int32_t tokens_predicted = 0;
-                int32_t tokens_evaluated = 0;
-
-                if (result_json.contains("choices")) {
-                    // OAI chat format
-                    const auto & choices = result_json.at("choices");
-                    if (!choices.empty()) {
-                        const auto & msg = choices[0].value("message", json::object());
-                        if (msg.contains("content") && !msg.at("content").is_null()) {
-                            completion_text = msg.at("content").get<std::string>();
-                        }
-                    }
-                    if (result_json.contains("usage")) {
-                        const auto & usage = result_json.at("usage");
-                        tokens_predicted = usage.value("completion_tokens", 0);
-                        tokens_evaluated = usage.value("prompt_tokens", 0);
-                    }
-                } else {
-                    // Native llama.cpp format
-                    completion_text = result_json.value("content", "");
-                    tokens_predicted = result_json.value("tokens_predicted", 0);
-                    tokens_evaluated = result_json.value("tokens_evaluated", 0);
-                }
-                reply->set_message(completion_text);
+                int32_t tokens_predicted = result_json.value("tokens_predicted", 0);
                 reply->set_tokens(tokens_predicted);
+                int32_t tokens_evaluated = result_json.value("tokens_evaluated", 0);
                 reply->set_prompt_tokens(tokens_evaluated);
 
-                // Timings: present in both formats as a top-level "timings" object
                 if (result_json.contains("timings")) {
-                    reply->set_timing_prompt_processing(result_json.at("timings").value("prompt_ms", 0.0));
-                    reply->set_timing_token_generation(result_json.at("timings").value("predicted_ms", 0.0));
+                    double timing_prompt_processing = result_json.at("timings").value("prompt_ms", 0.0);
+                    reply->set_timing_prompt_processing(timing_prompt_processing);
+                    double timing_token_generation = result_json.at("timings").value("predicted_ms", 0.0);
+                    reply->set_timing_token_generation(timing_token_generation);
                 }
 
-                // Logprobs: extract_logprobs_from_json handles both formats
+                // Extract and set logprobs if present
                 json logprobs_json = extract_logprobs_from_json(result_json);
                 if (!logprobs_json.empty() && !logprobs_json.is_null()) {
-                    reply->set_logprobs(logprobs_json.dump());
+                    std::string logprobs_str = logprobs_json.dump();
+                    reply->set_logprobs(logprobs_str);
                 }
 
                 // Populate chat deltas from the autoparser's final parsed message
@@ -2866,20 +2307,7 @@ public:
                 for (auto & res : all_results.results) {
                     GGML_ASSERT(dynamic_cast<server_task_result_cmpl_final*>(res.get()) != nullptr);
                     json res_json = res->to_json();
-                    // Handle both native and OAI chat formats
-                    std::string result_content;
-                    if (res_json.contains("choices")) {
-                        const auto & choices = res_json.at("choices");
-                        if (!choices.empty()) {
-                            const auto & msg = choices[0].value("message", json::object());
-                            if (msg.contains("content") && !msg.at("content").is_null()) {
-                                result_content = msg.at("content").get<std::string>();
-                            }
-                        }
-                    } else {
-                        result_content = res_json.value("content", "");
-                    }
-                    arr.push_back(result_content);
+                    arr.push_back(res_json.value("content", ""));
 
                     // Extract logprobs for each result
                     json logprobs_json = extract_logprobs_from_json(res_json);
@@ -2911,8 +2339,6 @@ public:
     }
 
     grpc::Status Embedding(ServerContext* context, const backend::PredictOptions* request, backend::EmbeddingResult* embeddingResult) override {
-        auto auth = checkAuth(context);
-        if (!auth.ok()) return auth;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -2938,9 +2364,7 @@ public:
             }
         }
 
-        // Honor the load-time embd_normalize set via options:embd_normalize.
-        // -1 none, 0 max-abs, 1 taxicab, 2 L2 (default), >2 p-norm.
-        int embd_normalize = params_base.embd_normalize;
+        int embd_normalize = 2; // default to Euclidean/L2 norm
         // create and queue the task
         auto rd = ctx_server.get_response_reader();
         {
@@ -3034,7 +2458,7 @@ public:
 
             tasks.reserve(documents.size());
             for (size_t i = 0; i < documents.size(); i++) {
-                auto tmp = format_prompt_rerank(ctx_server.impl->model_tgt, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i]);
+                auto tmp = format_prompt_rerank(ctx_server.impl->model, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i]);
                 server_task task = server_task(SERVER_TASK_TYPE_RERANK);
                 task.id = rd.queue_tasks.get_new_id();
                 task.index = i;
@@ -3095,9 +2519,7 @@ public:
         return grpc::Status::OK;
     }
 
-    grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
-        auto auth = checkAuth(context);
-        if (!auth.ok()) return auth;
+    grpc::Status TokenizeString(ServerContext* /*context*/, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -3167,13 +2589,6 @@ public:
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
 
-        // Report the active multimodal media marker so the Go layer can emit the
-        // same string when rendering prompts outside the tokenizer-template path.
-        // Only meaningful when an mtmd context was initialized (vision/audio models).
-        if (ctx_server.impl->mctx != nullptr) {
-            response->set_media_marker(get_media_marker());
-        }
-
         // Check if chat templates are initialized
         if (ctx_server.impl->chat_params.tmpls == nullptr) {
             // If templates are not initialized, we can't detect thinking support
@@ -3212,7 +2627,7 @@ public:
                 // Get template source and reconstruct a common_chat_template for analysis
                 std::string tmpl_src = common_chat_templates_source(ctx_server.impl->chat_params.tmpls.get());
                 if (!tmpl_src.empty()) {
-                    const auto * vocab = llama_model_get_vocab(ctx_server.impl->model_tgt);
+                    const auto * vocab = llama_model_get_vocab(ctx_server.impl->model);
                     std::string token_bos, token_eos;
                     if (vocab) {
                         auto bos_id = llama_vocab_bos(vocab);
@@ -3272,6 +2687,7 @@ public:
                         tf->set_id_field(ap.tools.format.id_field);
                         tf->set_fun_name_is_key(ap.tools.format.fun_name_is_key);
                         tf->set_tools_array_wrapped(ap.tools.format.tools_array_wrapped);
+                        tf->set_uses_python_dicts(ap.tools.format.uses_python_dicts);
                         tf->set_function_field(ap.tools.format.function_field);
 
                         tf->set_gen_id_field(ap.tools.format.gen_id_field);
@@ -3314,119 +2730,6 @@ public:
 
         return grpc::Status::OK;
     }
-
-    // runTranscriptionAsCompletion implements OAI /v1/audio/transcriptions on
-    // top of the existing chat-completion + multimodal-audio pipeline, exactly
-    // the way upstream llama.cpp's server does it (see
-    // tools/server/server-context.cpp post_transcriptions_oai → forwards into
-    // handle_completions_impl with a single user message attaching the audio
-    // file via the mtmd marker).
-    //
-    // We synthesize a backend::PredictOptions with one user message
-    // ("Transcribe audio to text" + optional language hint) and the audio
-    // bytes attached via the existing PredictOptions.audios field, then
-    // delegate to our own Predict() handler. This keeps every multimodal
-    // codepath identical to the chat path and avoids duplicating ~700 lines
-    // of task-construction logic.
-    grpc::Status runTranscriptionAsCompletion(grpc::ServerContext* context,
-                                              const backend::TranscriptRequest* request,
-                                              backend::Reply* out_reply) {
-        if (params_base.model.path.empty()) {
-            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
-        }
-        if (request->dst().empty()) {
-            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "dst (audio file path) is required");
-        }
-
-        // Read audio bytes from the path LocalAI's HTTP layer wrote.
-        std::ifstream f(request->dst(), std::ios::binary);
-        if (!f.is_open()) {
-            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "failed to open audio file: " + request->dst());
-        }
-        std::vector<unsigned char> bytes((std::istreambuf_iterator<char>(f)),
-                                          std::istreambuf_iterator<char>());
-        f.close();
-        if (bytes.empty()) {
-            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "audio file is empty: " + request->dst());
-        }
-
-        std::string b64 = base64_encode_bytes(bytes.data(), bytes.size());
-
-        // Build the same prompt upstream uses in convert_transcriptions_to_chatcmpl.
-        std::string user_prompt = "Transcribe audio to text";
-        if (!request->language().empty()) {
-            user_prompt += " (language: " + request->language() + ")";
-        }
-        if (!request->prompt().empty()) {
-            // Optional context hint from the caller.
-            user_prompt += "\n" + request->prompt();
-        }
-
-        backend::PredictOptions synthetic;
-        synthetic.set_usetokenizertemplate(true);
-        synthetic.set_temperature(request->temperature());
-        // Generation length: leave at 0 so parse_options uses -1 (model default).
-        // The model's stop tokens / EOS handle termination naturally for ASR.
-        backend::Message* msg = synthetic.add_messages();
-        msg->set_role("user");
-        msg->set_content(user_prompt);
-        synthetic.add_audios(b64);
-
-        return Predict(context, &synthetic, out_reply);
-    }
-
-    grpc::Status AudioTranscription(ServerContext* context,
-                                    const backend::TranscriptRequest* request,
-                                    backend::TranscriptResult* response) override {
-        auto auth = checkAuth(context);
-        if (!auth.ok()) return auth;
-
-        backend::Reply reply;
-        grpc::Status st = runTranscriptionAsCompletion(context, request, &reply);
-        if (!st.ok()) {
-            return st;
-        }
-        response->set_text(reply.message());
-        if (!request->language().empty()) {
-            response->set_language(request->language());
-        }
-        return grpc::Status::OK;
-    }
-
-    grpc::Status AudioTranscriptionStream(ServerContext* context,
-                                          const backend::TranscriptRequest* request,
-                                          grpc::ServerWriter<backend::TranscriptStreamResponse>* writer) override {
-        auto auth = checkAuth(context);
-        if (!auth.ok()) return auth;
-
-        // Buffered streaming: run the transcription as a normal chat
-        // completion, then emit one delta + one final event. Real
-        // token-by-token streaming would require refactoring PredictStream's
-        // 700-line writer-coupled body; the HTTP/SSE contract is identical
-        // either way, and clients that only consume the assembled text don't
-        // notice the difference.
-        backend::Reply reply;
-        grpc::Status st = runTranscriptionAsCompletion(context, request, &reply);
-        if (!st.ok()) {
-            return st;
-        }
-
-        const std::string& text = reply.message();
-        if (!text.empty()) {
-            backend::TranscriptStreamResponse delta_chunk;
-            delta_chunk.set_delta(text);
-            writer->Write(delta_chunk);
-        }
-
-        backend::TranscriptStreamResponse final_chunk;
-        backend::TranscriptResult* final_result = final_chunk.mutable_final_result();
-        final_result->set_text(text);
-        if (!request->language().empty()) {
-            final_result->set_language(request->language());
-        }
-        writer->Write(final_chunk);
-        return grpc::Status::OK;
-    }
 };
 
 
@@ -3458,18 +2761,10 @@ int main(int argc, char** argv) {
 
     ServerBuilder builder;
     builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
-
-    // Initialize bearer token auth if LOCALAI_GRPC_AUTH_TOKEN is set
-    const char* auth_token = std::getenv("LOCALAI_GRPC_AUTH_TOKEN");
-    if (auth_token != nullptr && auth_token[0] != '\0') {
-        g_grpc_auth_token = auth_token;
-        std::cout << "gRPC auth enabled via LOCALAI_GRPC_AUTH_TOKEN" << std::endl;
-    }
     builder.RegisterService(&service);
     builder.SetMaxMessageSize(50 * 1024 * 1024); // 50MB
     builder.SetMaxSendMessageSize(50 * 1024 * 1024); // 50MB
     builder.SetMaxReceiveMessageSize(50 * 1024 * 1024); // 50MB
-
     std::unique_ptr<Server> server(builder.BuildAndStart());
    // run the HTTP server in a thread - see comment below
     std::thread t([&]()

backend/cpp/llama-cpp/package.sh

@@ -24,9 +24,6 @@ if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
     cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
     cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
     cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
 elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
     # ARM64 architecture
     echo "Detected ARM64 architecture, copying ARM64 libraries..."
@@ -36,9 +33,6 @@ elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
     cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
     cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
     cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
 else
     echo "Error: Could not detect architecture"
     exit 1

backend/cpp/llama-cpp/run.sh

@@ -46,10 +46,6 @@ if [ "$(uname)" == "Darwin" ]; then
 	#export DYLD_FALLBACK_LIBRARY_PATH=$CURDIR/lib:$DYLD_FALLBACK_LIBRARY_PATH
 else
 	export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
-	# Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files)
-	if [ -d "$CURDIR/lib/rocblas/library" ]; then
-		export ROCBLAS_TENSILE_LIBPATH=$CURDIR/lib/rocblas/library
-	fi
 fi
 
 # If there is a lib/ld.so, use it

backend/cpp/turboquant/Makefile

@@ -1,81 +0,0 @@
-
-# Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
-# Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-TURBOQUANT_VERSION?=5aeb2fdbe26cd4c534c6fa15de73cb5749bd0403
-LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
-
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-ONEAPI_VARS?=/opt/intel/oneapi/setvars.sh
-TARGET?=--target grpc-server
-JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
-ARCH?=$(shell uname -m)
-
-CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
-LLAMA_CPP_DIR := $(CURRENT_MAKEFILE_DIR)/../llama-cpp
-
-GREEN := \033[0;32m
-RESET := \033[0m
-
-# turboquant is a llama.cpp fork. Rather than duplicating grpc-server.cpp / CMakeLists.txt /
-# prepare.sh we reuse the ones in backend/cpp/llama-cpp, and only swap which repo+sha the
-# fetch step pulls. Each flavor target copies ../llama-cpp into a sibling ../turboquant-<flavor>-build
-# directory, then invokes llama-cpp's own build-llama-cpp-grpc-server with LLAMA_REPO/LLAMA_VERSION
-# overridden to point at the fork.
-PATCHES_DIR := $(CURRENT_MAKEFILE_DIR)/patches
-
-# Each flavor target:
-#   1. copies backend/cpp/llama-cpp/ (grpc-server.cpp + prepare.sh + CMakeLists.txt + Makefile)
-#      into a sibling turboquant-<flavor>-build directory;
-#   2. clones the turboquant fork into turboquant-<flavor>-build/llama.cpp via the copy's
-#      own `llama.cpp` target, overriding LLAMA_REPO/LLAMA_VERSION;
-#   3. applies patches from backend/cpp/turboquant/patches/ to the cloned fork sources
-#      (needed until the fork catches up with upstream server-context.cpp changes);
-#   4. runs the copy's `grpc-server` target, which produces the binary we copy up as
-#      turboquant-<flavor>.
-define turboquant-build
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build
-	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build purge
-	# Augment the copied grpc-server.cpp's KV-cache allow-list with the
-	# fork's turbo2/turbo3/turbo4 types. We patch the *copy*, never the
-	# original under backend/cpp/llama-cpp/, so the stock llama-cpp build
-	# stays compiling against vanilla upstream.
-	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
-	$(info $(GREEN)I turboquant build info:$(1)$(RESET))
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build llama.cpp
-	bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/llama.cpp $(PATCHES_DIR)
-	CMAKE_ARGS="$(CMAKE_ARGS) $(2)" TARGET="$(3)" \
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server turboquant-$(1)
-endef
-
-turboquant-avx2:
-	$(call turboquant-build,avx2,-DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
-
-turboquant-avx512:
-	$(call turboquant-build,avx512,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
-
-turboquant-avx:
-	$(call turboquant-build,avx,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
-
-turboquant-fallback:
-	$(call turboquant-build,fallback,-DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
-
-turboquant-grpc:
-	$(call turboquant-build,grpc,-DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server --target rpc-server)
-
-turboquant-rpc-server: turboquant-grpc
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-grpc-build/llama.cpp/build/bin/rpc-server turboquant-rpc-server
-
-package:
-	bash package.sh
-
-purge:
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-*-build
-	rm -rf turboquant-* package
-
-clean: purge

backend/cpp/turboquant/apply-patches.sh

@@ -1,50 +0,0 @@
-#!/bin/bash
-# Apply the turboquant patch series to a cloned llama-cpp-turboquant checkout.
-#
-# The turboquant fork branched from upstream llama.cpp before a few API changes
-# that the shared backend/cpp/llama-cpp/grpc-server.cpp depends on. We carry
-# those upstream commits as patch files under backend/cpp/turboquant/patches/
-# and apply them here so the reused grpc-server source compiles against the
-# fork unmodified.
-#
-# Drop the corresponding patch from patches/ whenever the fork catches up with
-# upstream — the build will fail fast if a patch stops applying, which is the
-# signal to retire it.
-
-set -euo pipefail
-
-if [[ $# -ne 2 ]]; then
-    echo "usage: $0 <llama.cpp-src-dir> <patches-dir>" >&2
-    exit 2
-fi
-
-SRC_DIR=$1
-PATCHES_DIR=$2
-
-if [[ ! -d "$SRC_DIR" ]]; then
-    echo "source dir does not exist: $SRC_DIR" >&2
-    exit 2
-fi
-
-if [[ ! -d "$PATCHES_DIR" ]]; then
-    echo "no patches dir at $PATCHES_DIR, nothing to apply"
-    exit 0
-fi
-
-shopt -s nullglob
-patches=("$PATCHES_DIR"/*.patch)
-shopt -u nullglob
-
-if [[ ${#patches[@]} -eq 0 ]]; then
-    echo "no .patch files in $PATCHES_DIR, nothing to apply"
-    exit 0
-fi
-
-cd "$SRC_DIR"
-
-for patch in "${patches[@]}"; do
-    echo "==> applying $patch"
-    git apply --verbose "$patch"
-done
-
-echo "all turboquant patches applied successfully"

backend/cpp/turboquant/package.sh

@@ -1,57 +0,0 @@
-#!/bin/bash
-
-# Script to copy the appropriate libraries based on architecture
-# This script is used in the final stage of the Dockerfile
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-# Create lib directory
-mkdir -p $CURDIR/package/lib
-
-cp -avrf $CURDIR/turboquant-* $CURDIR/package/
-cp -rfv $CURDIR/run.sh $CURDIR/package/
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    # x86_64 architecture
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    # ARM64 architecture
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/