chore(recon): re-pin to squashed release commits (voice 1db1759, face e22260d)

Both engine repos were squashed to a single release commit and force-pushed, which
orphaned the previous pins (voice 30beecd, face 6107a24). Re-pin to the new
single-commit SHAs. These also fold in a real fix: the persistent graph-cache
fingerprint now includes op_params, so two structurally identical GGML_OP_CUSTOM
graphs (e.g. a blocked 3x3 vs a blocked 1x1 strided conv) can no longer false-hit
the cache and replay the wrong kernel. voice CI is now green (was failing
conv1x1_s2 with an out-of-bounds write on the GGML_NATIVE=OFF build); face stays
green. WeSpeaker embed parity 1.0 vs golden.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

.agents/adding-backends.md

@@ -34,7 +34,7 @@ The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `
 
 **Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
 
-**Path-filter registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/lib/backend-filter.mjs:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
+**`scripts/changed-backends.js` registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/changed-backends.js:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
 
 ```js
 if (item.dockerfile.endsWith("<your-dockerfile-suffix>")) {
@@ -54,9 +54,7 @@ for (const e of m.include.filter(e => e.backend === '<your-backend>')) {
 }"
 ```
 
-A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/lib/backend-filter.mjs`.
-
-If your backend consumes a *shared* build input that lives outside its own directory (a new script under `scripts/build/`, a new file copied into every image), add a rule to `SHARED_BUILD_INPUTS` in the same file — the per-backend prefix match cannot see those, and a miss ships your change to no image at all. See `scripts/lib/backend-filter_test.mjs` for the pattern; `make test-ci-scripts` runs it.
+A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/changed-backends.js`.
 
 **`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=<sha>` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`):
 
@@ -104,24 +102,6 @@ Multi-arch backends are NOT a single matrix entry with `platforms: 'linux/amd64,
 
 Entries whose `dockerfile` is `./backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` must also set a `builder-base-image` field pointing at a prebuilt base from `quay.io/go-skynet/ci-cache:base-grpc-*` (CI builds these via `.github/workflows/base-images.yml`). The mapping is by `(build-type, platforms)` — see existing entries for the pattern. CI uses these prebuilt bases to skip the gRPC compile (~25–35 min cold). Local `make backends/<name>` ignores `builder-base-image` and uses the from-source path inside the Dockerfile, so you don't need quay access for local builds.
 
-### Cover every OS the project supports (Linux **and** Darwin)
-
-`.github/backend-matrix.yml` has two matrices, and they are the source of truth for which OS a backend ships on:
-
-- `include:` — the **Linux** matrix (x86_64 + arm64; CPU and CUDA / ROCm / SYCL / Vulkan).
-- `includeDarwin:` — the **macOS / Apple Silicon** matrix (arm64; Metal where the engine supports it, otherwise a native arm64 CPU build).
-
-**A new backend must target every OS it can build for — do not ship Linux-only by default.** A backend that appears only under `include:` is silently unavailable on macOS even when its code would run there. Most C/C++/GGML engines build on Darwin out of the box (ggml defaults `GGML_METAL=ON` on Apple, so a plain build is Metal-enabled), and many Python backends do too (CPU / MPS wheels). If a backend genuinely cannot support an OS (e.g. CUDA-only, no CPU variant), state that in the PR description instead of omitting it silently.
-
-Wiring a backend into `includeDarwin:` is more than the matrix entry:
-
-1. **`includeDarwin:` entry** — `tag-suffix: "-metal-darwin-arm64-<backend>"`, `build-type: "metal"`, `lang: "go"` for go+ggml backends; omit `build-type` for the bespoke C++ ones (llama-cpp / ds4 / privacy-filter). Match an existing entry of the same shape.
-2. **`backend/index.yaml`** — add `metal:` to the backend's `capabilities` map (main and `-development`) and concrete `metal-<backend>` / `metal-<backend>-development` image entries pointing at the `-metal-darwin-arm64-<backend>` images.
-3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/lib/backend-filter.mjs` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
-4. **C++ proto gotcha** — if the backend compiles the generated gRPC/protobuf in a separate CMake target (e.g. `hw_grpc_proto`), that target must link `protobuf::libprotobuf` + `gRPC::grpc++` so the Homebrew include dirs propagate; otherwise macOS fails with `google/protobuf/runtime_version.h not found` (Linux hides this because apt headers sit in `/usr/include`).
-
-The CI path filter only builds a backend on a PR when a file under its directory changes, so a darwin-only YAML edit builds nothing — touch a file under `backend/<lang>/<backend>/` (a one-line comment is enough) in the same PR.
-
 ## 3. Add Backend Metadata to `backend/index.yaml`
 
 **Step 3a: Add Meta Definition**
@@ -218,69 +198,6 @@ docker-build-backends: ... docker-build-<backend-name>
 - If the backend is in `backend/python/<backend-name>/` but uses `.` as context in the workflow file, use `.` context
 - Check similar backends to determine the correct context
 
-## Engine preference for gallery model variants
-
-A gallery entry can declare `variants`, alternative builds of the same weights,
-and LocalAI picks one per host: it drops builds whose backend cannot run here or
-that do not fit memory, then ranks the survivors by **engine preference
-first, serving feature second, size third** (`SelectVariant` in
-`core/gallery/resolve_variant.go`).
-
-Ask whether your backend should outrank another one on some hardware. If it
-should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`,
-best engine first for that capability:
-
-```go
- 	{Nvidia, []string{engineVLLM, engineSGLang, engineLlamaCpp}},
-+	{Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}},
-```
-
-That is the ENGINE NAME table, matched as a substring of a gallery entry's
-`backend:` value. Two sibling tables in the same file speak different
-vocabularies and are matched against different things:
-
-| Table | Vocabulary | Matched against | Consumer |
-|-------|-----------|-----------------|----------|
-| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` |
-| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking |
-| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine |
-
-**Putting a token in the wrong table matches nothing and does not error**: every
-candidate scores equal and the next sort key decides, so the preference silently
-stops existing. The block comment above all three tables spells the contract out.
-
-The serving feature table is the odd one: it is not keyed by capability, because
-no hardware prefers a plain build over an equivalent faster build of the same
-weights. It reads a declared tag and nothing else. The entry name was the
-original signal and is gone: a naming convention is not a contract, and names
-are author-supplied free text where a short marker like `mtp` turns up inside
-unrelated words or on weights whose entry enables nothing.
-`overrides.options` was rejected for the mirror-image reason: `spec_type:` is
-llama.cpp's config vocabulary, whereas a cross-backend ranking decision must
-work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`.
-
-**If your backend can serve the same weights faster** (speculative decoding,
-multi-token prediction), say so in the docs for its gallery entries so curators
-tag them: the tagging rule and the per-backend evidence table live in
-[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to
-appear in the token table itself; it ranks builds, not engines.
-
-Leaving your backend out is a valid choice when no ordering can be justified for
-it. It then ranks below every known engine and selection falls back to size,
-which is the behaviour that predates preference.
-
-**Leaving a whole capability out is not.** A missing row gives that host an
-empty preference list, so size alone decides among everything that survives the
-filters, and the filter will not save you: `IsBackendCompatible` derives hardware
-support from the engine NAME, so `vllm` and `sglang` carry no darwin, cuda, rocm
-or sycl token and are never dropped on a host with no GPU. That is why `default`
-(no usable accelerator, including a GPU under the 4 GiB VRAM floor) and
-`darwin-x86` both have rows putting `llama-cpp` first. Every capability
-`getSystemCapabilities()` can return needs a row unless every engine really is
-equally at home there. When you add one, enumerate the engines you are demoting
-rather than relying on them falling through unmatched: unmatched engines all tie
-with each other, so size decides among them.
-
 ## Documenting the backend (README + docs)
 
 A backend is not "added" until it is discoverable. Update the user-facing docs:
@@ -308,7 +225,6 @@ After adding a new backend, verify:
 
 - [ ] Backend directory structure is complete with all necessary files
 - [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
-- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case (in `scripts/lib/backend-filter.mjs`) for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
 - [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
 - [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
 - [ ] Tag suffixes match between workflow file and index.yaml
@@ -316,8 +232,6 @@ After adding a new backend, verify:
 - [ ] No YAML syntax errors (check with linter)
 - [ ] No Makefile syntax errors (check with linter)
 - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
-- [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`.
-- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide.
 - [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`)
 - [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md`
 

.agents/adding-gallery-models.md

@@ -91,108 +91,6 @@ To add a variant (e.g., different quantization), use YAML merge:
       uri: huggingface://<gguf-org>/<gguf-repo>/<filename>-Q8_0.gguf
 ```
 
-## Offering several builds of one model (`variants`)
-
-When the same model is published in more than one quantization, or is also
-servable by another engine, add each build as its own ordinary gallery entry and
-then point one of them at the others with `variants`:
-
-```yaml
-- !!merge <<: *chatml
-  name: "nanbeige4.1-3b-q4"
-  # ... the usual urls / overrides / files for the Q4 build ...
-  variants:
-    - model: nanbeige4.1-3b-q8
-```
-
-Rules:
-
-- The declaring entry is a **complete, normal entry**. It keeps its own
-  `files`/`overrides` and stays installable on every host and by every older
-  LocalAI release, which simply ignore `variants`.
-- A variant references another gallery entry **by name**. That entry must exist
-  and must not declare `variants` of its own.
-- **A referenced entry keeps its own gallery row by default.** It is hidden only
-  in the collapsed listing (`collapse_variants=true`, which the web UI requests
-  by default), where the declaring entry stands in for it. Searching there still
-  matches the referenced entry and answers with the entry declaring it, so
-  referencing an entry never makes it unfindable; turning the collapse off
-  returns it under its own name.
-- **Order carries no meaning.** Do not try to encode a preference; write the
-  list in whatever order reads best.
-- **A variant may be smaller than the declaring entry.** Offering a downgrade
-  for small hosts is a normal shape: the declaring entry's own build competes
-  like every other candidate, so a large host keeps the large build.
-- **Do not describe hardware.** At install time LocalAI drops variants whose
-  backend cannot run on the host, then drops those that do not fit available
-  memory. The declaring entry's own build is exempt from both filters, so
-  selection always terminates on something installable. Sizes are measured live
-  from the weights and cached, so nothing has to be written down.
-- **Engine preference outranks size.** Among the builds that survive the
-  filters, the host's preferred engine wins first and only then does the larger
-  footprint win. On NVIDIA a vLLM build beats a larger llama.cpp one; on Apple
-  silicon an MLX build beats a larger GGUF one; on a host with no preference for
-  either engine the larger build wins, since a bigger footprint is a higher
-  quality quantization of the same weights. Predict what a user gets by asking
-  which engine the host prefers before asking which build is biggest. The
-  per-capability order lives in `engineNamePreferenceRules`
-  (`pkg/system/capabilities.go`); see
-  [adding-backends.md](adding-backends.md) for how a backend gets into it.
-- **Serving feature preference sits between engine and size.** Among builds on
-  an equally preferred engine, one that speculates or predicts several tokens
-  per step beats the plain build of the same weights, because it answers faster
-  for the same output: a `dflash` build beats an `mtp` one, and either beats a
-  plain build. The order lives in `servingFeaturePreferenceTokens`
-  (`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and
-  **nothing else**: not the entry name, not `overrides.options`. See
-  [the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately
-  outranks it: a serving feature makes the right engine faster, it does not make
-  a wrong engine right. Fit still outranks both, so a drafter pairing (strictly
-  larger than the plain build, since it ships a drafter alongside it) is dropped
-  on a host too small for it before this order is ever consulted.
-- A variant is nothing but a name; there is no per-variant memory field. When
-  the measured size for a build is wrong, correct it on the referenced entry by
-  setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers
-  a declared size over its own guesswork, so the fix applies everywhere the size
-  is shown or compared rather than only to variant selection.
-
-Users can override the automatic choice with `variant` on `POST /models/apply`,
-`local-ai models install --variant`, or the `install_model` MCP tool. See
-`docs/content/features/model-gallery.md`.
-
-The gallery lint specs live in `core/gallery`, so run that suite after adding a
-`variants` list.
-
-### The `dflash` / `mtp` tagging rule
-
-**Tag an entry `dflash` or `mtp` when the entry actually configures that
-feature. Variant ranking reads the tag and nothing else.**
-
-Decide by looking at what the entry configures, in whatever vocabulary its
-backend uses:
-
-| Backend | Configures the feature when it declares |
-|---------|------------------------------------------|
-| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` |
-| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` |
-| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` |
-
-That check is curation-time only. `spec_type` is llama.cpp's config vocabulary,
-and a cross-backend ranking decision must not depend on one backend's option
-syntax, which is precisely why the ranker reads the tag instead of the options.
-
-Two mistakes the rule exists to prevent:
-
-- **Weights that carry the heads are not an entry that enables them.** The
-  NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so
-  they enable no speculative decoding and must NOT be tagged. Tagging them wins
-  them the feature axis without being any faster.
-- **A name is not a declaration.** An entry whose name spells `-mtp` while
-  configuring nothing gets no tag, and an entry that configures the feature is
-  tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads
-  the name, so an untagged build that does enable the feature is simply ranked
-  as plain rather than promoted on a marker nobody meant.
-
 ## Available template configs
 
 Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture:

.agents/ci-caching.md

@@ -114,24 +114,6 @@ Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynami
 - **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
 - **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
 
-### Shared build inputs
-
-The per-backend prefix match only sees files under a backend's own directory, so a change to shared build infrastructure would rebuild *nothing* — an empty matrix, every job green, and the change reaching no image. That silently un-shipped PR #10946 (a partial-cuDNN packaging fix in `scripts/build/package-gpu-libs.sh`), which merged 1h48m after the weekly cron and so sat unbuilt for a week.
-
-`SHARED_BUILD_INPUTS` in `scripts/lib/backend-filter.mjs` closes that hole. Each rule maps a shared path to the narrowest set of matrix entries it can honestly invalidate, since a full matrix is 417 Linux + 56 Darwin builds:
-
-| Changed path | Rebuilds |
-|---|---|
-| `backend/backend.proto` | everything (all languages compile or copy it) |
-| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
-| `backend/python/common/` | Python, Linux + Darwin |
-| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
-| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
-| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
-| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
-
-Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
-
 The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
 
 ## The `DEPS_REFRESH` cache-buster (Python backends)

.agents/coding-style.md

@@ -65,7 +65,6 @@ This is enforced by `forbidigo` (see `.golangci.yml`): `http.DefaultClient` and
 
 The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.
 
-- **Docs-with-code rule**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. The PR template carries a checklist item for this.
 - **Feature Documentation**: If you add a new feature (like a new backend or API endpoint), create a new markdown file in `docs/content/features/` explaining what it is, how to configure it, and how to use it.
 - **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
 - **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.

.docker/bonsai-compile.sh

@@ -1,39 +0,0 @@
-#!/usr/bin/env bash
-# Shared compile logic for backend/Dockerfile.bonsai.
-# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
-
-set -euxo pipefail
-
-export CCACHE_DIR=/root/.ccache
-ccache --max-size=5G || true
-ccache -z || true
-
-export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
-
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
-  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
-  rm -rf /LocalAI/backend/cpp/bonsai-*-build
-fi
-
-cd /LocalAI/backend/cpp/bonsai
-
-if [ -z "${BUILD_TYPE:-}" ]; then
-  # Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries.
-  # arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme).
-  if [ "${TARGETARCH}" = "arm64" ]; then
-    apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
-    export CC=gcc-14 CXX=g++-14
-  fi
-  make bonsai-cpu-all
-else
-  # GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator
-  # does the compute. Keeps the GPU compile from also building the CPU variant matrix and
-  # avoids the gcc-14 apt step on GPU base images such as nvidia l4t.
-  make bonsai-fallback
-fi
-make bonsai-grpc
-make bonsai-rpc-server
-
-ccache -s || true

.docker/llama-cpp-compile.sh

@@ -17,29 +17,19 @@ if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
   rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
 fi
 
-cd /LocalAI/backend/cpp/llama-cpp
-if [ -z "${BUILD_TYPE:-}" ]; then
-  # Pure CPU image (BUILD_TYPE empty): one build with ggml CPU_ALL_VARIANTS replaces the
-  # per-microarch binaries (x86: avx/avx2/avx512/fallback; arm64: armv8.x/armv9.x). ggml
-  # dlopens the best libggml-cpu-*.so at runtime by probing host CPU features.
-  #
-  # arm64: the CPU_ALL_VARIANTS table includes armv9.2 SME variants whose -march=...+sme is
-  # rejected by the Ubuntu 24.04 default gcc-13. gcc-14 accepts it, so build the arm64
-  # variants with it (the host never *selects* SME unless it has it, but every variant must
-  # still compile).
-  if [ "${TARGETARCH}" = "arm64" ]; then
-    apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
-    export CC=gcc-14 CXX=g++-14
-  fi
-  make llama-cpp-cpu-all
-else
-  # GPU build (cublas/hipblas/sycl/vulkan/...): the accelerator does the compute, so a
-  # single fallback CPU build is enough - no per-microarch CPU variants needed. (This also
-  # keeps the heavy GPU backend compile from also building the whole CPU variant matrix,
-  # and avoids the gcc-14 apt step on GPU base images such as nvidia l4t.)
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  cd /LocalAI/backend/cpp/llama-cpp
   make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
+else
+  cd /LocalAI/backend/cpp/llama-cpp
+  make llama-cpp-avx
+  make llama-cpp-avx2
+  make llama-cpp-avx512
+  make llama-cpp-fallback
+  make llama-cpp-grpc
+  make llama-cpp-rpc-server
 fi
-make llama-cpp-grpc
-make llama-cpp-rpc-server
 
 ccache -s || true

.docker/turboquant-compile.sh

@@ -19,21 +19,17 @@ fi
 
 cd /LocalAI/backend/cpp/turboquant
 
-if [ -z "${BUILD_TYPE:-}" ]; then
-  # Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries.
-  # arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme).
-  if [ "${TARGETARCH}" = "arm64" ]; then
-    apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
-    export CC=gcc-14 CXX=g++-14
-  fi
-  make turboquant-cpu-all
-else
-  # GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator
-  # does the compute. Keeps the GPU compile from also building the CPU variant matrix and
-  # avoids the gcc-14 apt step on GPU base images such as nvidia l4t.
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
   make turboquant-fallback
+  make turboquant-grpc
+  make turboquant-rpc-server
+else
+  make turboquant-avx
+  make turboquant-avx2
+  make turboquant-avx512
+  make turboquant-fallback
+  make turboquant-grpc
+  make turboquant-rpc-server
 fi
-make turboquant-grpc
-make turboquant-rpc-server
 
 ccache -s || true

.githooks/pre-commit

@@ -7,11 +7,8 @@
 # Runs only the checks relevant to what's staged:
 #   - Go files          -> make lint + make test-coverage-check
 #   - core/http/react-ui -> make test-ui-coverage-check (Playwright e2e + gate)
-#   - realtime state machines / specs -> make test-realtime-conformance
-#       (respcoord/**, turncoord/**, or formal-verification/** -- a pure .fizz
-#        spec edit must still re-verify the design, detected separately from Go)
-# A commit touching none of these is skipped entirely (other docs/YAML can't
-# change lint findings, Go coverage, the UI, or the realtime conformance gate).
+# A commit touching neither is skipped entirely (docs/YAML/etc. can't change
+# lint findings, Go coverage, or the UI).
 #
 # To bypass for a single commit (e.g. a WIP checkpoint): git commit --no-verify
 set -eu
@@ -23,13 +20,11 @@ staged="$(git diff --cached --name-only --diff-filter=ACMRD)"
 
 go_changed=0
 ui_changed=0
-rt_changed=0
 if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
 if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
-if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
 
-if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
-	echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
+if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ]; then
+	echo "pre-commit: no Go or React UI changes staged — skipping."
 	exit 0
 fi
 
@@ -62,11 +57,4 @@ if [ "$ui_changed" -eq 1 ]; then
 	make test-ui-coverage-check
 fi
 
-if [ "$rt_changed" -eq 1 ]; then
-	echo "pre-commit ▶ realtime state-machine conformance (make test-realtime-conformance) —"
-	echo "             Go transition/rapid tests under -race + FizzBee model check of the"
-	echo "             authoritative specs. Fail-closed: needs FizzBee (make install-fizzbee)."
-	make test-realtime-conformance
-fi
-
 echo "pre-commit ✓ all relevant checks passed"

.github/PULL_REQUEST_TEMPLATE.md

@@ -7,7 +7,6 @@ This PR fixes #
 
 **[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
 - [ ] Yes, I signed my commits.
-- [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable
  
 <!--
 Thank you for contributing to LocalAI! 

.github/backend-matrix.yml

@@ -2,28 +2,6 @@
 # Matrix data for backend container image builds.
 # Consumed by scripts/changed-backends.js for both backend.yml and backend_pr.yml.
 # This file is NOT a workflow — it has no top-level 'on:' or 'jobs:'.
-#
-# OS / platform coverage — READ THIS WHEN ADDING A BACKEND
-# --------------------------------------------------------
-# This file is the source of truth for which OS each backend is built and
-# published for. A backend ships ONLY for the matrices it appears in:
-#   - Linux  -> the `include:` matrix below (x86_64 + arm64; CPU and
-#               CUDA / ROCm / SYCL / Vulkan variants).
-#   - macOS  -> the `includeDarwin:` matrix (Apple Silicon / arm64; Metal where
-#               the engine supports it, otherwise a native arm64 CPU build).
-#
-# New backends must target EVERY OS they can build for, not just Linux. A backend
-# listed only under `include:` is silently unavailable on macOS even when its code
-# would run there. Most C/C++/GGML engines build on Darwin (ggml defaults
-# GGML_METAL=ON on Apple, so a plain build is Metal-enabled), and many Python
-# backends do too (CPU / MPS). If a backend genuinely cannot support an OS, say so
-# in its PR description rather than silently omitting it.
-#
-# Adding a backend to `includeDarwin:` is more than one line — see the darwin
-# checklist in .agents/adding-backends.md (includeDarwin entry, the index.yaml
-# `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD
-# branch for C/C++ backends, and the inferBackendPathDarwin case in
-# scripts/lib/backend-filter.mjs so the path filter actually builds it).
 
 # Linux matrix (consumed by backend-jobs).
 include:
@@ -452,22 +430,6 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-12-amd64'
-    # bigger-runner: same rationale as -gpu-nvidia-cuda-12-llama-cpp above
-    # (observed 6h5m wall-clock on v4.2.1, just past the 6h job timeout).
-    runs-on: 'bigger-runner'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -494,19 +456,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-longcat-video'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "longcat-video"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -819,19 +768,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-moss-transcribe-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -858,19 +794,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-moss-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1111,21 +1034,6 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64'
-    # bigger-runner: observed 6h5m wall-clock on v4.2.1 — at the GHA timeout.
-    runs-on: 'bigger-runner'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1154,20 +1062,6 @@ include:
     backend: "turboquant"
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-arm64'
-    base-image: "ubuntu:24.04"
-    runs-on: 'ubuntu-24.04-arm'
-    ubuntu-version: '2404'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1220,19 +1114,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-longcat-video'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "longcat-video"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1441,19 +1322,6 @@ include:
     backend: "vllm-omni"
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
-  - build-type: 'l4t'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-longcat-video'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    ubuntu-version: '2404'
-    backend: "longcat-video"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
   - build-type: 'l4t'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1831,19 +1699,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-moss-transcribe-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1883,19 +1738,6 @@ include:
     backend: "parakeet-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-moss-transcribe-cpp'
-    base-image: "ubuntu:24.04"
-    ubuntu-version: '2404'
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1922,19 +1764,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-moss-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1987,19 +1816,6 @@ include:
     backend: "qwen3-tts-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-moss-tts-cpp'
-    base-image: "ubuntu:24.04"
-    ubuntu-version: '2404'
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2067,20 +1883,6 @@ include:
     dockerfile: "./backend/Dockerfile.llama-cpp"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'hipblas'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-rocm-hipblas-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-rocm-amd64'
-    runs-on: 'ubuntu-latest'
-    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -2345,20 +2147,6 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f32'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f32-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -2387,20 +2175,6 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f16'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f16-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'intel'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -2853,21 +2627,6 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-amd64'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -2883,21 +2642,6 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-arm64'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3040,20 +2784,6 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2204'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-arm64-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-l4t-cuda-12-arm64'
-    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2204'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3100,22 +2830,6 @@ include:
     context: "./"
     ubuntu-version: '2404'
   # Stablediffusion-ggml
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-amd64'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
-  # Stablediffusion-ggml
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3132,22 +2846,6 @@ include:
     context: "./"
     ubuntu-version: '2404'
   # Stablediffusion-ggml
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-bonsai'
-    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-arm64'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "bonsai"
-    dockerfile: "./backend/Dockerfile.bonsai"
-    context: "./"
-    ubuntu-version: '2404'
-  # Stablediffusion-ggml
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3877,115 +3575,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  # moss-transcribe-cpp
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-moss-transcribe-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-moss-transcribe-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'sycl_f32'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f32-moss-transcribe-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'sycl_f16'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f16-moss-transcribe-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-moss-transcribe-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-moss-transcribe-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-arm64-moss-transcribe-cpp'
-    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2204'
-  - build-type: 'hipblas'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-rocm-hipblas-moss-transcribe-cpp'
-    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
-    runs-on: 'ubuntu-latest'
-    skip-drivers: 'false'
-    backend: "moss-transcribe-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   # ced
   - build-type: 'cublas'
     cuda-major-version: "12"
@@ -4568,35 +4157,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  # moss-tts-cpp
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-moss-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-moss-tts-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   # omnivoice-cpp
   - build-type: ''
     cuda-major-version: ""
@@ -4639,19 +4199,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f32'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f32-moss-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4678,19 +4225,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'sycl_f16'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-sycl-f16-moss-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4718,20 +4252,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-moss-tts-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4760,20 +4280,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'vulkan'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-vulkan-moss-tts-cpp'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -4801,19 +4307,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2204'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    skip-drivers: 'false'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-arm64-moss-tts-cpp'
-    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-    runs-on: 'ubuntu-24.04-arm'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2204'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -4840,19 +4333,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'hipblas'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-rocm-hipblas-moss-tts-cpp'
-    base-image: "rocm/dev-ubuntu-24.04:6.4.4"
-    runs-on: 'ubuntu-latest'
-    skip-drivers: 'false'
-    backend: "moss-tts-cpp"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -5150,6 +4630,7 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # rfdetr
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -5164,35 +4645,6 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
-  # cloud-proxy
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-cloud-proxy'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "cloud-proxy"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-cloud-proxy'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "cloud-proxy"
-    dockerfile: "./backend/Dockerfile.golang"
-    context: "./"
-    ubuntu-version: '2404'
   # rfdetr
   - build-type: ''
     cuda-major-version: ""
@@ -5746,10 +5198,6 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-parakeet-cpp"
     build-type: "metal"
     lang: "go"
-  - backend: "moss-transcribe-cpp"
-    tag-suffix: "-metal-darwin-arm64-moss-transcribe-cpp"
-    build-type: "metal"
-    lang: "go"
   - backend: "ced"
     tag-suffix: "-metal-darwin-arm64-ced"
     build-type: "metal"
@@ -5770,10 +5218,6 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-qwen3-tts-cpp"
     build-type: "metal"
     lang: "go"
-  - backend: "moss-tts-cpp"
-    tag-suffix: "-metal-darwin-arm64-moss-tts-cpp"
-    build-type: "metal"
-    lang: "go"
   - backend: "omnivoice-cpp"
     tag-suffix: "-metal-darwin-arm64-omnivoice-cpp"
     build-type: "metal"
@@ -5782,37 +5226,6 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-vibevoice-cpp"
     build-type: "metal"
     lang: "go"
-  # Vision/utility C++/ggml backends (go+cgo). Their Makefiles already carry a
-  # Darwin/Metal path (GGML_METAL=ON when build-type=metal); this just builds and
-  # publishes the metal image so Apple Silicon can install them.
-  - backend: "depth-anything-cpp"
-    tag-suffix: "-metal-darwin-arm64-depth-anything-cpp"
-    build-type: "metal"
-    lang: "go"
-  - backend: "locate-anything-cpp"
-    tag-suffix: "-metal-darwin-arm64-locate-anything-cpp"
-    build-type: "metal"
-    lang: "go"
-  - backend: "rfdetr-cpp"
-    tag-suffix: "-metal-darwin-arm64-rfdetr-cpp"
-    build-type: "metal"
-    lang: "go"
-  - backend: "sam3-cpp"
-    tag-suffix: "-metal-darwin-arm64-sam3-cpp"
-    build-type: "metal"
-    lang: "go"
-  # privacy-filter (PII/NER) is a C++/ggml backend built by a bespoke darwin
-  # script (make backends/privacy-filter-darwin); ggml defaults Metal ON on Apple
-  # so the build is Metal-enabled. lang=go drives runner/toolchain selection only.
-  - backend: "privacy-filter"
-    tag-suffix: "-metal-darwin-arm64-privacy-filter"
-    lang: "go"
-  # LocalVQE has no Metal path; on Apple Silicon it builds CPU-only (GGML_METAL
-  # OFF) but is still a native arm64 image. Uses the darwin/metal build profile.
-  - backend: "localvqe"
-    tag-suffix: "-metal-darwin-arm64-localvqe"
-    build-type: "metal"
-    lang: "go"
   - backend: "voxtral"
     tag-suffix: "-metal-darwin-arm64-voxtral"
     build-type: "metal"
@@ -5829,6 +5242,9 @@ includeDarwin:
   - backend: "qwen-tts"
     tag-suffix: "-metal-darwin-arm64-qwen-tts"
     build-type: "mps"
+  - backend: "fish-speech"
+    tag-suffix: "-metal-darwin-arm64-fish-speech"
+    build-type: "mps"
   - backend: "voxcpm"
     tag-suffix: "-metal-darwin-arm64-voxcpm"
     build-type: "mps"
@@ -5862,19 +5278,6 @@ includeDarwin:
   - backend: "kitten-tts"
     tag-suffix: "-metal-darwin-arm64-kitten-tts"
     build-type: "mps"
-  # vLLM on Apple Silicon via vllm-metal (MLX). The install is custom
-  # (backend/python/vllm/install.sh has a darwin branch); lang stays python so
-  # backend_build_darwin.yml drives it through build-darwin-python-backend ->
-  # scripts/build/python-darwin.sh, which runs the backend's install.sh.
-  - backend: "vllm"
-    tag-suffix: "-metal-darwin-arm64-vllm"
-    build-type: "mps"
-  - backend: "trl"
-    tag-suffix: "-metal-darwin-arm64-trl"
-    build-type: "mps"
-  - backend: "liquid-audio"
-    tag-suffix: "-metal-darwin-arm64-liquid-audio"
-    build-type: "mps"
   - backend: "piper"
     tag-suffix: "-metal-darwin-arm64-piper"
     build-type: "metal"
@@ -5891,18 +5294,10 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-sherpa-onnx"
     build-type: "metal"
     lang: "go"
-  - backend: "supertonic"
-    tag-suffix: "-metal-darwin-arm64-supertonic"
-    build-type: "metal"
-    lang: "go"
   - backend: "local-store"
     tag-suffix: "-metal-darwin-arm64-local-store"
     build-type: "metal"
     lang: "go"
-  - backend: "cloud-proxy"
-    tag-suffix: "-metal-darwin-arm64-cloud-proxy"
-    build-type: "metal"
-    lang: "go"
   - backend: "llama-cpp-quantization"
     tag-suffix: "-metal-darwin-arm64-llama-cpp-quantization"
     build-type: "mps"

.github/bump_deps.sh

@@ -9,19 +9,7 @@ if [ -z "$FILE" ]; then
     FILE="Makefile"
 fi
 
-# -L so a renamed/transferred upstream repo (GitHub answers 301) still
-# resolves instead of handing us the redirect body, and -f so an HTTP error
-# aborts the run rather than letting an error page reach sed below.
-LAST_COMMIT=$(curl -sfL -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")
-
-# Guard the sed input: anything that is not a bare 40-hex SHA (an API error
-# body, an empty response) would otherwise be spliced into the Makefile pin —
-# either corrupting it silently or blowing up sed with an unterminated
-# expression, which is how this job failed for a renamed repo.
-if ! [[ "$LAST_COMMIT" =~ ^[0-9a-f]{40}$ ]]; then
-    echo "Refusing to bump $VAR: expected a 40-char commit SHA for $REPO@$BRANCH, got: $LAST_COMMIT" >&2
-    exit 1
-fi
+LAST_COMMIT=$(curl -s -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")
 
 # Read $VAR from Makefile (only first match)
 set +e

.github/bump_vllm_metal.sh

@@ -1,55 +0,0 @@
-#!/bin/bash
-# Bump the single vllm-metal pin (VLLM_METAL_VERSION) in the vLLM backend's
-# darwin (Apple Silicon) install path. The macOS/Metal build
-# (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is
-# version-locked to a specific vLLM source release. install.sh derives that vLLM
-# version at build time from vllm-metal's own installer (`vllm_v=`) at the pinned
-# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh,
-# which bumps the Linux cu130 wheel pin.
-#
-# This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the
-# darwin build can only use the exact vLLM version vllm-metal supports, so it may
-# lag the Linux pin (requirements-cublas13-after.txt) until vllm-metal catches up.
-set -xe
-REPO=$1   # vllm-project/vllm-metal
-FILE=$2   # backend/python/vllm/install.sh
-VAR=$3    # VLLM_METAL_VERSION (used for the workflow's output file names)
-
-if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
-    echo "usage: $0 <repo> <install-file> <var-name>" >&2
-    exit 1
-fi
-
-# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
-# /releases/latest returns the newest one (with its cp312 wheel asset).
-LATEST_TAG=$(curl -sS -H "Accept: application/vnd.github+json" \
-    "https://api.github.com/repos/$REPO/releases/latest" \
-    | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
-
-# The coupled vLLM source version lives in vllm-metal's installer at that tag.
-NEW_VLLM_VERSION=$(curl -fsSL \
-    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
-    | grep -oE 'vllm_v="[0-9]+\.[0-9]+\.[0-9]+"' | head -1 | cut -d'"' -f2)
-
-if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then
-    echo "Could not resolve vllm-metal tag ($LATEST_TAG) or its vllm_v ($NEW_VLLM_VERSION)." >&2
-    exit 1
-fi
-
-set +e
-CURRENT_TAG=$(grep -oE 'VLLM_METAL_VERSION="[^"]*"' "$FILE" | head -1 | cut -d'"' -f2)
-set -e
-
-# Rewrite the single pin. install.sh derives VLLM_VERSION from this tag at build
-# time, so there is nothing else to touch. peter-evans/create-pull-request opens
-# no PR on a clean tree, so a no-op rewrite (already current) is safe.
-sed -i "$FILE" \
-    -e "s|VLLM_METAL_VERSION=\"[^\"]*\"|VLLM_METAL_VERSION=\"$LATEST_TAG\"|"
-
-if [ -z "$CURRENT_TAG" ]; then
-    echo "Could not find VLLM_METAL_VERSION=\"...\" in $FILE." >&2
-    exit 0
-fi
-
-echo "vllm-metal ${CURRENT_TAG} -> ${LATEST_TAG} (builds vLLM ${NEW_VLLM_VERSION}): https://github.com/$REPO/releases/tag/${LATEST_TAG}" >> "${VAR}_message.txt"
-echo "${LATEST_TAG}" >> "${VAR}_commit.txt"

.github/ci/variantproposals/body.go

@@ -1,133 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"strings"
-)
-
-// RenderBody writes the pull request body.
-//
-// The body is the product of this job, not the diff. Grouping is a judgement
-// call that has gone wrong in both directions before, so a reviewer has to be
-// able to accept or reject each family from the body alone, without opening
-// HuggingFace to work out whether two entries hold the same weights.
-func RenderBody(r *Result, ledgerPath string) string {
-	var b strings.Builder
-
-	b.WriteString("## Proposed gallery variant groupings\n\n")
-	b.WriteString("This is a proposal, not a decision. The gallery agent adds one build per model and never joins an existing family, so entries that are alternative builds of the same weights drift apart as the gallery grows. This job re-applies the grouping heuristics from the manual sweeps and asks a human to confirm.\n\n")
-	b.WriteString("Each family below lists the parent, the variants, and the evidence that they are the same weights. **Reject anything whose evidence you do not believe.**\n\n")
-	b.WriteString(fmt.Sprintf("To decline a family permanently, add one line to `%s` in this pull request and close it:\n\n", ledgerPath))
-	b.WriteString("```yaml\npairs:\n  - {parent: some-model, variant: some-model-thing, reason: \"different finetune\"}\n```\n\n")
-
-	b.WriteString(fmt.Sprintf("### Proposed families (%d)\n\n", len(r.Families)))
-	if len(r.Families) == 0 {
-		b.WriteString("None.\n\n")
-	}
-	for _, f := range r.Families {
-		b.WriteString(fmt.Sprintf("#### `%s`\n\n", f.Parent))
-		b.WriteString("| variant | signals | evidence |\n|---|---|---|\n")
-		for _, p := range f.Proposals {
-			b.WriteString(fmt.Sprintf("| `%s` | %s | %s |\n", p.Variant, joinSignals(p.Evidence.Signals), describeEvidence(p.Evidence)))
-		}
-		b.WriteString("\n")
-	}
-
-	b.WriteString(fmt.Sprintf("### Declined by the ledger (%d)\n\n", len(r.Suppressed)))
-	if len(r.Suppressed) == 0 {
-		b.WriteString("Nothing the heuristics found was already on the ledger.\n\n")
-	} else {
-		b.WriteString("Candidates the heuristics found and the ledger has already settled. They are listed so the ledger's effect stays visible rather than silently shrinking the job's output.\n\n")
-		for _, s := range r.Suppressed {
-			b.WriteString(fmt.Sprintf("- `%s` + `%s`: %s\n", s.A, s.B, s.Reason))
-		}
-		b.WriteString("\n")
-	}
-
-	if len(r.AliasSkipped) > 0 {
-		b.WriteString(fmt.Sprintf("### Aliases, not variants (%d)\n\n", len(r.AliasSkipped)))
-		b.WriteString("These entries install byte for byte the same payload. An alias exists so clients can send a particular name; folding it under another entry would hide that name.\n\n")
-		for _, s := range r.AliasSkipped {
-			b.WriteString(fmt.Sprintf("- `%s` + `%s`: %s\n", s.A, s.B, s.Reason))
-		}
-		b.WriteString("\n")
-	}
-
-	if len(r.Refusals) > 0 {
-		b.WriteString(fmt.Sprintf("### Found but refused (%d)\n\n", len(r.Refusals)))
-		b.WriteString("Candidates the heuristics found but the authoring rules would not let this job write. They need a human edit or a rule change.\n\n")
-		for _, ref := range r.Refusals {
-			b.WriteString(fmt.Sprintf("- %s: %s\n", codeList(ref.Members), ref.Reason))
-		}
-		b.WriteString("\n")
-	}
-
-	b.WriteString("---\n\nOpened by `.github/ci/variantproposals`. Heuristics and the rejection ledger live there and in the ledger file; a wrong proposal is a bug in one of the two.\n")
-	return b.String()
-}
-
-func joinSignals(signals []Signal) string {
-	if len(signals) == 0 {
-		return "inferred through another member of the family"
-	}
-	out := make([]string, 0, len(signals))
-	for _, s := range signals {
-		out = append(out, "`"+string(s)+"`")
-	}
-	return strings.Join(out, ", ")
-}
-
-func describeEvidence(e Evidence) string {
-	var parts []string
-	if e.SharedStem != "" {
-		parts = append(parts, fmt.Sprintf("same name once quantization markers are stripped: `%s`", e.SharedStem))
-	}
-	if e.SharedFile != "" {
-		parts = append(parts, fmt.Sprintf("same primary weight filename once quantization markers are stripped: `%s`", e.SharedFile))
-	}
-	if e.SharedRepo != "" {
-		parts = append(parts, fmt.Sprintf("same upstream repo `%s`", e.SharedRepo))
-	}
-	if len(e.QuantTokens) > 0 {
-		parts = append(parts, "differing quantization tokens: `"+strings.Join(e.QuantTokens, "`, `")+"`")
-	}
-	if len(parts) == 0 {
-		return "reached this family through another member"
-	}
-	return strings.Join(parts, "; ")
-}
-
-func codeList(names []string) string {
-	out := make([]string, 0, len(names))
-	for _, n := range names {
-		out = append(out, "`"+n+"`")
-	}
-	return strings.Join(out, " + ")
-}
-
-// RenderSummary is the terminal-facing digest of a run, so the workflow log
-// says what happened without anyone opening the pull request.
-func RenderSummary(r *Result) string {
-	var b strings.Builder
-	fmt.Fprintf(&b, "families proposed: %d\n", len(r.Families))
-	for _, f := range r.Families {
-		names := make([]string, 0, len(f.Proposals))
-		for _, p := range f.Proposals {
-			names = append(names, p.Variant)
-		}
-		fmt.Fprintf(&b, "  %s <- %s\n", f.Parent, strings.Join(names, ", "))
-	}
-	fmt.Fprintf(&b, "declined by ledger: %d\n", len(r.Suppressed))
-	for _, s := range r.Suppressed {
-		fmt.Fprintf(&b, "  %s\n", s)
-	}
-	fmt.Fprintf(&b, "aliases skipped: %d\n", len(r.AliasSkipped))
-	for _, s := range r.AliasSkipped {
-		fmt.Fprintf(&b, "  %s\n", s)
-	}
-	fmt.Fprintf(&b, "refused: %d\n", len(r.Refusals))
-	for _, ref := range r.Refusals {
-		fmt.Fprintf(&b, "  %s: %s\n", strings.Join(ref.Members, " + "), ref.Reason)
-	}
-	return b.String()
-}

.github/ci/variantproposals/edit.go

@@ -1,120 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"regexp"
-	"sort"
-	"strings"
-)
-
-var (
-	inlineName   = regexp.MustCompile(`^- (?:&\S+ )?name:`)
-	keyName      = regexp.MustCompile(`^  name:`)
-	keyVariants  = regexp.MustCompile(`^  variants:\s*(.*)$`)
-	variantItem  = regexp.MustCompile(`^    - `)
-	unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`)
-)
-
-// ApplyFamilies writes the proposed variant lists into the index text.
-//
-// The edit is textual on purpose. Re-serialising the index through a YAML
-// marshaller would reflow 40,000 lines, drop the anchors and merge keys the
-// gallery relies on, and produce a diff no reviewer could read, which would
-// make the pull request worthless even when the proposals inside it are right.
-func ApplyFamilies(ix *Index, families []Family) ([]string, error) {
-	byName, _ := ix.ByName()
-
-	type edit struct {
-		at      int
-		remove  int
-		insert  []string
-		ordinal int
-	}
-	var edits []edit
-
-	for _, f := range families {
-		entry, ok := byName[strings.ToLower(f.Parent)]
-		if !ok {
-			return nil, fmt.Errorf("parent %q is not in the index", f.Parent)
-		}
-		items := make([]string, 0, len(f.Proposals))
-		for _, p := range f.Proposals {
-			items = append(items, "    - model: "+quoteName(p.Variant))
-		}
-
-		at, remove, err := insertionPoint(ix, entry)
-		if err != nil {
-			return nil, err
-		}
-		insert := items
-		if remove > 0 || !hasVariantsKey(ix, entry) {
-			insert = append([]string{"  variants:"}, items...)
-		}
-		edits = append(edits, edit{at: at, remove: remove, insert: insert, ordinal: entry.Index})
-	}
-
-	// Applying from the bottom up keeps every line number computed against the
-	// original text valid while earlier edits are still pending.
-	sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at })
-
-	lines := append([]string(nil), ix.Lines...)
-	for _, e := range edits {
-		tail := append([]string(nil), lines[e.at+e.remove:]...)
-		lines = append(lines[:e.at], append(append([]string(nil), e.insert...), tail...)...)
-	}
-	return lines, nil
-}
-
-func hasVariantsKey(ix *Index, e *GalleryEntry) bool {
-	for i := e.StartLine; i < e.EndLine; i++ {
-		if keyVariants.MatchString(ix.Lines[i]) {
-			return true
-		}
-	}
-	return false
-}
-
-// insertionPoint reports where new variant items belong, and how many existing
-// lines the insertion replaces.
-//
-// An entry with no variants key gets one right after its name, which is where
-// the hand-written families put it. An entry with an empty "variants: []" has
-// that line replaced by a block. An entry with a block gets its items appended.
-func insertionPoint(ix *Index, e *GalleryEntry) (at int, remove int, err error) {
-	for i := e.StartLine; i < e.EndLine; i++ {
-		m := keyVariants.FindStringSubmatch(ix.Lines[i])
-		if m == nil {
-			continue
-		}
-		if strings.TrimSpace(m[1]) == "[]" {
-			return i, 1, nil
-		}
-		if strings.TrimSpace(m[1]) != "" {
-			return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1]))
-		}
-		last := i
-		for j := i + 1; j < e.EndLine && variantItem.MatchString(ix.Lines[j]); j++ {
-			last = j
-		}
-		return last + 1, 0, nil
-	}
-
-	if inlineName.MatchString(ix.Lines[e.StartLine]) {
-		return e.StartLine + 1, 0, nil
-	}
-	for i := e.StartLine; i < e.EndLine; i++ {
-		if keyName.MatchString(ix.Lines[i]) {
-			return i + 1, 0, nil
-		}
-	}
-	return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name)
-}
-
-// quoteName quotes a variant reference when the name would otherwise change
-// meaning as bare YAML. Config-suffixed names carry a ":" and always need it.
-func quoteName(name string) string {
-	if unsafeInName.MatchString(name) {
-		return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"`
-	}
-	return name
-}

.github/ci/variantproposals/edit_test.go

@@ -1,153 +0,0 @@
-package main
-
-import (
-	"strings"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("ApplyFamilies", func() {
-	apply := func(ix *Index, families []Family) []string {
-		lines, err := ApplyFamilies(ix, families)
-		ExpectWithOffset(1, err).ToNot(HaveOccurred())
-		return lines
-	}
-
-	// insertedLines is what a reviewer would see in the diff. A textual editor
-	// that reflowed the file would show thousands here, which is the failure
-	// this whole approach exists to avoid.
-	insertedLines := func(before, after []string) int {
-		remaining := map[string]int{}
-		for _, l := range before {
-			remaining[l]++
-		}
-		n := 0
-		for _, l := range after {
-			if remaining[l] > 0 {
-				remaining[l]--
-				continue
-			}
-			n++
-		}
-		return n
-	}
-
-	It("adds a variants block right after the entry's name and touches nothing else", func() {
-		ix := indexOf(
-			entryYAML("foo-model", "acme/repo", "foo-model-Q4_K_M.gguf", "aa"),
-			entryYAML("foo-model-q8_0", "acme/repo", "foo-model-Q8_0.gguf", "bb"),
-		)
-		out := apply(ix, []Family{{Parent: "foo-model", Proposals: []Proposal{{Variant: "foo-model-q8_0"}}}})
-
-		Expect(out[0]).To(Equal("- name: foo-model"))
-		Expect(out[1]).To(Equal("  variants:"))
-		Expect(out[2]).To(Equal("    - model: foo-model-q8_0"))
-		Expect(len(out)).To(Equal(len(ix.Lines) + 2))
-		Expect(insertedLines(ix.Lines, out)).To(Equal(2))
-	})
-
-	It("appends to a variants block that already exists", func() {
-		ix := indexOf(`- name: partial
-  variants:
-    - model: partial-q8_0
-  url: u
-  overrides:
-    parameters:
-      model: partial-Q4_K_M.gguf
-`, entryYAML("partial-f16", "acme/repo", "partial-f16.gguf", "cc"))
-		out := apply(ix, []Family{{Parent: "partial", Proposals: []Proposal{{Variant: "partial-f16"}}}})
-
-		Expect(out[1]).To(Equal("  variants:"))
-		Expect(out[2]).To(Equal("    - model: partial-q8_0"))
-		Expect(out[3]).To(Equal("    - model: partial-f16"))
-		Expect(out[4]).To(Equal("  url: u"))
-	})
-
-	It("replaces an explicit empty list rather than leaving two variants keys", func() {
-		ix := indexOf(`- name: emptied
-  variants: []
-  url: u
-`, entryYAML("emptied-q8_0", "acme/repo", "emptied-Q8_0.gguf", "cc"))
-		out := apply(ix, []Family{{Parent: "emptied", Proposals: []Proposal{{Variant: "emptied-q8_0"}}}})
-
-		Expect(strings.Join(out[:4], "\n")).To(Equal("- name: emptied\n  variants:\n    - model: emptied-q8_0\n  url: u"))
-		Expect(strings.Count(strings.Join(out, "\n"), "variants:")).To(Equal(1))
-	})
-
-	It("quotes a config-suffixed name so the reference stays a string", func() {
-		ix := indexOf(
-			entryYAML("phi-2-chat", "acme/repo", "phi-2-chat-Q4_K_M.gguf", "aa"),
-			entryYAML("phi-2-chat:Q8_0", "acme/repo", "phi-2-chat-Q8_0.gguf", "bb"),
-		)
-		out := apply(ix, []Family{{Parent: "phi-2-chat", Proposals: []Proposal{{Variant: "phi-2-chat:Q8_0"}}}})
-		Expect(out[2]).To(Equal(`    - model: "phi-2-chat:Q8_0"`))
-
-		// The result has to still be a gallery, and the reference has to
-		// resolve to the entry it names.
-		reparsed, err := ParseIndex(strings.Join(out, "\n"))
-		Expect(err).ToNot(HaveOccurred())
-		Expect(reparsed.Entries[0].Variants).To(ConsistOf(VariantRef{Model: "phi-2-chat:Q8_0"}))
-	})
-
-	It("keeps line numbers correct when several entries are edited at once", func() {
-		ix := indexOf(
-			entryYAML("alpha", "acme/repo", "alpha-Q4_K_M.gguf", "aa"),
-			entryYAML("alpha-q8_0", "acme/repo", "alpha-Q8_0.gguf", "bb"),
-			entryYAML("beta", "acme/repo", "beta-Q4_K_M.gguf", "cc"),
-			entryYAML("beta-q8_0", "acme/repo", "beta-Q8_0.gguf", "dd"),
-		)
-		out := apply(ix, []Family{
-			{Parent: "alpha", Proposals: []Proposal{{Variant: "alpha-q8_0"}}},
-			{Parent: "beta", Proposals: []Proposal{{Variant: "beta-q8_0"}}},
-		})
-
-		reparsed, err := ParseIndex(strings.Join(out, "\n"))
-		Expect(err).ToNot(HaveOccurred())
-		Expect(reparsed.Entries).To(HaveLen(4))
-		Expect(reparsed.Entries[0].Variants).To(ConsistOf(VariantRef{Model: "alpha-q8_0"}))
-		Expect(reparsed.Entries[2].Variants).To(ConsistOf(VariantRef{Model: "beta-q8_0"}))
-		Expect(reparsed.Entries[1].Variants).To(BeEmpty())
-		Expect(reparsed.Entries[3].Variants).To(BeEmpty())
-	})
-
-	It("fails loudly rather than editing an entry it cannot find", func() {
-		ix := indexOf(entryYAML("only", "acme/repo", "only-Q4_K_M.gguf", "aa"))
-		_, err := ApplyFamilies(ix, []Family{{Parent: "missing", Proposals: []Proposal{{Variant: "x"}}}})
-		Expect(err).To(MatchError(ContainSubstring("not in the index")))
-	})
-})
-
-var _ = Describe("ParseIndex", func() {
-	It("records the anchor an entry defines and the anchor an entry merges", func() {
-		ix := indexOf(`- &anc
-  name: anchored
-  url: u
-`, `- !!merge <<: *anc
-  name: child
-`)
-		Expect(ix.Entries[0].AnchorName).To(Equal("anc"))
-		Expect(ix.Entries[1].MergesFrom).To(Equal("anc"))
-		Expect(ix.MergeChildren("anc")).To(HaveLen(1))
-	})
-
-	It("carries merged values into the child, so an inherited variants key is visible", func() {
-		ix := indexOf(`- &anc
-  name: anchored
-  url: u
-  variants:
-    - model: something
-`, `- !!merge <<: *anc
-  name: child
-`)
-		Expect(ix.Entries[1].HasVariants()).To(BeTrue())
-	})
-
-	It("refuses a list item that decodes to nothing", func() {
-		// Every line number the editor works from comes from pairing decoded
-		// entries with top level list items. If those two views can disagree,
-		// the editor writes into the wrong entry, so the parse refuses instead.
-		_, err := ParseIndex("- name: one\n  url: u\n-\n")
-		Expect(err).To(MatchError(ContainSubstring("empty")))
-	})
-})

.github/ci/variantproposals/index.go

@@ -1,286 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"os"
-	"regexp"
-	"sort"
-	"strings"
-
-	"gopkg.in/yaml.v3"
-)
-
-// File is the subset of a gallery file entry the proposer reads.
-type File struct {
-	Filename string `yaml:"filename"`
-	URI      string `yaml:"uri"`
-	SHA256   string `yaml:"sha256"`
-}
-
-// VariantRef mirrors the gallery's variant reference.
-type VariantRef struct {
-	Model string `yaml:"model"`
-}
-
-// GalleryEntry is one gallery entry, carrying both the semantics the heuristics need
-// and the text range the editor needs.
-//
-// The two views are kept together deliberately. The editor must not round-trip
-// the index through a YAML marshaller: the gallery is 40,000 lines and a
-// reflowed diff cannot be reviewed, which defeats the entire point of a job
-// whose output is a human decision.
-type GalleryEntry struct {
-	Name       string         `yaml:"name"`
-	URL        string         `yaml:"url"`
-	ConfigFile map[string]any `yaml:"config_file"`
-	Overrides  map[string]any `yaml:"overrides"`
-	Files      []File         `yaml:"files"`
-	Variants   []VariantRef   `yaml:"variants"`
-
-	// Index is the entry's position in gallery order.
-	Index int `yaml:"-"`
-	// StartLine and EndLine bound the entry's lines, zero based and half open.
-	StartLine int `yaml:"-"`
-	EndLine   int `yaml:"-"`
-	// AnchorName is set when the entry defines a YAML anchor. Adding a variants
-	// key to such an entry is inherited by everything that merges it, which is
-	// why proposals involving anchors get special treatment.
-	AnchorName string `yaml:"-"`
-	// MergesFrom is the anchor this entry pulls in with "!!merge <<:".
-	MergesFrom string `yaml:"-"`
-}
-
-// Index is a parsed gallery index: entries plus the exact lines they came from.
-type Index struct {
-	Lines   []string
-	Entries []*GalleryEntry
-}
-
-var (
-	entryStart  = regexp.MustCompile(`^-(?: |$)`)
-	anchorStart = regexp.MustCompile(`^- &(\S+)`)
-	mergeStart  = regexp.MustCompile(`^- !!merge <<: \*(\S+)`)
-)
-
-// LoadIndex reads and parses a gallery index file.
-func LoadIndex(path string) (*Index, error) {
-	data, err := os.ReadFile(path)
-	if err != nil {
-		return nil, err
-	}
-	return ParseIndex(string(data))
-}
-
-// ParseIndex builds an Index from the raw text of a gallery index.
-//
-// The YAML decode and the textual scan are cross checked against each other: if
-// they disagree on how many entries there are, every line number the editor
-// would use is suspect, so the run fails rather than editing the wrong entry.
-func ParseIndex(text string) (*Index, error) {
-	var entries []*GalleryEntry
-	if err := yaml.Unmarshal([]byte(text), &entries); err != nil {
-		return nil, fmt.Errorf("decoding gallery index: %w", err)
-	}
-
-	lines := strings.Split(text, "\n")
-	var starts []int
-	for i, line := range lines {
-		if entryStart.MatchString(line) {
-			starts = append(starts, i)
-		}
-	}
-	if len(starts) != len(entries) {
-		return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number", len(entries), len(starts))
-	}
-
-	for i, e := range entries {
-		if e == nil {
-			return nil, fmt.Errorf("gallery index list item %d is empty; refusing to edit by line number", i)
-		}
-		e.Index = i
-		e.StartLine = starts[i]
-		if i+1 < len(starts) {
-			e.EndLine = starts[i+1]
-		} else {
-			e.EndLine = len(lines)
-		}
-		if m := anchorStart.FindStringSubmatch(lines[e.StartLine]); m != nil {
-			e.AnchorName = m[1]
-		}
-		if m := mergeStart.FindStringSubmatch(lines[e.StartLine]); m != nil {
-			e.MergesFrom = m[1]
-		}
-	}
-
-	return &Index{Lines: lines, Entries: entries}, nil
-}
-
-// MergeChildren lists the entries that pull in the given anchor.
-func (ix *Index) MergeChildren(anchor string) []*GalleryEntry {
-	var out []*GalleryEntry
-	for _, e := range ix.Entries {
-		if e.MergesFrom == anchor {
-			out = append(out, e)
-		}
-	}
-	return out
-}
-
-// ByName indexes entries by lowercased name. A name appearing twice keeps the
-// first occurrence, matching the gallery's own first-match-wins resolution, and
-// the duplicates are returned so the caller can refuse to touch them: a
-// proposal naming an ambiguous entry cannot be reviewed.
-func (ix *Index) ByName() (map[string]*GalleryEntry, map[string]int) {
-	byName := make(map[string]*GalleryEntry, len(ix.Entries))
-	counts := make(map[string]int, len(ix.Entries))
-	for _, e := range ix.Entries {
-		key := strings.ToLower(e.Name)
-		counts[key]++
-		if _, seen := byName[key]; !seen {
-			byName[key] = e
-		}
-	}
-	dupes := map[string]int{}
-	for name, n := range counts {
-		if n > 1 {
-			dupes[name] = n
-		}
-	}
-	return byName, dupes
-}
-
-// Installable reports whether installing this entry would put anything on disk.
-// A variant target that installs nothing is a dead end for the selector, so it
-// is never proposed as one.
-func (e *GalleryEntry) Installable() bool {
-	return e.URL != "" || len(e.ConfigFile) > 0 || len(e.Overrides) > 0 || len(e.Files) > 0
-}
-
-// HasVariants reports whether the entry already offers builds of its own. Such
-// an entry cannot be a variant target: nesting is what the gallery's own
-// resolution refuses.
-func (e *GalleryEntry) HasVariants() bool {
-	return len(e.Variants) > 0
-}
-
-// auxiliaryFile matches the shared side files that several unrelated models
-// legitimately hand out the same copy of. Grouping on one of these is how an
-// earlier sweep linked four wan-2.1 entries to each other and Z-Image-Turbo to
-// qwen3-4b: they shared a text encoder, not weights.
-var auxiliaryFile = regexp.MustCompile(`(?i)(mmproj|vae|clip|t5|umt5|text_?encoder|tokenizer|\bae\b|^ae\.|scheduler|config)`)
-
-// IsAuxiliaryFile reports whether a filename is a side file rather than the
-// model's own weights.
-func IsAuxiliaryFile(filename string) bool {
-	base := filename
-	if i := strings.LastIndex(base, "/"); i >= 0 {
-		base = base[i+1:]
-	}
-	return auxiliaryFile.MatchString(base)
-}
-
-// PrimaryWeightFile returns the filename of the entry's own weights, and
-// whether one could be identified unambiguously.
-//
-// The declared overrides.parameters.model wins because that is the file the
-// backend is actually pointed at. Falling back to the file list only works when
-// exactly one non-auxiliary file is present; anything else is ambiguous, and
-// guessing is precisely the failure mode this heuristic has already had.
-func (e *GalleryEntry) PrimaryWeightFile() (string, bool) {
-	if params, ok := e.Overrides["parameters"].(map[string]any); ok {
-		if model, ok := params["model"].(string); ok && model != "" && !IsAuxiliaryFile(model) {
-			return model, true
-		}
-	}
-	var candidates []string
-	for _, f := range e.Files {
-		if f.Filename == "" || IsAuxiliaryFile(f.Filename) {
-			continue
-		}
-		candidates = append(candidates, f.Filename)
-	}
-	if len(candidates) == 1 {
-		return candidates[0], true
-	}
-	return "", false
-}
-
-// SourceRepo returns the upstream repository the entry's files come from, as a
-// coarse "host + owner + repo" key.
-func (e *GalleryEntry) SourceRepo() string {
-	for _, f := range e.Files {
-		if f.URI == "" {
-			continue
-		}
-		return repoKey(f.URI)
-	}
-	return ""
-}
-
-func repoKey(uri string) string {
-	u := strings.ToLower(uri)
-	u = strings.TrimPrefix(u, "huggingface://")
-	u = strings.TrimPrefix(u, "https://huggingface.co/")
-	u = strings.TrimPrefix(u, "http://huggingface.co/")
-	parts := strings.Split(u, "/")
-	if len(parts) >= 2 {
-		return parts[0] + "/" + parts[1]
-	}
-	return u
-}
-
-// SameInstallPayload reports whether two entries install byte for byte the same
-// thing.
-//
-// Entries like this are aliases, not variants. whisper-1 exists so a client
-// speaking the OpenAI API can send that name and get whisper-base; folding it
-// under whisper-base as a variant would hide the very name clients send.
-func SameInstallPayload(a, b *GalleryEntry) bool {
-	if a.URL != b.URL {
-		return false
-	}
-	if !sameYAML(a.Overrides, b.Overrides) || !sameYAML(a.ConfigFile, b.ConfigFile) {
-		return false
-	}
-	return sameChecksums(a.Files, b.Files)
-}
-
-func sameChecksums(a, b []File) bool {
-	if len(a) != len(b) || len(a) == 0 {
-		return false
-	}
-	ha := make([]string, 0, len(a))
-	hb := make([]string, 0, len(b))
-	for _, f := range a {
-		if f.SHA256 == "" {
-			return false
-		}
-		ha = append(ha, f.SHA256)
-	}
-	for _, f := range b {
-		if f.SHA256 == "" {
-			return false
-		}
-		hb = append(hb, f.SHA256)
-	}
-	sort.Strings(ha)
-	sort.Strings(hb)
-	for i := range ha {
-		if ha[i] != hb[i] {
-			return false
-		}
-	}
-	return true
-}
-
-func sameYAML(a, b any) bool {
-	ba, err := yaml.Marshal(a)
-	if err != nil {
-		return false
-	}
-	bb, err := yaml.Marshal(b)
-	if err != nil {
-		return false
-	}
-	return string(ba) == string(bb)
-}

.github/ci/variantproposals/ledger.go

@@ -1,180 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"os"
-	"sort"
-	"strings"
-
-	"gopkg.in/yaml.v3"
-)
-
-// Ledger records the grouping decisions a human has already made against the
-// proposer, so a declined candidate stays declined instead of coming back every
-// night until reviewers stop reading the job's pull requests.
-//
-// It is checked in next to the gallery and is meant to be edited inside the
-// proposal pull request itself: declining a family is adding one flow-mapping
-// line under pairs or groups and closing the PR.
-type Ledger struct {
-	// Tokens are name segments that mark a distinct model rather than another
-	// build of the same one: finetune names, language codes, product suffixes.
-	// A candidate whose two names differ by any of these is never proposed.
-	Tokens []LedgerToken `yaml:"tokens"`
-	// Pairs are individual candidates a human considered and declined. Order
-	// does not matter: the pair is matched both ways round.
-	Pairs []LedgerPair `yaml:"pairs"`
-	// Groups decline every pair drawn from a set at once, for families like a
-	// per-language release where listing each pair would be unreadable.
-	Groups []LedgerGroup `yaml:"groups"`
-}
-
-type LedgerToken struct {
-	Token  string `yaml:"token"`
-	Reason string `yaml:"reason"`
-}
-
-type LedgerPair struct {
-	Parent  string `yaml:"parent"`
-	Variant string `yaml:"variant"`
-	Reason  string `yaml:"reason"`
-}
-
-type LedgerGroup struct {
-	Members []string `yaml:"members"`
-	Reason  string   `yaml:"reason"`
-}
-
-// LoadLedger reads a ledger file. A missing file is not an error: a gallery
-// that has declined nothing yet is a legitimate state, and failing the job over
-// it would only teach people to keep an empty file around.
-func LoadLedger(path string) (*Ledger, error) {
-	data, err := os.ReadFile(path)
-	if os.IsNotExist(err) {
-		return &Ledger{}, nil
-	}
-	if err != nil {
-		return nil, err
-	}
-	return ParseLedger(data)
-}
-
-func ParseLedger(data []byte) (*Ledger, error) {
-	l := &Ledger{}
-	if err := yaml.Unmarshal(data, l); err != nil {
-		return nil, fmt.Errorf("parsing ledger: %w", err)
-	}
-	for i, t := range l.Tokens {
-		if strings.TrimSpace(t.Token) == "" {
-			return nil, fmt.Errorf("ledger tokens[%d] has an empty token", i)
-		}
-	}
-	for i, p := range l.Pairs {
-		if strings.TrimSpace(p.Parent) == "" || strings.TrimSpace(p.Variant) == "" {
-			return nil, fmt.Errorf("ledger pairs[%d] needs both parent and variant", i)
-		}
-	}
-	return l, nil
-}
-
-// Suppression is a ledger hit: why a candidate was not proposed, in words a
-// reviewer can check against the ledger file.
-type Suppression struct {
-	A      string
-	B      string
-	Reason string
-}
-
-func (s Suppression) String() string {
-	return fmt.Sprintf("%s + %s: %s", s.A, s.B, s.Reason)
-}
-
-// Suppresses reports whether the ledger has already declined pairing these two
-// entries, and why.
-//
-// The token rule is applied to the segments the two names do not share. Two
-// builds of the same weights differ only in quantization markers, so any
-// ledgered token showing up in that difference is by construction a claim that
-// the entries are different models.
-func (l *Ledger) Suppresses(a, b string) (Suppression, bool) {
-	la, lb := strings.ToLower(a), strings.ToLower(b)
-	for _, p := range l.Pairs {
-		lp, lv := strings.ToLower(p.Parent), strings.ToLower(p.Variant)
-		if (lp == la && lv == lb) || (lp == lb && lv == la) {
-			return Suppression{A: a, B: b, Reason: p.Reason}, true
-		}
-	}
-	for _, g := range l.Groups {
-		var seenA, seenB bool
-		for _, m := range g.Members {
-			lm := strings.ToLower(m)
-			if lm == la {
-				seenA = true
-			}
-			if lm == lb {
-				seenB = true
-			}
-		}
-		if seenA && seenB {
-			return Suppression{A: a, B: b, Reason: g.Reason}, true
-		}
-	}
-	diff := differingSegments(la, lb)
-	for _, t := range l.Tokens {
-		token := strings.ToLower(strings.TrimSpace(t.Token))
-		if _, ok := diff[token]; ok {
-			reason := t.Reason
-			if reason == "" {
-				reason = fmt.Sprintf("names differ by %q", token)
-			}
-			return Suppression{A: a, B: b, Reason: fmt.Sprintf("%s (token %q)", reason, token)}, true
-		}
-	}
-	return Suppression{}, false
-}
-
-// segments splits a name into the atoms the token rules are written against.
-func segments(name string) []string {
-	fields := strings.FieldsFunc(strings.ToLower(name), func(r rune) bool {
-		return r == '-' || r == '_' || r == '.' || r == ':' || r == '/'
-	})
-	return fields
-}
-
-// differingSegments returns the set of segments present in exactly one of the
-// two names.
-func differingSegments(a, b string) map[string]struct{} {
-	setA := map[string]int{}
-	for _, s := range segments(a) {
-		setA[s]++
-	}
-	setB := map[string]int{}
-	for _, s := range segments(b) {
-		setB[s]++
-	}
-	diff := map[string]struct{}{}
-	for s := range setA {
-		if setB[s] == 0 {
-			diff[s] = struct{}{}
-		}
-	}
-	for s := range setB {
-		if setA[s] == 0 {
-			diff[s] = struct{}{}
-		}
-	}
-	return diff
-}
-
-// SortedSuppressions gives the ledger's effect on one run in a stable order, so
-// the pull request body reads the same way for the same gallery.
-func SortedSuppressions(in []Suppression) []Suppression {
-	out := append([]Suppression(nil), in...)
-	sort.Slice(out, func(i, j int) bool {
-		if out[i].A != out[j].A {
-			return out[i].A < out[j].A
-		}
-		return out[i].B < out[j].B
-	})
-	return out
-}

.github/ci/variantproposals/main.go

@@ -1,65 +0,0 @@
-// Command variant-proposals looks for gallery entries that are alternative
-// builds of the same weights but are not grouped under one another, and writes
-// a proposal for a human to accept or reject.
-//
-// It never decides. Grouping has gone wrong repeatedly in both directions, so
-// the job's value is catching drift and surfacing candidates with their
-// evidence, not automating the call. The scheduled workflow feeds its output to
-// a pull request in the same shape as .github/checksum_checker.sh.
-package main
-
-import (
-	"flag"
-	"fmt"
-	"os"
-	"strings"
-)
-
-func main() {
-	index := flag.String("index", "gallery/index.yaml", "path to the gallery index")
-	ledger := flag.String("ledger", "gallery/variant-exclusions.yaml", "path to the rejection ledger")
-	bodyOut := flag.String("body-out", "", "write the pull request body here")
-	apply := flag.Bool("apply", false, "write the proposed groupings back into the index")
-	flag.Parse()
-
-	if err := run(*index, *ledger, *bodyOut, *apply); err != nil {
-		fmt.Fprintln(os.Stderr, "variant-proposals:", err)
-		os.Exit(1)
-	}
-}
-
-func run(indexPath, ledgerPath, bodyOut string, apply bool) error {
-	ix, err := LoadIndex(indexPath)
-	if err != nil {
-		return err
-	}
-	ledger, err := LoadLedger(ledgerPath)
-	if err != nil {
-		return err
-	}
-
-	result := Propose(ix, ledger)
-	fmt.Print(RenderSummary(result))
-
-	if !result.HasProposals() {
-		// An empty pull request every night is how a proposal job gets muted.
-		fmt.Println("nothing to propose")
-		return nil
-	}
-
-	if bodyOut != "" {
-		if err := os.WriteFile(bodyOut, []byte(RenderBody(result, ledgerPath)), 0o644); err != nil {
-			return err
-		}
-	}
-
-	if !apply {
-		return nil
-	}
-
-	lines, err := ApplyFamilies(ix, result.Families)
-	if err != nil {
-		return err
-	}
-	return os.WriteFile(indexPath, []byte(strings.Join(lines, "\n")), 0o644)
-}

.github/ci/variantproposals/propose.go

@@ -1,615 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"regexp"
-	"sort"
-	"strings"
-)
-
-// Signal names the grouping heuristic that linked two entries.
-type Signal string
-
-const (
-	// SignalName is "same name once quantization markers are stripped".
-	SignalName Signal = "name-modulo-quant"
-	// SignalConfigSuffix is the ":" convention, foo:q8_0 as a build of foo.
-	SignalConfigSuffix Signal = "config-suffix"
-	// SignalWeightFile is "same primary weight filename once quantization
-	// markers are stripped", auxiliary files excluded.
-	SignalWeightFile Signal = "weight-filename"
-)
-
-// Evidence is what a reviewer needs in order to agree or disagree without
-// opening HuggingFace: what the two entries share, and what differs.
-type Evidence struct {
-	Signals     []Signal
-	SharedStem  string
-	SharedFile  string
-	SharedRepo  string
-	QuantTokens []string
-}
-
-// Proposal is one variant target offered to one parent.
-type Proposal struct {
-	Variant  string
-	Evidence Evidence
-}
-
-// Family is a complete proposal: one parent gaining one or more variants.
-type Family struct {
-	Parent    string
-	Proposals []Proposal
-}
-
-// Refusal is a family the heuristics found but the rules would not let through.
-// Refusals are reported rather than dropped: a candidate the job keeps refusing
-// is either a rule worth revisiting or a gallery bug worth fixing.
-type Refusal struct {
-	Members []string
-	Reason  string
-}
-
-// Result is one run of the proposer.
-type Result struct {
-	Families     []Family
-	Refusals     []Refusal
-	Suppressed   []Suppression
-	AliasSkipped []Suppression
-}
-
-// HasProposals reports whether the run found anything to open a pull request
-// about. A job that opens an empty pull request every night is a job people
-// filter out of their inbox.
-func (r *Result) HasProposals() bool {
-	return len(r.Families) > 0
-}
-
-// sizeToken matches a parameter-count marker: 8b, 1.7b, a3b for an active
-// expert count, e2b for the Gemma effective sizes, 8x7b for a mixture.
-//
-// This is a structural rule rather than a ledger entry because it is about the
-// shape of the token, not about any one model. Different parameter sizes were
-// mis-grouped by an earlier sweep and the failure is systematic.
-var sizeToken = regexp.MustCompile(`^(?:[0-9]+(?:\.[0-9]+)?[bm]|[ae][0-9]+(?:\.[0-9]+)?b|[0-9]+x[0-9]+(?:\.[0-9]+)?b)$`)
-
-func differsByParameterSize(a, b string) (string, bool) {
-	for seg := range differingSegments(a, b) {
-		if sizeToken.MatchString(seg) {
-			return seg, true
-		}
-	}
-	return "", false
-}
-
-// genericFileStem lists weight filenames too generic to be evidence of
-// anything. Two entries both shipping "model.safetensors" share a convention,
-// not a set of weights.
-var genericFileStem = map[string]struct{}{
-	"model": {}, "weights": {}, "pytorch_model": {}, "diffusion_pytorch_model": {},
-	"consolidated": {}, "ggml-model": {}, "model-00001-of-00002": {},
-}
-
-// minFileStemLength keeps short, collision-prone filename stems from linking
-// unrelated entries.
-const minFileStemLength = 6
-
-type pair struct {
-	a, b     int
-	evidence Evidence
-}
-
-// Propose runs the grouping heuristics over a gallery index and returns what it
-// would offer a human, what it refused, and what the ledger silenced.
-//
-// Nothing here touches the network or git, and the index is not modified.
-func Propose(ix *Index, ledger *Ledger) *Result {
-	if ledger == nil {
-		ledger = &Ledger{}
-	}
-	result := &Result{}
-
-	byName, dupes := ix.ByName()
-
-	// Existing relationships. A target already claimed must not be claimed
-	// again, and two entries already in one family need no proposal.
-	claimedBy := map[string]string{}
-	familyOf := map[string]string{}
-	for _, e := range ix.Entries {
-		if !e.HasVariants() {
-			continue
-		}
-		familyOf[strings.ToLower(e.Name)] = strings.ToLower(e.Name)
-		for _, v := range e.Variants {
-			target := strings.ToLower(v.Model)
-			if _, taken := claimedBy[target]; !taken {
-				claimedBy[target] = strings.ToLower(e.Name)
-			}
-			familyOf[target] = strings.ToLower(e.Name)
-		}
-	}
-
-	candidates := map[[2]int]*Evidence{}
-
-	addPair := func(i, j int, sig Signal, apply func(*Evidence)) {
-		if i == j {
-			return
-		}
-		if i > j {
-			i, j = j, i
-		}
-		key := [2]int{i, j}
-		ev, ok := candidates[key]
-		if !ok {
-			ev = &Evidence{}
-			candidates[key] = ev
-		}
-		for _, s := range ev.Signals {
-			if s == sig {
-				apply(ev)
-				return
-			}
-		}
-		ev.Signals = append(ev.Signals, sig)
-		apply(ev)
-	}
-
-	// Signal 1 and 2: entries sharing a name stem.
-	byStem := map[string][]int{}
-	for _, e := range ix.Entries {
-		if e.Name == "" {
-			continue
-		}
-		byStem[NameStem(e.Name)] = append(byStem[NameStem(e.Name)], e.Index)
-	}
-	for stem, members := range byStem {
-		if len(members) < 2 {
-			continue
-		}
-		for i := 0; i < len(members); i++ {
-			for j := i + 1; j < len(members); j++ {
-				a, b := ix.Entries[members[i]], ix.Entries[members[j]]
-				sig := SignalName
-				if HasConfigSuffix(a.Name) || HasConfigSuffix(b.Name) {
-					sig = SignalConfigSuffix
-				}
-				// The bare parent carries no marker in its name, so the
-				// evidence would read "differs by q8_0" and say nothing about
-				// what the parent is. The weight filenames fill that in.
-				fa, _ := a.PrimaryWeightFile()
-				fb, _ := b.PrimaryWeightFile()
-				addPair(members[i], members[j], sig, func(ev *Evidence) {
-					ev.SharedStem = stem
-					ev.QuantTokens = quantDifference(a.Name, b.Name, fa, fb)
-				})
-			}
-		}
-	}
-
-	// Signal 3: entries whose own weight file is the same file at a different
-	// quantization. Auxiliary files never take part.
-	byFile := map[string][]int{}
-	for _, e := range ix.Entries {
-		primary, ok := e.PrimaryWeightFile()
-		if !ok {
-			continue
-		}
-		stem := FileStem(primary)
-		if len(stem) < minFileStemLength {
-			continue
-		}
-		if _, generic := genericFileStem[stem]; generic {
-			continue
-		}
-		byFile[stem] = append(byFile[stem], e.Index)
-	}
-	for stem, members := range byFile {
-		if len(members) < 2 {
-			continue
-		}
-		for i := 0; i < len(members); i++ {
-			for j := i + 1; j < len(members); j++ {
-				a, b := ix.Entries[members[i]], ix.Entries[members[j]]
-				// The filename alone is not evidence. Publishers reuse the
-				// upstream filename for finetunes and for models that merely
-				// embed the base weights: bert-embeddings, an ultravox audio
-				// model and a roleplay finetune all ship a file called
-				// llama-3.2-1b-instruct-q4_k_m.gguf. Requiring the same
-				// upstream repository turns the signal back into what it
-				// claims to be, one repo publishing one file at two
-				// quantizations. Two repos holding the same weights is a fact
-				// no filename proves, so it stays a human call.
-				repo := a.SourceRepo()
-				if repo == "" || repo != b.SourceRepo() {
-					continue
-				}
-				fa, _ := a.PrimaryWeightFile()
-				fb, _ := b.PrimaryWeightFile()
-				addPair(members[i], members[j], SignalWeightFile, func(ev *Evidence) {
-					ev.SharedFile = stem
-					ev.SharedRepo = repo
-					if len(ev.QuantTokens) == 0 {
-						ev.QuantTokens = quantDifference(fa, fb)
-					}
-				})
-			}
-		}
-	}
-
-	// Filter candidates. Everything dropped here is dropped for a reason a
-	// reviewer can read back off the ledger or the rules.
-	var kept []pair
-	for key, ev := range candidates {
-		a, b := ix.Entries[key[0]], ix.Entries[key[1]]
-		la, lb := strings.ToLower(a.Name), strings.ToLower(b.Name)
-		if la == lb {
-			continue
-		}
-		if dupes[la] > 0 || dupes[lb] > 0 {
-			result.Refusals = append(result.Refusals, Refusal{
-				Members: []string{a.Name, b.Name},
-				Reason:  "one of these names appears more than once in the gallery, so a variant reference to it is ambiguous",
-			})
-			continue
-		}
-		if fa, fb := familyOf[la], familyOf[lb]; fa != "" && fa == fb {
-			continue
-		}
-		if seg, differs := differsByParameterSize(la, lb); differs {
-			result.Suppressed = append(result.Suppressed, Suppression{
-				A: a.Name, B: b.Name, Reason: fmt.Sprintf("different parameter sizes (segment %q)", seg),
-			})
-			continue
-		}
-		if s, ok := ledger.Suppresses(a.Name, b.Name); ok {
-			result.Suppressed = append(result.Suppressed, s)
-			continue
-		}
-		if SameInstallPayload(a, b) {
-			result.AliasSkipped = append(result.AliasSkipped, Suppression{
-				A: a.Name, B: b.Name,
-				Reason: "identical install payload; these are aliases of one build, not alternative builds",
-			})
-			continue
-		}
-		kept = append(kept, pair{a: key[0], b: key[1], evidence: *ev})
-	}
-
-	sort.Slice(kept, func(i, j int) bool {
-		if kept[i].a != kept[j].a {
-			return kept[i].a < kept[j].a
-		}
-		return kept[i].b < kept[j].b
-	})
-
-	// Components. A pair from either signal joins the same family, so a chain
-	// of alternative builds discovered by different signals stays one family
-	// rather than two overlapping ones that would double claim a target.
-	parent := map[int]int{}
-	var find func(int) int
-	find = func(x int) int {
-		if p, ok := parent[x]; ok && p != x {
-			parent[x] = find(p)
-			return parent[x]
-		}
-		if _, ok := parent[x]; !ok {
-			parent[x] = x
-		}
-		return parent[x]
-	}
-	union := func(x, y int) {
-		rx, ry := find(x), find(y)
-		if rx != ry {
-			parent[ry] = rx
-		}
-	}
-	evidenceFor := map[[2]int]Evidence{}
-	for _, p := range kept {
-		union(p.a, p.b)
-		evidenceFor[[2]int{p.a, p.b}] = p.evidence
-	}
-
-	components := map[int][]int{}
-	for _, p := range kept {
-		for _, m := range []int{p.a, p.b} {
-			root := find(m)
-			if !contains(components[root], m) {
-				components[root] = append(components[root], m)
-			}
-		}
-	}
-
-	roots := make([]int, 0, len(components))
-	for r := range components {
-		roots = append(roots, r)
-	}
-	sort.Ints(roots)
-
-	proposedTargets := map[string]string{}
-	for _, root := range roots {
-		members := components[root]
-		sort.Ints(members)
-		family, refusal := buildFamily(ix, members, evidenceFor, claimedBy, proposedTargets, byName)
-		if refusal != nil {
-			result.Refusals = append(result.Refusals, *refusal)
-			continue
-		}
-		if family == nil {
-			continue
-		}
-		for _, p := range family.Proposals {
-			proposedTargets[strings.ToLower(p.Variant)] = family.Parent
-		}
-		result.Families = append(result.Families, *family)
-	}
-
-	sort.Slice(result.Families, func(i, j int) bool { return result.Families[i].Parent < result.Families[j].Parent })
-	result.Suppressed = SortedSuppressions(result.Suppressed)
-	result.AliasSkipped = SortedSuppressions(result.AliasSkipped)
-	result.Refusals = dedupeRefusals(result.Refusals)
-	return result
-}
-
-// dedupeRefusals collapses the same refusal reached from both orderings of a
-// pair, and sorts what is left. A reviewer reading the same complaint twice
-// learns to skim the section.
-func dedupeRefusals(in []Refusal) []Refusal {
-	seen := map[string]struct{}{}
-	var out []Refusal
-	for _, r := range in {
-		members := append([]string(nil), r.Members...)
-		sort.Strings(members)
-		key := strings.Join(members, "\x00") + "\x00" + r.Reason
-		if _, dup := seen[key]; dup {
-			continue
-		}
-		seen[key] = struct{}{}
-		out = append(out, r)
-	}
-	sort.Slice(out, func(i, j int) bool {
-		if a, b := strings.Join(out[i].Members, ","), strings.Join(out[j].Members, ","); a != b {
-			return a < b
-		}
-		return out[i].Reason < out[j].Reason
-	})
-	return out
-}
-
-func contains(xs []int, x int) bool {
-	for _, v := range xs {
-		if v == x {
-			return true
-		}
-	}
-	return false
-}
-
-// buildFamily turns a connected component into a proposal, or refuses it.
-func buildFamily(ix *Index, members []int, evidenceFor map[[2]int]Evidence, claimedBy map[string]string, proposedTargets map[string]string, byName map[string]*GalleryEntry) (*Family, *Refusal) {
-	names := make([]string, 0, len(members))
-	for _, m := range members {
-		names = append(names, ix.Entries[m].Name)
-	}
-
-	parentIdx, err := selectParent(ix, members)
-	if err != nil {
-		return nil, &Refusal{Members: names, Reason: err.Error()}
-	}
-	parentEntry := ix.Entries[parentIdx]
-	parentName := strings.ToLower(parentEntry.Name)
-
-	// A parent that is itself somebody's variant would create a chain, which
-	// the gallery's own resolution refuses to install.
-	if owner, claimed := claimedBy[parentName]; claimed {
-		return nil, &Refusal{Members: names, Reason: fmt.Sprintf("the natural parent %q is already a variant of %q; proposing it as a parent would nest variants", parentEntry.Name, owner)}
-	}
-	if owner, claimed := proposedTargets[parentName]; claimed {
-		return nil, &Refusal{Members: names, Reason: fmt.Sprintf("the natural parent %q is already proposed as a variant of %q; proposing it as a parent would nest variants", parentEntry.Name, owner)}
-	}
-
-	// Adding a variants key to an anchor is inherited by every entry that
-	// merges it, silently grouping models nobody proposed. Handling that means
-	// editing each merging child too, which is a larger change than this job
-	// should make unsupervised, so it refuses and hands the reviewer the list.
-	if parentEntry.AnchorName != "" {
-		children := ix.MergeChildren(parentEntry.AnchorName)
-		if len(children) > 0 {
-			childNames := make([]string, 0, len(children))
-			for _, c := range children {
-				childNames = append(childNames, c.Name)
-			}
-			return nil, &Refusal{
-				Members: names,
-				Reason: fmt.Sprintf("the parent %q defines YAML anchor &%s, and a variants key added there is inherited by the %d entries that merge it (%s). Grouping this family by hand also means adding an explicit `variants: []` to each of those entries",
-					parentEntry.Name, parentEntry.AnchorName, len(children), strings.Join(childNames, ", ")),
-			}
-		}
-	}
-
-	existing := map[string]struct{}{}
-	for _, v := range parentEntry.Variants {
-		existing[strings.ToLower(v.Model)] = struct{}{}
-	}
-
-	family := &Family{Parent: parentEntry.Name}
-	for _, m := range members {
-		if m == parentIdx {
-			continue
-		}
-		target := ix.Entries[m]
-		lower := strings.ToLower(target.Name)
-		if _, already := existing[lower]; already {
-			continue
-		}
-		if target.HasVariants() {
-			return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q already offers variants of its own, so it cannot itself be a variant target", target.Name)}
-		}
-		if !target.Installable() {
-			return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q has no url, config_file, overrides or files, so it is not independently installable", target.Name)}
-		}
-		if owner, claimed := claimedBy[lower]; claimed && owner != parentName {
-			return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q is already a variant of %q; a target claimed by two parents is not something the gallery resolves predictably", target.Name, owner)}
-		}
-		if owner, claimed := proposedTargets[lower]; claimed && owner != parentEntry.Name {
-			return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q is already proposed as a variant of %q in this same run", target.Name, owner)}
-		}
-		family.Proposals = append(family.Proposals, Proposal{
-			Variant:  target.Name,
-			Evidence: lookupEvidence(evidenceFor, parentIdx, m),
-		})
-	}
-
-	if len(family.Proposals) == 0 {
-		return nil, nil
-	}
-	sort.Slice(family.Proposals, func(i, j int) bool { return family.Proposals[i].Variant < family.Proposals[j].Variant })
-	return family, nil
-}
-
-func lookupEvidence(evidenceFor map[[2]int]Evidence, a, b int) Evidence {
-	if a > b {
-		a, b = b, a
-	}
-	if ev, ok := evidenceFor[[2]int{a, b}]; ok {
-		return ev
-	}
-	// The two entries reached the same family through a third one. Say so
-	// rather than inventing evidence that was never observed for this pair.
-	return Evidence{Signals: []Signal{SignalName}}
-}
-
-// selectParent picks the entry the others should hang off.
-//
-// The bare name wins when there is one: it is the name a user types and the one
-// documentation links to. Otherwise the smallest build wins, judged by the
-// quantization token in the entry's own weight filename, so the default install
-// is the one most hosts can actually run.
-func selectParent(ix *Index, members []int) (int, error) {
-	// The family's own stem: the one the most members reduce to, shortest name
-	// breaking a tie. An entry named exactly that is the bare entry.
-	stemCount := map[string]int{}
-	for _, m := range members {
-		stemCount[NameStem(ix.Entries[m].Name)]++
-	}
-	// Only a stem two or more members reduce to is the family's own stem. A
-	// stem reached by exactly one member is just that member's name, and
-	// treating it as the family stem would crown whichever name happens to be
-	// shortest rather than whichever build is the base one.
-	familyStem := ""
-	for stem, n := range stemCount {
-		if n < 2 {
-			continue
-		}
-		if familyStem == "" || n > stemCount[familyStem] ||
-			(n == stemCount[familyStem] && len(stem) < len(familyStem)) ||
-			(n == stemCount[familyStem] && len(stem) == len(familyStem) && stem < familyStem) {
-			familyStem = stem
-		}
-	}
-
-	var bare []int
-	for _, m := range members {
-		e := ix.Entries[m]
-		if HasConfigSuffix(e.Name) {
-			continue
-		}
-		if strings.ToLower(e.Name) == familyStem {
-			bare = append(bare, m)
-		}
-	}
-	if len(bare) == 1 {
-		return bare[0], nil
-	}
-	if len(bare) > 1 {
-		names := make([]string, 0, len(bare))
-		for _, m := range bare {
-			names = append(names, ix.Entries[m].Name)
-		}
-		return 0, fmt.Errorf("more than one entry is named exactly %q (%s), so which one is the base build is a judgement this job will not make", familyStem, strings.Join(names, ", "))
-	}
-
-	// No shared stem to be named after. An entry whose name every other member
-	// extends is still recognisably the base one, and this is the only handle
-	// left for families whose weights carry no readable quantization token at
-	// all, such as the ONNX builds.
-	if prefix, ok := uniquePrefixMember(ix, members); ok {
-		return prefix, nil
-	}
-
-	best := -1
-	bestWidth := 1 << 20
-	for _, m := range members {
-		e := ix.Entries[m]
-		width := unknownWidth
-		if primary, ok := e.PrimaryWeightFile(); ok {
-			width = BuildWidth(primary)
-		}
-		// Members are visited in gallery order, so a strict comparison leaves
-		// the earliest entry holding a tie and the choice is deterministic.
-		if width < bestWidth {
-			best, bestWidth = m, width
-		}
-	}
-	if best < 0 {
-		return 0, fmt.Errorf("no member could be identified as the smallest build")
-	}
-	if bestWidth == unknownWidth {
-		names := make([]string, 0, len(members))
-		for _, m := range members {
-			names = append(names, ix.Entries[m].Name)
-		}
-		return 0, fmt.Errorf("no member declares a weight file whose quantization can be read (%s), so the smallest build cannot be identified", strings.Join(names, ", "))
-	}
-	return best, nil
-}
-
-// uniquePrefixMember reports the single member whose name every other member's
-// name starts with, if there is exactly one.
-func uniquePrefixMember(ix *Index, members []int) (int, bool) {
-	found := -1
-	for _, m := range members {
-		name := strings.ToLower(ix.Entries[m].Name)
-		isPrefix := true
-		for _, other := range members {
-			if other == m {
-				continue
-			}
-			if !strings.HasPrefix(strings.ToLower(ix.Entries[other].Name), name) {
-				isPrefix = false
-				break
-			}
-		}
-		if !isPrefix {
-			continue
-		}
-		if found >= 0 {
-			return 0, false
-		}
-		found = m
-	}
-	return found, found >= 0
-}
-
-// quantDifference lists the quantization tokens that tell two names apart. It
-// is the compact form of the evidence: "these differ only by q4_k_m vs q8_0".
-func quantDifference(names ...string) []string {
-	var out []string
-	seen := map[string]struct{}{}
-	for _, name := range names {
-		// Filenames arrive here too, so the extension goes first and "/" counts
-		// as a separator. "_" deliberately does not: it holds "q4_k_m" together.
-		trimmed := weightExtension.ReplaceAllString(name, "")
-		for _, seg := range strings.FieldsFunc(strings.ToLower(trimmed), func(r rune) bool { return r == '-' || r == '/' }) {
-			if !IsQuantToken(seg) {
-				continue
-			}
-			if _, ok := seen[seg]; ok {
-				continue
-			}
-			seen[seg] = struct{}{}
-			out = append(out, seg)
-		}
-	}
-	sort.Strings(out)
-	return out
-}

.github/ci/variantproposals/propose_test.go

@@ -1,429 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"strings"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-// entryYAML writes one gallery entry with a single weight file, which is the
-// shape almost every real entry has. Specs that need something else write the
-// YAML out by hand.
-func entryYAML(name, repo, filename, sha string) string {
-	return fmt.Sprintf(`- name: %s
-  url: github:mudler/LocalAI/gallery/virtual.yaml@master
-  overrides:
-    parameters:
-      model: %s
-  files:
-    - filename: %s
-      uri: huggingface://%s/%s
-      sha256: %s
-`, name, filename, filename, repo, filename, sha)
-}
-
-func indexOf(entries ...string) *Index {
-	ix, err := ParseIndex(strings.Join(entries, ""))
-	ExpectWithOffset(1, err).ToNot(HaveOccurred())
-	return ix
-}
-
-// familyNames flattens a result into "parent <- variant, variant" strings, the
-// form the specs assert against.
-func familyNames(r *Result) []string {
-	out := make([]string, 0, len(r.Families))
-	for _, f := range r.Families {
-		names := make([]string, 0, len(f.Proposals))
-		for _, p := range f.Proposals {
-			names = append(names, p.Variant)
-		}
-		out = append(out, f.Parent+" <- "+strings.Join(names, ", "))
-	}
-	return out
-}
-
-func refusalReasons(r *Result) string {
-	var b strings.Builder
-	for _, ref := range r.Refusals {
-		b.WriteString(strings.Join(ref.Members, " + ") + ": " + ref.Reason + "\n")
-	}
-	return b.String()
-}
-
-func suppressionReasons(r *Result) string {
-	var b strings.Builder
-	for _, s := range r.Suppressed {
-		b.WriteString(s.String() + "\n")
-	}
-	return b.String()
-}
-
-var _ = Describe("Propose", func() {
-	Describe("the grouping signals", func() {
-		It("groups entries whose names differ only by a quantization marker", func() {
-			ix := indexOf(
-				entryYAML("foo-model", "acme/foo-GGUF", "foo-model-Q4_K_M.gguf", "aa"),
-				entryYAML("foo-model-q8_0", "acme/foo-GGUF", "foo-model-Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(ConsistOf("foo-model <- foo-model-q8_0"))
-			Expect(r.Families[0].Proposals[0].Evidence.Signals).To(ContainElement(SignalName))
-			Expect(r.Families[0].Proposals[0].Evidence.SharedStem).To(Equal("foo-model"))
-			Expect(r.Families[0].Proposals[0].Evidence.QuantTokens).To(ContainElements("q4_k_m", "q8_0"))
-		})
-
-		It("groups entries that use the colon config-suffix convention", func() {
-			ix := indexOf(
-				entryYAML("bar-model", "acme/bar-GGUF", "bar-model-Q4_K_M.gguf", "aa"),
-				entryYAML("bar-model:grammar-functioncall", "acme/bar-GGUF", "bar-model-Q4_K_M-grammar.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(ConsistOf("bar-model <- bar-model:grammar-functioncall"))
-			Expect(r.Families[0].Proposals[0].Evidence.Signals).To(ContainElement(SignalConfigSuffix))
-		})
-
-		It("groups entries whose own weight file is the same file at another quantization", func() {
-			// The names share no stem, so only the filename signal can link
-			// these two.
-			ix := indexOf(
-				entryYAML("omni-cpp", "Serveurperso/Omni-GGUF", "omnivoice-base-Q8_0.gguf", "aa"),
-				entryYAML("omni-cpp-hq", "Serveurperso/Omni-GGUF", "omnivoice-base-BF16.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(ConsistOf("omni-cpp <- omni-cpp-hq"))
-			ev := r.Families[0].Proposals[0].Evidence
-			Expect(ev.Signals).To(ConsistOf(SignalWeightFile))
-			Expect(ev.SharedFile).To(Equal("omnivoice-base"))
-			Expect(ev.SharedRepo).To(Equal("serveurperso/omni-gguf"))
-		})
-
-		It("does not let a shared auxiliary file link unrelated models", func() {
-			// Both entries ship the same text encoder. That is a packaging
-			// convention, not evidence of shared weights: this is how an
-			// earlier sweep linked four wan-2.1 entries to each other.
-			ix := indexOf(`- name: wan-2.1-t2v
-  url: u
-  files:
-    - filename: wan-2.1-t2v-Q4_K_M.gguf
-      uri: huggingface://acme/wan/wan-2.1-t2v-Q4_K_M.gguf
-      sha256: aa
-    - filename: umt5-xxl-encoder-Q8_0.gguf
-      uri: huggingface://acme/wan/umt5-xxl-encoder-Q8_0.gguf
-      sha256: cc
-`, `- name: z-image-turbo
-  url: u
-  files:
-    - filename: z-image-turbo-Q4_K_M.gguf
-      uri: huggingface://acme/wan/z-image-turbo-Q4_K_M.gguf
-      sha256: bb
-    - filename: umt5-xxl-encoder-Q8_0.gguf
-      uri: huggingface://acme/wan/umt5-xxl-encoder-Q8_0.gguf
-      sha256: cc
-`)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-		})
-
-		It("does not treat a shared filename in two different repos as evidence", func() {
-			// A finetune republished under the base model's filename is the
-			// most common way this signal misfires.
-			ix := indexOf(
-				entryYAML("llama-3.2-3b-instruct", "hugging-quants/Llama-3.2-3B-Instruct-GGUF", "llama-3.2-3b-instruct-q4_k_m.gguf", "aa"),
-				entryYAML("llama-3.2-3b-shiro-roleplay", "someone/Shiro-GGUF", "Llama-3.2-3B-Instruct.Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-		})
-	})
-
-	Describe("what must never be proposed", func() {
-		It("does not group different parameter sizes that share a prefix", func() {
-			ix := indexOf(
-				entryYAML("qwen3-tts-cpp-0.6b-base", "Serveurperso/Qwen3-TTS-GGUF", "qwen3-tts-talker-Q4_K_M.gguf", "aa"),
-				entryYAML("qwen3-tts-cpp-1.7b-base", "Serveurperso/Qwen3-TTS-GGUF", "qwen3-tts-talker-Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(suppressionReasons(r)).To(ContainSubstring("different parameter sizes"))
-		})
-
-		It("does not group the Gemma effective sizes", func() {
-			ix := indexOf(
-				entryYAML("gemma-4-e2b-it", "google/gemma-GGUF", "gemma-4-it-Q4_K_M.gguf", "aa"),
-				entryYAML("gemma-4-e4b-it", "google/gemma-GGUF", "gemma-4-it-Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(suppressionReasons(r)).To(ContainSubstring("different parameter sizes"))
-		})
-
-		It("does not group entries with a byte-identical install payload", func() {
-			// whisper-1 exists so OpenAI-compatible clients can send that name.
-			// Folding it under whisper-base would hide the name they send.
-			payload := `  url: github:mudler/LocalAI/gallery/whisper-base.yaml@master
-  overrides:
-    parameters:
-      model: ggml-whisper-base.bin
-  files:
-    - filename: ggml-whisper-base.bin
-      uri: huggingface://ggerganov/whisper.cpp/ggml-base.bin
-      sha256: aa
-`
-			ix := indexOf("- name: whisper-base\n"+payload, "- name: whisper-1\n"+payload)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(r.AliasSkipped).To(HaveLen(1))
-			Expect(r.AliasSkipped[0].Reason).To(ContainSubstring("aliases"))
-		})
-
-		DescribeTable("declines the categories the ledger records",
-			func(nameA, nameB string, ledgerYAML string) {
-				ix := indexOf(
-					entryYAML(nameA, "acme/repo", "shared-weights-Q4_K_M.gguf", "aa"),
-					entryYAML(nameB, "acme/repo", "shared-weights-Q8_0.gguf", "bb"),
-				)
-				ledger, err := ParseLedger([]byte(ledgerYAML))
-				Expect(err).ToNot(HaveOccurred())
-
-				// Without the ledger these would be proposed, which is what
-				// makes the ledger load bearing rather than decorative.
-				Expect(familyNames(Propose(ix, nil))).ToNot(BeEmpty())
-
-				r := Propose(ix, ledger)
-				Expect(familyNames(r)).To(BeEmpty())
-				Expect(r.Suppressed).To(HaveLen(1))
-			},
-			Entry("a finetune", "base-model", "base-model-abliterated",
-				"tokens:\n  - {token: abliterated, reason: finetune}\n"),
-			Entry("a distill", "base-model", "base-model-distilled",
-				"tokens:\n  - {token: distilled, reason: distilled}\n"),
-			Entry("English-only versus multilingual ASR", "whisper-small", "whisper-small-en",
-				"pairs:\n  - {parent: whisper-small, variant: whisper-small-en, reason: English-only versus multilingual}\n"),
-			Entry("two products sharing a prefix", "vibevoice-cpp", "vibevoice-cpp-asr",
-				"pairs:\n  - {parent: vibevoice-cpp, variant: vibevoice-cpp-asr, reason: different products}\n"),
-			Entry("a per-language release", "kokoros-de", "kokoros-ja",
-				"groups:\n  - {members: [kokoros, kokoros-de, kokoros-ja], reason: different languages}\n"),
-		)
-
-		It("reports the ledger's reason so its effect stays visible", func() {
-			ix := indexOf(
-				entryYAML("base-model", "acme/repo", "shared-weights-Q4_K_M.gguf", "aa"),
-				entryYAML("base-model-heretic", "acme/repo", "shared-weights-Q8_0.gguf", "bb"),
-			)
-			ledger, err := ParseLedger([]byte("tokens:\n  - {token: heretic, reason: \"finetune, not a re-quantization\"}\n"))
-			Expect(err).ToNot(HaveOccurred())
-			r := Propose(ix, ledger)
-			Expect(suppressionReasons(r)).To(ContainSubstring("finetune, not a re-quantization"))
-			Expect(suppressionReasons(r)).To(ContainSubstring(`token "heretic"`))
-		})
-	})
-
-	Describe("parent selection", func() {
-		It("picks the bare-named entry when one exists", func() {
-			ix := indexOf(
-				entryYAML("base-model-q8_0", "acme/repo", "base-model-Q8_0.gguf", "aa"),
-				entryYAML("base-model", "acme/repo", "base-model-Q4_K_M.gguf", "bb"),
-				entryYAML("base-model-f16", "acme/repo", "base-model-f16.gguf", "cc"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(ConsistOf("base-model <- base-model-f16, base-model-q8_0"))
-		})
-
-		It("picks the smallest build when no entry is bare-named", func() {
-			ix := indexOf(
-				entryYAML("ced-base-f16", "acme/repo", "ced-base-f16.gguf", "aa"),
-				entryYAML("ced-base-q8", "acme/repo", "ced-base-Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(ConsistOf("ced-base-q8 <- ced-base-f16"))
-		})
-
-		It("judges the smallest build by the quantization in the model filename, not the name", func() {
-			// The names carry no marker at all; only the filenames say which
-			// build is which.
-			ix := indexOf(
-				entryYAML("thing-hq", "acme/repo", "thing-weights-BF16.gguf", "aa"),
-				entryYAML("thing-lite", "acme/repo", "thing-weights-Q4_K_M.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(ConsistOf("thing-lite <- thing-hq"))
-		})
-	})
-
-	Describe("the rules a proposal has to respect", func() {
-		It("refuses to nest: a target that already offers variants of its own", func() {
-			ix := indexOf(
-				entryYAML("nest-model", "acme/repo", "nest-model-Q4_K_M.gguf", "aa"),
-				`- name: nest-model-q8_0
-  url: u
-  variants:
-    - model: nest-model-q8_0-mtp
-  overrides:
-    parameters:
-      model: nest-model-Q8_0.gguf
-  files:
-    - filename: nest-model-Q8_0.gguf
-      uri: huggingface://acme/repo/nest-model-Q8_0.gguf
-      sha256: bb
-`,
-				entryYAML("nest-model-q8_0-mtp", "other/repo", "nest-model-mtp.gguf", "cc"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(refusalReasons(r)).To(ContainSubstring("already offers variants of its own"))
-		})
-
-		It("refuses to nest: a parent that is already somebody else's variant", func() {
-			ix := indexOf(
-				`- name: outer
-  url: u
-  variants:
-    - model: middle
-  overrides:
-    parameters:
-      model: outer-Q4_K_M.gguf
-  files:
-    - filename: outer-Q4_K_M.gguf
-      uri: huggingface://acme/repo/outer-Q4_K_M.gguf
-      sha256: aa
-`,
-				entryYAML("middle", "acme/other", "middle-Q4_K_M.gguf", "bb"),
-				entryYAML("middle-q8_0", "acme/other", "middle-Q8_0.gguf", "cc"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(refusalReasons(r)).To(ContainSubstring("would nest variants"))
-		})
-
-		It("refuses to let two parents claim one target", func() {
-			ix := indexOf(
-				`- name: claimant
-  url: u
-  variants:
-    - model: contested-q8_0
-  overrides:
-    parameters:
-      model: claimant-Q4_K_M.gguf
-  files:
-    - filename: claimant-Q4_K_M.gguf
-      uri: huggingface://acme/repo/claimant-Q4_K_M.gguf
-      sha256: aa
-`,
-				entryYAML("contested", "acme/other", "contested-Q4_K_M.gguf", "bb"),
-				entryYAML("contested-q8_0", "acme/other", "contested-Q8_0.gguf", "cc"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(refusalReasons(r)).To(ContainSubstring("already a variant of"))
-		})
-
-		It("refuses a target that is not independently installable", func() {
-			ix := indexOf(
-				entryYAML("stub-model", "acme/repo", "stub-model-Q4_K_M.gguf", "aa"),
-				"- name: stub-model-q8_0\n  description: a stanza nobody finished\n",
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(refusalReasons(r)).To(ContainSubstring("not independently installable"))
-		})
-
-		It("refuses a family whose parent defines a merge anchor, naming the entries that would inherit", func() {
-			ix := indexOf(
-				`- &anchored
-  name: anchored-model
-  url: u
-  overrides:
-    parameters:
-      model: anchored-Q4_K_M.gguf
-  files:
-    - filename: anchored-Q4_K_M.gguf
-      uri: huggingface://acme/repo/anchored-Q4_K_M.gguf
-      sha256: aa
-`,
-				`- !!merge <<: *anchored
-  name: anchored-child
-  variants: []
-  overrides:
-    parameters:
-      model: unrelated-child-Q4_K_M.gguf
-  files:
-    - filename: unrelated-child-Q4_K_M.gguf
-      uri: huggingface://other/repo/unrelated-child-Q4_K_M.gguf
-      sha256: cc
-`,
-				entryYAML("anchored-model-q8_0", "acme/repo", "anchored-Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(refusalReasons(r)).To(ContainSubstring("defines YAML anchor &anchored"))
-			Expect(refusalReasons(r)).To(ContainSubstring("anchored-child"))
-			Expect(refusalReasons(r)).To(ContainSubstring("variants: []"))
-		})
-
-		It("refuses an entry whose name is not unique in the gallery", func() {
-			ix := indexOf(
-				entryYAML("twin", "acme/repo", "twin-Q4_K_M.gguf", "aa"),
-				entryYAML("twin", "acme/repo", "twin-Q4_K_M.gguf", "aa"),
-				entryYAML("twin-q8_0", "acme/repo", "twin-Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(BeEmpty())
-			Expect(refusalReasons(r)).To(ContainSubstring("appears more than once"))
-		})
-
-		It("says nothing about a pair that is already grouped", func() {
-			ix := indexOf(
-				`- name: settled
-  url: u
-  variants:
-    - model: settled-q8_0
-  overrides:
-    parameters:
-      model: settled-Q4_K_M.gguf
-  files:
-    - filename: settled-Q4_K_M.gguf
-      uri: huggingface://acme/repo/settled-Q4_K_M.gguf
-      sha256: aa
-`,
-				entryYAML("settled-q8_0", "acme/repo", "settled-Q8_0.gguf", "bb"),
-			)
-			r := Propose(ix, nil)
-			Expect(r.HasProposals()).To(BeFalse())
-			Expect(r.Refusals).To(BeEmpty())
-			Expect(r.Suppressed).To(BeEmpty())
-		})
-
-		It("adds only the missing members to a family that already exists", func() {
-			ix := indexOf(
-				`- name: partial
-  url: u
-  variants:
-    - model: partial-q8_0
-  overrides:
-    parameters:
-      model: partial-Q4_K_M.gguf
-  files:
-    - filename: partial-Q4_K_M.gguf
-      uri: huggingface://acme/repo/partial-Q4_K_M.gguf
-      sha256: aa
-`,
-				entryYAML("partial-q8_0", "acme/repo", "partial-Q8_0.gguf", "bb"),
-				entryYAML("partial-f16", "acme/repo", "partial-f16.gguf", "cc"),
-			)
-			r := Propose(ix, nil)
-			Expect(familyNames(r)).To(ConsistOf("partial <- partial-f16"))
-		})
-	})
-
-	It("does not modify the index it was given", func() {
-		text := entryYAML("foo-model", "acme/foo-GGUF", "foo-model-Q4_K_M.gguf", "aa") +
-			entryYAML("foo-model-q8_0", "acme/foo-GGUF", "foo-model-Q8_0.gguf", "bb")
-		ix, err := ParseIndex(text)
-		Expect(err).ToNot(HaveOccurred())
-		before := strings.Join(ix.Lines, "\n")
-		Propose(ix, nil)
-		Expect(strings.Join(ix.Lines, "\n")).To(Equal(before))
-	})
-})

.github/ci/variantproposals/quant.go

@@ -1,151 +0,0 @@
-package main
-
-import (
-	"regexp"
-	"strconv"
-	"strings"
-)
-
-// Quantization and precision markers that distinguish one build of a set of
-// weights from another build of the same weights. Stripping them from a name
-// is what lets the proposer notice that two entries are the same model.
-//
-// qat and apex are in this list on a maintainer ruling: they are quantization
-// techniques applied to published weights, not separate weights. Names that use
-// "apex" to mean a finetune are handled by the rejection ledger instead, because
-// no amount of pattern matching can tell the two uses apart.
-const quantAlternation = `q[2-8](?:_[0-9a-z]+)*|pq[2-8](?:_[0-9a-z]+)*|iq[1-9][0-9a-z]*(?:_[0-9a-z]+)*|i1|` +
-	`f16|f32|bf16|fp16|fp32|fp8|fp4|nvfp4|mxfp4(?:_moe)*|awq|gptq|qat|apex|gguf|ggml|[0-9]+bit|g[0-9]+`
-
-// quantSegment matches a whole hyphen-delimited segment of an entry name.
-// Names separate their parts with "-" and keep quantization tokens internally
-// joined with "_", so a segment is the right unit here: "q4_k_m" arrives whole.
-var quantSegment = regexp.MustCompile(`^(?:` + quantAlternation + `)$`)
-
-// quantFileSuffix matches a trailing quantization token in a weight filename.
-// Filenames mix "-", "_" and "." as separators, so unlike entry names they
-// cannot be split into segments up front without tearing "Q4_K_M" apart.
-var quantFileSuffix = regexp.MustCompile(`(?i)[-_.](?:` + quantAlternation + `)$`)
-
-var weightExtension = regexp.MustCompile(`(?i)\.(gguf|ggml|safetensors|bin|pt|pth|onnx)$`)
-
-// IsQuantToken reports whether a single name segment is a quantization or
-// precision marker rather than part of the model's identity.
-func IsQuantToken(segment string) bool {
-	return quantSegment.MatchString(strings.ToLower(segment))
-}
-
-// NameStem reduces an entry name to the identity it shares with its alternative
-// builds: the config suffix after ":" is dropped, then trailing quantization
-// segments are stripped.
-//
-// It implements the first two grouping signals together because they answer the
-// same question. "foo:q8_0" and "foo-q8_0" are both alternative builds of "foo",
-// and the caller that needs to report which convention was used can compare the
-// name against the stem itself.
-//
-// At least one segment always survives, so a name made entirely of quantization
-// tokens does not collapse to the empty stem and swallow every other such name.
-func NameStem(name string) string {
-	base := strings.ToLower(strings.TrimSpace(name))
-	if i := strings.Index(base, ":"); i >= 0 {
-		base = base[:i]
-	}
-	segments := strings.Split(base, "-")
-	for len(segments) > 1 && quantSegment.MatchString(segments[len(segments)-1]) {
-		segments = segments[:len(segments)-1]
-	}
-	return strings.Join(segments, "-")
-}
-
-// HasConfigSuffix reports whether a name uses the ":" convention for naming a
-// config variant of another entry.
-func HasConfigSuffix(name string) bool {
-	return strings.Contains(name, ":")
-}
-
-// FileStem reduces a weight filename to the identity shared by its other
-// quantizations: directories, extension and trailing quantization tokens go.
-//
-// This is the third grouping signal. It is the one that has misfired before, so
-// callers must filter auxiliary files out before handing a filename here: a
-// shared text encoder is not evidence of shared weights.
-func FileStem(filename string) string {
-	base := filename
-	if i := strings.LastIndex(base, "/"); i >= 0 {
-		base = base[i+1:]
-	}
-	base = weightExtension.ReplaceAllString(base, "")
-	for {
-		stripped := quantFileSuffix.ReplaceAllString(base, "")
-		if stripped == base {
-			break
-		}
-		base = stripped
-	}
-	return strings.ToLower(base)
-}
-
-// bitsPerWeight ranks quantization tokens so the smallest build of a family can
-// be identified when no bare-named entry exists to be the parent.
-//
-// The figures are nominal bits per weight, not measured file sizes. Ranking is
-// all that is asked of them, and a nominal figure is available from the name
-// alone without downloading anything.
-func bitsPerWeight(token string) (int, bool) {
-	t := strings.ToLower(token)
-	switch {
-	case t == "i1":
-		return 1, true
-	case strings.HasPrefix(t, "nvfp4"), strings.HasPrefix(t, "mxfp4"), t == "fp4":
-		return 4, true
-	case t == "fp8":
-		return 8, true
-	case t == "f16", t == "bf16", t == "fp16":
-		return 16, true
-	case t == "f32", t == "fp32":
-		return 32, true
-	case t == "awq", t == "gptq":
-		return 4, true
-	}
-	if m := regexp.MustCompile(`^p?q([1-9])`).FindStringSubmatch(t); m != nil {
-		n, _ := strconv.Atoi(m[1])
-		return n, true
-	}
-	if m := regexp.MustCompile(`^iq([1-9])`).FindStringSubmatch(t); m != nil {
-		n, _ := strconv.Atoi(m[1])
-		return n, true
-	}
-	if m := regexp.MustCompile(`^([0-9]+)bit$`).FindStringSubmatch(t); m != nil {
-		n, _ := strconv.Atoi(m[1])
-		return n, true
-	}
-	return 0, false
-}
-
-// unknownWidth sorts after every recognised quantization so an entry whose
-// build cannot be read from its filename never wins the "smallest build" tie
-// break by accident.
-const unknownWidth = 1 << 10
-
-// BuildWidth reports the nominal bits per weight of the build a filename holds.
-// An unreadable filename gets unknownWidth.
-func BuildWidth(filename string) int {
-	base := filename
-	if i := strings.LastIndex(base, "/"); i >= 0 {
-		base = base[i+1:]
-	}
-	base = weightExtension.ReplaceAllString(base, "")
-	best := unknownWidth
-	for {
-		m := quantFileSuffix.FindString(base)
-		if m == "" {
-			break
-		}
-		if bits, ok := bitsPerWeight(m[1:]); ok && bits < best {
-			best = bits
-		}
-		base = base[:len(base)-len(m)]
-	}
-	return best
-}

.github/ci/variantproposals/quant_test.go

@@ -1,92 +0,0 @@
-package main
-
-import (
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("quantization markers", func() {
-	DescribeTable("NameStem strips the markers that distinguish builds, not models",
-		func(name, expected string) {
-			Expect(NameStem(name)).To(Equal(expected))
-		},
-		Entry("plain q4", "foo-model-q4_k_m", "foo-model"),
-		Entry("q8_0", "foo-model-q8_0", "foo-model"),
-		Entry("q5_1", "foo-model-q5_1", "foo-model"),
-		Entry("q2 with group size", "ternary-bonsai-8b-q2-g64", "ternary-bonsai-8b"),
-		Entry("iq variant", "ideogram-4-iq4nl-ggml", "ideogram-4"),
-		Entry("i1 imatrix", "orca-agent-v0.1-i1", "orca-agent-v0.1"),
-		Entry("f16", "ced-base-f16", "ced-base"),
-		Entry("bf16", "some-model-bf16", "some-model"),
-		Entry("fp8", "some-model-fp8", "some-model"),
-		Entry("nvfp4", "qwen3.6-27b-nvfp4", "qwen3.6-27b"),
-		Entry("mxfp4_moe", "huihui-qwen3-vl-30b-a3b-instruct-abliterated-mxfp4_moe", "huihui-qwen3-vl-30b-a3b-instruct-abliterated"),
-		Entry("pq2", "ternary-bonsai-8b-pq2", "ternary-bonsai-8b"),
-		Entry("awq", "some-model-awq", "some-model"),
-		Entry("gptq", "some-model-gptq", "some-model"),
-		Entry("Nbit", "qwen3-8b-mlx-4bit", "qwen3-8b-mlx"),
-		Entry("gguf", "some-model-gguf", "some-model"),
-		Entry("ggml", "flux.1-dev-ggml", "flux.1-dev"),
-		Entry("qat is a quantization technique", "gemma-3-27b-it-qat", "gemma-3-27b-it"),
-		Entry("apex is a quantization technique", "qwen3.6-35b-a3b-apex", "qwen3.6-35b-a3b"),
-		Entry("stacked markers", "gemma-4-e2b-it-qat-q4_0", "gemma-4-e2b-it"),
-		Entry("the config suffix is dropped", "phi-2-chat:Q8_0", "phi-2-chat"),
-		Entry("a non-quant config suffix is dropped too", "meta-llama-3.1-8b-instruct:grammar-functioncall", "meta-llama-3.1-8b-instruct"),
-	)
-
-	DescribeTable("NameStem leaves alone what identifies a different model",
-		func(name, expected string) {
-			Expect(NameStem(name)).To(Equal(expected))
-		},
-		Entry("parameter size", "qwen3-tts-cpp-0.6b-base", "qwen3-tts-cpp-0.6b-base"),
-		Entry("language suffix", "kokoros-de", "kokoros-de"),
-		Entry("English-only ASR", "whisper-small-en", "whisper-small-en"),
-		Entry("finetune", "qwen3-30b-a3b-abliterated", "qwen3-30b-a3b-abliterated"),
-		Entry("product suffix", "vibevoice-cpp-asr", "vibevoice-cpp-asr"),
-	)
-
-	It("never strips a name down to nothing", func() {
-		Expect(NameStem("q4_k_m")).To(Equal("q4_k_m"))
-		Expect(NameStem("f16-q8_0")).To(Equal("f16"))
-	})
-
-	DescribeTable("FileStem reduces a weight filename to the weights it holds",
-		func(filename, expected string) {
-			Expect(FileStem(filename)).To(Equal(expected))
-		},
-		Entry("directory and extension go", "bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0.gguf", "ternary-bonsai-8b"),
-		Entry("underscored quant token stays whole", "Llama-3.2-1B-Instruct-Q4_K_M.gguf", "llama-3.2-1b-instruct"),
-		Entry("dot separated quant token", "Llama-3.2-3B-Instruct.Q4_K_M.gguf", "llama-3.2-3b-instruct"),
-		Entry("group size suffix", "Ternary-Bonsai-8B-Q2_0_g64.gguf", "ternary-bonsai-8b"),
-		Entry("bf16", "omnivoice-cpp-hq/omnivoice-base-BF16.gguf", "omnivoice-base"),
-		Entry("safetensors", "some/dir/Model-Name-fp8.safetensors", "model-name"),
-	)
-
-	DescribeTable("BuildWidth reads the nominal width out of a filename",
-		func(filename string, expected int) {
-			Expect(BuildWidth(filename)).To(Equal(expected))
-		},
-		Entry("q4", "foo-Q4_K_M.gguf", 4),
-		Entry("q8", "foo-Q8_0.gguf", 8),
-		Entry("q2", "foo-Q2_0.gguf", 2),
-		Entry("f16", "foo-f16.gguf", 16),
-		Entry("bf16", "foo-BF16.gguf", 16),
-		Entry("iq3", "foo-iq3_xxs.gguf", 3),
-		Entry("nothing readable sorts last", "foo.gguf", unknownWidth),
-	)
-
-	It("treats an auxiliary file as never being the model's own weights", func() {
-		for _, f := range []string{
-			"mmproj-model-f16.gguf",
-			"dir/vae-BF16.gguf",
-			"clip_l.safetensors",
-			"umt5-xxl-encoder-Q8_0.gguf",
-			"t5xxl_fp16.safetensors",
-			"ae.safetensors",
-			"omnivoice-tokenizer-Q8_0.gguf",
-		} {
-			Expect(IsAuxiliaryFile(f)).To(BeTrue(), "expected %q to be auxiliary", f)
-		}
-		Expect(IsAuxiliaryFile("gemma-3-27b-it-Q4_K_M.gguf")).To(BeFalse())
-	})
-})

.github/ci/variantproposals/variantproposals_suite_test.go

@@ -1,13 +0,0 @@
-package main
-
-import (
-	"testing"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-func TestVariantProposals(t *testing.T) {
-	RegisterFailHandler(Fail)
-	RunSpecs(t, "gallery variant proposals")
-}

.github/dependabot.yml

@@ -45,16 +45,6 @@ updates:
     directory: "/backend/python/diffusers"
     schedule:
       interval: "weekly"
-    # torch and transformers are deliberately pinned in this backend (see
-    # backend/python/diffusers/requirements-*.txt and issue #9979), and the
-    # l4t12 variant resolves them from the Jetson pip index
-    # (https://pypi.jetson-ai-lab.io/jp6/cu129/). dependabot cannot authenticate
-    # against that index and fails the whole weekly update with a
-    # private_source_authentication_failure. Ignore the two pinned deps we don't
-    # want bumped anyway so the job stays green.
-    ignore:
-    - dependency-name: "torch"
-    - dependency-name: "transformers"
   - package-ecosystem: "pip"
     directory: "/backend/python/exllama"
     schedule:

.github/workflows/backend.yml

@@ -32,33 +32,19 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     outputs:
+      matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
       matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
       matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
       merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
+      merge-matrix-singlearch: ${{ steps.set-matrix.outputs['merge-matrix-singlearch'] }}
+      has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
       has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
       has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
       has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
-      # Single-arch backends are sharded across SINGLEARCH_SHARDS matrix jobs to
-      # stay under GitHub's 256-jobs-per-matrix limit (see changed-backends.js).
-      matrix-singlearch-1: ${{ steps.set-matrix.outputs['matrix-singlearch-1'] }}
-      merge-matrix-singlearch-1: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-1'] }}
-      has-backends-singlearch-1: ${{ steps.set-matrix.outputs['has-backends-singlearch-1'] }}
-      has-merges-singlearch-1: ${{ steps.set-matrix.outputs['has-merges-singlearch-1'] }}
-      matrix-singlearch-2: ${{ steps.set-matrix.outputs['matrix-singlearch-2'] }}
-      merge-matrix-singlearch-2: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-2'] }}
-      has-backends-singlearch-2: ${{ steps.set-matrix.outputs['has-backends-singlearch-2'] }}
-      has-merges-singlearch-2: ${{ steps.set-matrix.outputs['has-merges-singlearch-2'] }}
-      matrix-singlearch-3: ${{ steps.set-matrix.outputs['matrix-singlearch-3'] }}
-      merge-matrix-singlearch-3: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-3'] }}
-      has-backends-singlearch-3: ${{ steps.set-matrix.outputs['has-backends-singlearch-3'] }}
-      has-merges-singlearch-3: ${{ steps.set-matrix.outputs['has-merges-singlearch-3'] }}
-      matrix-singlearch-4: ${{ steps.set-matrix.outputs['matrix-singlearch-4'] }}
-      merge-matrix-singlearch-4: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-4'] }}
-      has-backends-singlearch-4: ${{ steps.set-matrix.outputs['has-backends-singlearch-4'] }}
-      has-merges-singlearch-4: ${{ steps.set-matrix.outputs['has-merges-singlearch-4'] }}
+      has-merges-singlearch: ${{ steps.set-matrix.outputs['has-merges-singlearch'] }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
 
       - name: Setup Bun
         uses: oven-sh/setup-bun@v2
@@ -123,9 +109,9 @@ jobs:
   # take their full ~6h cold without blocking manifest assembly for the
   # multi-arch backends whose per-arch digests would otherwise sit untagged
   # on quay long enough to be GC'd.
-  backend-jobs-singlearch-1:
+  backend-jobs-singlearch:
     needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-1'] == 'true'
+    if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
     uses: ./.github/workflows/backend_build.yml
     with:
       tag-latest: ${{ matrix.tag-latest }}
@@ -152,100 +138,7 @@ jobs:
     strategy:
       fail-fast: false
       max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-1']) }}
-
-  backend-jobs-singlearch-2:
-    needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-2'] == 'true'
-    uses: ./.github/workflows/backend_build.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-2']) }}
-
-  backend-jobs-singlearch-3:
-    needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-3'] == 'true'
-    uses: ./.github/workflows/backend_build.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-3']) }}
-
-  backend-jobs-singlearch-4:
-    needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-4'] == 'true'
-    uses: ./.github/workflows/backend_build.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-4']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
 
   # Apply tags to per-arch digests via `imagetools create`. Split into two
   # jobs that mirror the build split so each merge waits ONLY on its
@@ -281,12 +174,10 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
 
-  # One merge shard per build shard: backend-merge-jobs-singlearch-<n> needs only
-  # backend-jobs-singlearch-<n>, preserving the "merge waits only on its own
-  # build" property while staying under the 256-jobs-per-matrix limit.
-  backend-merge-jobs-singlearch-1:
-    needs: [generate-matrix, backend-jobs-singlearch-1]
-    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-1'] == 'true' }}
+  backend-merge-jobs-singlearch:
+    needs: [generate-matrix, backend-jobs-singlearch]
+    # See note on backend-merge-jobs-multiarch above for !cancelled().
+    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch'] == 'true' }}
     uses: ./.github/workflows/backend_merge.yml
     with:
       tag-latest: ${{ matrix.tag-latest }}
@@ -298,55 +189,7 @@ jobs:
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
     strategy:
       fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-1']) }}
-
-  backend-merge-jobs-singlearch-2:
-    needs: [generate-matrix, backend-jobs-singlearch-2]
-    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-2'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-2']) }}
-
-  backend-merge-jobs-singlearch-3:
-    needs: [generate-matrix, backend-jobs-singlearch-3]
-    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-3'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-3']) }}
-
-  backend-merge-jobs-singlearch-4:
-    needs: [generate-matrix, backend-jobs-singlearch-4]
-    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-4'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-4']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch']) }}
 
   backend-jobs-darwin:
     needs: generate-matrix

.github/workflows/backend_build.yml

@@ -101,7 +101,7 @@ jobs:
     steps:
 
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
 

.github/workflows/backend_build_darwin.yml

@@ -57,7 +57,7 @@ jobs:
       HOMEBREW_NO_ANALYTICS: '1'
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
 
@@ -82,7 +82,7 @@ jobs:
       # as the Linux registry cache.
       - name: Restore Homebrew cache
         id: brew-cache
-        uses: actions/cache/restore@v6
+        uses: actions/cache/restore@v4
         with:
           path: |
             ~/Library/Caches/Homebrew/downloads
@@ -99,7 +99,6 @@ jobs:
             /opt/homebrew/Cellar/xxhash
             /opt/homebrew/Cellar/zstd
             /opt/homebrew/Cellar/nlohmann-json
-            /opt/homebrew/Cellar/opus
           key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
 
       - name: Dependencies
@@ -114,12 +113,7 @@ jobs:
           # nlohmann-json is header-only and required by the ds4 backend
           # (dsml_renderer.cpp includes <nlohmann/json.hpp>); on Linux it comes
           # from the apt-installed nlohmann-json3-dev in the build image.
-          # opus + pkg-config are required by the opus go backend: its
-          # Makefile/package.sh call `pkg-config --cflags/--libs opus` to build
-          # libopusshim.dylib and to locate libopus.dylib for bundling. brew's
-          # pkg-config defaults its search path to the Homebrew prefix so the
-          # opus.pc is found.
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd nlohmann-json opus pkg-config
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd nlohmann-json
           # Force-reinstall ccache so brew re-validates its full runtime-dep
           # closure on every run. This is the durable fix: when the upstream
           # ccache formula gains a new transitive dep (as it has multiple times
@@ -138,11 +132,11 @@ jobs:
           # and decides "already installed" without re-linking, so on a cache-
           # hit run the formulas aren't on PATH. Force-link them; --overwrite
           # tolerates pre-existing symlinks from earlier installs.
-          brew link --overwrite protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd nlohmann-json opus pkg-config 2>/dev/null || true
+          brew link --overwrite protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd nlohmann-json 2>/dev/null || true
 
       - name: Save Homebrew cache
         if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v6
+        uses: actions/cache/save@v4
         with:
           path: |
             ~/Library/Caches/Homebrew/downloads
@@ -159,7 +153,6 @@ jobs:
             /opt/homebrew/Cellar/xxhash
             /opt/homebrew/Cellar/zstd
             /opt/homebrew/Cellar/nlohmann-json
-            /opt/homebrew/Cellar/opus
           key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
 
       # ---- ccache for llama.cpp CMake builds ----
@@ -178,7 +171,7 @@ jobs:
       - name: Restore ccache
         if: inputs.backend == 'llama-cpp'
         id: ccache-cache
-        uses: actions/cache/restore@v6
+        uses: actions/cache/restore@v4
         with:
           path: ~/Library/Caches/ccache
           key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
@@ -211,7 +204,7 @@ jobs:
       - name: Restore Python wheel cache
         if: inputs.lang == 'python'
         id: pyenv-cache
-        uses: actions/cache/restore@v6
+        uses: actions/cache/restore@v4
         with:
           path: |
             ~/Library/Caches/pip
@@ -235,17 +228,8 @@ jobs:
         run: |
           make backends/ds4-darwin
 
-      # privacy-filter is a C++/ggml backend like ds4 - a single grpc-server with
-      # otool dylib bundling - so it gets its own bespoke darwin script rather than
-      # the generic build-darwin-go-backend path.
-      - name: Build privacy-filter backend (Darwin Metal)
-        if: inputs.backend == 'privacy-filter'
-        run: |
-          make protogen-go
-          make backends/privacy-filter-darwin
-
       - name: Build ${{ inputs.backend }}-darwin
-        if: inputs.backend != 'llama-cpp' && inputs.backend != 'ds4' && inputs.backend != 'privacy-filter'
+        if: inputs.backend != 'llama-cpp' && inputs.backend != 'ds4'
         run: |
           make protogen-go
           BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
@@ -256,14 +240,14 @@ jobs:
 
       - name: Save ccache
         if: inputs.backend == 'llama-cpp' && github.event_name != 'pull_request'
-        uses: actions/cache/save@v6
+        uses: actions/cache/save@v4
         with:
           path: ~/Library/Caches/ccache
           key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
 
       - name: Save Python wheel cache
         if: inputs.lang == 'python' && github.event_name != 'pull_request' && steps.pyenv-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v6
+        uses: actions/cache/save@v4
         with:
           path: |
             ~/Library/Caches/pip

.github/workflows/backend_merge.yml

@@ -49,7 +49,7 @@ jobs:
       # Sparse checkout: the merge job needs `.github/scripts/` (for the
       # keepalive cleanup script) but none of the source tree.
       - name: Checkout (.github/scripts only)
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           sparse-checkout: |
             .github/scripts

.github/workflows/backend_pr.yml

@@ -11,33 +11,19 @@ jobs:
   generate-matrix:
     runs-on: ubuntu-latest
     outputs:
+      matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
       matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
       matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
       merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
+      merge-matrix-singlearch: ${{ steps.set-matrix.outputs['merge-matrix-singlearch'] }}
+      has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
       has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
       has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
       has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
-      # Single-arch backends are sharded across SINGLEARCH_SHARDS matrix jobs to
-      # stay under GitHub's 256-jobs-per-matrix limit (see changed-backends.js).
-      matrix-singlearch-1: ${{ steps.set-matrix.outputs['matrix-singlearch-1'] }}
-      merge-matrix-singlearch-1: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-1'] }}
-      has-backends-singlearch-1: ${{ steps.set-matrix.outputs['has-backends-singlearch-1'] }}
-      has-merges-singlearch-1: ${{ steps.set-matrix.outputs['has-merges-singlearch-1'] }}
-      matrix-singlearch-2: ${{ steps.set-matrix.outputs['matrix-singlearch-2'] }}
-      merge-matrix-singlearch-2: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-2'] }}
-      has-backends-singlearch-2: ${{ steps.set-matrix.outputs['has-backends-singlearch-2'] }}
-      has-merges-singlearch-2: ${{ steps.set-matrix.outputs['has-merges-singlearch-2'] }}
-      matrix-singlearch-3: ${{ steps.set-matrix.outputs['matrix-singlearch-3'] }}
-      merge-matrix-singlearch-3: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-3'] }}
-      has-backends-singlearch-3: ${{ steps.set-matrix.outputs['has-backends-singlearch-3'] }}
-      has-merges-singlearch-3: ${{ steps.set-matrix.outputs['has-merges-singlearch-3'] }}
-      matrix-singlearch-4: ${{ steps.set-matrix.outputs['matrix-singlearch-4'] }}
-      merge-matrix-singlearch-4: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-4'] }}
-      has-backends-singlearch-4: ${{ steps.set-matrix.outputs['has-backends-singlearch-4'] }}
-      has-merges-singlearch-4: ${{ steps.set-matrix.outputs['has-merges-singlearch-4'] }}
+      has-merges-singlearch: ${{ steps.set-matrix.outputs['has-merges-singlearch'] }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
 
       - name: Setup Bun
         uses: oven-sh/setup-bun@v2
@@ -85,10 +71,10 @@ jobs:
       fail-fast: true
       max-parallel: 8
       matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
-  backend-jobs-singlearch-1:
+  backend-jobs-singlearch:
     needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-1'] == 'true'
     uses: ./.github/workflows/backend_build.yml
+    if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
     with:
       tag-latest: ${{ matrix.tag-latest }}
       tag-suffix: ${{ matrix.tag-suffix }}
@@ -112,94 +98,7 @@ jobs:
     strategy:
       fail-fast: true
       max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-1']) }}
-
-  backend-jobs-singlearch-2:
-    needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-2'] == 'true'
-    uses: ./.github/workflows/backend_build.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: true
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-2']) }}
-
-  backend-jobs-singlearch-3:
-    needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-3'] == 'true'
-    uses: ./.github/workflows/backend_build.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: true
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-3']) }}
-
-  backend-jobs-singlearch-4:
-    needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch-4'] == 'true'
-    uses: ./.github/workflows/backend_build.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-      build-type: ${{ matrix.build-type }}
-      cuda-major-version: ${{ matrix.cuda-major-version }}
-      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-      platforms: ${{ matrix.platforms }}
-      platform-tag: ${{ matrix.platform-tag || '' }}
-      runs-on: ${{ matrix.runs-on }}
-      builder-base-image: ${{ matrix.builder-base-image || '' }}
-      base-image: ${{ matrix.base-image }}
-      backend: ${{ matrix.backend }}
-      dockerfile: ${{ matrix.dockerfile }}
-      skip-drivers: ${{ matrix.skip-drivers }}
-      context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
-      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: true
-      max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-4']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
   backend-merge-jobs-multiarch:
     needs: [generate-matrix, backend-jobs-multiarch]
     # backend_merge.yml's push-side steps are all gated on
@@ -219,9 +118,9 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
 
-  backend-merge-jobs-singlearch-1:
-    needs: [generate-matrix, backend-jobs-singlearch-1]
-    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-1'] == 'true' }}
+  backend-merge-jobs-singlearch:
+    needs: [generate-matrix, backend-jobs-singlearch]
+    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch'] == 'true' }}
     uses: ./.github/workflows/backend_merge.yml
     with:
       tag-latest: ${{ matrix.tag-latest }}
@@ -231,49 +130,7 @@ jobs:
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
     strategy:
       fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-1']) }}
-
-  backend-merge-jobs-singlearch-2:
-    needs: [generate-matrix, backend-jobs-singlearch-2]
-    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-2'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-2']) }}
-
-  backend-merge-jobs-singlearch-3:
-    needs: [generate-matrix, backend-jobs-singlearch-3]
-    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-3'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-3']) }}
-
-  backend-merge-jobs-singlearch-4:
-    needs: [generate-matrix, backend-jobs-singlearch-4]
-    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-4'] == 'true' }}
-    uses: ./.github/workflows/backend_merge.yml
-    with:
-      tag-latest: ${{ matrix.tag-latest }}
-      tag-suffix: ${{ matrix.tag-suffix }}
-    secrets:
-      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-    strategy:
-      fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-4']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch']) }}
   backend-jobs-darwin:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build_darwin.yml

.github/workflows/base-images.yml

@@ -127,7 +127,7 @@ jobs:
             # the original l4t matrix entry which set skip-drivers: 'true'.
             skip-drivers: 'true'
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
         with:
           submodules: false
       - name: Free disk space

.github/workflows/build-test.yaml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -25,7 +25,7 @@ jobs:
     runs-on: macos-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -47,7 +47,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
       - name: Configure apt mirror on runner

.github/workflows/bump-inference-defaults.yml

@@ -14,7 +14,7 @@ jobs:
   bump:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
 
       - uses: actions/setup-go@v5
         with:

.github/workflows/bump_deps.yaml

@@ -22,18 +22,10 @@ jobs:
             variable: "TURBOQUANT_VERSION"
             branch: "feature/turboquant-kv-cache"
             file: "backend/cpp/turboquant/Makefile"
-          - repository: "PrismML-Eng/llama.cpp"
-            variable: "BONSAI_VERSION"
-            branch: "prism"
-            file: "backend/cpp/bonsai/Makefile"
           - repository: "antirez/ds4"
             variable: "DS4_VERSION"
             branch: "main"
             file: "backend/cpp/ds4/Makefile"
-          - repository: "meituan-longcat/LongCat-Video"
-            variable: "LONGCAT_VIDEO_VERSION"
-            branch: "main"
-            file: "backend/python/longcat-video/Makefile"
           - repository: "localai-org/privacy-filter.cpp"
             variable: "PRIVACY_FILTER_VERSION"
             branch: "master"
@@ -50,15 +42,11 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
-          - repository: "localai-org/moss-transcribe.cpp"
-            variable: "MOSS_VERSION"
-            branch: "master"
-            file: "backend/go/moss-transcribe-cpp/Makefile"
-          - repository: "localai-org/ced.cpp"
+          - repository: "mudler/ced.cpp"
             variable: "CED_VERSION"
-            branch: "main"
+            branch: "master"
             file: "backend/go/ced/Makefile"
-          - repository: "localai-org/voice-detect.cpp"
+          - repository: "mudler/voice-detect.cpp"
             variable: "VOICEDETECT_VERSION"
             branch: "master"
             file: "backend/go/voice-detect/Makefile"
@@ -90,7 +78,7 @@ jobs:
             variable: "SAM3_VERSION"
             branch: "main"
             file: "backend/go/sam3-cpp/Makefile"
-          - repository: "localai-org/rf-detr.cpp"
+          - repository: "mudler/rf-detr.cpp"
             variable: "RFDETR_VERSION"
             branch: "main"
             file: "backend/go/rfdetr-cpp/Makefile"
@@ -112,7 +100,7 @@ jobs:
             file: "backend/go/vibevoice-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
       - name: Bump dependencies 🔧
         id: bump
         run: |
@@ -148,7 +136,7 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
       - name: Bump vLLM cu130 wheel pin 🔧
         id: bump
         run: |
@@ -174,39 +162,3 @@ jobs:
           branch: "update/VLLM_VERSION"
           body: ${{ steps.bump.outputs.message }}
           signoff: true
-
-  bump-vllm-metal:
-    # The darwin (Apple Silicon) vLLM build installs vllm-metal, which is locked
-    # to a specific vLLM source release. install.sh pins both VLLM_METAL_VERSION
-    # (the wheel release) and VLLM_VERSION (the vLLM it builds against); this job
-    # tracks vllm-project/vllm-metal and rewrites both atomically. Separate from
-    # bump-vllm-wheel because darwin follows vllm-metal, not vllm/vllm latest.
-    if: github.repository == 'mudler/LocalAI'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v7
-      - name: Bump vllm-metal pin 🔧
-        id: bump
-        run: |
-          bash .github/bump_vllm_metal.sh vllm-project/vllm-metal backend/python/vllm/install.sh VLLM_METAL_VERSION
-          {
-            echo 'message<<EOF'
-            cat "VLLM_METAL_VERSION_message.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          {
-            echo 'commit<<EOF'
-            cat "VLLM_METAL_VERSION_commit.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          rm -rfv VLLM_METAL_VERSION_message.txt VLLM_METAL_VERSION_commit.txt
-      - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
-        with:
-          token: ${{ secrets.UPDATE_BOT_TOKEN }}
-          push-to-fork: ci-forks/LocalAI
-          commit-message: ':arrow_up: Update vllm-project/vllm-metal (darwin)'
-          title: 'chore: :arrow_up: Update vllm-metal (darwin) to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_METAL_VERSION"
-          body: ${{ steps.bump.outputs.message }}
-          signoff: true

.github/workflows/bump_docs.yaml

@@ -13,7 +13,7 @@ jobs:
           - repository: "mudler/LocalAI"
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
       - name: Bump dependencies 🔧
         run: |
           bash .github/bump_docs.sh ${{ matrix.repository }}

.github/workflows/checksum_checker.yaml

@@ -8,7 +8,7 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
       - name: Configure apt mirror on runner
         uses: ./.github/actions/configure-apt-mirror
       - name: Install dependencies

.github/workflows/deploy-explorer.yaml

@@ -16,7 +16,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - uses: actions/setup-go@v5

.github/workflows/gallery-agent.yaml

@@ -31,7 +31,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
 

.github/workflows/gallery_variant_proposals.yaml

@@ -1,54 +0,0 @@
-name: Propose gallery variant groupings
-on:
-  schedule:
-    - cron: 0 4 * * 1
-  workflow_dispatch:
-jobs:
-  variant_proposals:
-    if: github.repository == 'mudler/LocalAI'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v7
-      - uses: actions/setup-go@v5
-        with:
-          go-version-file: go.mod
-          cache: false
-
-      # The heuristics are the risky part of this job. A regression in them
-      # produces confident, wrong proposals, which is worse than no job at all.
-      - name: Test the proposer
-        run: go test ./.github/ci/variantproposals/
-
-      - name: Propose groupings 🔧
-        id: propose
-        run: |
-          rm -f /tmp/variant-proposals-body.md
-          go run ./.github/ci/variantproposals \
-            -index gallery/index.yaml \
-            -ledger gallery/variant-exclusions.yaml \
-            -body-out /tmp/variant-proposals-body.md \
-            -apply
-          if [ -s /tmp/variant-proposals-body.md ]; then
-            echo "have_proposals=true" >> "$GITHUB_OUTPUT"
-            {
-              echo 'body<<VARIANT_PROPOSAL_BODY_EOF'
-              cat /tmp/variant-proposals-body.md
-              echo VARIANT_PROPOSAL_BODY_EOF
-            } >> "$GITHUB_OUTPUT"
-          else
-            echo "have_proposals=false" >> "$GITHUB_OUTPUT"
-          fi
-
-      # No body file means the proposer found nothing. Opening an empty pull
-      # request every run is how a proposal job gets muted by its reviewers.
-      - name: Create Pull Request
-        if: steps.propose.outputs.have_proposals == 'true'
-        uses: peter-evans/create-pull-request@v8
-        with:
-          token: ${{ secrets.UPDATE_BOT_TOKEN }}
-          push-to-fork: ci-forks/LocalAI
-          commit-message: 'chore(model-gallery): propose variant groupings'
-          title: 'chore(model-gallery): propose variant groupings for review'
-          branch: "propose/variant-groupings"
-          body: ${{ steps.propose.outputs.body }}
-          signoff: true

.github/workflows/generate_intel_image.yaml

@@ -44,7 +44,7 @@ jobs:
         uses: docker/setup-buildx-action@master
 
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
 
       - name: Cache Intel images
         uses: docker/build-push-action@v7

.github/workflows/gh-pages.yml

@@ -28,7 +28,7 @@ jobs:
       HUGO_VERSION: "0.146.3"
     steps:
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0  # needed for enableGitInfo
           submodules: true

.github/workflows/image_build.yml

@@ -80,7 +80,7 @@ jobs:
     steps:
 
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
 
       - name: Configure apt mirror on runner
         id: apt_mirror

.github/workflows/image_merge.yml

@@ -36,7 +36,7 @@ jobs:
       # Sparse checkout: needed for .github/scripts/ (the keepalive cleanup
       # script). Skips the rest of the source tree.
       - name: Checkout (.github/scripts only)
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           sparse-checkout: |
             .github/scripts

.github/workflows/lint.yml

@@ -20,7 +20,7 @@ jobs:
   golangci-lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
         with:
           # Full history so golangci-lint's new-from-merge-base can reach
           # origin/master and compute the diff against it.
@@ -46,23 +46,3 @@ jobs:
           touch core/http/react-ui/dist/index.html
       - name: lint
         run: make lint
-
-  build-scripts:
-    # The image packaging scripts encode invariants that only surface inside a
-    # container build (a missing transitive dep, a partial cuDNN family). Their
-    # shell tests need nothing but bash + gcc + ldd, so run them on every PR
-    # rather than waiting on a multi-GB cross-arch backend image build.
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v7
-      - name: run packaging script tests
-        run: make test-build-scripts
-
-      # The backend matrix path filter fails silently: a miss emits an empty
-      # matrix, every job goes green, and the change reaches no image (#10946).
-      # Its tests need only node, so they ride along with this job.
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '20'
-      - name: run CI script tests
-        run: make test-ci-scripts

.github/workflows/realtime-conformance.yml

@@ -1,69 +0,0 @@
----
-name: 'realtime-conformance'
-
-# Verifies the realtime state-machine implementations conform to their formal
-# designs (docs/design/realtime-state-machines.md, formal-verification/). BOTH
-# layers are enforced and the gate is fail-closed: the Go conformance layer
-# (respcoord + turncoord transition/rapid tests under -race) AND the FizzBee model check of
-# the authoritative specs. FizzBee is pinned + checksum-verified
-# (formal-verification/fizzbee.sha256), so a failed install fails the job rather
-# than silently skipping verification.
-
-on:
-  pull_request:
-    paths:
-      - 'core/http/endpoints/openai/coordinator/**'
-      - 'core/http/endpoints/openai/respcoord/**'
-      - 'core/http/endpoints/openai/turncoord/**'
-      - 'core/http/endpoints/openai/conncoord/**'
-      - 'core/http/endpoints/openai/compactcoord/**'
-      - 'core/http/endpoints/openai/ttscoord/**'
-      - 'formal-verification/**'
-      - 'scripts/realtime-conformance.sh'
-      - 'scripts/install-fizzbee.sh'
-      - '.github/workflows/realtime-conformance.yml'
-  push:
-    branches:
-      - master
-    paths:
-      - 'core/http/endpoints/openai/coordinator/**'
-      - 'core/http/endpoints/openai/respcoord/**'
-      - 'core/http/endpoints/openai/turncoord/**'
-      - 'core/http/endpoints/openai/conncoord/**'
-      - 'core/http/endpoints/openai/compactcoord/**'
-      - 'core/http/endpoints/openai/ttscoord/**'
-      - 'formal-verification/**'
-      - 'scripts/realtime-conformance.sh'
-
-concurrency:
-  group: realtime-conformance-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  conformance:
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        go-version: ['1.26.x']
-    steps:
-      - name: Clone
-        uses: actions/checkout@v7
-      - name: Setup Go ${{ matrix.go-version }}
-        uses: actions/setup-go@v5
-        with:
-          go-version: ${{ matrix.go-version }}
-          cache: false
-      - name: Cache FizzBee
-        uses: actions/cache@v6
-        with:
-          path: .tools/fizzbee
-          key: fizzbee-v0.5.2-${{ runner.os }}-${{ hashFiles('formal-verification/fizzbee.sha256') }}
-      - name: Install FizzBee (pinned, checksum-verified)
-        # No `|| true`: a failed/forged download must fail the job, not silently
-        # drop the design verification. install-fizzbee.sh is a no-op if the
-        # cached binary is already present and valid.
-        run: ./scripts/install-fizzbee.sh
-      - name: Run conformance gate (fail-closed)
-        # No skip env: both the Go conformance and the FizzBee model check are
-        # required. The gate auto-detects .tools/fizzbee/fizz.
-        run: make test-realtime-conformance

.github/workflows/release.yaml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -24,35 +24,20 @@ jobs:
           args: release --clean
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          MACOS_SIGN_P12: ${{ secrets.MACOS_CERTIFICATE }}
-          MACOS_SIGN_PASSWORD: ${{ secrets.MACOS_CERTIFICATE_PWD }}
-          MACOS_NOTARY_KEY: ${{ secrets.MACOS_NOTARY_KEY }}
-          MACOS_NOTARY_KEY_ID: ${{ secrets.MACOS_NOTARY_KEY_ID }}
-          MACOS_NOTARY_ISSUER_ID: ${{ secrets.MACOS_NOTARY_ISSUER_ID }}
   launcher-build-darwin:
     runs-on: macos-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
       - name: Set up Go
         uses: actions/setup-go@v5
         with:
           go-version: 1.23
-      - name: Import signing certificate
-        env:
-          MACOS_CERTIFICATE: ${{ secrets.MACOS_CERTIFICATE }}
-          MACOS_CERTIFICATE_PWD: ${{ secrets.MACOS_CERTIFICATE_PWD }}
-          MACOS_CI_KEYCHAIN_PWD: ${{ secrets.MACOS_CI_KEYCHAIN_PWD }}
-        run: bash contrib/macos/sign-and-notarize.sh import-cert
-      - name: Build, sign and notarize the DMG
-        env:
-          MACOS_SIGN_IDENTITY: ${{ secrets.MACOS_SIGN_IDENTITY }}
-          MACOS_NOTARY_KEY: ${{ secrets.MACOS_NOTARY_KEY }}
-          MACOS_NOTARY_KEY_ID: ${{ secrets.MACOS_NOTARY_KEY_ID }}
-          MACOS_NOTARY_ISSUER_ID: ${{ secrets.MACOS_NOTARY_ISSUER_ID }}
-        run: make release-launcher-darwin
+      - name: Build launcher for macOS ARM64
+        run: |
+          make build-launcher-darwin
       - name: Upload DMG to Release
         uses: softprops/action-gh-release@v3
         with:
@@ -61,7 +46,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           fetch-depth: 0
       - name: Configure apt mirror on runner

.github/workflows/secscan.yaml

@@ -14,7 +14,7 @@ jobs:
       GO111MODULE: on
     steps:
       - name: Checkout Source
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         if: ${{ github.actor != 'dependabot[bot]' }}
       - name: Run Gosec Security Scanner
         if: ${{ github.actor != 'dependabot[bot]' }}

.github/workflows/stalebot.yml

@@ -11,7 +11,7 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@1e223db275d687790206a7acac4d1a11bd6fe629 # v9
+      - uses: actions/stale@eb5cf3af3ac0a1aa4c9c45633dd1ae542a27a899 # v9
         with:
           stale-issue-message: 'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
           stale-pr-message: 'This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.'

.github/workflows/test-extra.yml

@@ -50,7 +50,7 @@ jobs:
       parakeet-cpp: ${{ steps.detect.outputs.parakeet-cpp }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
       - name: Setup Bun
         uses: oven-sh/setup-bun@v2
       - name: Install dependencies
@@ -67,7 +67,7 @@ jobs:
   #   runs-on: ubuntu-latest
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v7
+  #       uses: actions/checkout@v6
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -90,7 +90,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -113,7 +113,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -137,7 +137,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -158,7 +158,7 @@ jobs:
   #  runs-on: ubuntu-latest
   #  steps:
   #    - name: Clone
-  #      uses: actions/checkout@v7
+  #      uses: actions/checkout@v6
   #      with:
   #        submodules: true
   #    - name: Dependencies
@@ -178,7 +178,7 @@ jobs:
   #   runs-on: ubuntu-latest
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v7
+  #       uses: actions/checkout@v6
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -240,7 +240,7 @@ jobs:
   #           sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
   #           df -h
   #     - name: Clone
-  #       uses: actions/checkout@v7
+  #       uses: actions/checkout@v6
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -265,7 +265,7 @@ jobs:
   #   runs-on: ubuntu-latest
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v7
+  #       uses: actions/checkout@v6
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -288,7 +288,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -309,7 +309,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -330,7 +330,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -351,7 +351,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -373,7 +373,7 @@ jobs:
   #   timeout-minutes: 45
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v7
+  #       uses: actions/checkout@v6
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -394,7 +394,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -415,7 +415,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -436,7 +436,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -462,7 +462,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -484,7 +484,7 @@ jobs:
     timeout-minutes: 30
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -513,7 +513,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -530,7 +530,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -552,7 +552,7 @@ jobs:
     timeout-minutes: 20
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -579,7 +579,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -587,7 +587,7 @@ jobs:
         with:
           go-version: '1.25.4'
       - name: Setup Node.js
-        uses: actions/setup-node@v7
+        uses: actions/setup-node@v6
         with:
           node-version: '22'
       - name: Build sherpa-onnx backend image and run realtime e2e tests
@@ -604,7 +604,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -625,7 +625,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -645,7 +645,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -664,7 +664,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -681,7 +681,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -698,7 +698,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -741,7 +741,7 @@ jobs:
   #   timeout-minutes: 90
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v7
+  #       uses: actions/checkout@v6
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -783,7 +783,7 @@ jobs:
   #   timeout-minutes: 90
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v7
+  #       uses: actions/checkout@v6
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -808,7 +808,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -840,7 +840,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -876,7 +876,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -915,7 +915,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -952,7 +952,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -987,7 +987,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -1008,16 +1008,12 @@ jobs:
   # image + working dir.
   tests-vibevoice-cpp-grpc-transcription:
     needs: detect-changes
-    # Skip on release tag pushes: the ASR Q4_K model is ~10 GB and cannot be
-    # pulled from HF within the inner `go test -timeout 30m` budget on a CI
-    # runner, so every tag build hung and timed out. Still runs on PRs/branch
-    # pushes that touch vibevoice-cpp so regressions are caught off the release path.
-    if: (needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true') && !startsWith(github.ref, 'refs/tags/')
+    if: needs.detect-changes.outputs.vibevoice-cpp == 'true' || needs.detect-changes.outputs.run-all == 'true'
     runs-on: bigger-runner
     timeout-minutes: 150
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -1046,7 +1042,7 @@ jobs:
     timeout-minutes: 60
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go
@@ -1062,7 +1058,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -1095,7 +1091,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -1118,7 +1114,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies
@@ -1144,7 +1140,7 @@ jobs:
     timeout-minutes: 90
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies

.github/workflows/test.yml

@@ -21,7 +21,7 @@ jobs:
         go-version: ['1.26.x']
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Free disk space
@@ -48,7 +48,7 @@ jobs:
           sudo apt-get update
           sudo apt-get install curl ffmpeg libopus-dev
       - name: Setup Node.js
-        uses: actions/setup-node@v7
+        uses: actions/setup-node@v6
         with:
           node-version: '22'
       - name: Build React UI
@@ -84,7 +84,7 @@ jobs:
         go-version: ['1.26.x']
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Setup Go ${{ matrix.go-version }}
@@ -100,7 +100,7 @@ jobs:
           brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus ffmpeg
           pip install --user --no-cache-dir grpcio-tools grpcio
       - name: Setup Node.js
-        uses: actions/setup-node@v7
+        uses: actions/setup-node@v6
         with:
           node-version: '22'
       - name: Build React UI
@@ -121,19 +121,3 @@ jobs:
           detached: true
           connect-timeout-seconds: 180
           limit-access-to-actor: true
-
-  # Fast standalone unit tests for the backends' pure C++ helpers - currently the
-  # llama-cpp message reconstruction (backend/cpp/llama-cpp/message_content.h),
-  # which guards the OpenAI chat content normalization (mudler/LocalAI#10524,
-  # #7324, #7528). The runner discovers every *_test.cpp under backend/cpp/, so
-  # new pure-C++ unit tests are picked up with no CI changes. These need only the
-  # C++ stdlib + nlohmann/json, so they run on every PR without the full
-  # llama.cpp + gRPC backend build. (The same suite is also wired as an opt-in
-  # CMake/ctest target, -DLLAMA_GRPC_BUILD_TESTS=ON, for in-backend-build runs.)
-  tests-backend-cpp:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v7
-      - name: Run backend C++ unit tests
-        run: make test-backend-cpp

.github/workflows/tests-aio.yml

@@ -62,7 +62,7 @@ jobs:
           sudo rm -rfv build || true
           df -h
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Dependencies

.github/workflows/tests-e2e.yml

@@ -21,7 +21,7 @@ jobs:
         go-version: ['1.25.x']
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Configure apt mirror on runner
@@ -47,7 +47,7 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y build-essential libopus-dev
       - name: Setup Node.js
-        uses: actions/setup-node@v7
+        uses: actions/setup-node@v6
         with:
           node-version: '22'
       - name: Build React UI

.github/workflows/tests-pii-ner-e2e.yml

@@ -1,97 +0,0 @@
----
-name: 'PII NER tier E2E (live GGUF, CPU)'
-
-# Runs the real privacy-filter GGUF NER tier end-to-end on CPU — the gap the
-# hermetic tests/e2e suite cannot cover (it only exercises the in-process
-# pattern tier). Heavy (builds the C++ backend image + downloads a ~2.7 GB
-# GGUF), so it is path-filtered on PRs and otherwise runs nightly / on demand.
-#
-# This drives the container-level harness (tests/e2e-backends) via
-# `make test-extra-backend-privacy-filter`: it builds the privacy-filter image,
-# downloads the model, loads it on CPU, and asserts byte-correct, UTF-8-aligned
-# TokenClassify spans. The complementary HTTP-path specs in tests/e2e
-# (e2e_pii_ner_test.go) Skip unless PII_NER_MODEL_GGUF is wired.
-
-on:
-  workflow_dispatch:
-  schedule:
-    - cron: '0 3 * * *'
-  push:
-    branches:
-      - master
-    paths:
-      - 'backend/cpp/privacy-filter/**'
-      - 'backend/Dockerfile.privacy-filter'
-      - 'core/services/routing/pii/**'
-      - 'core/services/routing/piidetector/**'
-      - 'core/backend/token_classify.go'
-      - 'core/http/endpoints/localai/pii.go'
-      - 'core/schema/pii.go'
-      - 'tests/e2e-backends/**'
-      - 'tests/e2e/e2e_pii_ner_test.go'
-      - 'tests/e2e/e2e_suite_test.go'
-      - '.github/workflows/tests-pii-ner-e2e.yml'
-  pull_request:
-    paths:
-      - 'backend/cpp/privacy-filter/**'
-      - 'backend/Dockerfile.privacy-filter'
-      - 'core/services/routing/pii/**'
-      - 'core/services/routing/piidetector/**'
-      - 'core/backend/token_classify.go'
-      - 'core/http/endpoints/localai/pii.go'
-      - 'core/schema/pii.go'
-      - 'tests/e2e-backends/**'
-      - 'tests/e2e/e2e_pii_ner_test.go'
-      - 'tests/e2e/e2e_suite_test.go'
-      - '.github/workflows/tests-pii-ner-e2e.yml'
-
-concurrency:
-  group: ci-tests-pii-ner-e2e-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
-  tests-pii-ner-e2e:
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        go-version: ['1.25.x']
-    steps:
-      - name: Clone
-        uses: actions/checkout@v7
-        with:
-          submodules: true
-      - name: Free disk space
-        run: |
-          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL || true
-          sudo docker image prune --all --force || true
-          df -h
-      - name: Configure apt mirror on runner
-        uses: ./.github/actions/configure-apt-mirror
-      - name: Setup Go ${{ matrix.go-version }}
-        uses: actions/setup-go@v5
-        with:
-          go-version: ${{ matrix.go-version }}
-          cache: false
-      - name: Proto Dependencies
-        run: |
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential
-      # Builds local-ai-backend:privacy-filter, downloads the GGUF, loads it on
-      # CPU and runs the token_classify capability spec (byte-offset contract).
-      - name: Run live PII NER backend E2E
-        run: PATH="$PATH:$HOME/go/bin" make test-extra-backend-privacy-filter
-      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
-        uses: mxschmitt/action-tmate@v3.23
-        with:
-          detached: true
-          connect-timeout-seconds: 180
-          limit-access-to-actor: true

.github/workflows/tests-ui-e2e.yml

@@ -23,7 +23,7 @@ jobs:
         go-version: ['1.26.x']
     steps:
       - name: Clone
-        uses: actions/checkout@v7
+        uses: actions/checkout@v6
         with:
           submodules: true
       - name: Configure apt mirror on runner
@@ -34,7 +34,7 @@ jobs:
           go-version: ${{ matrix.go-version }}
           cache: false
       - name: Setup Node.js
-        uses: actions/setup-node@v7
+        uses: actions/setup-node@v6
         with:
           node-version: '22'
       - name: Setup Bun

.github/workflows/update_swagger.yaml

@@ -10,7 +10,7 @@ jobs:
       fail-fast: false
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v7
+      - uses: actions/checkout@v6
       - name: Configure apt mirror on runner
         uses: ./.github/actions/configure-apt-mirror
       - uses: actions/setup-go@v5

.gitignore

@@ -41,12 +41,7 @@ models/*
 test-models/
 test-dir/
 tests/e2e-aio/backends
-# The mock backend binary built by `make build-mock-backend`. Anchored to its
-# full path: a bare `mock-backend` also matched the *directory* holding the
-# source, so git would not descend into it and adding a file there needed -f.
-# tests/e2e/mock-backend/.gitignore covers the same binary; kept here too so
-# the artifact stays ignored if that scoped file is ever removed.
-/tests/e2e/mock-backend/mock-backend
+mock-backend
 
 release/
 
@@ -96,18 +91,3 @@ core/http/react-ui/test-results/
 
 # Local worktrees
 .worktrees/
-
-# SDD / brainstorm scratch (agent-driven development)
-.superpowers/
-
-# Local Apple signing material (never commit)
-.certs/
-
-# Pinned dev tools (e.g. FizzBee for the realtime-conformance gate)
-.tools/
-
-# FizzBee model-check artifacts: the parser emits <spec>.json next to each
-# .fizz and the checker writes run dirs under out/. Both are regenerated by
-# the realtime-conformance gate; only the .fizz sources are authoritative.
-formal-verification/*.json
-formal-verification/out/

.goreleaser.yaml

@@ -9,8 +9,7 @@ source:
   enabled: true
   name_template: '{{ .ProjectName }}-{{ .Tag }}-source'
 builds:
-  - id: local-ai
-    main: ./cmd/local-ai
+  - main: ./cmd/local-ai
     env:
       - CGO_ENABLED=0
     ldflags:
@@ -36,19 +35,3 @@ snapshot:
   version_template: "{{ .Tag }}-next"
 changelog:
   use: github-native
-# Sign + notarize the macOS server binary via the quill backend (runs on Linux,
-# no macOS runner needed). Disabled automatically when MACOS_SIGN_P12 is unset
-# (forks / PRs), so those builds stay unsigned and green.
-notarize:
-  macos:
-    - enabled: '{{ isEnvSet "MACOS_SIGN_P12" }}'
-      ids:
-        - local-ai
-      sign:
-        certificate: "{{.Env.MACOS_SIGN_P12}}"
-        password: "{{.Env.MACOS_SIGN_PASSWORD}}"
-      notarize:
-        issuer_id: "{{.Env.MACOS_NOTARY_ISSUER_ID}}"
-        key_id: "{{.Env.MACOS_NOTARY_KEY_ID}}"
-        key: "{{.Env.MACOS_NOTARY_KEY}}"
-        wait: true

.impeccable/live/config.json

@@ -1,6 +0,0 @@
-{
-  "files": ["core/http/react-ui/index.html"],
-  "insertBefore": "</body>",
-  "commentSyntax": "html",
-  "cspChecked": true
-}

AGENTS.md

@@ -39,10 +39,8 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 - **Logging**: Use `github.com/mudler/xlog` (same API as slog)
 - **Go style**: Prefer `any` over `interface{}`
 - **Comments**: Explain *why*, not *what*
-- **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
+- **Docs**: Update `docs/content/` when adding features or changing config
 - **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
 - **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
 - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
-- **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml` `metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
-- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).
 - **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI

Dockerfile

@@ -12,16 +12,12 @@ ARG APT_MIRROR
 ARG APT_PORTS_MIRROR
 ENV DEBIAN_FRONTEND=noninteractive
 
-# hwdata ships /usr/share/hwdata/pci.ids. Without it, the ghw library we use
-# for hardware detection cannot resolve PCI vendor IDs and fails to enumerate
-# GPUs at all, so the image reports "No GPU detected" (see issue #10941).
 RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
     APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
     apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates curl wget espeak-ng libgomp1 \
-        ffmpeg libopenblas0 libopenblas-dev libopus0 sox \
-        hwdata && \
+        ffmpeg libopenblas0 libopenblas-dev libopus0 sox && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -175,17 +171,6 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ]; then \
     ln -s /opt/rocm-**/lib/llvm/lib/libomp.so /usr/lib/libomp.so \
     ; fi
 
-# ROCm's bundled libdrm_amdgpu is built with a hardcoded fallback lookup path
-# for the ASIC ID table (/opt/amdgpu/share/libdrm/amdgpu.ids), which only exists
-# if AMD's full amdgpu graphics/DKMS stack is installed. This compute-only image
-# doesn't have it, so hipblas/rocBLAS log "No such file or directory" on every
-# model load and can fail to identify the GPU. Point it at the equivalent file
-# Ubuntu's libdrm-common package already ships.
-RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ -f /usr/share/libdrm/amdgpu.ids ] && [ ! -e /opt/amdgpu/share/libdrm/amdgpu.ids ]; then \
-    mkdir -p /opt/amdgpu/share/libdrm && \
-    ln -s /usr/share/libdrm/amdgpu.ids /opt/amdgpu/share/libdrm/amdgpu.ids \
-    ; fi
-
 RUN expr "${BUILD_TYPE}" = intel && echo "intel" > /run/localai/capability || echo "not intel"
 
 # Cuda
@@ -393,12 +378,7 @@ RUN go install github.com/mikefarah/yq/v4@latest
 # If you cannot find a more suitable place for an addition, this layer is a suitable place for it.
 FROM requirements-drivers
 
-# Optional override for the HEALTHCHECK target. Left empty so healthcheck.sh
-# derives the endpoint from the mode the container is actually running — the
-# same image runs `local-ai run` (HTTP on 8080) and `local-ai worker` (HTTP on
-# the gRPC base port minus one), and a hardcoded default marked every worker
-# permanently unhealthy (#10987). Set it to pin an explicit URL.
-ENV HEALTHCHECK_ENDPOINT=""
+ENV HEALTHCHECK_ENDPOINT=http://localhost:8080/readyz
 
 ARG CUDA_MAJOR_VERSION=12
 ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
@@ -408,7 +388,6 @@ ENV NVIDIA_VISIBLE_DEVICES=all
 WORKDIR /
 
 COPY ./entrypoint.sh .
-COPY ./scripts/build/healthcheck.sh .
 
 # Copy the binary
 COPY --from=builder /build/local-ai ./
@@ -419,22 +398,9 @@ RUN --mount=from=builder,src=/build/,dst=/mnt/build \
 # Make sure the models directory exists
 RUN mkdir -p /models /backends /data
 
-# Define the health check command.
-#
-# --start-period is the knob for slow starts, not --timeout/--retries. Since
-# #10949 a frontend's startup preload materializes HuggingFace artifacts before
-# the HTTP server binds (31 GB observed on a live cluster), so a healthy replica
-# can legitimately fail probes for a long time. Failures inside the start period
-# leave the container `starting` instead of burning retries, and the period ends
-# early on the first success — so a generous value costs a fast-starting
-# container nothing. A process that actually died is handled by the restart
-# policy, not by health.
-#
-# --timeout is a per-probe deadline: 10m meant a wedged probe could hang for ten
-# minutes and stretch detection without bound. A localhost curl that has not
-# answered in 10s is itself the fault being detected.
-HEALTHCHECK --start-period=60m --interval=1m --timeout=10s --retries=3 \
-  CMD /healthcheck.sh
+# Define the health check command
+HEALTHCHECK --interval=1m --timeout=10m --retries=10 \
+  CMD curl -f ${HEALTHCHECK_ENDPOINT} || exit 1
 
 VOLUME /models /backends /configuration /data
 EXPOSE 8080

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -103,7 +103,7 @@ COVERAGE_E2E_LABELS?=!real-models
 COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go
 
 
-.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-coverage-baseline test-ui-coverage-check install-hooks build vendor lint lint-all
+.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-ui test-ui-coverage-baseline test-ui-coverage-check install-hooks build vendor lint lint-all
 
 all: help
 
@@ -201,27 +201,6 @@ test: prepare-test
 	OPUS_SHIM_LIBRARY=$(abspath ./pkg/opus/shim/libopusshim.so) \
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) --fail-fast -v -r $(TEST_PATHS)
 
-## Compiles and runs the standalone C++ unit tests for the backends (pure
-## helpers that depend only on the stdlib + nlohmann/json, no full backend
-## build). Discovers every *_test.cpp under backend/cpp/ - see
-## backend/cpp/run-unit-tests.sh. Set NLOHMANN_INCLUDE to skip the header fetch.
-test-backend-cpp:
-	bash backend/cpp/run-unit-tests.sh
-
-## Runs the shell-level regression tests for the image packaging scripts
-## (scripts/build/*_test.sh). These guard invariants that only ever break
-## inside a container build - a missing transitive dep, a partial cuDNN
-## family - and that no Go test can observe. Needs only bash + gcc + ldd.
-test-build-scripts:
-	@set -e; for t in scripts/build/*_test.sh; do echo "== $$t"; bash "$$t"; done
-
-## Runs the unit tests for the CI helper scripts under scripts/lib/. Currently
-## the backend matrix path filter, whose failure mode is invisible in CI: it
-## emits an empty matrix, every job goes green, and the change ships to no
-## image at all (see PR #10946). Plain `node --test`, no dependencies.
-test-ci-scripts:
-	@set -e; for t in scripts/lib/*_test.mjs; do echo "== $$t"; node --test "$$t"; done
-
 ## Runs the core suite ($(TEST_PATHS)) with statement-coverage instrumentation
 ## and writes a merged profile to $(COVERAGE_PROFILE). Deliberately omits
 ## --fail-fast so a single failure doesn't truncate the coverage number, and
@@ -419,23 +398,6 @@ test-realtime: build-mock-backend
 	@echo 'Running realtime e2e tests (mock backend)'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime && !real-models" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
 
-# Verify the realtime state-machine implementations conform to their formal
-# designs (Go transition/rapid tests under -race + FizzBee model check of the
-# authoritative specs). See docs/design/realtime-state-machines.md (Part 6) and
-# docs/design/specs/README.md.
-test-realtime-conformance:
-	GOCMD=$(GOCMD) ./scripts/realtime-conformance.sh
-
-# Verify the shared model-loader shutdown behavior independently of any API
-# modality (focused loader/gRPC/distributed/worker tests under -race + FizzBee).
-test-model-lifecycle-conformance:
-	GOCMD=$(GOCMD) ./scripts/model-lifecycle-conformance.sh
-
-# Install the pinned, checksum-verified FizzBee model checker (into .tools/,
-# gitignored) used by the conformance targets. Idempotent; no-op if present.
-install-fizzbee:
-	./scripts/install-fizzbee.sh
-
 # Container-based real-model realtime testing. Build env vars / pipeline
 # definition kept here so test-realtime-models-docker can drive a fully wired
 # pipeline (VAD + STT + LLM + TTS) from inside a containerised runner.
@@ -584,7 +546,6 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/chatterbox
 	$(MAKE) -C backend/python/vllm
 	$(MAKE) -C backend/python/vllm-omni
-	$(MAKE) -C backend/python/longcat-video
 	$(MAKE) -C backend/python/sglang
 	$(MAKE) -C backend/python/vibevoice
 	$(MAKE) -C backend/python/liquid-audio
@@ -614,7 +575,6 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/chatterbox test
 	$(MAKE) -C backend/python/vllm test
 	$(MAKE) -C backend/python/vllm-omni test
-	$(MAKE) -C backend/python/longcat-video test
 	$(MAKE) -C backend/python/vibevoice test
 	$(MAKE) -C backend/python/liquid-audio test
 	$(MAKE) -C backend/python/moonshine test
@@ -666,9 +626,6 @@ test-extra: prepare-test-extra
 ## suite against it.
 ##
 BACKEND_TEST_MODEL_URL?=https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q8_0.gguf
-## Suite timeout for `go test`. Wrappers whose model download alone can eat
-## most of the default (multi-GB models on a slow HF CDN day) override this.
-BACKEND_TEST_TIMEOUT?=30m
 
 ## Generic target — runs the suite against whatever BACKEND_IMAGE points at.
 ## Depends on protogen-go so pkg/grpc/proto is generated before `go test`.
@@ -696,7 +653,7 @@ test-extra-backend: protogen-go
 	BACKEND_TEST_FACE_IMAGE_3_URL="$$BACKEND_TEST_FACE_IMAGE_3_URL" \
 	BACKEND_TEST_FACE_IMAGE_3_FILE="$$BACKEND_TEST_FACE_IMAGE_3_FILE" \
 	BACKEND_TEST_VERIFY_DISTANCE_CEILING="$$BACKEND_TEST_VERIFY_DISTANCE_CEILING" \
-	go test -v -timeout $(BACKEND_TEST_TIMEOUT) ./tests/e2e-backends/...
+	go test -v -timeout 30m ./tests/e2e-backends/...
 
 ## Convenience wrappers: build the image, then exercise it.
 test-extra-backend-llama-cpp: docker-build-llama-cpp
@@ -719,16 +676,6 @@ test-extra-backend-turboquant: docker-build-turboquant
 	BACKEND_TEST_CACHE_TYPE_V=turbo3 \
 	$(MAKE) test-extra-backend
 
-## bonsai: exercises the llama.cpp-fork backend with a real Q1_0 (1-bit) model —
-## the PrismML Bonsai-8B GGUF, whose weight quant is *only* decodable by the fork's
-## Q1_0 kernels. Loading it is what makes this backend distinct from stock llama-cpp;
-## a standard-quant model would only test the upstream code path the llama-cpp backend
-## already covers.
-test-extra-backend-bonsai: docker-build-bonsai
-	BACKEND_IMAGE=local-ai-backend:bonsai \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/prism-ml/Bonsai-8B-gguf/resolve/main/Bonsai-8B-Q1_0.gguf \
-	$(MAKE) test-extra-backend
-
 ## Audio transcription wrapper for the llama-cpp backend.
 ## Drives the new AudioTranscription / AudioTranscriptionStream RPCs against
 ## ggml-org/Qwen3-ASR-0.6B-GGUF (a small ASR model that requires its mmproj
@@ -743,16 +690,6 @@ test-extra-backend-llama-cpp-transcription: docker-build-llama-cpp
 	BACKEND_TEST_CTX_SIZE=2048 \
 	$(MAKE) test-extra-backend
 
-## privacy-filter: the PII/NER token-classification backend. Exercises the
-## TokenClassify RPC and asserts byte-correct, UTF-8-aligned span offsets
-## against the openai-privacy-filter multilingual GGUF (CPU-runnable, ~50M
-## active params). This is the live-backend coverage for the PII NER tier.
-test-extra-backend-privacy-filter: docker-build-privacy-filter
-	BACKEND_IMAGE=local-ai-backend:privacy-filter \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/LocalAI-io/privacy-filter-multilingual-GGUF/resolve/main/privacy-filter-multilingual-f16.gguf \
-	BACKEND_TEST_CAPS=health,load,token_classify \
-	$(MAKE) test-extra-backend
-
 ## vllm is resolved from a HuggingFace model id (no file download) and
 ## exercises Predict + streaming + tool-call extraction via the hermes parser.
 ## Requires a host CPU with the SIMD instructions the prebuilt vllm CPU
@@ -1046,7 +983,6 @@ test-extra-backend-vibevoice-cpp-tts: docker-build-vibevoice-cpp
 ## post-image disk budget.
 test-extra-backend-vibevoice-cpp-transcription: docker-build-vibevoice-cpp
 	BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
-	BACKEND_TEST_TIMEOUT=120m \
 	BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-asr-q4_k.gguf#vibevoice-asr-q4_k.gguf' \
 	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf' \
 	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
@@ -1074,19 +1010,7 @@ test-extra-backend-whisper-transcription: docker-build-whisper
 ## is reachable.
 test-extra-backend-parakeet-cpp-transcription: docker-build-parakeet-cpp
 	BACKEND_IMAGE=local-ai-backend:parakeet-cpp \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/realtime_eou_120m-v1-f16.gguf \
-	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
-	BACKEND_TEST_CAPS=health,load,transcription \
-	$(MAKE) test-extra-backend
-
-## Audio transcription wrapper for the moss-transcribe-cpp (moss-transcribe.cpp
-## ggml port) backend. Mirrors test-extra-backend-parakeet-cpp-transcription:
-## drives the AudioTranscription RPC against a published MOSS GGUF using the JFK
-## 11s clip from whisper.cpp's CI samples. Not part of the default test suite -
-## run explicitly once the pinned model URL is reachable.
-test-extra-backend-moss-transcribe-cpp-transcription: docker-build-moss-transcribe-cpp
-	BACKEND_IMAGE=local-ai-backend:moss-transcribe-cpp \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/moss-transcribe.cpp-gguf/resolve/main/moss-transcribe-q5_k.gguf \
+	BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/tdt_ctc-110m-f16.gguf \
 	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
 	BACKEND_TEST_CAPS=health,load,transcription \
 	$(MAKE) test-extra-backend
@@ -1195,10 +1119,6 @@ backends/ds4-darwin: build
 	bash ./scripts/build/ds4-darwin.sh
 	./local-ai backends install "ocifile://$(abspath ./backend-images/ds4.tar)"
 
-backends/privacy-filter-darwin: build
-	bash ./scripts/build/privacy-filter-darwin.sh
-	./local-ai backends install "ocifile://$(abspath ./backend-images/privacy-filter.tar)"
-
 build-darwin-python-backend: build
 	bash ./scripts/build/python-darwin.sh
 
@@ -1240,10 +1160,6 @@ BACKEND_IK_LLAMA_CPP = ik-llama-cpp|ik-llama-cpp|.|false|false
 # turboquant is a llama.cpp fork with TurboQuant KV-cache quantization.
 # Reuses backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
 BACKEND_TURBOQUANT = turboquant|turboquant|.|false|false
-# bonsai is a llama.cpp fork (PrismML) adding the Q1_0 (1-bit) and Q2_0 (ternary)
-# weight-quant kernels the Bonsai / Ternary-Bonsai models ship in. Reuses
-# backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
-BACKEND_BONSAI = bonsai|bonsai|.|false|false
 # ds4 is antirez/ds4, a DeepSeek V4 Flash-specific inference engine.
 # Single-model; hardware-only validation lives at tests/e2e-backends/
 # (BACKEND_BINARY mode); see docs/superpowers/plans/2026-05-11-ds4-backend.md.
@@ -1263,12 +1179,10 @@ BACKEND_STABLEDIFFUSION_GGML = stablediffusion-ggml|golang|.|--progress=plain|tr
 BACKEND_WHISPER = whisper|golang|.|false|true
 BACKEND_CRISPASR = crispasr|golang|.|false|true
 BACKEND_PARAKEET_CPP = parakeet-cpp|golang|.|false|true
-BACKEND_MOSS_TRANSCRIBE_CPP = moss-transcribe-cpp|golang|.|false|true
 BACKEND_DEPTH_ANYTHING_CPP = depth-anything-cpp|golang|.|false|true
 BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
-BACKEND_MOSS_TTS_CPP = moss-tts-cpp|golang|.|false|true
 BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
 BACKEND_LOCALVQE = localvqe|golang|.|false|true
@@ -1290,7 +1204,6 @@ BACKEND_NEUTTS = neutts|python|.|false|true
 BACKEND_KOKORO = kokoro|python|.|false|true
 BACKEND_VLLM = vllm|python|.|false|true
 BACKEND_VLLM_OMNI = vllm-omni|python|.|false|true
-BACKEND_LONGCAT_VIDEO = longcat-video|python|.|--progress=plain|true
 BACKEND_SGLANG = sglang|python|.|false|true
 BACKEND_DIFFUSERS = diffusers|python|.|--progress=plain|true
 BACKEND_CHATTERBOX = chatterbox|python|.|false|true
@@ -1348,7 +1261,6 @@ endef
 $(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_IK_LLAMA_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_TURBOQUANT)))
-$(eval $(call generate-docker-build-target,$(BACKEND_BONSAI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DS4)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PRIVACY_FILTER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PIPER)))
@@ -1360,7 +1272,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_STABLEDIFFUSION_GGML)))
 $(eval $(call generate-docker-build-target,$(BACKEND_WHISPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_CRISPASR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PARAKEET_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TRANSCRIBE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DEPTH_ANYTHING_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VOXTRAL)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OPUS)))
@@ -1377,7 +1288,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VLLM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VLLM_OMNI)))
-$(eval $(call generate-docker-build-target,$(BACKEND_LONGCAT_VIDEO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_SGLANG)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DIFFUSERS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_CHATTERBOX)))
@@ -1395,7 +1305,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
@@ -1415,7 +1324,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-privacy-filter
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1530,37 +1439,13 @@ docs: docs/static/gallery.html
 ########################################################
 
 ## fyne cross-platform build
-# Build LocalAI.app from the launcher via fyne (metadata read from cmd/launcher/FyneApp.toml).
-# Signing happens via contrib/macos/sign-and-notarize.sh, which is a no-op when the signing
-# secrets are unset, so unsigned local/fork builds keep working.
-build-launcher-darwin:
-	rm -rf dist/LocalAI.app cmd/launcher/LocalAI.app
-	mkdir -p dist
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)
-	mv cmd/launcher/LocalAI.app dist/LocalAI.app
-	bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app
-
-# Notarize + staple the .app itself, then wrap it into a drag-to-Applications
-# DMG via hdiutil and sign the DMG. The app is stapled BEFORE packaging so the
-# bundle carries its own ticket and verifies offline (a dmg-only staple leaves
-# the app relying on an online Gatekeeper check, which fails offline / once the
-# app is copied out of the dmg). No-op without notary secrets.
-dmg-launcher-darwin: build-launcher-darwin
-	bash contrib/macos/sign-and-notarize.sh notarize-app dist/LocalAI.app
-	rm -rf dist/dmg dist/LocalAI.dmg
-	mkdir -p dist/dmg
-	cp -R dist/LocalAI.app dist/dmg/LocalAI.app
-	ln -s /Applications dist/dmg/Applications
-	hdiutil create -volname "LocalAI" -srcfolder dist/dmg -ov -format UDZO dist/LocalAI.dmg
-	bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.dmg
-
-# Submit the DMG to Apple notarization and staple the ticket (no-op without notary secrets).
-notarize-launcher-darwin: dmg-launcher-darwin
-	bash contrib/macos/sign-and-notarize.sh notarize dist/LocalAI.dmg
-
-# Single entrypoint for CI: build -> sign app -> notarize+staple app -> dmg -> sign dmg -> notarize+staple dmg.
-release-launcher-darwin: notarize-launcher-darwin
-	@echo "dist/LocalAI.dmg is ready"
+build-launcher-darwin: build-launcher
+	go run github.com/tiagomelo/macos-dmg-creator/cmd/createdmg@latest \
+	--appName "LocalAI" \
+	--appBinaryPath "$(LAUNCHER_BINARY_NAME)" \
+	--bundleIdentifier "com.localai.launcher" \
+	--iconPath "core/http/static/logo.png" \
+	--outputDir "dist/"
 
 build-launcher-linux:
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv launcher.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz

README.md

@@ -177,7 +177,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
 
 ## Latest News
 
-- **June 2026**: New native biometric backends from the LocalAI team: [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavier Python `insightface` and `speaker-recognition` backends ([PR #10441](https://github.com/mudler/LocalAI/pull/10441)).
+- **June 2026**: New native biometric backends from the LocalAI team: [voice-detect.cpp](https://github.com/mudler/voice-detect.cpp) for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavier Python `insightface` and `speaker-recognition` backends ([PR #10441](https://github.com/mudler/LocalAI/pull/10441)).
 - **June 2026**: New [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (a tiny Go client for the Realtime API with a full talk-back voice loop and tool calling), plus [streaming of the realtime LLM / TTS / transcription pipeline stages](https://github.com/mudler/LocalAI/pull/10176) and [configurable WebRTC ICE candidates](https://github.com/mudler/LocalAI/pull/10231).
 - **June 2026**: Big speech push: the [parakeet.cpp](https://github.com/mudler/parakeet.cpp) ASR engine gains [NeMo-faithful segment timestamps](https://github.com/mudler/LocalAI/pull/10207), a [multilingual streaming Nemotron-3.5 model](https://github.com/mudler/LocalAI/pull/10199), [dynamic batching for concurrent transcription](https://github.com/mudler/LocalAI/pull/10112) and [CUDA graphs](https://github.com/mudler/LocalAI/pull/10273); the new [CrispASR backend](https://github.com/mudler/LocalAI/pull/10099) adds multi-architecture ASR + TTS, and [60 Piper TTS voices across 42 languages](https://github.com/mudler/LocalAI/pull/10296) land in the gallery (plus [per-request TTS instructions and params](https://github.com/mudler/LocalAI/pull/10172)).
 - **June 2026**: New backends and models: [locate-anything.cpp](https://github.com/mudler/LocalAI/pull/10264) for open-vocabulary object detection via ggml, [Ideogram4 image generation](https://github.com/mudler/LocalAI/pull/10201) in stablediffusion-ggml, [llama.cpp video input](https://github.com/mudler/LocalAI/pull/10216), and the [Gemma 4 QAT family with MTP speculative-decoding pairs](https://github.com/mudler/LocalAI/pull/10215). Plus an [interactive CLI chat mode](https://github.com/mudler/LocalAI/pull/10226) and [RAG source citations in agent responses](https://github.com/mudler/LocalAI/pull/10228).
@@ -232,17 +232,12 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 | Backend | What it does |
 |---------|-------------|
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
-| [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
-| [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
-| [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
-| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
-| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
+| [ced.cpp](https://github.com/mudler/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
+| [voxtral.c](https://github.com/mudler/voxtral.c) | Voxtral Realtime 4B speech-to-text in pure C |
 | [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
-| [rf-detr.cpp](https://github.com/localai-org/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation |
+| [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation |
 | [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) |
 | [depth-anything.cpp](https://github.com/mudler/depth-anything.cpp) | Depth Anything 3 monocular metric depth + camera pose estimation |
-| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend |
-| [free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp) | Pose-free 3D reconstruction (FreeSplatter): turns a handful of plain photos into 3D Gaussians, no camera poses or GPU required |
 | [privacy-filter.cpp](https://github.com/localai-org/privacy-filter.cpp) | Standalone GGML PII/NER token-classification engine powering LocalAI's PII redaction tier |
 | [LocalVQE](https://github.com/localai-org/LocalVQE) | Joint acoustic echo cancellation, noise suppression, and dereverberation |
 | [local-store](https://github.com/mudler/LocalAI) | Local-first vector database for embeddings (shipped in-tree) |

backend/Dockerfile.bonsai

@@ -1,160 +0,0 @@
-ARG BASE_IMAGE=ubuntu:24.04
-# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
-# when no prebuilt base is supplied. The builder-prebuilt stage is only
-# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
-# content here is harmless — BuildKit prunes the unreferenced builder.
-ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
-# BUILDER_TARGET selects which builder stage the final scratch image copies
-# package output from. Declared at global scope (before any FROM) so it's
-# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
-# `make backends/bonsai` on the from-source path.
-ARG BUILDER_TARGET=builder-fromsource
-ARG APT_MIRROR=""
-ARG APT_PORTS_MIRROR=""
-
-
-# ============================================================================
-# Stage: builder-fromsource — self-contained build path.
-# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
-# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
-# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
-# default; local `make backends/bonsai`).
-#
-# The install script is the same one that backend/Dockerfile.base-grpc-builder
-# runs, so the result is bit-equivalent to the prebuilt-base path
-# (builder-prebuilt below).
-# ============================================================================
-FROM ${BASE_IMAGE} AS builder-fromsource
-ARG BUILD_TYPE
-ARG CUDA_MAJOR_VERSION
-ARG CUDA_MINOR_VERSION
-ARG CMAKE_FROM_SOURCE=false
-# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
-ARG CMAKE_VERSION=3.31.10
-ARG GRPC_VERSION=v1.65.0
-ARG GRPC_MAKEFLAGS="-j4 -Otarget"
-ARG SKIP_DRIVERS=false
-ARG TARGETARCH
-ARG TARGETVARIANT
-ARG GO_VERSION=1.25.4
-ARG UBUNTU_VERSION=2404
-ARG APT_MIRROR
-ARG APT_PORTS_MIRROR
-ARG AMDGPU_TARGETS=""
-ARG BACKEND=rerankers
-# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ARG CMAKE_ARGS
-
-ENV BUILD_TYPE=${BUILD_TYPE} \
-    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
-    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
-    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
-    CMAKE_VERSION=${CMAKE_VERSION} \
-    GRPC_VERSION=${GRPC_VERSION} \
-    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
-    SKIP_DRIVERS=${SKIP_DRIVERS} \
-    TARGETARCH=${TARGETARCH} \
-    UBUNTU_VERSION=${UBUNTU_VERSION} \
-    APT_MIRROR=${APT_MIRROR} \
-    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
-    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
-    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
-    CMAKE_ARGS=${CMAKE_ARGS} \
-    DEBIAN_FRONTEND=noninteractive
-
-# CUDA on PATH (no-op when CUDA isn't installed)
-ENV PATH=/usr/local/cuda/bin:${PATH}
-# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
-ENV PATH=/opt/rocm/bin:${PATH}
-
-WORKDIR /build
-
-# Install everything via the shared script — the same one that
-# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
-# this from-source path are bit-equivalent.
-RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
-    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
-    bash /usr/local/sbin/install-base-deps
-
-# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
-# CMake's find_package finds it at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
-# for rationale. bonsai is a llama.cpp fork that reuses
-# backend/cpp/llama-cpp source via a thin wrapper Makefile, so MOST TUs
-# are content-identical to the upstream llama-cpp build. Sharing a cache
-# id with llama-cpp could give cross-fork hits — but for now keep them
-# separate so a regression in one doesn't poison the other. Revisit
-# sharing after measuring the actual hit rate.
-#
-# The compile body is shared with builder-prebuilt via .docker/bonsai-compile.sh.
-RUN --mount=type=bind,source=.docker/bonsai-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=bonsai-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-
-# Copy libraries using a script to handle architecture differences
-RUN make -BC /LocalAI/backend/cpp/bonsai package
-
-
-# ============================================================================
-# Stage: builder-prebuilt — uses the pre-built base from
-# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
-# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
-# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
-# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
-# builder-base-image).
-# ============================================================================
-FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
-
-ARG BUILD_TYPE
-ENV BUILD_TYPE=${BUILD_TYPE}
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
-# AMDGPU_TARGETS must be forwarded into the env here too — backend/cpp/llama-cpp/Makefile
-# (which the bonsai Makefile reuses via a sibling build dir) errors out when the var
-# is empty on a hipblas build, and the prebuilt path is what CI exercises most of the
-# time. The builder-fromsource stage above already does this; mirror it here.
-ARG AMDGPU_TARGETS
-ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
-ARG TARGETARCH
-ARG TARGETVARIANT
-
-# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
-# /usr/local. Mirror what the from-source path does so the compile step
-# can find gRPC at the canonical prefix the Makefile expects.
-RUN cp -a /opt/grpc/. /usr/local/
-
-COPY . /LocalAI
-
-RUN --mount=type=bind,source=.docker/bonsai-compile.sh,target=/usr/local/sbin/compile.sh \
-    --mount=type=cache,target=/root/.ccache,id=bonsai-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    bash /usr/local/sbin/compile.sh
-
-RUN make -BC /LocalAI/backend/cpp/bonsai package
-
-
-# ============================================================================
-# Final stage — copies package output from one of the two builders.
-# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
-#
-# BuildKit doesn't support variable expansion in `COPY --from=` directly,
-# so we resolve the ARG by aliasing the chosen builder to a fixed stage
-# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
-# BUILDER_TARGET itself is declared as a global ARG at the top of this
-# file (required for use in FROM), so we just re-import it into this
-# stage's scope before the FROM directive.
-# ============================================================================
-FROM ${BUILDER_TARGET} AS builder
-
-FROM scratch
-
-
-# Copy all available binaries (the build process only creates the appropriate ones for the target architecture)
-COPY --from=builder /LocalAI/backend/cpp/bonsai/package/. ./

backend/Dockerfile.python

@@ -224,11 +224,7 @@ ARG DEPS_REFRESH=initial
 
 RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
 
-# Package GPU libraries into the backend's lib directory.
-#
-# Must stay after the venv is built above: package-gpu-libs.sh inspects
-# /${BACKEND}/venv to decide whether this backend already carries a complete
-# cuDNN from pip, and bundles one only when it does not (issue #10905).
+# Package GPU libraries into the backend's lib directory
 RUN mkdir -p /${BACKEND}/lib && \
     TARGET_LIB_DIR="/${BACKEND}/lib" BUILD_TYPE="${BUILD_TYPE}" CUDA_MAJOR_VERSION="${CUDA_MAJOR_VERSION}" \
     bash /package-gpu-libs.sh "/${BACKEND}/lib"

backend/README.md

@@ -46,7 +46,6 @@ The backend system provides language-specific Dockerfiles that handle the build
 - **vllm**: High-performance LLM inference
 - **mlx**: Apple Silicon optimization
 - **diffusers**: Stable Diffusion models
-- **longcat-video**: CUDA text/image-to-video and speech-driven avatar generation
 - **Audio**: coqui, faster-whisper, kitten-tts
 - **Vision**: mlx-vlm, rfdetr
 - **Specialized**: rerankers, chatterbox, kokoro

backend/backend.proto

@@ -18,18 +18,6 @@ service Backend {
   rpc GenerateVideo(GenerateVideoRequest) returns (Result) {}
   rpc AudioTranscription(TranscriptRequest) returns (TranscriptResult) {}
   rpc AudioTranscriptionStream(TranscriptRequest) returns (stream TranscriptStreamResponse) {}
-  // AudioTranscriptionLive is the bidirectional live-microphone ASR RPC. The
-  // first message MUST carry a Config; subsequent messages carry Audio frames
-  // (mono float PCM at config.sample_rate, 16 kHz default). After a
-  // successful open the backend replies with a single ready ack
-  // (TranscriptLiveResponse{ready:true}); backends or models without
-  // cache-aware streaming support return UNIMPLEMENTED instead. Newly
-  // finalized text streams back as deltas; eou=true marks the model's
-  // end-of-utterance token. One stream spans many utterances (the decoder
-  // resets itself after each EOU). Closing the send side finalizes: the
-  // backend flushes the decoder tail and emits a terminal message carrying
-  // final_result. A second Config mid-stream resets the decode session.
-  rpc AudioTranscriptionLive(stream TranscriptLiveRequest) returns (stream TranscriptLiveResponse) {}
   rpc TTS(TTSRequest) returns (Result) {}
   rpc TTSStream(TTSRequest) returns (stream Reply) {}
   rpc SoundGeneration(SoundGenerationRequest) returns (Result) {}
@@ -136,10 +124,6 @@ message MetricsResponse {
 message TokenClassifyRequest {
   string text = 1;
   float threshold = 2;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 3;
 }
 
 // TokenClassifyEntity is one detected entity span. Byte offsets are
@@ -177,10 +161,6 @@ message ScoreRequest {
   // candidates differ in length and the consumer wants a per-token
   // measure comparable across them (PMI-style scoring).
   bool length_normalize = 4;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 5;
 }
 
 // CandidateScore is one row in the ScoreResponse, matching by index
@@ -212,10 +192,6 @@ message RerankRequest {
   string query = 1;
   repeated string documents = 2;
   int32 top_n = 3;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 4;
 }
 
 message RerankResult {
@@ -327,39 +303,6 @@ message PredictOptions {
   int32 TopLogprobs = 51;  // Number of top logprobs to return per token (maps to OpenAI top_logprobs parameter)
   map<string, string> Metadata = 52;  // Generic per-request metadata (e.g., enable_thinking)
   float MinP = 53;  // Minimum probability sampling threshold (0.0 = disabled)
-
-  // ModelIdentity names the model this request is for, so a backend can reject
-  // a request that reached it by mistake instead of answering from whatever
-  // model it happens to hold. In distributed mode a worker can recycle a
-  // stopped backend's gRPC port for a different model's backend, and a
-  // liveness-only health probe cannot tell that apart from a valid cached
-  // route (#10952).
-  //
-  // The value is the controller's ModelConfig.Model, the SAME expression that
-  // produces ModelOptions.Model at LoadModel time, so the two are equal by
-  // construction rather than by convention.
-  //
-  // Empty means "no identity supplied": backends MUST skip the check. That
-  // keeps an old controller talking to a new backend working, and covers
-  // callers that legitimately synthesize a PredictOptions internally.
-  //
-  // Do NOT reuse TTSRequest.model or SoundGenerationRequest.model for this
-  // purpose. FileStagingClient already rewrites those to worker-local absolute
-  // paths (core/services/nodes/file_staging_client.go), so in distributed mode
-  // they already differ from the load-time value and comparing them would
-  // reject valid requests. Extending identity to those RPCs needs a separate
-  // field carrying the untranslated value - which is exactly what
-  // TTSRequest.ModelIdentity and SoundGenerationRequest.ModelIdentity are.
-  //
-  // Every other request message that reaches a backend through the distributed
-  // router now carries the same ModelIdentity field, populated from the same
-  // ModelConfig.Model. FileStagingClient rewrites Src/Dst/Voice/Model/
-  // StartImage/EndImage/Audio and never ModelIdentity, so what the backend
-  // compares is always what the controller sent.
-  string ModelIdentity = 54;
-
-  // 24 was never assigned; reserve it so it is not silently reused.
-  reserved 24;
 }
 
 // ToolCallDelta represents an incremental tool call update from the C++ parser.
@@ -529,10 +472,6 @@ message TranscriptRequest {
   float temperature = 8;
   repeated string timestamp_granularities = 9;
   bool stream = 10;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 11;
 }
 
 message TranscriptResult {
@@ -540,10 +479,6 @@ message TranscriptResult {
   string text = 2;
   string language = 3;
   float duration = 4;
-  // True when the decode ended on the model's end-of-utterance special token
-  // (<EOU>/<EOB>, emitted by cache-aware streaming models such as
-  // parakeet_realtime_eou_120m-v1). The marker itself is stripped from text.
-  bool eou = 5;
 }
 
 message TranscriptStreamResponse {
@@ -551,34 +486,6 @@ message TranscriptStreamResponse {
   TranscriptResult final_result = 2;
 }
 
-// === AudioTranscriptionLive messages =====================================
-
-message TranscriptLiveRequest {
-  oneof payload {
-    TranscriptLiveConfig config = 1;
-    TranscriptLiveAudio  audio  = 2;
-  }
-}
-
-message TranscriptLiveConfig {
-  string language = 1;             // "" => model default
-  int32 sample_rate = 2;           // 0 => 16000; backends may reject others
-  map<string, string> params = 3;  // backend-specific tuning
-}
-
-message TranscriptLiveAudio {
-  repeated float pcm = 1;          // mono PCM in [-1,1] at config.sample_rate
-}
-
-message TranscriptLiveResponse {
-  bool ready = 1;                       // open ack: sent once, before any delta
-  string delta = 2;                     // newly-finalized text since previous response
-  bool eou = 3;                         // <EOU> fired during this feed (the user yielded the turn)
-  repeated TranscriptWord words = 4;    // words finalized by this feed (stream-relative ns)
-  TranscriptResult final_result = 5;    // terminal message only, after the send side closes
-  bool eob = 6;                         // <EOB> fired: a backchannel ("uh-huh") ended — NOT a turn boundary
-}
-
 message TranscriptWord {
   int64 start = 1;
   int64 end = 2;
@@ -611,10 +518,6 @@ message GenerateImageRequest {
 
   // Reference images for models that support them (e.g., Flux Kontext)
   repeated string ref_images = 12;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 13;
 }
 
 message GenerateVideoRequest {
@@ -630,14 +533,6 @@ message GenerateVideoRequest {
   float cfg_scale = 10;    // Classifier-free guidance scale
   int32 step = 11;         // Number of inference steps
   string dst = 12;        // Output path for the generated video
-  string audio = 13;      // Path to staged audio for audio-conditioned video
-  // Backend-specific per-request generation parameters. Values are strings
-  // and are validated/coerced by the selected backend.
-  map<string, string> params = 14;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 15;
 }
 
 message TTSRequest {
@@ -655,26 +550,10 @@ message TTSRequest {
   // (e.g. Chatterbox exaggeration/cfg_weight/temperature). Values are strings and
   // coerced by the backend; unset leaves the backend's configured defaults.
   map<string, string> params = 7;
-  // ModelIdentity is a SEPARATE field from `model` above and carries the
-  // UNTRANSLATED controller-side ModelConfig.Model, so a backend can reject a
-  // request that reached it through a stale distributed route (#10952).
-  //
-  // `model` cannot be reused for this: FileStagingClient.TTS/.TTSStream and the
-  // SoundGeneration path rewrite it into a worker-local absolute path
-  // (core/services/nodes/file_staging_client.go), while the load-time value is
-  // untranslated. In distributed mode - exactly the configuration this guards -
-  // the two already differ, so comparing them would reject valid requests.
-  //
-  // Empty means "no identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 8;
 }
 
 message VADRequest {
   repeated float audio = 1;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 2;
 }
 
 message VADSegment {
@@ -706,10 +585,6 @@ message DiarizeRequest {
   float  min_duration_on = 8;          // discard segments shorter than this (seconds); 0 = backend default
   float  min_duration_off = 9;         // merge gaps shorter than this (seconds); 0 = backend default
   bool   include_text = 10;            // when the backend can emit per-segment transcript for free, ask it to populate `text`
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 11;
 }
 
 message DiarizeSegment {
@@ -744,18 +619,6 @@ message SoundGenerationRequest {
   optional string language = 14;
   optional string timesignature = 15;
   optional bool instrumental = 17;
-  // ModelIdentity is a SEPARATE field from `model` above and carries the
-  // UNTRANSLATED controller-side ModelConfig.Model, so a backend can reject a
-  // request that reached it through a stale distributed route (#10952).
-  //
-  // `model` cannot be reused for this: FileStagingClient.TTS/.TTSStream and the
-  // SoundGeneration path rewrite it into a worker-local absolute path
-  // (core/services/nodes/file_staging_client.go), while the load-time value is
-  // untranslated. In distributed mode - exactly the configuration this guards -
-  // the two already differ, so comparing them would reject valid requests.
-  //
-  // Empty means "no identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 18;
 }
 
 message TokenizationResponse {
@@ -795,10 +658,6 @@ message DetectOptions {
   repeated float points = 3;   // Point coordinates as [x1, y1, label1, x2, y2, label2, ...] (label: 1=pos, 0=neg)
   repeated float boxes = 4;    // Box coordinates as [x1, y1, x2, y2, ...]
   float threshold = 5;         // Detection confidence threshold
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 6;
 }
 
 message Detection {
@@ -821,10 +680,6 @@ message SoundDetectionRequest {
   string src = 1;       // audio file path (LocalAI writes the upload to disk)
   int32 top_k = 2;      // number of top tags to return (0 = all classes)
   float threshold = 3;  // optional: drop tags scoring below this
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 4;
 }
 
 message SoundClass {
@@ -849,10 +704,6 @@ message DepthRequest {
   bool include_points = 7;         // back-project to a 3D point cloud (DualDPT)
   float points_conf_thresh = 8;    // keep points with confidence >= this threshold
   repeated string exports = 9;     // requested exports: "glb", "colmap"
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 10;
 }
 
 message DepthResponse {
@@ -884,10 +735,6 @@ message FaceVerifyRequest {
   string img2 = 2;              // base64-encoded image
   float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
   bool   anti_spoofing = 4;     // run MiniFASNet liveness on each image; failed liveness forces verified=false
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 5;
 }
 
 message FaceVerifyResponse {
@@ -909,10 +756,6 @@ message FaceAnalyzeRequest {
   string          img = 1;          // base64-encoded image
   repeated string actions = 2;      // subset of ["age","gender","emotion","race"]; empty = all-supported
   bool            anti_spoofing = 3;
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 4;
 }
 
 message FaceAnalysis {
@@ -945,10 +788,6 @@ message VoiceVerifyRequest {
   string audio2 = 2;            // path to second audio clip
   float  threshold = 3;         // cosine-distance threshold; 0 = use backend default
   bool   anti_spoofing = 4;     // reserved for future AASIST bolt-on
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 5;
 }
 
 message VoiceVerifyResponse {
@@ -963,10 +802,6 @@ message VoiceVerifyResponse {
 message VoiceAnalyzeRequest {
   string          audio = 1;        // path to audio clip
   repeated string actions = 2;      // subset of ["age","gender","emotion"]; empty = all-supported
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 3;
 }
 
 message VoiceAnalysis {
@@ -985,10 +820,6 @@ message VoiceAnalyzeResponse {
 
 message VoiceEmbedRequest {
   string audio = 1;              // path to audio clip
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 2;
 }
 
 message VoiceEmbedResponse {
@@ -1083,10 +914,6 @@ message AudioTransformRequest {
   string reference_path = 2;         // optional auxiliary; empty => zero-fill
   string dst = 3;                    // required, output file path
   map<string, string> params = 4;    // backend-specific tuning
-  // ModelIdentity names the model this request is for; see
-  // PredictOptions.ModelIdentity for the full rationale. Empty means "no
-  // identity supplied" and backends MUST skip the check.
-  string ModelIdentity = 5;
 }
 
 message AudioTransformResult {
@@ -1385,3 +1212,4 @@ message ForwardReply {
   repeated ForwardHeader headers = 2;
   bytes body_chunk = 3;
 }
+

backend/cpp/bonsai/Makefile

@@ -1,105 +0,0 @@
-
-# Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp.
-# Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-BONSAI_VERSION?=9fcaed763ccda38ea81068ad9d7f991aaddca451
-LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp
-
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-ONEAPI_VARS?=/opt/intel/oneapi/setvars.sh
-TARGET?=--target grpc-server
-JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
-ARCH?=$(shell uname -m)
-
-CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
-LLAMA_CPP_DIR := $(CURRENT_MAKEFILE_DIR)/../llama-cpp
-
-GREEN := \033[0;32m
-RESET := \033[0m
-
-# bonsai is a llama.cpp fork (PrismML) adding the Q1_0 (1-bit) and Q2_0 (ternary)
-# weight-quantization kernels that the Bonsai / Ternary-Bonsai models ship in. Rather
-# than duplicating grpc-server.cpp / CMakeLists.txt / prepare.sh we reuse the ones in
-# backend/cpp/llama-cpp, and only swap which repo+sha the fetch step pulls. Each flavor
-# target copies ../llama-cpp into a sibling ../bonsai-<flavor>-build directory, then
-# invokes llama-cpp's own build with LLAMA_REPO/LLAMA_VERSION overridden to point at the
-# fork.
-#
-# The Q1_0/Q2_0 additions are model *weight* types decoded inside libllama, transparent
-# to the reused gRPC server, so (unlike turboquant's KV-cache types) no grpc-server.cpp
-# allow-list patch is needed. The fork branched from upstream before a few API changes
-# the shared grpc-server.cpp depends on; those are carried as patch files under
-# backend/cpp/bonsai/patches/ and applied to the cloned fork by apply-patches.sh.
-PATCHES_DIR := $(CURRENT_MAKEFILE_DIR)/patches
-
-define bonsai-build
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build
-	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build
-	# Drop patches vendored for upstream llama.cpp: the fork tree diverges, so
-	# they reject there. Fork-specific patches live in backend/cpp/bonsai/patches/
-	# and are applied by apply-patches.sh below.
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
-	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build llama.cpp
-	bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/llama.cpp $(PATCHES_DIR)
-	CMAKE_ARGS="$(CMAKE_ARGS) $(2)" TARGET="$(3)" \
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server bonsai-$(1)
-endef
-
-bonsai-avx2:
-	$(call bonsai-build,avx2,-DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
-
-bonsai-avx512:
-	$(call bonsai-build,avx512,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
-
-bonsai-avx:
-	$(call bonsai-build,avx,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
-
-bonsai-fallback:
-	$(call bonsai-build,fallback,-DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
-
-# Single-build CPU backend via ggml CPU_ALL_VARIANTS (mirrors llama-cpp-cpu-all).
-# bonsai reuses backend/cpp/llama-cpp's CMakeLists.txt (hw_grpc_proto STATIC) and
-# Makefile (SHARED_LIBS make-var + EXTRA_CMAKE_ARGS), so this passes the same overrides
-# through to the copied build: SHARED_LIBS=ON, the DL flags, and --target ggml (which
-# pulls in the per-microarch libggml-cpu-*.so via ggml's add_dependencies). The .so set
-# is collected for package.sh to bundle into package/lib.
-bonsai-cpu-all:
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build
-	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build
-	# Drop patches vendored for upstream llama.cpp: the fork tree diverges, so
-	# they reject there. Fork-specific patches live in backend/cpp/bonsai/patches/
-	# and are applied by apply-patches.sh below.
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
-	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build llama.cpp
-	bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/llama.cpp $(PATCHES_DIR)
-	SHARED_LIBS=ON EXTRA_CMAKE_ARGS="-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON" TARGET="--target grpc-server --target ggml" \
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server bonsai-cpu-all
-	rm -rf ggml-shared-libs && mkdir -p ggml-shared-libs
-	find $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/llama.cpp/build \( -name '*.so*' -o -name '*.dylib' \) -exec cp -av {} ggml-shared-libs/ \;
-	@echo "Collected ggml shared backends:" && ls -la ggml-shared-libs/
-
-bonsai-grpc:
-	$(call bonsai-build,grpc,-DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server --target rpc-server)
-
-bonsai-rpc-server: bonsai-grpc
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-grpc-build/llama.cpp/build/bin/rpc-server bonsai-rpc-server
-
-package:
-	bash package.sh
-
-purge:
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-*-build
-	rm -rf bonsai-* package
-
-clean: purge

backend/cpp/bonsai/apply-patches.sh

@@ -1,48 +0,0 @@
-#!/bin/bash
-# Apply the bonsai patch series to a cloned PrismML llama.cpp (prism branch) checkout.
-#
-# The prism fork branched from upstream llama.cpp before a number of API changes that the
-# shared backend/cpp/llama-cpp/grpc-server.cpp depends on. We carry those upstream commits
-# as patch files under backend/cpp/bonsai/patches/ and apply them here so the reused
-# grpc-server source compiles against the fork unmodified.
-#
-# Drop the corresponding patch from patches/ whenever the fork catches up with upstream —
-# the build will fail fast if a patch stops applying, which is the signal to retire it.
-
-set -euo pipefail
-
-if [[ $# -ne 2 ]]; then
-    echo "usage: $0 <llama.cpp-src-dir> <patches-dir>" >&2
-    exit 2
-fi
-
-SRC_DIR=$1
-PATCHES_DIR=$2
-
-if [[ ! -d "$SRC_DIR" ]]; then
-    echo "source dir does not exist: $SRC_DIR" >&2
-    exit 2
-fi
-
-if [[ ! -d "$PATCHES_DIR" ]]; then
-    echo "no patches dir at $PATCHES_DIR, nothing to apply"
-    exit 0
-fi
-
-shopt -s nullglob
-patches=("$PATCHES_DIR"/*.patch)
-shopt -u nullglob
-
-if [[ ${#patches[@]} -eq 0 ]]; then
-    echo "no .patch files in $PATCHES_DIR, nothing to apply"
-    exit 0
-fi
-
-cd "$SRC_DIR"
-
-for patch in "${patches[@]}"; do
-    echo "==> applying $patch"
-    git apply --verbose "$patch"
-done
-
-echo "all bonsai patches applied successfully"

backend/cpp/bonsai/package.sh

@@ -1,66 +0,0 @@
-#!/bin/bash
-
-# Script to copy the appropriate libraries based on architecture
-# This script is used in the final stage of the Dockerfile
-
-set -e
-
-CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
-
-# Create lib directory
-mkdir -p $CURDIR/package/lib
-
-cp -avrf $CURDIR/bonsai-* $CURDIR/package/
-cp -rfv $CURDIR/run.sh $CURDIR/package/
-
-# Bundle the ggml shared backends from the CPU_ALL_VARIANTS build into package/lib. ggml
-# discovers the per-microarch libggml-cpu-*.so by scanning the executable directory, which
-# (via the bundled lib/ld.so that run.sh launches through) resolves to lib/. See the
-# matching comment in backend/cpp/llama-cpp/package.sh. No-op on the fallback/ROCm builds.
-if [ -d "$CURDIR/ggml-shared-libs" ]; then
-    echo "Bundling ggml shared backends (CPU_ALL_VARIANTS)..."
-    cp -avf $CURDIR/ggml-shared-libs/*.so* $CURDIR/package/lib/
-fi
-
-# Detect architecture and copy appropriate libraries
-if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    # x86_64 architecture
-    echo "Detected x86_64 architecture, copying x86_64 libraries..."
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    # ARM64 architecture
-    echo "Detected ARM64 architecture, copying ARM64 libraries..."
-    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
-    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
-    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
-    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
-    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
-else
-    echo "Error: Could not detect architecture"
-    exit 1
-fi
-
-# Package GPU libraries based on BUILD_TYPE
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
-echo "Packaging completed successfully"
-ls -liah $CURDIR/package/
-ls -liah $CURDIR/package/lib/

backend/cpp/bonsai/patches/README.md

@@ -1,19 +0,0 @@
-# bonsai fork skew patches
-
-The `bonsai` backend reuses `backend/cpp/llama-cpp/grpc-server.cpp` (written against
-LocalAI's pinned *upstream* llama.cpp) but compiles it against the PrismML `prism` fork,
-which branched from upstream some commits earlier. Any upstream API change that the shared
-gRPC server depends on, but that the fork does not yet carry, is back-ported here as a
-`*.patch` file and applied to the cloned fork checkout by `../apply-patches.sh`.
-
-CI treats both this directory and `backend/cpp/llama-cpp/` as Bonsai inputs, since
-the wrapper copies and builds the shared llama.cpp backend sources.
-
-Rules:
-
-- One upstream commit (or minimal hunk) per patch, named `NNNN-short-description.patch`.
-- Patches are applied with `git apply` from the fork's checkout root.
-- `apply-patches.sh` fails fast if a patch stops applying cleanly — that is the signal the
-  fork has caught up (or diverged), so re-cut or drop the patch.
-- Keep this set as small as possible; the long-term fix is the fork rebasing onto a newer
-  upstream (or Q1_0/Q2_0 landing in mainline llama.cpp, retiring this backend entirely).

backend/cpp/bonsai/run.sh

@@ -1,56 +0,0 @@
-#!/bin/bash
-set -ex
-
-# Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath "$0")")
-
-cd /
-
-echo "CPU info:"
-grep -e "model\sname" /proc/cpuinfo | head -1
-grep -e "flags" /proc/cpuinfo | head -1
-
-BINARY=bonsai-fallback
-
-# x86/arm64 ship a single bonsai-cpu-all built with ggml CPU_ALL_VARIANTS: ggml's
-# backend registry dlopens the best libggml-cpu-*.so for this host, so no shell-side
-# probing. ROCm ships only bonsai-fallback, so fall back to it when cpu-all is absent.
-if [ -e "$CURDIR"/bonsai-cpu-all ]; then
-	BINARY=bonsai-cpu-all
-fi
-
-if [ -n "$LLAMACPP_GRPC_SERVERS" ]; then
-	if [ -e "$CURDIR"/bonsai-grpc ]; then
-		BINARY=bonsai-grpc
-	fi
-fi
-
-# Extend ld library path with the dir where this script is located/lib
-if [ "$(uname)" == "Darwin" ]; then
-	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
-else
-	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
-	# Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files)
-	if [ -d "$CURDIR/lib/rocblas/library" ]; then
-		export ROCBLAS_TENSILE_LIBPATH="$CURDIR"/lib/rocblas/library
-	fi
-	# Same for hipBLASLt (rocblaslt): the bundled libhipblaslt.so resolves its
-	# TensileLibrary_lazy_gfx*.dat kernel data relative to itself, so point it at
-	# the bundled data or it falls back to slow generic kernels (issue #10660).
-	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
-		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
-	fi
-fi
-
-# If there is a lib/ld.so, use it
-if [ -f "$CURDIR"/lib/ld.so ]; then
-	echo "Using lib/ld.so"
-	echo "Using binary: $BINARY"
-	exec "$CURDIR"/lib/ld.so "$CURDIR"/$BINARY "$@"
-fi
-
-echo "Using binary: $BINARY"
-exec "$CURDIR"/$BINARY "$@"
-
-# We should never reach this point, however just in case we do, run fallback
-exec "$CURDIR"/bonsai-fallback "$@"

backend/cpp/ds4/grpc-server.cpp

@@ -51,11 +51,6 @@ namespace {
 
 // Global state - ds4 is single-engine-per-process by design.
 std::mutex g_engine_mu;
-// The ModelOptions.Model this process loaded, compared against
-// PredictOptions.ModelIdentity so a request that arrived through a stale
-// distributed route is rejected rather than answered from the wrong model
-// (#10952). Guarded by g_engine_mu like the rest of the engine state.
-std::string g_loaded_model_identity;
 ds4_engine *g_engine = nullptr;
 ds4_session *g_session = nullptr;
 int g_ctx_size = 32768;
@@ -567,24 +562,6 @@ static void build_prompt(ds4_engine *engine, const backend::PredictOptions *requ
     ds4_chat_append_assistant_prefix(engine, out, think);
 }
 
-// check_model_identity mirrors pkg/grpc/server.go and
-// backend/python/common/model_identity.py. Either side empty means "skip": the
-// request side is empty for a controller that predates the field, the loaded
-// side when such a controller performed the load. A false rejection is worse
-// than the miss it prevents. Callers must already hold g_engine_mu.
-static GStatus check_model_identity(const backend::PredictOptions *request) {
-    if (request == nullptr || request->modelidentity().empty()) return GStatus::OK;
-    if (g_loaded_model_identity.empty() ||
-        g_loaded_model_identity == request->modelidentity()) {
-        return GStatus::OK;
-    }
-    // NOT_FOUND plus this exact sentinel is the cross-language contract the
-    // router matches on (grpcerrors.ModelMismatchSentinel).
-    return GStatus(StatusCode::NOT_FOUND,
-                   "ds4: model identity mismatch: loaded \"" + g_loaded_model_identity +
-                   "\", requested \"" + request->modelidentity() + "\"");
-}
-
 class DS4Backend final : public backend::Backend::Service {
 public:
     GStatus Health(ServerContext *, const backend::HealthMessage *,
@@ -739,7 +716,6 @@ public:
         }
 
         result->set_success(true);
-        g_loaded_model_identity = request->model();
         result->set_message("loaded " + model_path);
         return GStatus::OK;
     }
@@ -748,7 +724,6 @@ public:
                           backend::TokenizationResponse *response) override {
         std::lock_guard<std::mutex> lock(g_engine_mu);
         if (!g_engine) return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
-        if (GStatus id = check_model_identity(request); !id.ok()) return id;
         ds4_tokens out = {};
         ds4_tokenize_text(g_engine, request->prompt().c_str(), &out);
         for (int i = 0; i < out.len; ++i) response->add_tokens(out.v[i]);
@@ -763,7 +738,6 @@ public:
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
-        if (GStatus id = check_model_identity(request); !id.ok()) return id;
         if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
             return GStatus(StatusCode::UNAVAILABLE, route_err);
         }
@@ -863,7 +837,6 @@ public:
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
-        if (GStatus id = check_model_identity(request); !id.ok()) return id;
         if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
             return GStatus(StatusCode::UNAVAILABLE, route_err);
         }

backend/cpp/ds4/package.sh

@@ -1,14 +1,12 @@
 #!/bin/bash
-set -euo pipefail
+set -e
 CURDIR=$(dirname "$(realpath "$0")")
 REPO_ROOT="${CURDIR}/../../.."
-PACKAGE_DIR="$CURDIR/package"
 
-rm -rf "$PACKAGE_DIR"
-mkdir -p "$PACKAGE_DIR/lib"
-cp -avf "$CURDIR/grpc-server" "$PACKAGE_DIR/"
-cp -avf "$CURDIR/ds4-worker"  "$PACKAGE_DIR/"
-cp -rfv "$CURDIR/run.sh"      "$PACKAGE_DIR/"
+mkdir -p "$CURDIR/package/lib"
+cp -avf "$CURDIR/grpc-server" "$CURDIR/package/"
+cp -avf "$CURDIR/ds4-worker"  "$CURDIR/package/"
+cp -rfv "$CURDIR/run.sh"      "$CURDIR/package/"
 
 UNAME_S=$(uname -s)
 if [ "$UNAME_S" = "Darwin" ]; then
@@ -18,54 +16,25 @@ if [ "$UNAME_S" = "Darwin" ]; then
 fi
 
 if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 "$PACKAGE_DIR/lib/ld.so"
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 "$CURDIR/package/lib/ld.so"
+    LIBDIR=/lib/x86_64-linux-gnu
 elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    cp -arfLv /lib/ld-linux-aarch64.so.1 "$PACKAGE_DIR/lib/ld.so"
+    cp -arfLv /lib/ld-linux-aarch64.so.1 "$CURDIR/package/lib/ld.so"
+    LIBDIR=/lib/aarch64-linux-gnu
 else
     echo "package.sh: unknown architecture" >&2; exit 1
 fi
 
-# Bundle the complete dependency closure for both executables. In particular,
-# grpc-server links the distro gRPC/protobuf/absl stack; copying only the core
-# C/C++ runtime libraries leaves the scratch image unable to start.
-{
-    ldd "$CURDIR/grpc-server"
-    ldd "$CURDIR/ds4-worker"
-} | awk '$2 == "=>" && $3 ~ /^\// { print $3 }' | sort -u | \
-while read -r so; do
-    cp -arfLv "$so" "$PACKAGE_DIR/lib/"
+for lib in libc.so.6 libgcc_s.so.1 libstdc++.so.6 libm.so.6 libgomp.so.1 \
+           libdl.so.2 librt.so.1 libpthread.so.0; do
+    cp -arfLv "$LIBDIR/$lib" "$CURDIR/package/lib/$lib"
 done
 
 GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
 if [ -f "$GPU_LIB_SCRIPT" ]; then
-    # shellcheck source=/dev/null
-    source "$GPU_LIB_SCRIPT" "$PACKAGE_DIR/lib"
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
     package_gpu_libs
 fi
 
-# Resolve every dependency through the same loader and library path used by
-# the from-scratch image. The loader can still search host defaults, so reject
-# any absolute dependency path that escapes the package instead of accepting a
-# false-positive validation against a library that scratch will not contain.
-validate_packaged_binary() {
-    local binary="$1"
-    local resolution
-    resolution=$("$PACKAGE_DIR/lib/ld.so" \
-        --library-path "$PACKAGE_DIR/lib" \
-        --list "$PACKAGE_DIR/$binary")
-
-    printf '%s\n' "$resolution" | awk -v prefix="$PACKAGE_DIR/lib/" '
-        $2 == "=>" && $3 ~ /^\// && index($3, prefix) != 1 {
-            print "package.sh: dependency resolved outside package: " $0 > "/dev/stderr"
-            invalid = 1
-        }
-        END { exit invalid }
-    '
-}
-
-for binary in grpc-server ds4-worker; do
-    validate_packaged_binary "$binary"
-done
-
 echo "ds4 package contents:"
-ls -lah "$PACKAGE_DIR/" "$PACKAGE_DIR/lib/"
+ls -lah "$CURDIR/package/" "$CURDIR/package/lib/"

backend/cpp/ik-llama-cpp/CMakeLists.txt

@@ -1,6 +1,15 @@
-## Multimodal support is provided by the in-tree `mtmd` library target
-## (examples/mtmd/), which the grpc-server links and includes below. clip/llava
-## were pruned upstream; the high-level mtmd_* / mtmd_helper_* API is used instead.
+## Clip/LLaVA library for multimodal support — built locally from copied sources
+set(TARGET myclip)
+add_library(${TARGET} clip.cpp clip.h llava.cpp llava.h)
+install(TARGETS ${TARGET} LIBRARY)
+target_include_directories(myclip PUBLIC .)
+target_include_directories(myclip PUBLIC ../..)
+target_include_directories(myclip PUBLIC ../../common)
+target_link_libraries(${TARGET} PRIVATE common ggml llama ${CMAKE_THREAD_LIBS_INIT})
+target_compile_features(${TARGET} PRIVATE cxx_std_11)
+if (NOT MSVC)
+    target_compile_options(${TARGET} PRIVATE -Wno-cast-qual)
+endif()
 
 set(TARGET grpc-server)
 set(CMAKE_CXX_STANDARD 17)
@@ -58,16 +67,12 @@ add_library(hw_grpc_proto
   ${hw_proto_hdrs} )
 
 add_executable(${TARGET} grpc-server.cpp json.hpp)
-# mtmd public headers (mtmd.h / mtmd-helper.h) live in examples/mtmd/.
-# Linking the mtmd target also propagates this include dir, but we add it
-# explicitly for clarity.
-target_include_directories(${TARGET} PRIVATE ../mtmd)
-target_link_libraries(${TARGET} PRIVATE common llama mtmd ${CMAKE_THREAD_LIBS_INIT} absl::flags hw_grpc_proto
+target_link_libraries(${TARGET} PRIVATE common llama myclip ${CMAKE_THREAD_LIBS_INIT} absl::flags hw_grpc_proto
   absl::flags_parse
   gRPC::${_REFLECTION}
   gRPC::${_GRPC_GRPCPP}
   protobuf::${_PROTOBUF_LIBPROTOBUF})
-target_compile_features(${TARGET} PRIVATE cxx_std_17)
+target_compile_features(${TARGET} PRIVATE cxx_std_11)
 if(TARGET BUILD_INFO)
   add_dependencies(${TARGET} BUILD_INFO)
 endif()

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=9d07d8681ece159a89fb4e16a1f9c9f3a5fac20f
+IK_LLAMA_VERSION?=6c00e87ac84404af588ad2e65935bd6f079c696f
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/ik-llama-cpp/grpc-server.cpp

@@ -11,8 +11,8 @@
 #include <memory>
 #include <string>
 #include <getopt.h>
-#include "mtmd.h"
-#include "mtmd-helper.h"
+#include "clip.h"
+#include "llava.h"
 #include "log.h"
 #include "common.h"
 #include "json.hpp"
@@ -45,9 +45,7 @@ using backend::HealthMessage;
 
 ///// LLAMA.CPP server code below
 
-// Match mtmd.h and ik_llama's server/common headers, which all use
-// nlohmann::ordered_json; a plain nlohmann::json alias collides at global scope.
-using json = nlohmann::ordered_json;
+using json = nlohmann::json;
 
 struct server_params
 {
@@ -221,11 +219,6 @@ struct llama_client_slot
 
     // multimodal
     std::vector<slot_image> images;
-    // Full prompt with mtmd media markers (mtmd_default_marker()) substituted in
-    // place of the legacy [img-N] tags, covering the text up to and including the
-    // last image. The text after the last image is kept in params.input_suffix and
-    // decoded through the normal token path so the sampling loop is unchanged.
-    std::string mtmd_prompt;
 
     // stats
     size_t sent_count = 0;
@@ -259,14 +252,14 @@ struct llama_client_slot
 
         for (slot_image & img : images)
         {
-            if (img.bitmap) {
-                mtmd_bitmap_free(img.bitmap);
-                img.bitmap = nullptr;
+            free(img.image_embedding);
+            if (img.img_data) {
+                clip_image_u8_free(img.img_data);
             }
+            img.prefix_prompt = "";
         }
 
         images.clear();
-        mtmd_prompt = "";
     }
 
     bool has_budget(gpt_params &global_params) {
@@ -403,13 +396,46 @@ struct llama_metrics {
     }
 };
 
+struct llava_embd_batch {
+    std::vector<llama_pos>      pos;
+    std::vector<int32_t>        n_seq_id;
+    std::vector<llama_seq_id>   seq_id_0;
+    std::vector<llama_seq_id *> seq_ids;
+    std::vector<int8_t>         logits;
+    llama_batch batch;
+    llava_embd_batch(float * embd, int32_t n_tokens, llama_pos pos_0, llama_seq_id seq_id) {
+        pos     .resize(n_tokens);
+        n_seq_id.resize(n_tokens);
+        seq_ids .resize(n_tokens + 1);
+        logits  .resize(n_tokens);
+        seq_id_0.resize(1);
+        seq_id_0[0] = seq_id;
+        seq_ids [n_tokens] = nullptr;
+        batch = {
+            /*n_tokens       =*/ n_tokens,
+            /*tokens         =*/ nullptr,
+            /*embd           =*/ embd,
+            /*pos            =*/ pos.data(),
+            /*n_seq_id       =*/ n_seq_id.data(),
+            /*seq_id         =*/ seq_ids.data(),
+            /*logits         =*/ logits.data(),
+        };
+        for (int i = 0; i < n_tokens; i++) {
+            batch.pos     [i] = pos_0 + i;
+            batch.n_seq_id[i] = 1;
+            batch.seq_id  [i] = seq_id_0.data();
+            batch.logits  [i] = false;
+        }
+    }
+};
+
 struct llama_server_context
 {
     llama_model *model = nullptr;
     llama_context *ctx = nullptr;
     const llama_vocab * vocab = nullptr;
 
-    mtmd_context *mctx = nullptr;
+    clip_ctx *clp_ctx = nullptr;
 
     gpt_params params;
 
@@ -465,6 +491,11 @@ struct llama_server_context
         if (!params.mmproj.path.empty()) {
             multimodal = true;
             LOG_INFO("Multi Modal Mode Enabled", {});
+            clp_ctx = clip_model_load(params.mmproj.path.c_str(), /*verbosity=*/ 1);
+            if(clp_ctx == nullptr) {
+                LOG_ERR("unable to load clip model: %s", params.mmproj.path.c_str());
+                return false;
+            }
 
             if (params.n_ctx < 2048) { // request larger context for the image embedding
                 params.n_ctx = 2048;
@@ -481,24 +512,10 @@ struct llama_server_context
         }
 
         if (multimodal) {
-            // mtmd_init_from_file requires the already-loaded text model, so it must
-            // run AFTER llama_init_from_gpt_params. It validates the projector
-            // against the model internally and returns nullptr on dim mismatch, so
-            // the explicit clip_n_mmproj_embd check is no longer needed.
-            mtmd_context_params mparams = mtmd_context_params_default();
-            mparams.use_gpu         = params.mmproj_use_gpu;
-            mparams.print_timings   = false;
-            mparams.n_threads       = params.n_threads_mtmd != -1 ? params.n_threads_mtmd
-                                      : params.n_threads_batch != -1 ? params.n_threads_batch
-                                                                     : params.n_threads;
-            mparams.verbosity       = GGML_LOG_LEVEL_INFO;
-            mparams.flash_attn_type = params.flash_attn ? LLAMA_FLASH_ATTN_TYPE_ENABLED
-                                                        : LLAMA_FLASH_ATTN_TYPE_DISABLED;
-            mparams.image_min_tokens = params.image_min_tokens;
-            mparams.image_max_tokens = params.image_max_tokens;
-            mctx = mtmd_init_from_file(params.mmproj.path.c_str(), model, mparams);
-            if (mctx == nullptr) {
-                LOG_ERR("unable to load multimodal projector: %s", params.mmproj.path.c_str());
+            const int n_embd_clip = clip_n_mmproj_embd(clp_ctx);
+            const int n_embd_llm  = llama_model_n_embd(model);
+            if (n_embd_clip != n_embd_llm) {
+                LOG("%s: embedding dim of the multimodal projector (%d) is not equal to that of LLaMA (%d). Make sure that you use the correct mmproj file.\n", __func__, n_embd_clip, n_embd_llm);
                 llama_free(ctx);
                 llama_free_model(model);
                 return false;
@@ -848,8 +865,8 @@ struct llama_server_context
 
                     slot_image img_sl;
                     img_sl.id = img.count("id") != 0 ? img["id"].get<int>() : slot->images.size();
-                    img_sl.bitmap = mtmd_helper_bitmap_init_from_buf(mctx, image_buffer.data(), image_buffer.size());
-                    if (img_sl.bitmap == nullptr)
+                    img_sl.img_data = clip_image_u8_init();
+                    if (!clip_image_load_from_bytes(image_buffer.data(), image_buffer.size(), img_sl.img_data))
                     {
                         LOG_ERR("%s: failed to load image, slot_id: %d, img_sl_id: %d",
                              __func__,
@@ -862,74 +879,50 @@ struct llama_server_context
                         {"slot_id",   slot->id},
                         {"img_sl_id", img_sl.id}
                     });
+                    img_sl.request_encode_image = true;
                     slot->images.push_back(img_sl);
                 }
-                // Translate the legacy [img-N] tags into mtmd media markers, in
-                // order, and collect the matching bitmaps in marker order so they
-                // line up with the markers passed to mtmd_tokenize(). The text after
-                // the last image stays in input_suffix and is decoded through the
-                // normal token path, so the sampling loop is unchanged.
-                // example: system prompt [img-102] user [img-103] describe [img-134]
+                // process prompt
+                // example: system prompt [img-102] user [img-103] describe [img-134] -> [{id: 102, prefix: 'system prompt '}, {id: 103, prefix: ' user '}, {id: 134, prefix: ' describe '}]}
                 if (slot->images.size() > 0 && !slot->prompt.is_array())
                 {
-                    const std::string marker = mtmd_default_marker();
                     std::string prompt = slot->prompt.get<std::string>();
-                    std::string built_prompt;
-                    std::vector<slot_image> ordered;
-                    size_t pos = 0, copy_from = 0;
+                    size_t pos = 0, begin_prefix = 0;
                     std::string pattern = "[img-";
-
-                    auto free_images = [&]() {
-                        for (slot_image &img : slot->images) {
-                            if (img.bitmap) {
-                                mtmd_bitmap_free(img.bitmap);
-                                img.bitmap = nullptr;
-                            }
-                        }
-                        slot->images.clear();
-                    };
-
                     while ((pos = prompt.find(pattern, pos)) != std::string::npos) {
-                        size_t tag_begin = pos;
+                        size_t end_prefix = pos;
                         pos += pattern.length();
                         size_t end_pos = prompt.find(']', pos);
-                        if (end_pos == std::string::npos) {
-                            break;
-                        }
-                        std::string image_id = prompt.substr(pos, end_pos - pos);
-                        try
+                        if (end_pos != std::string::npos)
                         {
-                            int img_id = std::stoi(image_id);
-                            bool found = false;
-                            for (slot_image &img : slot->images)
+                            std::string image_id = prompt.substr(pos, end_pos - pos);
+                            try
                             {
-                                if (img.id == img_id) {
-                                    found = true;
-                                    // text before this tag, then the media marker
-                                    built_prompt += prompt.substr(copy_from, tag_begin - copy_from);
-                                    built_prompt += marker;
-                                    copy_from = end_pos + 1;
-                                    ordered.push_back(img);
-                                    break;
+                                int img_id = std::stoi(image_id);
+                                bool found = false;
+                                for (slot_image &img : slot->images)
+                                {
+                                    if (img.id == img_id) {
+                                        found = true;
+                                        img.prefix_prompt = prompt.substr(begin_prefix, end_prefix - begin_prefix);
+                                        begin_prefix = end_pos + 1;
+                                        break;
+                                    }
                                 }
-                            }
-                            if (!found) {
-                                LOG("ERROR: Image with id: %i, not found.\n", img_id);
-                                free_images();
+                                if (!found) {
+                                    LOG("ERROR: Image with id: %i, not found.\n", img_id);
+                                    slot->images.clear();
+                                    return false;
+                                }
+                            } catch (const std::invalid_argument& e) {
+                                LOG("Invalid image number id in prompt\n");
+                                slot->images.clear();
                                 return false;
                             }
-                        } catch (const std::invalid_argument& e) {
-                            LOG("Invalid image number id in prompt\n");
-                            free_images();
-                            return false;
                         }
-                        pos = end_pos + 1;
                     }
-                    // bitmaps are consumed in marker order by mtmd_tokenize()
-                    slot->images = ordered;
-                    slot->mtmd_prompt = built_prompt;
                     slot->prompt = "";
-                    slot->params.input_suffix = prompt.substr(copy_from);
+                    slot->params.input_suffix = prompt.substr(begin_prefix);
                     slot->params.cache_prompt = false; // multimodal doesn't support cache prompt
                 }
             }
@@ -1183,10 +1176,21 @@ struct llama_server_context
 
     bool process_images(llama_client_slot &slot) const
     {
-        // With the mtmd pipeline, image encoding is no longer eager: the bitmaps
-        // are tokenized and encoded together with the surrounding text inside
-        // ingest_images() via mtmd_tokenize() + mtmd_helper_eval_chunks(). This
-        // just reports whether the slot carries any images to process.
+        for (slot_image &img : slot.images)
+        {
+            if (!img.request_encode_image)
+            {
+                continue;
+            }
+
+            if (!llava_image_embed_make_with_clip_img(clp_ctx, params.n_threads, img.img_data, &img.image_embedding, &img.image_tokens)) {
+                LOG("Error processing the given image");
+                return false;
+            }
+
+            img.request_encode_image = false;
+        }
+
         return slot.images.size() > 0;
     }
 
@@ -1431,70 +1435,69 @@ struct llama_server_context
         }
     }
 
-    // Tokenize the multimodal prompt (text interleaved with media markers) together
-    // with the slot's bitmaps, then decode the resulting chunks into the llama
-    // context via the high-level mtmd helper. The helper runs llama_decode() on the
-    // text chunks and mtmd_encode() + llama_decode() on the image chunks, handling
-    // batching and any pre/post decode setup (e.g. non-causal attention for gemma3).
-    // Advances slot.n_past by the number of positions consumed, then leaves the
-    // post-image suffix tokens in `batch` so the normal decode + sampling loop
-    // produces the first generated token.
+    // for multiple images processing
     bool ingest_images(llama_client_slot &slot, int n_batch)
     {
-        if (mctx == nullptr)
-        {
-            LOG("%s : multimodal context is not initialized\n", __func__);
-            return false;
-        }
+        int image_idx = 0;
 
-        // bitmaps stay owned by slot.images (freed on reset()); pass non-owning ptrs
-        std::vector<const mtmd_bitmap *> bitmaps;
-        bitmaps.reserve(slot.images.size());
-        for (const slot_image &img : slot.images)
+        while (image_idx < (int) slot.images.size())
         {
-            bitmaps.push_back(img.bitmap);
-        }
+            slot_image &img = slot.images[image_idx];
 
-        mtmd_input_text inp_txt;
-        inp_txt.text          = slot.mtmd_prompt.c_str();
-        inp_txt.add_special   = add_bos_token;
-        inp_txt.parse_special = true;
+            // process prefix prompt
+            for (int32_t i = 0; i < (int32_t) batch.n_tokens; i += n_batch)
+            {
+                const int32_t n_tokens = std::min(n_batch, (int32_t) (batch.n_tokens - i));
+                llama_batch batch_view = {
+                    n_tokens,
+                    batch.token    + i,
+                    nullptr,
+                    batch.pos      + i,
+                    batch.n_seq_id + i,
+                    batch.seq_id   + i,
+                    batch.logits   + i,
+                };
+                if (llama_decode(ctx, batch_view))
+                {
+                    LOG("%s : failed to eval\n", __func__);
+                    return false;
+                }
+            }
 
-        mtmd::input_chunks chunks(mtmd_input_chunks_init());
-        int32_t res = mtmd_tokenize(mctx,
-                                    chunks.ptr.get(),
-                                    &inp_txt,
-                                    bitmaps.data(),
-                                    bitmaps.size());
-        if (res != 0)
-        {
-            LOG("%s : failed to tokenize multimodal prompt, res = %d\n", __func__, res);
-            return false;
-        }
+            // process image with llm
+            for (int i = 0; i < img.image_tokens; i += n_batch)
+            {
+                int n_eval = img.image_tokens - i;
+                if (n_eval > n_batch)
+                {
+                    n_eval = n_batch;
+                }
 
-        const llama_pos start_pos = (llama_pos) system_tokens.size() + slot.n_past;
-        llama_pos new_n_past = start_pos;
-        if (mtmd_helper_eval_chunks(mctx,
-                                    ctx,
-                                    chunks.ptr.get(),
-                                    start_pos,
-                                    slot.id,
-                                    n_batch,
-                                    /*logits_last=*/ false,
-                                    &new_n_past) != 0)
-        {
-            LOG("%s : failed to eval multimodal chunks\n", __func__);
-            return false;
-        }
-        slot.n_past += (int32_t) (new_n_past - start_pos);
+                const int n_embd = llama_model_n_embd(model);
+                float * embd = img.image_embedding + i * n_embd;
+                llava_embd_batch llava_batch = llava_embd_batch(embd, n_eval, slot.n_past, 0);
+                if (llama_decode(ctx, llava_batch.batch))
+                {
+                    LOG("%s : failed to eval image\n", __func__);
+                    return false;
+                }
+                slot.n_past += n_eval;
+            }
+            image_idx++;
 
-        // queue the post-image suffix text for the normal decode + sampling path
-        common_batch_clear(batch);
-        std::vector<llama_token> suffix_tokens = tokenize(slot.params.input_suffix, false);
-        for (llama_token tok : suffix_tokens)
-        {
-            common_batch_add(batch, tok, system_tokens.size() + slot.n_past, { slot.id }, false);
-            slot.n_past += 1;
+            common_batch_clear(batch);
+
+            // append prefix of next image
+            const auto json_prompt = (image_idx >= (int) slot.images.size()) ?
+                slot.params.input_suffix : // no more images, then process suffix prompt
+                (json)(slot.images[image_idx].prefix_prompt);
+
+            std::vector<llama_token> append_tokens = tokenize(json_prompt, false); // has next image
+            for (int i = 0; i < (int) append_tokens.size(); ++i)
+            {
+                common_batch_add(batch, append_tokens[i], system_tokens.size() + slot.n_past, { slot.id }, true);
+                slot.n_past += 1;
+            }
         }
 
         return true;
@@ -1881,11 +1884,8 @@ struct llama_server_context
 
                     const bool has_images = process_images(slot);
 
-                    // For the multimodal path the whole pre-image / inter-image text is
-                    // tokenized and decoded inside ingest_images() via mtmd, so no prefix
-                    // tokens are queued here; the post-image suffix is appended by
-                    // ingest_images() for the normal decode + sampling loop.
-                    std::vector<llama_token> prefix_tokens = has_images ? std::vector<llama_token>() : prompt_tokens;
+                    // process the prefix of first image
+                    std::vector<llama_token> prefix_tokens = has_images ? tokenize(slot.images[0].prefix_prompt, add_bos_token) : prompt_tokens;
 
                     int32_t slot_npast = slot.n_past_se > 0 ? slot.n_past_se : slot.n_past;
 
@@ -2412,33 +2412,7 @@ static void params_parse(const backend::ModelOptions* request,
 
 // GRPC Server start
 class BackendServiceImpl final : public backend::Backend::Service {
-private:
-  // The ModelOptions.Model this process was loaded with. Compared against
-  // PredictOptions.ModelIdentity so a request that reached us through a stale
-  // distributed route is rejected instead of answered from the wrong model
-  // (#10952).
-  std::string loaded_model_identity;
-
 public:
-  // checkModelIdentity mirrors pkg/grpc/server.go and
-  // backend/python/common/model_identity.py. Either side being empty means
-  // "skip": the request side is empty for a controller that predates the field,
-  // and the loaded side is empty when such a controller performed the load. A
-  // false rejection is worse than the miss it prevents.
-  grpc::Status checkModelIdentity(const backend::PredictOptions* request) {
-    if (request == nullptr || request->modelidentity().empty()) {
-      return grpc::Status::OK;
-    }
-    if (loaded_model_identity.empty() || loaded_model_identity == request->modelidentity()) {
-      return grpc::Status::OK;
-    }
-    // NOT_FOUND plus this exact sentinel is the cross-language contract the
-    // router matches on (grpcerrors.ModelMismatchSentinel).
-    return grpc::Status(grpc::StatusCode::NOT_FOUND,
-                        "ik-llama-cpp: model identity mismatch: loaded \"" + loaded_model_identity +
-                        "\", requested \"" + request->modelidentity() + "\"");
-  }
-
   grpc::Status Health(ServerContext* context, const backend::HealthMessage* request, backend::Reply* reply) {
     // Implement Health RPC
     reply->set_message("OK");
@@ -2464,12 +2438,9 @@ public:
     result->set_message("Loading succeeded");
     result->set_success(true);
     loaded_model = true;
-    loaded_model_identity = request->model();
     return Status::OK;
   }
   grpc::Status PredictStream(grpc::ServerContext* context, const backend::PredictOptions* request, grpc::ServerWriter<backend::Reply>* writer) override {
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         json data = parse_options(true, request, llama);
         const int task_id = llama.queue_tasks.get_new_id();
         llama.queue_results.add_waiting_task_id(task_id);
@@ -2524,8 +2495,6 @@ public:
 
 
     grpc::Status Predict(ServerContext* context, const backend::PredictOptions* request, backend::Reply* reply) {
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         json data = parse_options(false, request, llama);
         const int task_id = llama.queue_tasks.get_new_id();
         llama.queue_results.add_waiting_task_id(task_id);
@@ -2563,8 +2532,6 @@ public:
 
     /// https://github.com/ggerganov/llama.cpp/blob/aa2341298924ac89778252015efcb792f2df1e20/examples/server/server.cpp#L2969
     grpc::Status Embedding(ServerContext* context, const backend::PredictOptions* request, backend::EmbeddingResult* embeddingResult) {
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         json data = parse_options(false, request, llama);
         const int task_id = llama.queue_tasks.get_new_id();
         llama.queue_results.add_waiting_task_id(task_id);
@@ -2589,8 +2556,6 @@ public:
     }
 
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response){
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
          json data = parse_options(false, request, llama);
 
          std::vector<llama_token> tokens = llama.tokenize(data["prompt"],false);

backend/cpp/ik-llama-cpp/patches/0002-clip-ggml-quantize-chunk-user-data.patch

@@ -0,0 +1,11 @@
+--- a/examples/llava/clip.cpp
++++ b/examples/llava/clip.cpp
+@@ -2494,7 +2494,7 @@
+             }
+             new_data = work.data();
+
+-            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr);
++            new_size = ggml_quantize_chunk(new_type, f32_data, new_data, 0, n_elms/cur->ne[0], cur->ne[0], nullptr, nullptr);
+         } else {
+             new_type = cur->type;
+             new_data = cur->data;

backend/cpp/ik-llama-cpp/prepare.sh

@@ -17,9 +17,28 @@ cp -r grpc-server.cpp llama.cpp/examples/grpc-server/
 cp -r utils.hpp llama.cpp/examples/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/examples/grpc-server/
 
-## Multimodal support is provided by the `mtmd` library target (examples/mtmd/),
-## which the grpc-server links and includes directly. No source copy is needed:
-## clip/llava were pruned upstream and the high-level mtmd_* API is used instead.
+## Copy clip/llava files for multimodal support (built as myclip library)
+cp -rfv llama.cpp/examples/llava/clip.h llama.cpp/examples/grpc-server/clip.h
+cp -rfv llama.cpp/examples/llava/clip.cpp llama.cpp/examples/grpc-server/clip.cpp
+cp -rfv llama.cpp/examples/llava/llava.cpp llama.cpp/examples/grpc-server/llava.cpp
+# Prepend llama.h include to llava.h
+echo '#include "llama.h"' > llama.cpp/examples/grpc-server/llava.h
+cat llama.cpp/examples/llava/llava.h >> llama.cpp/examples/grpc-server/llava.h
+# Copy clip-impl.h if it exists
+if [ -f llama.cpp/examples/llava/clip-impl.h ]; then
+    cp -rfv llama.cpp/examples/llava/clip-impl.h llama.cpp/examples/grpc-server/clip-impl.h
+fi
+# Copy stb_image.h
+if [ -f llama.cpp/vendor/stb/stb_image.h ]; then
+    cp -rfv llama.cpp/vendor/stb/stb_image.h llama.cpp/examples/grpc-server/stb_image.h
+elif [ -f llama.cpp/common/stb_image.h ]; then
+    cp -rfv llama.cpp/common/stb_image.h llama.cpp/examples/grpc-server/stb_image.h
+fi
+
+## Fix API compatibility in llava.cpp (llama_n_embd -> llama_model_n_embd)
+if [ -f llama.cpp/examples/grpc-server/llava.cpp ]; then
+    sed -i 's/llama_n_embd(/llama_model_n_embd(/g' llama.cpp/examples/grpc-server/llava.cpp
+fi
 
 set +e
 if grep -q "grpc-server" llama.cpp/examples/CMakeLists.txt; then

backend/cpp/ik-llama-cpp/run.sh

@@ -2,7 +2,7 @@
 set -ex
 
 # Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath "$0")")
+CURDIR=$(dirname "$(realpath $0)")
 
 cd /
 
@@ -13,28 +13,28 @@ grep -e "flags" /proc/cpuinfo | head -1
 # ik_llama.cpp requires AVX2 — default to avx2 binary
 BINARY=ik-llama-cpp-avx2
 
-if [ -e "$CURDIR"/ik-llama-cpp-fallback ] && ! grep -q -e "\savx2\s" /proc/cpuinfo ; then
+if [ -e $CURDIR/ik-llama-cpp-fallback ] && ! grep -q -e "\savx2\s" /proc/cpuinfo ; then
 	echo "CPU:    AVX2   NOT found, using fallback"
 	BINARY=ik-llama-cpp-fallback
 fi
 
 # Extend ld library path with the dir where this script is located/lib
 if [ "$(uname)" == "Darwin" ]; then
-	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
-	#export DYLD_FALLBACK_LIBRARY_PATH="$CURDIR"/lib:$DYLD_FALLBACK_LIBRARY_PATH
+	export DYLD_LIBRARY_PATH=$CURDIR/lib:$DYLD_LIBRARY_PATH
+	#export DYLD_FALLBACK_LIBRARY_PATH=$CURDIR/lib:$DYLD_FALLBACK_LIBRARY_PATH
 else
-	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+	export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
 fi
 
 # If there is a lib/ld.so, use it
-if [ -f "$CURDIR"/lib/ld.so ]; then
+if [ -f $CURDIR/lib/ld.so ]; then
 	echo "Using lib/ld.so"
 	echo "Using binary: $BINARY"
-	exec "$CURDIR"/lib/ld.so "$CURDIR"/$BINARY "$@"
+	exec $CURDIR/lib/ld.so $CURDIR/$BINARY "$@"
 fi
 
 echo "Using binary: $BINARY"
-exec "$CURDIR"/$BINARY "$@"
+exec $CURDIR/$BINARY "$@"
 
 # We should never reach this point, however just in case we do, run fallback
-exec "$CURDIR"/ik-llama-cpp-fallback "$@"
+exec $CURDIR/ik-llama-cpp-fallback "$@"

backend/cpp/ik-llama-cpp/utils.hpp

@@ -11,12 +11,9 @@
 
 #include "json.hpp"
 
-#include "mtmd.h"
+#include "clip.h"
 
-// mtmd.h and ik_llama's entire server/common stack (chat.h, server-common.h,
-// server-task.h, ...) declare `using json = nlohmann::ordered_json`, so match it
-// here: a plain `nlohmann::json` alias collides with mtmd.h's at global scope.
-using json = nlohmann::ordered_json;
+using json = nlohmann::json;
 
 extern bool server_verbose;
 
@@ -114,12 +111,13 @@ struct slot_image
 {
     int32_t id;
 
-    // mtmd bitmap (image/audio) decoded from the request buffer. Owned by the
-    // slot; freed via mtmd_bitmap_free() on reset. The high-level mtmd pipeline
-    // (mtmd_tokenize + mtmd_helper_eval_chunks) consumes these directly, so the
-    // legacy eager-encode fields (embedding/tokens) and per-image prefix prompt
-    // are no longer needed.
-    mtmd_bitmap * bitmap = nullptr;
+    bool request_encode_image = false;
+    float * image_embedding = nullptr;
+    int32_t image_tokens = 0;
+
+    clip_image_u8 * img_data;
+
+    std::string prefix_prompt; // before of this image
 };
 
 // completion token output with probabilities

backend/cpp/llama-cpp/CMakeLists.txt

@@ -50,13 +50,8 @@ add_custom_command(
         "${hw_proto}"
       DEPENDS "${hw_proto}")
 
-# hw_grpc_proto: force STATIC. Under the CPU_ALL_VARIANTS build BUILD_SHARED_LIBS=ON
-# (ggml/llama become shared), which would otherwise make this glue library a DSO. As a
-# DSO it references the hidden-visibility symbols in the static libprotobuf.a, which the
-# linker cannot satisfy ("hidden symbol ... in libprotobuf.a is referenced by DSO").
-# Keeping it STATIC links protobuf/gRPC directly into the grpc-server executable while
-# only ggml/llama stay shared. No effect on the static variants (already BUILD_SHARED_LIBS=OFF).
-add_library(hw_grpc_proto STATIC
+# hw_grpc_proto
+add_library(hw_grpc_proto
   ${hw_grpc_srcs}
   ${hw_grpc_hdrs}
   ${hw_proto_srcs}
@@ -87,27 +82,3 @@ target_compile_features(${TARGET} PRIVATE cxx_std_11)
 if(TARGET BUILD_INFO)
   add_dependencies(${TARGET} BUILD_INFO)
 endif()
-
-# Unit test for the message-content normalization helper (message_content.h).
-# Off by default so the normal backend build is untouched; enable with
-# -DLLAMA_GRPC_BUILD_TESTS=ON and run via ctest. It reuses llama.cpp's vendored
-# <nlohmann/json.hpp> (propagated by the common helpers library) so it has no
-# extra dependency beyond what the backend already builds against.
-option(LLAMA_GRPC_BUILD_TESTS "Build grpc-server unit tests" OFF)
-if(LLAMA_GRPC_BUILD_TESTS)
-    enable_testing()
-    add_executable(message_content_test message_content_test.cpp message_content.h)
-    target_include_directories(message_content_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
-    target_link_libraries(message_content_test PRIVATE ${_LLAMA_COMMON_TARGET})
-    target_compile_features(message_content_test PRIVATE cxx_std_17)
-    add_test(NAME message_content_test COMMAND message_content_test)
-
-    # Parent-death watcher test (parent_watch.h) — standard library only, but
-    # needs a threading runtime for std::thread.
-    find_package(Threads REQUIRED)
-    add_executable(parent_watch_test parent_watch_test.cpp parent_watch.h)
-    target_include_directories(parent_watch_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
-    target_link_libraries(parent_watch_test PRIVATE Threads::Threads)
-    target_compile_features(parent_watch_test PRIVATE cxx_std_17)
-    add_test(NAME parent_watch_test COMMAND parent_watch_test)
-endif()

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=571d0d540df04f25298d0e159e520d9fc62ed121
+LLAMA_VERSION?=e475fa2b5f9fb50c3d6fc3e7c6fdf1e004465b62
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=
@@ -10,16 +10,8 @@ TARGET?=--target grpc-server
 JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
 ARCH?=$(shell uname -m)
 
-# Shared libs default to OFF: we link static gRPC and the avx/avx2/avx512/fallback
-# variants are fully static. The CPU_ALL_VARIANTS build flips SHARED_LIBS=ON (ggml/llama
-# become shared so the dynamic CPU backends work; gRPC stays static via its imported
-# targets). SHARED_LIBS is a make variable, not an appended -D, so it survives the
-# recursive sub-make into the VARIANT build dir (which re-parses this Makefile) instead
-# of being re-clobbered by a second -DBUILD_SHARED_LIBS=OFF. EXTRA_CMAKE_ARGS is the hook
-# the CPU_ALL_VARIANTS target uses to inject -DGGML_BACKEND_DL/-DGGML_CPU_ALL_VARIANTS.
-SHARED_LIBS?=OFF
-EXTRA_CMAKE_ARGS?=
-CMAKE_ARGS+=-DBUILD_SHARED_LIBS=$(SHARED_LIBS) -DLLAMA_CURL=OFF $(EXTRA_CMAKE_ARGS)
+# Disable Shared libs as we are linking on static gRPC and we can't mix shared and static
+CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=OFF
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 ifeq ($(NATIVE),false)
@@ -128,39 +120,15 @@ llama-cpp-fallback: llama.cpp
 	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) VARIANT="llama-cpp-fallback-build" build-llama-cpp-grpc-server
 	cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-fallback-build/grpc-server llama-cpp-fallback
 
-# Single-build CPU backend using ggml's CPU_ALL_VARIANTS. Produces ONE grpc-server
-# plus a set of dlopen-able libggml-cpu-*.so (sandybridge/haswell/skylakex/...) that
-# ggml's backend registry selects from at runtime by probing host CPU features.
-# Replaces the avx/avx2/avx512/fallback multi-binary build on x86.
-#
-# CPU_ALL_VARIANTS requires GGML_BACKEND_DL, which requires BUILD_SHARED_LIBS=ON, so we
-# pass SHARED_LIBS=ON and the DL flags as make variables (NOT pre-expanded into the
-# CMAKE_ARGS env string): command-line make variables propagate through every recursive
-# sub-make, so the deepest VARIANT-dir build computes BUILD_SHARED_LIBS=ON consistently.
-# Only ggml/llama go shared - gRPC is found via its static imported targets, so the
-# grpc-server binary keeps static gRPC and only dynamically links ggml.
-#
-# TARGET adds "ggml": the per-microarch backends are runtime-dlopened, not link deps of
-# grpc-server, so they only build because each is an add_dependencies() of the ggml target.
-llama-cpp-cpu-all: llama.cpp
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp $(CURRENT_MAKEFILE_DIR)/../llama-cpp-cpu-all-build
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-cpu-all-build purge
-	$(info ${GREEN}I llama-cpp build info:cpu-all-variants${RESET})
-	$(MAKE) SHARED_LIBS=ON EXTRA_CMAKE_ARGS="-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON" TARGET="--target grpc-server --target ggml" VARIANT="llama-cpp-cpu-all-build" build-llama-cpp-grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-cpu-all-build/grpc-server llama-cpp-cpu-all
-	rm -rf ggml-shared-libs && mkdir -p ggml-shared-libs
-	find $(CURRENT_MAKEFILE_DIR)/../llama-cpp-cpu-all-build/llama.cpp/build \( -name '*.so*' -o -name '*.dylib' \) -exec cp -av {} ggml-shared-libs/ \;
-	@echo "Collected ggml shared backends:" && ls -la ggml-shared-libs/
-
 llama-cpp-grpc: llama.cpp
 	cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build purge
 	$(info ${GREEN}I llama-cpp build info:grpc${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" TARGET="--target grpc-server --target ggml-rpc-server" $(MAKE) VARIANT="llama-cpp-grpc-build" build-llama-cpp-grpc-server
+	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" TARGET="--target grpc-server --target rpc-server" $(MAKE) VARIANT="llama-cpp-grpc-build" build-llama-cpp-grpc-server
 	cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build/grpc-server llama-cpp-grpc
 
 llama-cpp-rpc-server: llama-cpp-grpc
-	cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build/llama.cpp/build/bin/ggml-rpc-server llama-cpp-rpc-server
+	cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build/llama.cpp/build/bin/rpc-server llama-cpp-rpc-server
 
 llama.cpp:
 	mkdir -p llama.cpp

backend/cpp/llama-cpp/grpc-server.cpp

@@ -30,19 +30,6 @@
 #define LOCALAI_HAS_SERVER_SCHEMA 1
 #include "server-schema.cpp"
 #endif
-// server-stream.cpp exists only in llama.cpp after the upstream refactor that
-// added the SSE stream-resumption layer (stream_session/stream_pipe_producer).
-// server-context.cpp calls into it (spipe->cleanup(), stream_aware_should_stop,
-// stream_session_attach_pipe), so its definitions must be part of this
-// translation unit or the link fails with "undefined reference to
-// stream_pipe_producer::cleanup()". The file is self-contained (its only
-// external symbols come from server-common, already pulled in above) and the
-// http route-handler factories it also defines are unused here but harmless.
-// __has_include keeps the source compatible with older pins/forks that predate
-// the split.
-#if __has_include("server-stream.cpp")
-#include "server-stream.cpp"
-#endif
 #include "server-context.cpp"
 
 // LocalAI
@@ -50,9 +37,7 @@
 #include "backend.pb.h"
 #include "backend.grpc.pb.h"
 #include "common.h"
-#include "arg.h"
 #include "chat-auto-parser.h"
-#include "message_content.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
 #include <grpcpp/grpcpp.h>
@@ -75,8 +60,6 @@
 #include <windows.h>
 #endif
 
-#include "parent_watch.h" // best-effort parent-death backstop (see header)
-
 
 using grpc::Server;
 using grpc::ServerBuilder;
@@ -609,28 +592,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
     params.checkpoint_min_step = 256;
 #endif
 
-    // Raw upstream llama-server flags collected from any option entry that
-    // starts with '-'. Applied once after the loop via common_params_parse.
-    std::vector<std::string> extra_argv;
-
-    auto add_device_options = [&](const std::string & devices) {
-        const std::regex regex{ R"([,]+)" };
-        std::sregex_token_iterator it{ devices.begin(), devices.end(), regex, -1 };
-        std::vector<std::string> split_arg{ it, {} };
-
-        for (std::string device : split_arg) {
-            const auto start = device.find_first_not_of(" \t\n\r");
-            if (start == std::string::npos) {
-                continue;
-            }
-            const auto end = device.find_last_not_of(" \t\n\r");
-            device = device.substr(start, end - start + 1);
-
-            extra_argv.push_back("--device");
-            extra_argv.push_back(device);
-        }
-    };
-
      // decode options. Options are in form optname:optvale, or if booleans only optname.
     for (int i = 0; i < request->options_size(); i++) {
         std::string opt = request->options(i);
@@ -762,10 +723,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
                 params.no_op_offload = false;
             }
-        } else if (!strcmp(optname, "device") || !strcmp(optname, "devices")) {
-            if (optval != NULL) {
-                add_device_options(optval_str);
-            }
         } else if (!strcmp(optname, "split_mode") || !strcmp(optname, "sm")) {
             // Accepts: none | layer | row | tensor (the latter requires a llama.cpp build
             // that includes ggml-org/llama.cpp#19378, FlashAttention enabled, and KV-cache
@@ -1123,31 +1080,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                 } catch (...) {}
             }
 
-        // --- main model MoE on CPU (upstream --cpu-moe / --n-cpu-moe) ---
-        } else if (!strcmp(optname, "cpu_moe")) {
-            // Bool-style flag: keep all MoE expert weights on CPU.
-            const bool enable = (optval == NULL) ||
-                optval_str == "true" || optval_str == "1" || optval_str == "yes" ||
-                optval_str == "on" || optval_str == "enabled";
-            if (enable) {
-                params.tensor_buft_overrides.push_back(llm_ffn_exps_cpu_override());
-            }
-        } else if (!strcmp(optname, "n_cpu_moe")) {
-            if (optval != NULL) {
-                try {
-                    int n = std::stoi(optval_str);
-                    if (n < 0) n = 0;
-                    // Keep override-name storage alive for the lifetime of the
-                    // params struct (mirrors upstream arg.cpp's function-local static).
-                    static std::list<std::string> buft_overrides_main;
-                    for (int i = 0; i < n; ++i) {
-                        buft_overrides_main.push_back(llm_ffn_exps_block_regex(i));
-                        params.tensor_buft_overrides.push_back(
-                            {buft_overrides_main.back().c_str(), ggml_backend_cpu_buffer_type()});
-                    }
-                } catch (...) {}
-            }
-
         // --- draft model tensor buffer overrides (upstream --spec-draft-override-tensor) ---
         } else if (!strcmp(optname, "draft_override_tensor") || !strcmp(optname, "spec_draft_override_tensor")) {
             // Format: <tensor regex>=<buffer type>,<tensor regex>=<buffer type>,...
@@ -1179,30 +1111,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                 else { cur.push_back(c); }
             }
             if (!cur.empty()) flush(cur);
-
-        // --- generic passthrough: any entry starting with '-' is a raw
-        //     upstream llama-server flag, forwarded verbatim to the parser. ---
-        } else if (optname[0] == '-') {
-            std::string flag = optname;
-            // These flags make upstream's parser exit() (printing usage /
-            // completion), which would kill the backend process. Skip them.
-            if (flag == "-h" || flag == "--help" || flag == "--usage" ||
-                flag == "--version" || flag == "--license" ||
-                flag == "--list-devices" || flag == "-cl" ||
-                flag == "--cache-list" ||
-                flag.rfind("--completion", 0) == 0) {
-                fprintf(stderr,
-                    "[llama-cpp] ignoring passthrough flag that would exit: %s\n",
-                    flag.c_str());
-            } else {
-                extra_argv.push_back(flag);
-                // Preserve the whole value after the first ':' so embedded
-                // colons (e.g. host:port) survive strtok's truncation of optval.
-                auto colon = opt.find(':');
-                if (colon != std::string::npos) {
-                    extra_argv.push_back(opt.substr(colon + 1));
-                }
-            }
         }
     }
 
@@ -1238,6 +1146,27 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         }
     }
 
+    if (!params.kv_overrides.empty()) {
+        params.kv_overrides.emplace_back();
+        params.kv_overrides.back().key[0] = 0;
+    }
+
+    // tensor_buft_overrides sentinel termination (mirrors upstream common/arg.cpp).
+    // Real entries are pushed during option parsing; here we pad/terminate so the
+    // model loader sees back().pattern == nullptr (GGML_ASSERT at common.cpp:1543)
+    // and so llama_params_fit has the placeholder slots it requires.
+    {
+        const size_t ntbo = llama_max_tensor_buft_overrides();
+        while (params.tensor_buft_overrides.size() < ntbo) {
+            params.tensor_buft_overrides.push_back({nullptr, nullptr});
+        }
+    }
+    // Terminate the draft tensor_buft_overrides list with a sentinel, mirroring
+    // the main-model handling above.
+    if (!params.speculative.draft.tensor_buft_overrides.empty()) {
+        params.speculative.draft.tensor_buft_overrides.push_back({nullptr, nullptr});
+    }
+
     // TODO: Add yarn
 
     if (!request->tensorsplit().empty()) {
@@ -1330,69 +1259,6 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             params.sampling.grammar_triggers.push_back(std::move(trigger));
         }
     }
-
-    // Apply any raw upstream flags last so an explicit passthrough flag wins
-    // over the LocalAI-resolved field it maps to (e.g. --ctx-size beats
-    // context_size). This is the same parser llama-server itself uses.
-    if (!extra_argv.empty()) {
-        // common_params_parser_init resets a few fields for the SERVER example
-        // (n_parallel -> -1, use_color). Snapshot n_parallel so an unrelated
-        // passthrough flag can't silently clobber LocalAI's resolved value.
-        const int saved_n_parallel = params.n_parallel;
-
-        std::vector<char *> argv;
-        std::string prog = "llama-server";
-        argv.push_back(prog.data());
-        for (auto & a : extra_argv) {
-            argv.push_back(a.data());
-        }
-
-        // ctx_arg.params is a reference, so this overlays the given flags onto
-        // `params` in place. Returns false on a recoverable parse error (and
-        // self-restores params); may exit() on a hard error, exactly as
-        // passing the same bad flag to llama-server would.
-        if (!common_params_parse((int)argv.size(), argv.data(), params,
-                                 LLAMA_EXAMPLE_SERVER)) {
-            fprintf(stderr,
-                "[llama-cpp] failed to parse passthrough options; ignoring them\n");
-        }
-
-        // Restore n_parallel unless a passthrough flag explicitly set it
-        // (parser_init's reset sentinel for SERVER is -1).
-        if (params.n_parallel == -1) {
-            params.n_parallel = saved_n_parallel;
-        }
-    }
-
-    // Terminate/pad the override vectors only after BOTH the named-option loop
-    // and the generic passthrough (common_params_parse above) have pushed their
-    // real entries, so back() is the null sentinel the model loader asserts on.
-    // Running these before the passthrough let a passthrough flag (--cpu-moe,
-    // --override-tensor, --override-kv, ...) append a real entry after the
-    // sentinel: a GGML_ASSERT crash for tensor_buft_overrides, a silent drop for
-    // kv_overrides. Double-termination is harmless (the while is a no-op if the
-    // passthrough parse already padded; an extra trailing null is ignored).
-
-    if (!params.kv_overrides.empty()) {
-        params.kv_overrides.emplace_back();
-        params.kv_overrides.back().key[0] = 0;
-    }
-
-    // tensor_buft_overrides sentinel termination (mirrors upstream common/arg.cpp).
-    // Real entries are pushed during option parsing; here we pad/terminate so the
-    // model loader sees back().pattern == nullptr (GGML_ASSERT at common.cpp:1543)
-    // and so llama_params_fit has the placeholder slots it requires.
-    {
-        const size_t ntbo = llama_max_tensor_buft_overrides();
-        while (params.tensor_buft_overrides.size() < ntbo) {
-            params.tensor_buft_overrides.push_back({nullptr, nullptr});
-        }
-    }
-    // Terminate the draft tensor_buft_overrides list with a sentinel, mirroring
-    // the main-model handling above.
-    if (!params.speculative.draft.tensor_buft_overrides.empty()) {
-        params.speculative.draft.tensor_buft_overrides.push_back({nullptr, nullptr});
-    }
 }
 
 
@@ -1401,40 +1267,10 @@ class BackendServiceImpl final : public backend::Backend::Service {
 private:
     server_context& ctx_server;
     common_params params_base; // Store copy of params_base, set after model load
-    // The ModelOptions.Model this process was loaded with. Compared against
-    // PredictOptions.ModelIdentity so a request that reached us through a stale
-    // distributed route is rejected instead of answered from the wrong model
-    // (#10952). Written under LoadModel, read by the inference RPCs.
-    std::string loaded_model_identity;
 
 public:
     BackendServiceImpl(server_context& ctx) : ctx_server(ctx) {}
 
-    // checkModelIdentity mirrors pkg/grpc/server.go and
-    // backend/python/common/model_identity.py. Either side being empty means
-    // "skip": the request side is empty for a controller that predates the
-    // field and for the synthetic PredictOptions this server builds internally
-    // for ASR, and the loaded side is empty when such a controller performed
-    // the load. A false rejection is worse than the miss it prevents.
-    // Templated over the request type: every guarded request message exposes
-    // modelidentity(), and one body keeps the rule identical across modalities
-    // rather than repeating it per RPC.
-    template <typename Request>
-    grpc::Status checkModelIdentity(const Request* request) {
-        if (request == nullptr || request->modelidentity().empty()) {
-            return grpc::Status::OK;
-        }
-        if (loaded_model_identity.empty() || loaded_model_identity == request->modelidentity()) {
-            return grpc::Status::OK;
-        }
-        // NOT_FOUND plus this exact sentinel is the cross-language contract the
-        // router matches on (grpcerrors.ModelMismatchSentinel). The code alone
-        // is not enough: NOT_FOUND is returned for unrelated reasons elsewhere.
-        return grpc::Status(grpc::StatusCode::NOT_FOUND,
-                            "llama-cpp: model identity mismatch: loaded \"" + loaded_model_identity +
-                            "\", requested \"" + request->modelidentity() + "\"");
-    }
-
     grpc::Status Health(ServerContext* context, const backend::HealthMessage* /*request*/, backend::Reply* reply) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
@@ -1565,7 +1401,6 @@ public:
         result->set_message("Loading succeeded");
         result->set_success(true);
         loaded_model = true;
-        loaded_model_identity = request->model();
         // Store copy of params_base for use in parse_options and other methods
         params_base = params;
 
@@ -1647,8 +1482,6 @@ public:
     grpc::Status PredictStream(grpc::ServerContext* context, const backend::PredictOptions* request, grpc::ServerWriter<backend::Reply>* writer) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -1687,20 +1520,242 @@ public:
 
                 for (int i = 0; i < request->messages_size(); i++) {
                     const auto& msg = request->messages(i);
-                    llama_grpc::ReconstructedMessageInput rin;
-                    rin.role = msg.role();
-                    rin.content = msg.content();
-                    rin.name = msg.name();
-                    rin.tool_call_id = msg.tool_call_id();
-                    rin.reasoning_content = msg.reasoning_content();
-                    rin.tool_calls = msg.tool_calls();
-                    rin.is_last_user_msg = (i == last_user_msg_idx);
-                    if (rin.is_last_user_msg) {
-                        for (int j = 0; j < request->images_size(); j++) rin.images.push_back(request->images(j));
-                        for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j));
-                        for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j));
+                    json msg_json;
+                    msg_json["role"] = msg.role();
+
+                    bool is_last_user_msg = (i == last_user_msg_idx);
+                    bool has_images_or_audio = (request->images_size() > 0 || request->audios_size() > 0 || request->videos_size() > 0);
+
+                    // Handle content - can be string, null, or array
+                    // For multimodal content, we'll embed images/audio from separate fields
+                    if (!msg.content().empty()) {
+                        // Try to parse content as JSON to see if it's already an array
+                        json content_val;
+                        try {
+                            content_val = json::parse(msg.content());
+                            // Handle null values - convert to empty string to avoid template errors
+                            if (content_val.is_null()) {
+                                content_val = "";
+                            }
+                        } catch (const json::parse_error&) {
+                            // Not JSON, treat as plain string
+                            content_val = msg.content();
+                        }
+
+                        // If content is an object (e.g., from tool call failures), convert to string
+                        if (content_val.is_object()) {
+                            content_val = content_val.dump();
+                        }
+
+                        // If content is a string and this is the last user message with images/audio, combine them
+                        if (content_val.is_string() && is_last_user_msg && has_images_or_audio) {
+                            json content_array = json::array();
+                            // Add text first
+                            content_array.push_back({{"type", "text"}, {"text", content_val.get<std::string>()}});
+                            // Add images
+                            if (request->images_size() > 0) {
+                                for (int j = 0; j < request->images_size(); j++) {
+                                    json image_chunk;
+                                    image_chunk["type"] = "image_url";
+                                    json image_url;
+                                    image_url["url"] = "data:image/jpeg;base64," + request->images(j);
+                                    image_chunk["image_url"] = image_url;
+                                    content_array.push_back(image_chunk);
+                                }
+                            }
+                            // Add audios
+                            if (request->audios_size() > 0) {
+                                for (int j = 0; j < request->audios_size(); j++) {
+                                    json audio_chunk;
+                                    audio_chunk["type"] = "input_audio";
+                                    json input_audio;
+                                    input_audio["data"] = request->audios(j);
+                                    input_audio["format"] = "wav"; // default, could be made configurable
+                                    audio_chunk["input_audio"] = input_audio;
+                                    content_array.push_back(audio_chunk);
+                                }
+                            }
+                            if (request->videos_size() > 0) {
+                                for (int j = 0; j < request->videos_size(); j++) {
+                                    json video_chunk;
+                                    video_chunk["type"] = "input_video";
+                                    json input_video;
+                                    input_video["data"] = request->videos(j);
+                                    video_chunk["input_video"] = input_video;
+                                    content_array.push_back(video_chunk);
+                                }
+                            }
+                            msg_json["content"] = content_array;
+                        } else {
+                            // Use content as-is (already array or not last user message)
+                            // Ensure null values are converted to empty string
+                            if (content_val.is_null()) {
+                                msg_json["content"] = "";
+                            } else {
+                                msg_json["content"] = content_val;
+                            }
+                        }
+                    } else if (is_last_user_msg && has_images_or_audio) {
+                        // If no content but this is the last user message with images/audio, create content array
+                        json content_array = json::array();
+                        if (request->images_size() > 0) {
+                            for (int j = 0; j < request->images_size(); j++) {
+                                json image_chunk;
+                                image_chunk["type"] = "image_url";
+                                json image_url;
+                                image_url["url"] = "data:image/jpeg;base64," + request->images(j);
+                                image_chunk["image_url"] = image_url;
+                                content_array.push_back(image_chunk);
+                            }
+                        }
+                        if (request->audios_size() > 0) {
+                            for (int j = 0; j < request->audios_size(); j++) {
+                                json audio_chunk;
+                                audio_chunk["type"] = "input_audio";
+                                json input_audio;
+                                input_audio["data"] = request->audios(j);
+                                input_audio["format"] = "wav"; // default, could be made configurable
+                                audio_chunk["input_audio"] = input_audio;
+                                content_array.push_back(audio_chunk);
+                            }
+                        }
+                        if (request->videos_size() > 0) {
+                            for (int j = 0; j < request->videos_size(); j++) {
+                                json video_chunk;
+                                video_chunk["type"] = "input_video";
+                                json input_video;
+                                input_video["data"] = request->videos(j);
+                                video_chunk["input_video"] = input_video;
+                                content_array.push_back(video_chunk);
+                            }
+                        }
+                        msg_json["content"] = content_array;
+                    } else if (msg.role() == "tool") {
+                        // Tool role messages must have content field set, even if empty
+                        // Jinja templates expect content to be a string, not null or object
+                        SRV_INF("[CONTENT DEBUG] PredictStream: Message %d is tool role, content_empty=%d\n", i, msg.content().empty() ? 1 : 0);
+                        if (msg.content().empty()) {
+                            msg_json["content"] = "";
+                            SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): empty content, set to empty string\n", i);
+                        } else {
+                            SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): content exists: %s\n",
+                                    i, msg.content().substr(0, std::min<size_t>(200, msg.content().size())).c_str());
+                            // Content exists, parse and ensure it's a string
+                            json content_val;
+                            try {
+                                content_val = json::parse(msg.content());
+                                SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): parsed JSON, type=%s\n",
+                                        i, content_val.is_null() ? "null" :
+                                           content_val.is_object() ? "object" :
+                                           content_val.is_string() ? "string" :
+                                           content_val.is_array() ? "array" : "other");
+                                // Handle null values - Jinja templates expect content to be a string, not null
+                                if (content_val.is_null()) {
+                                    msg_json["content"] = "";
+                                    SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): null content, converted to empty string\n", i);
+                                } else if (content_val.is_object()) {
+                                    // If content is an object (e.g., from tool call failures/errors), convert to string
+                                    msg_json["content"] = content_val.dump();
+                                    SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): object content, converted to string: %s\n",
+                                            i, content_val.dump().substr(0, std::min<size_t>(200, content_val.dump().size())).c_str());
+                                } else if (content_val.is_string()) {
+                                    msg_json["content"] = content_val.get<std::string>();
+                                    SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): string content, using as-is\n", i);
+                                } else {
+                                    // For arrays or other types, convert to string
+                                    msg_json["content"] = content_val.dump();
+                                    SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): %s content, converted to string\n",
+                                            i, content_val.is_array() ? "array" : "other type");
+                                }
+                            } catch (const json::parse_error&) {
+                                // Not JSON, treat as plain string
+                                msg_json["content"] = msg.content();
+                                SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (tool): not JSON, using as string\n", i);
+                            }
+                        }
+                    } else {
+                        // Ensure all messages have content set (fallback for any unhandled cases)
+                        // Jinja templates expect content to be present, default to empty string if not set
+                        if (!msg_json.contains("content")) {
+                            SRV_INF("[CONTENT DEBUG] PredictStream: Message %d (role=%s): no content field, adding empty string\n",
+                                    i, msg.role().c_str());
+                            msg_json["content"] = "";
+                        }
                     }
-                    messages_json.push_back(llama_grpc::build_reconstructed_message(rin));
+
+                    // Add optional fields for OpenAI-compatible message format
+                    if (!msg.name().empty()) {
+                        msg_json["name"] = msg.name();
+                    }
+                    if (!msg.tool_call_id().empty()) {
+                        msg_json["tool_call_id"] = msg.tool_call_id();
+                    }
+                    if (!msg.reasoning_content().empty()) {
+                        msg_json["reasoning_content"] = msg.reasoning_content();
+                    }
+                    if (!msg.tool_calls().empty()) {
+                        // Parse tool_calls JSON string and add to message
+                        try {
+                            json tool_calls = json::parse(msg.tool_calls());
+                            msg_json["tool_calls"] = tool_calls;
+                            SRV_INF("[TOOL CALLS DEBUG] PredictStream: Message %d has tool_calls: %s\n", i, tool_calls.dump().c_str());
+                            // IMPORTANT: If message has tool_calls but content is empty or not set,
+                            // set content to space " " instead of empty string "", because llama.cpp's
+                            // common_chat_msgs_to_json_oaicompat converts empty strings to null (line 312),
+                            // which causes template errors when accessing message.content[:tool_start_length]
+                            if (!msg_json.contains("content") || (msg_json.contains("content") && msg_json["content"].is_string() && msg_json["content"].get<std::string>().empty())) {
+                                SRV_INF("[CONTENT DEBUG] PredictStream: Message %d has tool_calls but empty content, setting to space\n", i);
+                                msg_json["content"] = " ";
+                            }
+                            // Log each tool call with name and arguments
+                            if (tool_calls.is_array()) {
+                                for (size_t tc_idx = 0; tc_idx < tool_calls.size(); tc_idx++) {
+                                    const auto& tc = tool_calls[tc_idx];
+                                    std::string tool_name = "unknown";
+                                    std::string tool_args = "{}";
+                                    if (tc.contains("function")) {
+                                        const auto& func = tc["function"];
+                                        if (func.contains("name")) {
+                                            tool_name = func["name"].get<std::string>();
+                                        }
+                                        if (func.contains("arguments")) {
+                                            tool_args = func["arguments"].is_string() ?
+                                                func["arguments"].get<std::string>() :
+                                                func["arguments"].dump();
+                                        }
+                                    } else if (tc.contains("name")) {
+                                        tool_name = tc["name"].get<std::string>();
+                                        if (tc.contains("arguments")) {
+                                            tool_args = tc["arguments"].is_string() ?
+                                                tc["arguments"].get<std::string>() :
+                                                tc["arguments"].dump();
+                                        }
+                                    }
+                                    SRV_INF("[TOOL CALLS DEBUG] PredictStream: Message %d, tool_call %zu: name=%s, arguments=%s\n",
+                                            i, tc_idx, tool_name.c_str(), tool_args.c_str());
+                                }
+                            }
+                        } catch (const json::parse_error& e) {
+                            SRV_WRN("Failed to parse tool_calls JSON: %s\n", e.what());
+                        }
+                    }
+
+                    // Debug: Log final content state before adding to array
+                    if (msg_json.contains("content")) {
+                        if (msg_json["content"].is_null()) {
+                            SRV_INF("[CONTENT DEBUG] PredictStream: Message %d FINAL STATE: content is NULL - THIS WILL CAUSE ERROR!\n", i);
+                        } else {
+                            SRV_INF("[CONTENT DEBUG] PredictStream: Message %d FINAL STATE: content type=%s, has_value=%d\n",
+                                    i, msg_json["content"].is_string() ? "string" :
+                                       msg_json["content"].is_array() ? "array" :
+                                       msg_json["content"].is_object() ? "object" : "other",
+                                    msg_json["content"].is_null() ? 0 : 1);
+                        }
+                    } else {
+                        SRV_INF("[CONTENT DEBUG] PredictStream: Message %d FINAL STATE: NO CONTENT FIELD - THIS WILL CAUSE ERROR!\n", i);
+                    }
+
+                    messages_json.push_back(msg_json);
                 }
 
                 // Final safety check: Ensure no message has null content (Jinja templates require strings)
@@ -1921,7 +1976,36 @@ public:
                 if (body_json.contains("messages") && body_json["messages"].is_array()) {
                     SRV_INF("[CONTENT DEBUG] PredictStream: Before oaicompat_chat_params_parse - checking %zu messages\n", body_json["messages"].size());
                     for (size_t idx = 0; idx < body_json["messages"].size(); idx++) {
-                        llama_grpc::normalize_template_message(body_json["messages"][idx]);
+                        auto& msg = body_json["messages"][idx];
+                        std::string role_str = msg.contains("role") ? msg["role"].get<std::string>() : "unknown";
+                        if (msg.contains("content")) {
+                            if (msg["content"].is_null()) {
+                                SRV_INF("[CONTENT DEBUG] PredictStream: BEFORE TEMPLATE - Message %zu (role=%s) has NULL content - FIXING!\n", idx, role_str.c_str());
+                                msg["content"] = ""; // Fix null content
+                            } else if (role_str == "tool" && msg["content"].is_array()) {
+                                // Tool messages must have string content, not array
+                                // oaicompat_chat_params_parse expects tool messages to have string content
+                                SRV_INF("[CONTENT DEBUG] PredictStream: BEFORE TEMPLATE - Message %zu (role=tool) has array content, converting to string\n", idx);
+                                msg["content"] = msg["content"].dump();
+                            } else if (!msg["content"].is_string() && !msg["content"].is_array()) {
+                                // If content is object or other non-string type, convert to string for templates
+                                SRV_INF("[CONTENT DEBUG] PredictStream: BEFORE TEMPLATE - Message %zu (role=%s) content is not string/array, converting\n", idx, role_str.c_str());
+                                if (msg["content"].is_object()) {
+                                    msg["content"] = msg["content"].dump();
+                                } else {
+                                    msg["content"] = "";
+                                }
+                            } else {
+                                SRV_INF("[CONTENT DEBUG] PredictStream: BEFORE TEMPLATE - Message %zu (role=%s): content type=%s\n",
+                                        idx, role_str.c_str(),
+                                        msg["content"].is_string() ? "string" :
+                                        msg["content"].is_array() ? "array" :
+                                        msg["content"].is_object() ? "object" : "other");
+                            }
+                        } else {
+                            SRV_INF("[CONTENT DEBUG] PredictStream: BEFORE TEMPLATE - Message %zu (role=%s) MISSING content field - ADDING!\n", idx, role_str.c_str());
+                            msg["content"] = ""; // Add missing content
+                        }
                     }
                 }
 
@@ -2216,8 +2300,6 @@ public:
     grpc::Status Predict(ServerContext* context, const backend::PredictOptions* request, backend::Reply* reply) override {
          auto auth = checkAuth(context);
          if (!auth.ok()) return auth;
-         auto identity = checkModelIdentity(request);
-         if (!identity.ok()) return identity;
          if (params_base.model.path.empty()) {
              return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
          }
@@ -2255,20 +2337,264 @@ public:
                 SRV_INF("[CONTENT DEBUG] Predict: Processing %d messages\n", request->messages_size());
                 for (int i = 0; i < request->messages_size(); i++) {
                     const auto& msg = request->messages(i);
-                    llama_grpc::ReconstructedMessageInput rin;
-                    rin.role = msg.role();
-                    rin.content = msg.content();
-                    rin.name = msg.name();
-                    rin.tool_call_id = msg.tool_call_id();
-                    rin.reasoning_content = msg.reasoning_content();
-                    rin.tool_calls = msg.tool_calls();
-                    rin.is_last_user_msg = (i == last_user_msg_idx);
-                    if (rin.is_last_user_msg) {
-                        for (int j = 0; j < request->images_size(); j++) rin.images.push_back(request->images(j));
-                        for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j));
-                        for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j));
+                    json msg_json;
+                    msg_json["role"] = msg.role();
+
+                    SRV_INF("[CONTENT DEBUG] Predict: Message %d: role=%s, content_empty=%d, content_length=%zu\n",
+                            i, msg.role().c_str(), msg.content().empty() ? 1 : 0, msg.content().size());
+                    if (!msg.content().empty()) {
+                        SRV_INF("[CONTENT DEBUG] Predict: Message %d content (first 200 chars): %s\n",
+                                i, msg.content().substr(0, std::min<size_t>(200, msg.content().size())).c_str());
                     }
-                    messages_json.push_back(llama_grpc::build_reconstructed_message(rin));
+
+                    bool is_last_user_msg = (i == last_user_msg_idx);
+                    bool has_images_or_audio = (request->images_size() > 0 || request->audios_size() > 0 || request->videos_size() > 0);
+
+                    // Handle content - can be string, null, or array
+                    // For multimodal content, we'll embed images/audio from separate fields
+                    if (!msg.content().empty()) {
+                        // Try to parse content as JSON to see if it's already an array
+                        json content_val;
+                        try {
+                            content_val = json::parse(msg.content());
+                            // Handle null values - convert to empty string to avoid template errors
+                            if (content_val.is_null()) {
+                                SRV_INF("[CONTENT DEBUG] Predict: Message %d parsed JSON is null, converting to empty string\n", i);
+                                content_val = "";
+                            }
+                        } catch (const json::parse_error&) {
+                            // Not JSON, treat as plain string
+                            content_val = msg.content();
+                        }
+
+                        // If content is an object (e.g., from tool call failures), convert to string
+                        if (content_val.is_object()) {
+                            SRV_INF("[CONTENT DEBUG] Predict: Message %d content is object, converting to string\n", i);
+                            content_val = content_val.dump();
+                        }
+
+                        // If content is a string and this is the last user message with images/audio, combine them
+                        if (content_val.is_string() && is_last_user_msg && has_images_or_audio) {
+                            json content_array = json::array();
+                            // Add text first
+                            content_array.push_back({{"type", "text"}, {"text", content_val.get<std::string>()}});
+                            // Add images
+                            if (request->images_size() > 0) {
+                                for (int j = 0; j < request->images_size(); j++) {
+                                    json image_chunk;
+                                    image_chunk["type"] = "image_url";
+                                    json image_url;
+                                    image_url["url"] = "data:image/jpeg;base64," + request->images(j);
+                                    image_chunk["image_url"] = image_url;
+                                    content_array.push_back(image_chunk);
+                                }
+                            }
+                            // Add audios
+                            if (request->audios_size() > 0) {
+                                for (int j = 0; j < request->audios_size(); j++) {
+                                    json audio_chunk;
+                                    audio_chunk["type"] = "input_audio";
+                                    json input_audio;
+                                    input_audio["data"] = request->audios(j);
+                                    input_audio["format"] = "wav"; // default, could be made configurable
+                                    audio_chunk["input_audio"] = input_audio;
+                                    content_array.push_back(audio_chunk);
+                                }
+                            }
+                            if (request->videos_size() > 0) {
+                                for (int j = 0; j < request->videos_size(); j++) {
+                                    json video_chunk;
+                                    video_chunk["type"] = "input_video";
+                                    json input_video;
+                                    input_video["data"] = request->videos(j);
+                                    video_chunk["input_video"] = input_video;
+                                    content_array.push_back(video_chunk);
+                                }
+                            }
+                            msg_json["content"] = content_array;
+                        } else {
+                            // Use content as-is (already array or not last user message)
+                            // Ensure null values are converted to empty string
+                            if (content_val.is_null()) {
+                                SRV_INF("[CONTENT DEBUG] Predict: Message %d content_val was null, setting to empty string\n", i);
+                                msg_json["content"] = "";
+                            } else {
+                                msg_json["content"] = content_val;
+                                SRV_INF("[CONTENT DEBUG] Predict: Message %d content set, type=%s\n",
+                                        i, content_val.is_string() ? "string" :
+                                           content_val.is_array() ? "array" :
+                                           content_val.is_object() ? "object" : "other");
+                            }
+                        }
+                    } else if (is_last_user_msg && has_images_or_audio) {
+                        // If no content but this is the last user message with images/audio, create content array
+                        json content_array = json::array();
+                        if (request->images_size() > 0) {
+                            for (int j = 0; j < request->images_size(); j++) {
+                                json image_chunk;
+                                image_chunk["type"] = "image_url";
+                                json image_url;
+                                image_url["url"] = "data:image/jpeg;base64," + request->images(j);
+                                image_chunk["image_url"] = image_url;
+                                content_array.push_back(image_chunk);
+                            }
+                        }
+                        if (request->audios_size() > 0) {
+                            for (int j = 0; j < request->audios_size(); j++) {
+                                json audio_chunk;
+                                audio_chunk["type"] = "input_audio";
+                                json input_audio;
+                                input_audio["data"] = request->audios(j);
+                                input_audio["format"] = "wav"; // default, could be made configurable
+                                audio_chunk["input_audio"] = input_audio;
+                                content_array.push_back(audio_chunk);
+                            }
+                        }
+                        if (request->videos_size() > 0) {
+                            for (int j = 0; j < request->videos_size(); j++) {
+                                json video_chunk;
+                                video_chunk["type"] = "input_video";
+                                json input_video;
+                                input_video["data"] = request->videos(j);
+                                video_chunk["input_video"] = input_video;
+                                content_array.push_back(video_chunk);
+                            }
+                        }
+                        msg_json["content"] = content_array;
+                        SRV_INF("[CONTENT DEBUG] Predict: Message %d created content array with media\n", i);
+                    } else if (!msg.tool_calls().empty()) {
+                        // Tool call messages may have null content, but templates expect string
+                        // IMPORTANT: Set to space " " instead of empty string "", because llama.cpp's
+                        // common_chat_msgs_to_json_oaicompat converts empty strings to null (line 312),
+                        // which causes template errors when accessing message.content[:tool_start_length]
+                        SRV_INF("[CONTENT DEBUG] Predict: Message %d has tool_calls, setting content to space (not empty string)\n", i);
+                        msg_json["content"] = " ";
+                    } else if (msg.role() == "tool") {
+                        // Tool role messages must have content field set, even if empty
+                        // Jinja templates expect content to be a string, not null or object
+                        SRV_INF("[CONTENT DEBUG] Predict: Message %d is tool role, content_empty=%d\n", i, msg.content().empty() ? 1 : 0);
+                        if (msg.content().empty()) {
+                            msg_json["content"] = "";
+                            SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): empty content, set to empty string\n", i);
+                        } else {
+                            SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): content exists: %s\n",
+                                    i, msg.content().substr(0, std::min<size_t>(200, msg.content().size())).c_str());
+                            // Content exists, parse and ensure it's a string
+                            json content_val;
+                            try {
+                                content_val = json::parse(msg.content());
+                                SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): parsed JSON, type=%s\n",
+                                        i, content_val.is_null() ? "null" :
+                                           content_val.is_object() ? "object" :
+                                           content_val.is_string() ? "string" :
+                                           content_val.is_array() ? "array" : "other");
+                                // Handle null values - Jinja templates expect content to be a string, not null
+                                if (content_val.is_null()) {
+                                    msg_json["content"] = "";
+                                    SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): null content, converted to empty string\n", i);
+                                } else if (content_val.is_object()) {
+                                    // If content is an object (e.g., from tool call failures/errors), convert to string
+                                    msg_json["content"] = content_val.dump();
+                                    SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): object content, converted to string: %s\n",
+                                            i, content_val.dump().substr(0, std::min<size_t>(200, content_val.dump().size())).c_str());
+                                } else if (content_val.is_string()) {
+                                    msg_json["content"] = content_val.get<std::string>();
+                                    SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): string content, using as-is\n", i);
+                                } else {
+                                    // For arrays or other types, convert to string
+                                    msg_json["content"] = content_val.dump();
+                                    SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): %s content, converted to string\n",
+                                            i, content_val.is_array() ? "array" : "other type");
+                                }
+                            } catch (const json::parse_error&) {
+                                // Not JSON, treat as plain string
+                                msg_json["content"] = msg.content();
+                                SRV_INF("[CONTENT DEBUG] Predict: Message %d (tool): not JSON, using as string\n", i);
+                            }
+                        }
+                    } else {
+                        // Ensure all messages have content set (fallback for any unhandled cases)
+                        // Jinja templates expect content to be present, default to empty string if not set
+                        if (!msg_json.contains("content")) {
+                            SRV_INF("[CONTENT DEBUG] Predict: Message %d (role=%s): no content field, adding empty string\n",
+                                    i, msg.role().c_str());
+                            msg_json["content"] = "";
+                        }
+                    }
+
+                    // Add optional fields for OpenAI-compatible message format
+                    if (!msg.name().empty()) {
+                        msg_json["name"] = msg.name();
+                    }
+                    if (!msg.tool_call_id().empty()) {
+                        msg_json["tool_call_id"] = msg.tool_call_id();
+                    }
+                    if (!msg.reasoning_content().empty()) {
+                        msg_json["reasoning_content"] = msg.reasoning_content();
+                    }
+                    if (!msg.tool_calls().empty()) {
+                        // Parse tool_calls JSON string and add to message
+                        try {
+                            json tool_calls = json::parse(msg.tool_calls());
+                            msg_json["tool_calls"] = tool_calls;
+                            SRV_INF("[TOOL CALLS DEBUG] Predict: Message %d has tool_calls: %s\n", i, tool_calls.dump().c_str());
+                            // IMPORTANT: If message has tool_calls but content is empty or not set,
+                            // set content to space " " instead of empty string "", because llama.cpp's
+                            // common_chat_msgs_to_json_oaicompat converts empty strings to null (line 312),
+                            // which causes template errors when accessing message.content[:tool_start_length]
+                            if (!msg_json.contains("content") || (msg_json.contains("content") && msg_json["content"].is_string() && msg_json["content"].get<std::string>().empty())) {
+                                SRV_INF("[CONTENT DEBUG] Predict: Message %d has tool_calls but empty content, setting to space\n", i);
+                                msg_json["content"] = " ";
+                            }
+                            // Log each tool call with name and arguments
+                            if (tool_calls.is_array()) {
+                                for (size_t tc_idx = 0; tc_idx < tool_calls.size(); tc_idx++) {
+                                    const auto& tc = tool_calls[tc_idx];
+                                    std::string tool_name = "unknown";
+                                    std::string tool_args = "{}";
+                                    if (tc.contains("function")) {
+                                        const auto& func = tc["function"];
+                                        if (func.contains("name")) {
+                                            tool_name = func["name"].get<std::string>();
+                                        }
+                                        if (func.contains("arguments")) {
+                                            tool_args = func["arguments"].is_string() ?
+                                                func["arguments"].get<std::string>() :
+                                                func["arguments"].dump();
+                                        }
+                                    } else if (tc.contains("name")) {
+                                        tool_name = tc["name"].get<std::string>();
+                                        if (tc.contains("arguments")) {
+                                            tool_args = tc["arguments"].is_string() ?
+                                                tc["arguments"].get<std::string>() :
+                                                tc["arguments"].dump();
+                                        }
+                                    }
+                                    SRV_INF("[TOOL CALLS DEBUG] Predict: Message %d, tool_call %zu: name=%s, arguments=%s\n",
+                                            i, tc_idx, tool_name.c_str(), tool_args.c_str());
+                                }
+                            }
+                        } catch (const json::parse_error& e) {
+                            SRV_WRN("Failed to parse tool_calls JSON: %s\n", e.what());
+                        }
+                    }
+
+                    // Debug: Log final content state before adding to array
+                    if (msg_json.contains("content")) {
+                        if (msg_json["content"].is_null()) {
+                            SRV_INF("[CONTENT DEBUG] Predict: Message %d FINAL STATE: content is NULL - THIS WILL CAUSE ERROR!\n", i);
+                        } else {
+                            SRV_INF("[CONTENT DEBUG] Predict: Message %d FINAL STATE: content type=%s, has_value=%d\n",
+                                    i, msg_json["content"].is_string() ? "string" :
+                                       msg_json["content"].is_array() ? "array" :
+                                       msg_json["content"].is_object() ? "object" : "other",
+                                    msg_json["content"].is_null() ? 0 : 1);
+                        }
+                    } else {
+                        SRV_INF("[CONTENT DEBUG] Predict: Message %d FINAL STATE: NO CONTENT FIELD - THIS WILL CAUSE ERROR!\n", i);
+                    }
+
+                    messages_json.push_back(msg_json);
                 }
 
                 // Final safety check: Ensure no message has null content (Jinja templates require strings)
@@ -2489,7 +2815,36 @@ public:
                 if (body_json.contains("messages") && body_json["messages"].is_array()) {
                     SRV_INF("[CONTENT DEBUG] Predict: Before oaicompat_chat_params_parse - checking %zu messages\n", body_json["messages"].size());
                     for (size_t idx = 0; idx < body_json["messages"].size(); idx++) {
-                        llama_grpc::normalize_template_message(body_json["messages"][idx]);
+                        auto& msg = body_json["messages"][idx];
+                        std::string role_str = msg.contains("role") ? msg["role"].get<std::string>() : "unknown";
+                        if (msg.contains("content")) {
+                            if (msg["content"].is_null()) {
+                                SRV_INF("[CONTENT DEBUG] Predict: BEFORE TEMPLATE - Message %zu (role=%s) has NULL content - FIXING!\n", idx, role_str.c_str());
+                                msg["content"] = ""; // Fix null content
+                            } else if (role_str == "tool" && msg["content"].is_array()) {
+                                // Tool messages must have string content, not array
+                                // oaicompat_chat_params_parse expects tool messages to have string content
+                                SRV_INF("[CONTENT DEBUG] Predict: BEFORE TEMPLATE - Message %zu (role=tool) has array content, converting to string\n", idx);
+                                msg["content"] = msg["content"].dump();
+                            } else if (!msg["content"].is_string() && !msg["content"].is_array()) {
+                                // If content is object or other non-string type, convert to string for templates
+                                SRV_INF("[CONTENT DEBUG] Predict: BEFORE TEMPLATE - Message %zu (role=%s) content is not string/array, converting\n", idx, role_str.c_str());
+                                if (msg["content"].is_object()) {
+                                    msg["content"] = msg["content"].dump();
+                                } else {
+                                    msg["content"] = "";
+                                }
+                            } else {
+                                SRV_INF("[CONTENT DEBUG] Predict: BEFORE TEMPLATE - Message %zu (role=%s): content type=%s\n",
+                                        idx, role_str.c_str(),
+                                        msg["content"].is_string() ? "string" :
+                                        msg["content"].is_array() ? "array" :
+                                        msg["content"].is_object() ? "object" : "other");
+                            }
+                        } else {
+                            SRV_INF("[CONTENT DEBUG] Predict: BEFORE TEMPLATE - Message %zu (role=%s) MISSING content field - ADDING!\n", idx, role_str.c_str());
+                            msg["content"] = ""; // Add missing content
+                        }
                     }
                 }
 
@@ -2750,8 +3105,6 @@ public:
     grpc::Status Embedding(ServerContext* context, const backend::PredictOptions* request, backend::EmbeddingResult* embeddingResult) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -2850,8 +3203,6 @@ public:
     }
 
     grpc::Status Rerank(ServerContext* context, const backend::RerankRequest* request, backend::RerankResult* rerankResult) override {
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         if (!params_base.embedding || params_base.pooling_type != LLAMA_POOLING_TYPE_RANK) {
             return grpc::Status(grpc::StatusCode::UNIMPLEMENTED, "This server does not support reranking. Start it with `--reranking` and without `--embedding`");
         }
@@ -2976,8 +3327,6 @@ public:
     grpc::Status Score(ServerContext* context, const backend::ScoreRequest* request, backend::ScoreResponse* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -3149,8 +3498,6 @@ public:
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
@@ -3436,8 +3783,6 @@ public:
                                     backend::TranscriptResult* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
 
         backend::Reply reply;
         grpc::Status st = runTranscriptionAsCompletion(context, request, &reply);
@@ -3456,8 +3801,6 @@ public:
                                           grpc::ServerWriter<backend::TranscriptStreamResponse>* writer) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
-        auto identity = checkModelIdentity(request);
-        if (!identity.ok()) return identity;
 
         // Buffered streaming: run the transcription as a normal chat
         // completion, then emit one delta + one final event. Real
@@ -3513,10 +3856,6 @@ int main(int argc, char** argv) {
     }
   }
 
-    // Best-effort backstop: self-terminate if the LocalAI process that spawned
-    // us dies without cleaning us up (see parent_watch.h).
-    llama_grpc::start_parent_death_watcher();
-
     server_context ctx_server;
     BackendServiceImpl service(ctx_server);
 

backend/cpp/llama-cpp/message_content.h

@@ -1,192 +0,0 @@
-#pragma once
-
-#include <string>
-#include <vector>
-
-#include <nlohmann/json.hpp>
-
-namespace llama_grpc {
-
-// Normalizes a proto message's content string into the JSON value used when
-// reconstructing OpenAI-format messages for the tokenizer (jinja) template.
-//
-// Shared by the streaming (PredictStream) and non-streaming (Predict) message
-// reconstruction paths so the two cannot drift.
-//
-// LocalAI's Go layer (schema.Messages.ToProto) always sends content as a plain
-// text string; multimodal media travels in separate proto fields, never inside
-// content. So user/system/developer content is *only ever* opaque text and must
-// NOT be JSON-sniffed: a prompt that merely looks like JSON (e.g. an ingredient
-// list ["1/4 cup sugar", ...]) would otherwise be reinterpreted as structured
-// content parts and rejected by oaicompat_chat_params_parse with
-// "unsupported content[].type" (https://github.com/mudler/LocalAI/issues/10524).
-// (developer is OpenAI's modern system alias - same "human-authored text" nature.)
-//
-// For assistant/tool messages we still collapse a literal JSON null/object
-// (tool-call bookkeeping) to a string, but we never turn a plain string into an
-// array/scalar. The array defense is therefore role-independent (arrays/scalars
-// fall through for every role); the role gate only governs the null/object case.
-inline nlohmann::ordered_json normalize_message_content(const std::string& role,
-                                                        const std::string& content) {
-    nlohmann::ordered_json content_val = content;
-    if (role != "user" && role != "system" && role != "developer") {
-        try {
-            nlohmann::ordered_json parsed = nlohmann::ordered_json::parse(content);
-            if (parsed.is_null()) {
-                content_val = "";
-            } else if (parsed.is_object()) {
-                content_val = parsed.dump();
-            }
-            // arrays / scalars: keep the original plain-text string as-is
-        } catch (const nlohmann::ordered_json::parse_error&) {
-            // Not JSON, already the plain string
-        }
-    }
-    return content_val;
-}
-
-// Final safety pass applied to each reconstructed OpenAI message right before it
-// is handed to oaicompat_chat_params_parse (jinja templating). Jinja templates
-// assume content is a string: a literal null breaks slicing such as
-// message.content[:N] (#7324), and a tool message with array content is rejected
-// (#7528). A multimodal user message legitimately carries a typed-part array
-// ({type:text}, {type:image_url}, ...), which must be left intact. Shared by the
-// streaming and non-streaming paths so this invariant cannot drift between them.
-inline void normalize_template_message(nlohmann::ordered_json& msg) {
-    if (!msg.contains("content")) {
-        msg["content"] = ""; // templates expect the field to exist
-        return;
-    }
-    nlohmann::ordered_json& content = msg["content"];
-    const std::string role = (msg.contains("role") && msg["role"].is_string())
-                                 ? msg["role"].get<std::string>()
-                                 : std::string();
-    if (content.is_null()) {
-        content = ""; // #7324: null would crash content[:N] slicing
-    } else if (role == "tool" && content.is_array()) {
-        content = content.dump(); // #7528: tool messages must have string content
-    } else if (!content.is_string() && !content.is_array()) {
-        if (content.is_object()) {
-            content = content.dump(); // tool-call bookkeeping object -> string
-        } else {
-            content = ""; // other scalar (number/bool) -> empty
-        }
-    }
-    // string, or a non-tool (multimodal) typed-part array: leave untouched
-}
-
-// One proto message's data, flattened to plain types so the reconstruction logic
-// can be shared and unit-tested without protobuf. The streaming and non-streaming
-// predict paths both populate this from proto::Message + the request's media.
-struct ReconstructedMessageInput {
-    std::string role;
-    std::string content;            // proto.Message.content (always a plain string)
-    std::string name;
-    std::string tool_call_id;
-    std::string reasoning_content;
-    std::string tool_calls;         // tool_calls as a JSON string, or empty
-    bool is_last_user_msg = false;  // attach request media to this message
-    std::vector<std::string> images; // base64 (jpeg)
-    std::vector<std::string> audios; // base64 (wav)
-    std::vector<std::string> videos; // base64
-};
-
-// Appends the request's media as OpenAI typed content parts. Imperative (not
-// brace-init) to avoid nlohmann's object-vs-array initializer-list ambiguity.
-inline void append_media_parts(nlohmann::ordered_json& content_array,
-                               const std::vector<std::string>& images,
-                               const std::vector<std::string>& audios,
-                               const std::vector<std::string>& videos) {
-    for (const auto& img : images) {
-        nlohmann::ordered_json image_chunk;
-        image_chunk["type"] = "image_url";
-        nlohmann::ordered_json image_url;
-        image_url["url"] = "data:image/jpeg;base64," + img;
-        image_chunk["image_url"] = image_url;
-        content_array.push_back(image_chunk);
-    }
-    for (const auto& aud : audios) {
-        nlohmann::ordered_json audio_chunk;
-        audio_chunk["type"] = "input_audio";
-        nlohmann::ordered_json input_audio;
-        input_audio["data"] = aud;
-        input_audio["format"] = "wav"; // default; could be made configurable
-        audio_chunk["input_audio"] = input_audio;
-        content_array.push_back(audio_chunk);
-    }
-    for (const auto& vid : videos) {
-        nlohmann::ordered_json video_chunk;
-        video_chunk["type"] = "input_video";
-        nlohmann::ordered_json input_video;
-        input_video["data"] = vid;
-        video_chunk["input_video"] = input_video;
-        content_array.push_back(video_chunk);
-    }
-}
-
-// Reconstructs a single OpenAI-format message (the object fed to
-// oaicompat_chat_params_parse) from a proto message. Shared by PredictStream and
-// Predict so the content/multimodal/tool_calls handling cannot drift between the
-// two stream modes (it previously lived as two ~150-line copies with a redundant
-// Predict-only tool_calls->" " branch). Guarantees content is always a string or
-// a typed-part array, never null/missing.
-inline nlohmann::ordered_json build_reconstructed_message(const ReconstructedMessageInput& in) {
-    nlohmann::ordered_json msg_json;
-    msg_json["role"] = in.role;
-    const bool has_media = !in.images.empty() || !in.audios.empty() || !in.videos.empty();
-
-    if (!in.content.empty()) {
-        nlohmann::ordered_json content_val = normalize_message_content(in.role, in.content);
-        if (content_val.is_string() && in.is_last_user_msg && has_media) {
-            // Last user message + media: build a typed-part array (text first).
-            nlohmann::ordered_json content_array = nlohmann::ordered_json::array();
-            nlohmann::ordered_json text_part;
-            text_part["type"] = "text";
-            text_part["text"] = content_val.get<std::string>();
-            content_array.push_back(text_part);
-            append_media_parts(content_array, in.images, in.audios, in.videos);
-            msg_json["content"] = content_array;
-        } else if (content_val.is_null()) {
-            msg_json["content"] = "";
-        } else {
-            msg_json["content"] = content_val;
-        }
-    } else if (in.is_last_user_msg && has_media) {
-        // No text but media on the last user message: media-only typed array.
-        nlohmann::ordered_json content_array = nlohmann::ordered_json::array();
-        append_media_parts(content_array, in.images, in.audios, in.videos);
-        msg_json["content"] = content_array;
-    } else {
-        // Empty content (any role, incl. tool/assistant): templates need a string.
-        msg_json["content"] = "";
-    }
-
-    if (!in.name.empty()) {
-        msg_json["name"] = in.name;
-    }
-    if (!in.tool_call_id.empty()) {
-        msg_json["tool_call_id"] = in.tool_call_id;
-    }
-    if (!in.reasoning_content.empty()) {
-        msg_json["reasoning_content"] = in.reasoning_content;
-    }
-    if (!in.tool_calls.empty()) {
-        try {
-            nlohmann::ordered_json tool_calls = nlohmann::ordered_json::parse(in.tool_calls);
-            msg_json["tool_calls"] = tool_calls;
-            // tool_calls + empty/blank content: use " " not "", because llama.cpp's
-            // common_chat_msgs_to_json_oaicompat turns "" into null, which breaks
-            // templates that slice message.content[:tool_start_length] (#7324).
-            if (!msg_json.contains("content") ||
-                (msg_json["content"].is_string() && msg_json["content"].get<std::string>().empty())) {
-                msg_json["content"] = " ";
-            }
-        } catch (const nlohmann::ordered_json::parse_error&) {
-            // Malformed tool_calls JSON: leave content as-is (prior behavior).
-        }
-    }
-
-    return msg_json;
-}
-
-}  // namespace llama_grpc

backend/cpp/llama-cpp/message_content_test.cpp

@@ -1,234 +0,0 @@
-// Unit tests for the shared message-reconstruction helpers (message_content.h).
-//
-// Build & run standalone (nlohmann/json single header on the include path):
-//   g++ -std=c++17 -I<dir-with-nlohmann> message_content_test.cpp -o t && ./t
-// or via CMake: -DLLAMA_GRPC_BUILD_TESTS=ON then ctest.
-//
-// Regression coverage for:
-//   #10524 - a user/system prompt that is itself a JSON-array string must stay
-//            plain text, never be reinterpreted as OpenAI structured parts.
-//   #7324  - assistant/tool null content -> "" (templates slice content[:N]);
-//            assistant+tool_calls+empty content -> " " (not "", which becomes null).
-//   #7528  - tool message array content must reach the template as a string.
-//   multimodal - last user message text + media -> typed-part array, media kept.
-
-#include <cassert>
-#include <iostream>
-#include <string>
-
-#include "message_content.h"
-
-using nlohmann::ordered_json;
-using llama_grpc::normalize_message_content;
-using llama_grpc::normalize_template_message;
-using llama_grpc::build_reconstructed_message;
-using llama_grpc::ReconstructedMessageInput;
-
-static int failures = 0;
-
-static void check(bool ok, const std::string& name, const std::string& detail = "") {
-    if (!ok) {
-        std::cerr << "FAIL " << name << (detail.empty() ? "" : ": " + detail) << "\n";
-        failures++;
-    }
-}
-
-// ---- normalize_message_content -------------------------------------------
-
-static void expect_norm_string(const char* name, const std::string& role,
-                               const std::string& content, const std::string& want) {
-    auto got = normalize_message_content(role, content);
-    if (!got.is_string()) {
-        check(false, name, "expected a JSON string, got " +
-                               std::string(got.is_array() ? "array" : got.is_object() ? "object" : "other") +
-                               " (" + got.dump() + ")");
-        return;
-    }
-    check(got.get<std::string>() == want, name, "expected \"" + want + "\", got \"" + got.get<std::string>() + "\"");
-}
-
-static void test_normalize() {
-    const std::string ingredients = R"(["1/4 cup brown sugar, packed","1 pound ground beef"])";
-
-    // #10524 - JSON-array text must stay a string. Role-INDEPENDENT array defense.
-    for (const char* role : {"user", "system", "developer", "function", "assistant", "tool"}) {
-        expect_norm_string((std::string("json_array_stays_text:") + role).c_str(), role, ingredients, ingredients);
-    }
-
-    // #10524 - user/system/developer JSON-object text stays verbatim (NOT re-dumped).
-    expect_norm_string("user_json_object_verbatim", "user", R"({"a":1})", R"({"a":1})");
-    expect_norm_string("system_json_object_verbatim", "system", R"({"a":1})", R"({"a":1})");
-    expect_norm_string("developer_json_object_verbatim", "developer", R"({"a":1})", R"({"a":1})");
-
-    // Plain text unchanged for all roles.
-    expect_norm_string("user_plain_text", "user", "hello world", "hello world");
-    expect_norm_string("assistant_non_json_text_kept", "assistant", "hi [unclosed", "hi [unclosed");
-
-    // #7324 boundary - user/system/developer literal "null" preserved (never parsed).
-    expect_norm_string("user_literal_null_stays", "user", "null", "null");
-    expect_norm_string("system_literal_null_stays", "system", "null", "null");
-    expect_norm_string("developer_literal_null_stays", "developer", "null", "null");
-
-    // #7324 - assistant/tool literal null collapses to empty string.
-    expect_norm_string("assistant_null_to_empty", "assistant", "null", "");
-    expect_norm_string("tool_null_to_empty", "tool", "null", "");
-
-    // #7324/#7528 - assistant/tool object bookkeeping stringified (stays a string).
-    check(normalize_message_content("assistant", R"({"tool":"x"})").is_string(), "assistant_object_stringified");
-    check(normalize_message_content("tool", R"({"error":"boom"})").is_string(), "tool_object_stringified");
-
-    // #10524-family - a bare scalar that parses as a JSON number stays the string.
-    expect_norm_string("assistant_scalar_number_stays_string", "assistant", "42", "42");
-
-    // baseline - empty content stays empty.
-    expect_norm_string("user_empty_stays_empty", "user", "", "");
-}
-
-// ---- normalize_template_message (BEFORE TEMPLATE sanitizer) ---------------
-
-static void test_template_sanitizer() {
-    // #7528 - a tool message with an ACTUAL array becomes a string.
-    {
-        ordered_json msg = {{"role", "tool"}, {"content", ordered_json::array({{{"type", "text"}, {"text", "r"}}})}};
-        normalize_template_message(msg);
-        check(msg["content"].is_string(), "before_template_tool_array_to_string", "got " + msg["content"].dump());
-    }
-    // #7324 - null content -> "" for any role.
-    {
-        ordered_json msg = {{"role", "assistant"}, {"content", nullptr}};
-        normalize_template_message(msg);
-        check(msg["content"].is_string() && msg["content"] == "", "before_template_null_to_empty");
-    }
-    // object content -> dumped string (would otherwise throw at the template).
-    {
-        ordered_json msg = {{"role", "assistant"}, {"content", {{"x", 1}}}};
-        normalize_template_message(msg);
-        check(msg["content"].is_string(), "before_template_object_to_string", "got " + msg["content"].dump());
-    }
-    // missing content field -> "".
-    {
-        ordered_json msg = {{"role", "user"}};
-        normalize_template_message(msg);
-        check(msg.contains("content") && msg["content"] == "", "before_template_missing_to_empty");
-    }
-    // multimodal: a well-typed user array must be left UNTOUCHED (role!=tool).
-    {
-        ordered_json parts = ordered_json::array();
-        parts.push_back({{"type", "text"}, {"text", "x"}});
-        ordered_json img; img["type"] = "image_url"; img["image_url"] = {{"url", "data:..."}};
-        parts.push_back(img);
-        ordered_json msg = {{"role", "user"}, {"content", parts}};
-        normalize_template_message(msg);
-        check(msg["content"].is_array() && msg["content"].size() == 2, "before_template_user_typed_array_preserved",
-              "got " + msg["content"].dump());
-    }
-    // a plain string is left untouched.
-    {
-        ordered_json msg = {{"role", "user"}, {"content", "hello"}};
-        normalize_template_message(msg);
-        check(msg["content"] == "hello", "before_template_string_untouched");
-    }
-}
-
-// ---- build_reconstructed_message ----------------------------------------
-
-static void test_reconstruction() {
-    const std::string ingredients = R"(["1/4 cup brown sugar","1 pound ground beef"])";
-
-    // #10524 end-state - user JSON-array text, no media -> string content.
-    {
-        ReconstructedMessageInput in;
-        in.role = "user"; in.content = ingredients;
-        auto m = build_reconstructed_message(in);
-        check(m["content"].is_string() && m["content"] == ingredients, "recon_user_json_array_string",
-              "got " + m["content"].dump());
-    }
-    // multimodal - user text + one image on last user msg -> typed array, image kept.
-    {
-        ReconstructedMessageInput in;
-        in.role = "user"; in.content = ingredients; in.is_last_user_msg = true;
-        in.images.push_back("BASE64IMG");
-        auto m = build_reconstructed_message(in);
-        check(m["content"].is_array() && m["content"].size() == 2, "recon_multimodal_text_plus_image",
-              "got " + m["content"].dump());
-        check(m["content"][0]["type"] == "text" && m["content"][0]["text"] == ingredients, "recon_multimodal_text_first");
-        check(m["content"][1]["type"] == "image_url", "recon_multimodal_image_kept");
-    }
-    // multimodal media-only - empty text + image on last user msg.
-    {
-        ReconstructedMessageInput in;
-        in.role = "user"; in.content = ""; in.is_last_user_msg = true;
-        in.images.push_back("BASE64IMG");
-        auto m = build_reconstructed_message(in);
-        check(m["content"].is_array() && m["content"].size() == 1 && m["content"][0]["type"] == "image_url",
-              "recon_media_only", "got " + m["content"].dump());
-    }
-    // #7528 - tool array-string content stays a string.
-    {
-        ReconstructedMessageInput in;
-        in.role = "tool"; in.content = R"(["a","b"])"; in.tool_call_id = "call_1";
-        auto m = build_reconstructed_message(in);
-        check(m["content"].is_string() && m["content"] == R"(["a","b"])", "recon_tool_array_string",
-              "got " + m["content"].dump());
-        check(m["tool_call_id"] == "call_1", "recon_tool_call_id_set");
-    }
-    // tool empty content -> "".
-    {
-        ReconstructedMessageInput in;
-        in.role = "tool"; in.content = "";
-        auto m = build_reconstructed_message(in);
-        check(m["content"].is_string() && m["content"] == "", "recon_tool_empty_to_string");
-    }
-    // #7324 - assistant + tool_calls + empty content -> " " (single space, not "").
-    {
-        ReconstructedMessageInput in;
-        in.role = "assistant"; in.content = "";
-        in.tool_calls = R"([{"id":"c1","type":"function","function":{"name":"f","arguments":"{}"}}])";
-        auto m = build_reconstructed_message(in);
-        check(m["content"].is_string() && m["content"] == " ", "recon_toolcalls_empty_content_space",
-              "got " + m["content"].dump());
-        check(m["tool_calls"].is_array() && m["tool_calls"].size() == 1, "recon_toolcalls_parsed");
-    }
-    // assistant + tool_calls + real content keeps the content.
-    {
-        ReconstructedMessageInput in;
-        in.role = "assistant"; in.content = "I'll call f";
-        in.tool_calls = R"([{"id":"c1","type":"function","function":{"name":"f","arguments":"{}"}}])";
-        auto m = build_reconstructed_message(in);
-        check(m["content"] == "I'll call f", "recon_toolcalls_with_content_kept");
-    }
-    // assistant null content -> "".
-    {
-        ReconstructedMessageInput in;
-        in.role = "assistant"; in.content = "null";
-        auto m = build_reconstructed_message(in);
-        check(m["content"] == "", "recon_assistant_null_to_empty");
-    }
-    // malformed tool_calls JSON must not throw; content preserved.
-    {
-        ReconstructedMessageInput in;
-        in.role = "assistant"; in.content = "hi"; in.tool_calls = "{not json";
-        auto m = build_reconstructed_message(in);
-        check(m["content"] == "hi" && !m.contains("tool_calls"), "recon_malformed_toolcalls_safe");
-    }
-    // optional fields: name + reasoning carried through.
-    {
-        ReconstructedMessageInput in;
-        in.role = "tool"; in.content = "result"; in.name = "get_weather"; in.reasoning_content = "thinking";
-        auto m = build_reconstructed_message(in);
-        check(m["name"] == "get_weather" && m["reasoning_content"] == "thinking", "recon_optional_fields");
-    }
-}
-
-int main() {
-    test_normalize();
-    test_template_sanitizer();
-    test_reconstruction();
-
-    if (failures == 0) {
-        std::cout << "OK: all message_content tests passed\n";
-        return 0;
-    }
-    std::cerr << failures << " test(s) failed\n";
-    return 1;
-}

backend/cpp/llama-cpp/package.sh

@@ -14,22 +14,6 @@ mkdir -p $CURDIR/package/lib
 cp -avrf $CURDIR/llama-cpp-* $CURDIR/package/
 cp -rfv $CURDIR/run.sh $CURDIR/package/
 
-# Bundle the ggml shared backends produced by the CPU_ALL_VARIANTS build (libggml-base.so,
-# libggml.so, libllama.so and the per-microarch libggml-cpu-*.so), all into package/lib.
-#
-# Two distinct resolution mechanisms both land here:
-#   - NEEDED deps (libggml-base/libggml/libllama): resolved by the dynamic linker via the
-#     LD_LIBRARY_PATH=$CURDIR/lib that run.sh exports.
-#   - The per-microarch libggml-cpu-*.so are NOT linked; ggml *discovers* them at runtime by
-#     scanning the executable's own directory (readlink /proc/self/exe). run.sh launches via
-#     the bundled $CURDIR/lib/ld.so, so /proc/self/exe -> .../lib/ld.so and ggml scans lib/.
-#     That is why the variants must sit in lib/ (next to ld.so), not just on the link path.
-# No-op on builds (arm64/darwin) that don't produce the all-variants set.
-if [ -d "$CURDIR/ggml-shared-libs" ]; then
-    echo "Bundling ggml shared backends (CPU_ALL_VARIANTS)..."
-    cp -avf $CURDIR/ggml-shared-libs/*.so* $CURDIR/package/lib/
-fi
-
 # Detect architecture and copy appropriate libraries
 if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
     # x86_64 architecture

backend/cpp/llama-cpp/parent_watch.h

@@ -1,179 +0,0 @@
-// Parent-death watcher (best-effort backstop) for the llama.cpp gRPC backend.
-//
-// LocalAI spawns this backend as a child process and, on a clean shutdown,
-// tears it down itself (SIGTERM -> grace -> SIGKILL). That graceful path only
-// runs when LocalAI receives a catchable signal and lives long enough to run
-// its handlers. If LocalAI is SIGKILLed (e.g. a supervising process's grace
-// period elapses first), that teardown never runs and this backend would be
-// reparented to init and linger, holding VRAM and its listen port.
-//
-// The watcher here is a best-effort backstop for exactly that case: it does
-// NOT replace the graceful teardown, it only covers the "parent vanished
-// without cleaning up" path. It detects reparenting: when the process that
-// spawned this backend dies, the kernel reparents us to the nearest sub-reaper
-// or to init (PID 1), so getppid() stops matching the value captured at
-// startup. This getppid() approach is portable across Linux/macOS (unlike the
-// Linux-only PR_SET_PDEATHSIG), which is why it is used here, mirroring the Go
-// backends' pkg/grpc/parentwatch.go. It is disabled on Windows, which has no
-// equivalent orphan-reparenting semantics.
-//
-// This header is intentionally dependency-free (C++ standard library only) so
-// it can be exercised by a standalone unit test (parent_watch_test.cpp) without
-// building the full llama.cpp + gRPC backend.
-#ifndef LLAMA_GRPC_PARENT_WATCH_H
-#define LLAMA_GRPC_PARENT_WATCH_H
-
-#include <algorithm>
-#include <cctype>
-#include <chrono>
-#include <cstdio>
-#include <cstdlib>
-#include <functional>
-#include <string>
-#include <thread>
-
-#if !defined(_WIN32)
-#include <unistd.h> // getppid(2), _exit(2)
-#endif
-
-namespace llama_grpc {
-
-// Env var names are shared verbatim with the Go and Python backends for
-// consistency across languages.
-inline const char *kEnvParentWatch()         { return "LOCALAI_BACKEND_PARENT_WATCH"; }
-inline const char *kEnvParentWatchInterval() { return "LOCALAI_BACKEND_PARENT_WATCH_INTERVAL"; }
-
-// Default poll interval in milliseconds. Matches the Go side's 2 * time.Second.
-inline long parent_watch_default_interval_ms() { return 2000; }
-
-namespace detail {
-inline std::string trim_lower(const std::string &in, bool lower) {
-    size_t a = in.find_first_not_of(" \t\r\n");
-    size_t b = in.find_last_not_of(" \t\r\n");
-    if (a == std::string::npos) {
-        return "";
-    }
-    std::string s = in.substr(a, b - a + 1);
-    if (lower) {
-        std::transform(s.begin(), s.end(), s.begin(),
-                       [](unsigned char c) { return std::tolower(c); });
-    }
-    return s;
-}
-} // namespace detail
-
-// parent_watch_enabled reports whether the watcher should run. Enabled by
-// default; a falsey value ("false"/"0"/"no"/"off", case-insensitive) disables
-// it, matching the Go implementation's exact semantics.
-inline bool parent_watch_enabled() {
-#if defined(_WIN32)
-    return false;
-#else
-    const char *v = std::getenv(kEnvParentWatch());
-    if (v == nullptr || v[0] == '\0') {
-        return true;
-    }
-    const std::string s = detail::trim_lower(v, true);
-    return !(s == "false" || s == "0" || s == "no" || s == "off");
-#endif
-}
-
-// parent_watch_interval_ms returns the poll interval in milliseconds. Accepts
-// Go-style duration strings ("500ms", "2s", "1m") for cross-language parity, or
-// a bare number interpreted as seconds. Defaults to
-// parent_watch_default_interval_ms().
-inline long parent_watch_interval_ms() {
-    const long def = parent_watch_default_interval_ms();
-    const char *v = std::getenv(kEnvParentWatchInterval());
-    if (v == nullptr || v[0] == '\0') {
-        return def;
-    }
-    const std::string s = detail::trim_lower(v, false);
-    if (s.empty()) {
-        return def;
-    }
-    size_t i = 0;
-    while (i < s.size() && (std::isdigit((unsigned char)s[i]) || s[i] == '.')) {
-        i++;
-    }
-    if (i == 0) {
-        return def;
-    }
-    double num = 0.0;
-    try {
-        num = std::stod(s.substr(0, i));
-    } catch (...) {
-        return def;
-    }
-    const std::string unit = s.substr(i);
-    long ms;
-    if (unit == "ms") {
-        ms = (long)num;
-    } else if (unit == "s" || unit.empty()) {
-        ms = (long)(num * 1000.0);
-    } else if (unit == "m") {
-        ms = (long)(num * 60000.0);
-    } else {
-        return def; // unrecognized unit
-    }
-    return ms > 0 ? ms : def;
-}
-
-#if !defined(_WIN32)
-// parent_died reports whether this process has been reparented away from the
-// parent it had when the watcher started. Reparenting is the standard POSIX
-// signal that the original parent (here, the LocalAI process that spawned this
-// backend) has exited: the orphan is handed to the nearest sub-reaper or to
-// init (PID 1), so getppid() no longer matches the value captured at startup.
-inline bool parent_died(pid_t orig_ppid) {
-    const pid_t ppid = getppid();
-    return ppid != orig_ppid || ppid == 1;
-}
-
-// watch_parent_death polls until parent_died reports the original parent is
-// gone, then invokes on_death. It blocks, so run it on its own thread.
-inline void watch_parent_death(pid_t orig_ppid, long interval_ms,
-                               const std::function<void()> &on_death) {
-    for (;;) {
-        std::this_thread::sleep_for(std::chrono::milliseconds(interval_ms));
-        if (parent_died(orig_ppid)) {
-            on_death();
-            return;
-        }
-    }
-}
-#endif
-
-// start_parent_death_watcher installs the best-effort safety net described in
-// the file header on the calling backend process. It is a no-op when disabled,
-// on Windows, or when the process is already orphaned at startup
-// (getppid() <= 1). This is a backstop alongside — never a replacement for —
-// LocalAI's graceful teardown.
-inline void start_parent_death_watcher() {
-#if !defined(_WIN32)
-    if (!parent_watch_enabled()) {
-        return;
-    }
-    const pid_t orig_ppid = getppid();
-    // A parent of 1 (or less) at startup means we were already orphaned (or
-    // launched directly under init) — there is no original parent to watch for.
-    if (orig_ppid <= 1) {
-        return;
-    }
-    const long interval_ms = parent_watch_interval_ms();
-    std::thread([orig_ppid, interval_ms]() {
-        watch_parent_death(orig_ppid, interval_ms, [orig_ppid]() {
-            fprintf(stderr,
-                    "backend parent process (pid %d) exited without stopping "
-                    "this backend; self-terminating to avoid orphaning\n",
-                    (int)orig_ppid);
-            fflush(stderr);
-            _exit(1);
-        });
-    }).detach();
-#endif
-}
-
-} // namespace llama_grpc
-
-#endif // LLAMA_GRPC_PARENT_WATCH_H

backend/cpp/llama-cpp/parent_watch_test.cpp

@@ -1,197 +0,0 @@
-// Unit tests for the parent-death watcher (parent_watch.h).
-//
-// Build & run standalone (C++ standard library only, no nlohmann/json needed):
-//   g++ -std=c++17 -pthread parent_watch_test.cpp -o t && ./t
-//
-// The core test (TestDetectsReparent) builds a genuine two-level process tree
-// (test -> middle -> grandchild), lets the middle process die, and asserts the
-// grandchild's watch_parent_death detects the reparenting and self-terminates —
-// mirroring the Go test in pkg/grpc/parentwatch_test.go, but with fork(2).
-//
-// On Windows this file compiles to a no-op success (the watcher is unsupported
-// there), matching parent_watch.h's platform gating.
-
-#include <cstdio>
-#include <cstdlib>
-#include <string>
-
-#include "parent_watch.h"
-
-static int failures = 0;
-
-static void check(bool ok, const std::string &name) {
-    if (!ok) {
-        failures++;
-        fprintf(stderr, "FAIL: %s\n", name.c_str());
-    } else {
-        fprintf(stderr, "ok:   %s\n", name.c_str());
-    }
-}
-
-// Env-parsing tests are platform-independent and always run.
-static void test_env_parsing() {
-    using namespace llama_grpc;
-
-    // Interval: default when unset.
-    unsetenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL");
-    check(parent_watch_interval_ms() == 2000, "interval default 2000ms");
-
-    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "500ms", 1);
-    check(parent_watch_interval_ms() == 500, "interval 500ms");
-
-    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "2s", 1);
-    check(parent_watch_interval_ms() == 2000, "interval 2s");
-
-    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "1m", 1);
-    check(parent_watch_interval_ms() == 60000, "interval 1m");
-
-    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "3", 1); // bare number -> seconds
-    check(parent_watch_interval_ms() == 3000, "interval bare 3 -> 3000ms");
-
-    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "garbage", 1);
-    check(parent_watch_interval_ms() == 2000, "interval garbage -> default");
-    unsetenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL");
-
-#if !defined(_WIN32)
-    // Enabled semantics (POSIX only; always false on Windows).
-    unsetenv("LOCALAI_BACKEND_PARENT_WATCH");
-    check(parent_watch_enabled(), "enabled by default");
-
-    for (const char *falsey : {"false", "0", "no", "off", "OFF", " False "}) {
-        setenv("LOCALAI_BACKEND_PARENT_WATCH", falsey, 1);
-        check(!parent_watch_enabled(), std::string("disabled by '") + falsey + "'");
-    }
-    setenv("LOCALAI_BACKEND_PARENT_WATCH", "true", 1);
-    check(parent_watch_enabled(), "enabled by 'true'");
-    setenv("LOCALAI_BACKEND_PARENT_WATCH", "1", 1);
-    check(parent_watch_enabled(), "enabled by '1'");
-    unsetenv("LOCALAI_BACKEND_PARENT_WATCH");
-#endif
-}
-
-#if !defined(_WIN32)
-
-#include <atomic>
-#include <ctime>
-#include <sys/stat.h>
-#include <sys/wait.h>
-#include <unistd.h>
-
-static bool file_exists(const std::string &p) {
-    struct stat st;
-    return ::stat(p.c_str(), &st) == 0;
-}
-
-static bool wait_for_file(const std::string &p, int timeout_ms) {
-    int waited = 0;
-    while (waited < timeout_ms) {
-        if (file_exists(p)) {
-            return true;
-        }
-        usleep(20 * 1000);
-        waited += 20;
-    }
-    return false;
-}
-
-static void write_file(const std::string &p, const std::string &content) {
-    FILE *f = fopen(p.c_str(), "w");
-    if (f) {
-        fwrite(content.data(), 1, content.size(), f);
-        fclose(f);
-    }
-}
-
-// Builds test -> middle -> grandchild via fork(2). The grandchild arms the REAL
-// watch_parent_death against middle; middle exits, orphaning the grandchild;
-// the watcher must detect the reparenting and self-terminate.
-static void test_detects_reparent() {
-    char tmpl[] = "/tmp/parentwatch_test_XXXXXX";
-    char *dir = mkdtemp(tmpl);
-    if (dir == nullptr) {
-        check(false, "mkdtemp");
-        return;
-    }
-    const std::string ready_file = std::string(dir) + "/ready";
-    const std::string exited_file = std::string(dir) + "/exited";
-
-    pid_t middle = fork();
-    if (middle < 0) {
-        check(false, "fork middle");
-        return;
-    }
-
-    if (middle == 0) {
-        // ---- middle process ----
-        pid_t grandchild = fork();
-        if (grandchild < 0) {
-            _exit(4);
-        }
-        if (grandchild == 0) {
-            // ---- grandchild process ----
-            pid_t orig_ppid = getppid(); // == middle
-            std::thread([&]() {
-                llama_grpc::watch_parent_death(orig_ppid, 50 /*ms*/, [&]() {
-                    write_file(exited_file, "1");
-                    _exit(7);
-                });
-            }).detach();
-
-            // Safety valve: never linger if something goes wrong.
-            std::thread([]() {
-                usleep(30 * 1000 * 1000);
-                _exit(2);
-            }).detach();
-
-            // Signal readiness only after the watcher captured orig_ppid.
-            write_file(ready_file, std::to_string(getpid()));
-            for (;;) {
-                pause();
-            }
-        }
-        // middle: wait until grandchild is ready, then exit to orphan it.
-        if (!wait_for_file(ready_file, 10000)) {
-            _exit(5);
-        }
-        _exit(0);
-    }
-
-    // ---- test (top) process ----
-    int status = 0;
-    waitpid(middle, &status, 0); // reap middle only; grandchild is orphaned
-
-    check(file_exists(ready_file), "grandchild signaled readiness");
-
-    bool detected = wait_for_file(exited_file, 10000);
-    check(detected, "watcher detected parent death and self-terminated");
-
-    // Best-effort cleanup: kill the grandchild if it somehow survived.
-    if (file_exists(ready_file)) {
-        FILE *f = fopen(ready_file.c_str(), "r");
-        if (f) {
-            int pid = 0;
-            if (fscanf(f, "%d", &pid) == 1 && pid > 1) {
-                kill(pid, SIGKILL);
-            }
-            fclose(f);
-        }
-    }
-    unlink(ready_file.c_str());
-    unlink(exited_file.c_str());
-    rmdir(dir);
-}
-
-#endif // !_WIN32
-
-int main() {
-    test_env_parsing();
-#if !defined(_WIN32)
-    test_detects_reparent();
-#endif
-    if (failures == 0) {
-        fprintf(stderr, "\nAll parent_watch tests passed.\n");
-        return 0;
-    }
-    fprintf(stderr, "\n%d parent_watch test(s) failed.\n", failures);
-    return 1;
-}

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch

@@ -1,814 +0,0 @@
-# Vendored from upstream llama.cpp PR #24523 (Preliminary MiniMax-M3 support).
-# Rebased against LLAMA_VERSION 00fa7cb284cbf133fc426733bd64238a3588a33e (also applies cleanly
-# to the later pin 505b1ed15ca80e2a19f12ff4ac365e40fb374053). LLAMA_VERSION is auto-bumped
-# nightly; if a bump rejects this patch, re-vendor from #24523 — or, once #24523 merges
-# upstream, delete this file and bump LLAMA_VERSION normally.
-# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
-diff --git a/common/chat.cpp b/common/chat.cpp
-index 22d2ee4..440be9a 100644
---- a/common/chat.cpp
-+++ b/common/chat.cpp
-@@ -2035,6 +2035,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
-     return data;
- }
- 
-+static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
-+                                                             const autoparser::generation_params & inputs) {
-+    common_chat_params data;
-+
-+    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
-+    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
-+    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
-+    data.supports_thinking  = true;
-+    data.thinking_start_tag = "<mm:think>";
-+    data.thinking_end_tag   = "</mm:think>";
-+
-+    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
-+    // params use the parameter name as the tag (<file_path>...</file_path>).
-+    const std::string NS           = "]<]minimax[>[";
-+    const std::string THINK_START  = "<mm:think>";
-+    const std::string THINK_END    = "</mm:think>";
-+    const std::string FC_START     = NS + "<tool_call>";
-+    const std::string FC_END       = NS + "</tool_call>";
-+    const std::string INVOKE_END   = NS + "</invoke>";
-+
-+    data.preserved_tokens   = {
-+        NS,
-+        "<tool_call>",
-+        "</tool_call>",
-+        THINK_START,
-+        THINK_END,
-+    };
-+
-+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
-+    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
-+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
-+
-+    const std::string GEN_PROMPT = data.generation_prompt;
-+
-+    if (inputs.has_continuation()) {
-+        const auto & msg = inputs.continue_msg;
-+
-+        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
-+        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
-+            data.generation_prompt += THINK_END + msg.render_content();
-+        }
-+
-+        data.prompt += data.generation_prompt;
-+    }
-+
-+    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
-+        auto generation_prompt = p.literal(GEN_PROMPT);
-+        auto end = p.end();
-+
-+        auto reasoning = p.eps();
-+        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
-+        if (extract_reasoning && inputs.enable_thinking) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
-+        } else if (extract_reasoning) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
-+        }
-+
-+        if (has_response_format) {
-+            auto response_format = p.rule("response-format",
-+                p.literal("```json") + p.space() +
-+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
-+                p.space() + p.literal("```"));
-+            return generation_prompt + reasoning + response_format + end;
-+        }
-+
-+        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
-+            return generation_prompt + reasoning + p.content(p.rest()) + end;
-+        }
-+
-+        auto tool_choice = p.choice();
-+        foreach_function(inputs.tools, [&](const json & tool) {
-+            const auto & function = tool.at("function");
-+            std::string  name     = function.at("name");
-+            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
-+            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
-+
-+            std::set<std::string> required;
-+            if (params.contains("required")) {
-+                params.at("required").get_to(required);
-+            }
-+
-+            auto schema_info = common_schema_info();
-+            schema_info.resolve_refs(params);
-+
-+            std::vector<common_peg_parser> required_parsers;
-+            std::vector<common_peg_parser> optional_parsers;
-+            for (const auto & [param_name, param_schema] : props.items()) {
-+                bool is_required = required.find(param_name) != required.end();
-+                bool is_string   = schema_info.resolves_to_string(param_schema);
-+
-+                const std::string p_close = NS + "</" + param_name + ">";
-+
-+                auto arg = p.tool_arg(
-+                    p.tool_arg_open(
-+                        p.literal(NS + "<") +
-+                        p.tool_arg_name(p.literal(param_name)) +
-+                        p.literal(">")) +
-+                    (is_string
-+                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
-+                                p.tool_arg_close(p.literal(p_close)), p_close)
-+                         : p.tool_arg_json_value(p.schema(p.json(),
-+                                                          "tool-" + name + "-arg-" + param_name + "-schema",
-+                                                          param_schema, false)) +
-+                           p.tool_arg_close(p.literal(p_close))));
-+
-+                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
-+                if (is_required) {
-+                    required_parsers.push_back(named_arg);
-+                } else {
-+                    optional_parsers.push_back(named_arg);
-+                }
-+            }
-+
-+            common_peg_parser args_seq = p.eps();
-+            for (size_t i = 0; i < required_parsers.size(); i++) {
-+                if (i > 0) {
-+                    args_seq = args_seq + p.space();
-+                }
-+                args_seq = args_seq + required_parsers[i];
-+            }
-+
-+            if (!optional_parsers.empty()) {
-+                common_peg_parser any_opt = p.choice();
-+                for (const auto & opt : optional_parsers) {
-+                    any_opt |= opt;
-+                }
-+                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
-+            }
-+
-+            common_peg_parser invoke_body = args_seq;
-+            auto func_parser = p.tool(
-+                p.tool_open(p.literal(NS + "<invoke name=\"") +
-+                            p.tool_name(p.literal(name)) + p.literal("\">")) +
-+                p.space() + invoke_body + p.space() +
-+                p.tool_close(p.literal(INVOKE_END)));
-+
-+            tool_choice |= p.rule("tool-" + name, func_parser);
-+        });
-+
-+        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
-+
-+        common_peg_parser tool_calls = p.eps();
-+        if (inputs.parallel_tool_calls) {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice +
-+                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
-+        } else {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
-+        }
-+
-+        if (!require_tools) {
-+            tool_calls = p.optional(tool_calls);
-+        }
-+
-+        auto content_before_tools = p.content(p.until(FC_START));
-+        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
-+    });
-+
-+    data.parser = parser.save();
-+
-+    if (include_grammar) {
-+        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
-+        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
-+            foreach_function(inputs.tools, [&](const json & tool) {
-+                const auto & function = tool.at("function");
-+                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
-+                builder.resolve_refs(schema);
-+            });
-+            if (has_response_format) {
-+                auto schema = inputs.json_schema;
-+                builder.resolve_refs(schema);
-+            }
-+            parser.build_grammar(builder, data.grammar_lazy);
-+        });
-+
-+        data.grammar_triggers = {
-+            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
-+        };
-+    }
-+
-+    return data;
-+}
-+
- // Cohere2 MoE (a.k.a. "North Code") parser.
- //
- // The assistant turn is fully marker-wrapped:
-@@ -2612,6 +2797,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
-         return common_chat_params_init_gigachat_v3(tmpl, params);
-     }
- 
-+    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
-+    // markup delimiters, so detect the template and use a dedicated parser.
-+    if (src.find("]<]minimax[>[") != std::string::npos &&
-+        src.find("<tool_call>") != std::string::npos &&
-+        src.find("<invoke name=") != std::string::npos) {
-+        LOG_DBG("Using specialized template: MiniMax-M3\n");
-+        return common_chat_params_init_minimax_m3(tmpl, params);
-+    }
-+
-     // DeepSeek V3.2 format detection: template defines dsml_token and uses it for tool calls.
-     // The template source contains the token as a variable assignment, not as a literal in markup.
-     if (src.find("dsml_token") != std::string::npos &&
-diff --git a/conversion/__init__.py b/conversion/__init__.py
-index 02ea638..71de528 100644
---- a/conversion/__init__.py
-+++ b/conversion/__init__.py
-@@ -155,6 +155,8 @@ TEXT_MODEL_MAP: dict[str, str] = {
-     "MiniCPMForCausalLM": "minicpm",
-     "MiniCPMV4_6ForConditionalGeneration": "minicpm",
-     "MiniMaxM2ForCausalLM": "minimax",
-+    "MiniMaxM3SparseForCausalLM": "minimax",
-+    "MiniMaxM3SparseForConditionalGeneration": "minimax",
-     "Ministral3ForCausalLM": "mistral3",
-     "Mistral3ForConditionalGeneration": "mistral3",
-     "MistralForCausalLM": "llama",
-diff --git a/conversion/base.py b/conversion/base.py
-index 0421aa4..224481a 100644
---- a/conversion/base.py
-+++ b/conversion/base.py
-@@ -1154,7 +1154,8 @@ class TextModel(ModelBase):
-                 or "projector." in name or "pre_mm_projector_norm" in name \
-                 or "image_newline" in name or "view_seperator" in name \
-                 or "patch_embed" in name or "patch_embedding" in name \
--                or "patch_merger." in name or "model.connector." in name:
-+                or "patch_merger." in name or "patch_merge_mlp" in name \
-+                or "model.connector." in name:
-             return None
- 
-         return super().filter_tensors(item)
-@@ -1201,7 +1202,7 @@ class TextModel(ModelBase):
-             self.gguf_writer.add_embedding_length(n_embd)
-             logger.info(f"gguf: embedding length = {n_embd}")
- 
--        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-+        if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
-             self.gguf_writer.add_feed_forward_length(n_ff)
-             logger.info(f"gguf: feed forward length = {n_ff}")
- 
-diff --git a/conversion/minimax.py b/conversion/minimax.py
-index 4857775..4f637f5 100644
---- a/conversion/minimax.py
-+++ b/conversion/minimax.py
-@@ -52,3 +52,67 @@ class MiniMaxM2Model(TextModel):
-             return
- 
-         yield from super().modify_tensors(data_torch, name, bid)
-+
-+
-+@ModelBase.register("MiniMaxM3SparseForCausalLM", "MiniMaxM3SparseForConditionalGeneration")
-+class MiniMaxM3Model(TextModel):
-+    # Text-only MiniMax-M3: MiniMax-M2 GQA + DeepSeek-V3 shared/leading-dense experts (swigluoai).
-+    model_arch = gguf.MODEL_ARCH.MINIMAXM3
-+    _experts_cache: dict[int, dict[str, Tensor]] = {}
-+
-+    def set_gguf_parameters(self):
-+        # feed_forward_length comes from dense_intermediate_size (base); experts use intermediate_size.
-+        super().set_gguf_parameters()
-+
-+        self.gguf_writer.add_expert_feed_forward_length(self.find_hparam(["intermediate_size"]))
-+        self.gguf_writer.add_rope_dimension_count(self.find_hparam(["rotary_dim"]))
-+        self.gguf_writer.add_expert_shared_count(self.find_hparam(["n_shared_experts"]))
-+        self.gguf_writer.add_expert_weights_scale(self.find_hparam(["routed_scaling_factor"]))
-+        self.gguf_writer.add_expert_weights_norm(True)
-+
-+        # leading dense layers: moe_layer_freq (ints) or mlp_layer_types (Transformers 5.12, strings)
-+        moe_layer_freq = self.find_hparam(["moe_layer_freq", "mlp_layer_types"])
-+        n_dense = 0
-+        for v in moe_layer_freq:
-+            if v == 0 or v == "dense":
-+                n_dense += 1
-+            else:
-+                break
-+        self.gguf_writer.add_leading_dense_block_count(n_dense)
-+
-+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None):
-+        # index_* (sparse-attn indexer) tensors are preserved but unused; the loader skips them
-+        if name.startswith("language_model."):
-+            name = name[len("language_model."):]
-+
-+        # Gemma-style (1+w) RMSNorm: bake +1 in so llama.cpp can use plain RMSNorm
-+        if name.endswith("norm.weight"):
-+            data_torch = data_torch + 1.0
-+
-+        # merge routed experts (w1/w2/w3); shared_experts.* passes through to *_shexp
-+        if "block_sparse_moe.experts." in name:
-+            n_experts = self.find_hparam(["num_local_experts", "num_experts"])
-+            assert bid is not None
-+
-+            expert_cache = self._experts_cache.setdefault(bid, {})
-+            expert_cache[name] = data_torch
-+            expert_weights = ["w1", "w2", "w3"]
-+
-+            if len(expert_cache) < n_experts * len(expert_weights):
-+                return
-+
-+            for w_name in expert_weights:
-+                datas: list[Tensor] = []
-+                for xid in range(n_experts):
-+                    ename = f"model.layers.{bid}.block_sparse_moe.experts.{xid}.{w_name}.weight"
-+                    datas.append(expert_cache[ename])
-+                    del expert_cache[ename]
-+
-+                data_torch = torch.stack(datas, dim=0)
-+                merged_name = f"model.layers.{bid}.block_sparse_moe.experts.{w_name}.weight"
-+                yield from super().modify_tensors(data_torch, merged_name, bid)
-+
-+            del self._experts_cache[bid]
-+            return
-+
-+        yield from super().modify_tensors(data_torch, name, bid)
-diff --git a/gguf-py/gguf/constants.py b/gguf-py/gguf/constants.py
-index 869e436..760e3dd 100644
---- a/gguf-py/gguf/constants.py
-+++ b/gguf-py/gguf/constants.py
-@@ -525,6 +525,7 @@ class MODEL_ARCH(IntEnum):
-     APERTUS          = auto()
-     COGVLM           = auto()
-     MINIMAXM2        = auto()
-+    MINIMAXM3        = auto()
-     RND1             = auto()
-     PANGU_EMBED      = auto()
-     MISTRAL3         = auto()
-@@ -613,6 +614,10 @@ class MODEL_TENSOR(IntEnum):
-     MOE_LATENT_UP        = auto() # nemotron 3 super
-     ATTN_Q_NORM          = auto()
-     ATTN_K_NORM          = auto()
-+    ATTN_INDEX_Q         = auto() # minimax-m3 sparse-attn indexer (unused)
-+    ATTN_INDEX_K         = auto()
-+    ATTN_INDEX_Q_NORM    = auto()
-+    ATTN_INDEX_K_NORM    = auto()
-     LAYER_OUT_NORM       = auto()
-     LAYER_OUT_SCALE      = auto()
-     PER_LAYER_TOKEN_EMBD = auto() # gemma3n
-@@ -1105,6 +1110,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
-     MODEL_ARCH.GROVEMOE:         "grovemoe",
-     MODEL_ARCH.APERTUS:          "apertus",
-     MODEL_ARCH.MINIMAXM2:        "minimax-m2",
-+    MODEL_ARCH.MINIMAXM3:        "minimax-m3",
-     MODEL_ARCH.COGVLM:           "cogvlm",
-     MODEL_ARCH.RND1:             "rnd1",
-     MODEL_ARCH.PANGU_EMBED:      "pangu-embedded",
-@@ -1163,6 +1169,10 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
-     MODEL_TENSOR.ATTN_GATE:                 "blk.{bid}.attn_gate",
-     MODEL_TENSOR.ATTN_Q_NORM:               "blk.{bid}.attn_q_norm",
-     MODEL_TENSOR.ATTN_K_NORM:               "blk.{bid}.attn_k_norm",
-+    MODEL_TENSOR.ATTN_INDEX_Q:              "blk.{bid}.attn_index_q",
-+    MODEL_TENSOR.ATTN_INDEX_K:              "blk.{bid}.attn_index_k",
-+    MODEL_TENSOR.ATTN_INDEX_Q_NORM:         "blk.{bid}.attn_index_q_norm",
-+    MODEL_TENSOR.ATTN_INDEX_K_NORM:         "blk.{bid}.attn_index_k_norm",
-     MODEL_TENSOR.ATTN_OUT_NORM:             "blk.{bid}.attn_output_norm",
-     MODEL_TENSOR.ATTN_POST_NORM:            "blk.{bid}.post_attention_norm",
-     MODEL_TENSOR.FFN_GATE_INP:              "blk.{bid}.ffn_gate_inp",
-@@ -4102,6 +4112,30 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.FFN_UP_EXP,
-         MODEL_TENSOR.FFN_EXP_PROBS_B,
-     ],
-+    MODEL_ARCH.MINIMAXM3: [
-+        MODEL_TENSOR.TOKEN_EMBD,
-+        MODEL_TENSOR.OUTPUT_NORM,
-+        MODEL_TENSOR.OUTPUT,
-+        MODEL_TENSOR.ATTN_NORM,
-+        MODEL_TENSOR.ATTN_Q,
-+        MODEL_TENSOR.ATTN_Q_NORM,
-+        MODEL_TENSOR.ATTN_K,
-+        MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_V,
-+        MODEL_TENSOR.ATTN_OUT,
-+        MODEL_TENSOR.FFN_NORM,
-+        MODEL_TENSOR.FFN_GATE_INP,
-+        MODEL_TENSOR.FFN_EXP_PROBS_B,
-+        MODEL_TENSOR.FFN_GATE_EXP,
-+        MODEL_TENSOR.FFN_DOWN_EXP,
-+        MODEL_TENSOR.FFN_UP_EXP,
-+        MODEL_TENSOR.FFN_GATE_SHEXP,
-+        MODEL_TENSOR.FFN_DOWN_SHEXP,
-+        MODEL_TENSOR.FFN_UP_SHEXP,
-+        MODEL_TENSOR.FFN_GATE,
-+        MODEL_TENSOR.FFN_DOWN,
-+        MODEL_TENSOR.FFN_UP,
-+    ],
-     MODEL_ARCH.COGVLM: [
-         MODEL_TENSOR.TOKEN_EMBD,
-         MODEL_TENSOR.OUTPUT_NORM,
-@@ -4128,6 +4162,10 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
-         MODEL_TENSOR.ATTN_Q_NORM,
-         MODEL_TENSOR.ATTN_K,
-         MODEL_TENSOR.ATTN_K_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_Q,
-+        MODEL_TENSOR.ATTN_INDEX_K,
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM,
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM,
-         MODEL_TENSOR.ATTN_V,
-         MODEL_TENSOR.ATTN_OUT,
-         MODEL_TENSOR.FFN_NORM,
-diff --git a/gguf-py/gguf/tensor_mapping.py b/gguf-py/gguf/tensor_mapping.py
-index 9efb36f..a62040b 100644
---- a/gguf-py/gguf/tensor_mapping.py
-+++ b/gguf-py/gguf/tensor_mapping.py
-@@ -717,6 +717,22 @@ class TensorNameMap:
-             "model.layers.{bid}.attention.key_layernorm",                     # apertus
-         ),
- 
-+        MODEL_TENSOR.ATTN_INDEX_Q: (
-+            "model.layers.{bid}.self_attn.index_q_proj",                      # minimax-m3 (sparse-attn indexer)
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K: (
-+            "model.layers.{bid}.self_attn.index_k_proj",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_Q_NORM: (
-+            "model.layers.{bid}.self_attn.index_q_norm",                      # minimax-m3
-+        ),
-+
-+        MODEL_TENSOR.ATTN_INDEX_K_NORM: (
-+            "model.layers.{bid}.self_attn.index_k_norm",                      # minimax-m3
-+        ),
-+
-         MODEL_TENSOR.ROPE_FREQS: (
-             "encoder.layers.{bid}.self_attention.rotary_emb.inv_freq",  # persimmon
-         ),
-diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
-index b890e66..cb8bfc8 100644
---- a/src/llama-arch.cpp
-+++ b/src/llama-arch.cpp
-@@ -125,6 +125,7 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
-     { LLM_ARCH_GROVEMOE,         "grovemoe"         },
-     { LLM_ARCH_APERTUS,          "apertus"          },
-     { LLM_ARCH_MINIMAX_M2,       "minimax-m2"       },
-+    { LLM_ARCH_MINIMAX_M3,       "minimax-m3"       },
-     { LLM_ARCH_COGVLM,           "cogvlm"           },
-     { LLM_ARCH_RND1,             "rnd1"             },
-     { LLM_ARCH_PANGU_EMBED,      "pangu-embedded"   },
-@@ -395,6 +396,10 @@ static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
-     { LLM_TENSOR_ATTN_POST_NORM,                         "blk.%d.post_attention_norm" },
-     { LLM_TENSOR_ATTN_Q_NORM,                            "blk.%d.attn_q_norm" },
-     { LLM_TENSOR_ATTN_K_NORM,                            "blk.%d.attn_k_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_Q,                           "blk.%d.attn_index_q" },
-+    { LLM_TENSOR_ATTN_INDEX_K,                           "blk.%d.attn_index_k" },
-+    { LLM_TENSOR_ATTN_INDEX_Q_NORM,                      "blk.%d.attn_index_q_norm" },
-+    { LLM_TENSOR_ATTN_INDEX_K_NORM,                      "blk.%d.attn_index_k_norm" },
-     { LLM_TENSOR_ATTN_GATE,                              "blk.%d.attn_gate" },
-     { LLM_TENSOR_FFN_POST_NORM,                          "blk.%d.post_ffw_norm" },
-     { LLM_TENSOR_FFN_POST_NORM_1,                        "blk.%d.post_ffw_norm_1" },
-@@ -761,6 +766,11 @@ static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
-     {LLM_TENSOR_FFN_NORM_EXPS,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_K_NORM,                {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-+    // minimax-m3 sparse-attn indexer: unused (GGML_OP_NONE) so the loader skips it
-+    {LLM_TENSOR_ATTN_INDEX_Q,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K,               {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_Q_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-+    {LLM_TENSOR_ATTN_INDEX_K_NORM,          {LLM_TENSOR_LAYER_REPEATING, GGML_OP_NONE}},
-     {LLM_TENSOR_LAYER_OUT_NORM,             {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_LAYER_OUT_SCALE,            {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-     {LLM_TENSOR_ATTN_Q_A_NORM,              {LLM_TENSOR_LAYER_REPEATING, GGML_OP_MUL}},
-@@ -998,6 +1008,7 @@ bool llm_arch_supports_sm_tensor(const llm_arch & arch) {
-         case LLM_ARCH_LFM2:
-         case LLM_ARCH_LFM2MOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_MISTRAL4:
-         case LLM_ARCH_KIMI_LINEAR:
-             return false;
-diff --git a/src/llama-arch.h b/src/llama-arch.h
-index a4f5091..2d50ead 100644
---- a/src/llama-arch.h
-+++ b/src/llama-arch.h
-@@ -144,6 +144,7 @@ enum llm_arch {
-     LLM_ARCH_TALKIE,
-     LLM_ARCH_MELLUM,
-     LLM_ARCH_EAGLE3,
-+    LLM_ARCH_MINIMAX_M3,
-     LLM_ARCH_DFLASH,
-     LLM_ARCH_UNKNOWN,
- };
-@@ -429,6 +430,10 @@ enum llm_tensor {
-     LLM_TENSOR_FFN_LATENT_UP,
-     LLM_TENSOR_ATTN_Q_NORM,
-     LLM_TENSOR_ATTN_K_NORM,
-+    LLM_TENSOR_ATTN_INDEX_Q,      // minimax-m3 sparse-attn indexer (unused)
-+    LLM_TENSOR_ATTN_INDEX_K,
-+    LLM_TENSOR_ATTN_INDEX_Q_NORM,
-+    LLM_TENSOR_ATTN_INDEX_K_NORM,
-     LLM_TENSOR_LAYER_OUT_NORM,
-     LLM_TENSOR_LAYER_OUT_SCALE,
-     LLM_TENSOR_POST_ATTN_NORM,
-diff --git a/src/llama-graph.cpp b/src/llama-graph.cpp
-index c8ecb0a..4c2c286 100644
---- a/src/llama-graph.cpp
-+++ b/src/llama-graph.cpp
-@@ -1719,6 +1719,16 @@ ggml_tensor * llm_graph_context::build_ffn(
-                 cur = ggml_reglu(ctx0, cur);
-                 cb(cur, "ffn_reglu", il);
-             } break;
-+        case LLM_FFN_SWIGLU_OAI:
-+            {
-+                // clamped SwiGLU: parallel gate path (cur=gate, tmp=up)
-+                GGML_ASSERT(gate && type_gate == LLM_FFN_PAR);
-+                constexpr float alpha = 1.702f;
-+                constexpr float limit = 7.0f;
-+                cur = ggml_swiglu_oai(ctx0, cur, tmp, alpha, limit);
-+                cb(cur, "ffn_swiglu_oai", il);
-+                type_gate = LLM_FFN_SEQ; // gate*up already fused; skip the par multiply
-+            } break;
-         default:
-             GGML_ABORT("fatal error");
-     }
-diff --git a/src/llama-graph.h b/src/llama-graph.h
-index c84cb6a..806ce7b 100644
---- a/src/llama-graph.h
-+++ b/src/llama-graph.h
-@@ -54,6 +54,7 @@ enum llm_ffn_op_type : int {
-     LLM_FFN_SWIGLU,
-     LLM_FFN_GEGLU,
-     LLM_FFN_REGLU,
-+    LLM_FFN_SWIGLU_OAI,
-     LLM_FFN_SWIGLU_OAI_MOE,
- };
- 
-diff --git a/src/llama-model.cpp b/src/llama-model.cpp
-index d874813..7bb71c0 100644
---- a/src/llama-model.cpp
-+++ b/src/llama-model.cpp
-@@ -280,6 +280,8 @@ static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params
-             return new llama_model_apertus(params);
-         case LLM_ARCH_MINIMAX_M2:
-             return new llama_model_minimax_m2(params);
-+        case LLM_ARCH_MINIMAX_M3:
-+            return new llama_model_minimax_m3(params);
-         case LLM_ARCH_COGVLM:
-             return new llama_model_cogvlm(params);
-         case LLM_ARCH_PANGU_EMBED:
-@@ -807,6 +809,7 @@ const char * llm_type_name(llm_type type) {
-         case LLM_TYPE_310B_A15B:     return "310B.A15B";
-         case LLM_TYPE_355B_A32B:     return "355B.A32B";
-         case LLM_TYPE_397B_A17B:     return "397B.A17B";
-+        case LLM_TYPE_428B_A23B:     return "428B.A23B";
-         case LLM_TYPE_685B_A37B:     return "685B.A37B";
-         case LLM_TYPE_744B_A40B:     return "744B.A40B";
-         case LLM_TYPE_E2B:           return "E2B";
-@@ -2532,6 +2535,7 @@ llama_rope_type llama_model_rope_type(const llama_model * model) {
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_APERTUS:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_COGVLM:
-         case LLM_ARCH_PANGU_EMBED:
-         case LLM_ARCH_AFMOE:
-diff --git a/src/llama-model.h b/src/llama-model.h
-index 45b054c..540e0d2 100644
---- a/src/llama-model.h
-+++ b/src/llama-model.h
-@@ -139,6 +139,7 @@ enum llm_type {
-     LLM_TYPE_310B_A15B, // /MiMo-V2-Flash
-     LLM_TYPE_355B_A32B, // GLM-4.5
-     LLM_TYPE_397B_A17B, // Qwen3.5
-+    LLM_TYPE_428B_A23B, // MiniMax M3
-     LLM_TYPE_685B_A37B, // DeepSeek V3.2
-     LLM_TYPE_744B_A40B, // GLM-5
-     LLM_TYPE_E2B,
-diff --git a/src/models/minimax-m3.cpp b/src/models/minimax-m3.cpp
-new file mode 100644
-index 0000000..137852a
---- /dev/null
-+++ b/src/models/minimax-m3.cpp
-@@ -0,0 +1,197 @@
-+#include "models.h"
-+
-+// MiniMax-M3, text-only: MiniMax-M2 GQA (per-head QK-norm, partial rotary) + DeepSeek-V3
-+// leading-dense/routed/shared experts (swigluoai). Sparse attn -> dense; vision + MTP dropped.
-+
-+void llama_model_minimax_m3::load_arch_hparams(llama_model_loader & ml) {
-+    ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
-+    ml.get_key(LLM_KV_LEADING_DENSE_BLOCK_COUNT,   hparams.n_layer_dense_lead, false);
-+    ml.get_key(LLM_KV_EXPERT_FEED_FORWARD_LENGTH,  hparams.n_ff_exp);
-+    ml.get_key(LLM_KV_EXPERT_SHARED_COUNT,         hparams.n_expert_shared);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE,        hparams.expert_weights_scale, false);
-+    ml.get_key(LLM_KV_EXPERT_WEIGHTS_NORM,         hparams.expert_weights_norm, false);
-+    ml.get_key(LLM_KV_EXPERT_GATING_FUNC,          hparams.expert_gating_func);
-+
-+    switch (hparams.n_layer()) {
-+        case 60: type = LLM_TYPE_428B_A23B; break;
-+        default: type = LLM_TYPE_UNKNOWN;
-+    }
-+}
-+
-+void llama_model_minimax_m3::load_arch_tensors(llama_model_loader &) {
-+    LLAMA_LOAD_LOCALS;
-+    const int64_t n_expert_shared = hparams.n_expert_shared;
-+    const int64_t n_ff_exp        = hparams.n_ff_exp;
-+
-+    tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
-+
-+    output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
-+    output      = create_tensor(tn(LLM_TENSOR_OUTPUT,      "weight"), {n_embd, n_vocab}, 0);
-+
-+    for (int i = 0; i < n_layer; ++i) {
-+        auto & layer = layers[i];
-+
-+        create_tensor_qkv(layer, i, n_embd, n_embd_head_k * n_head, n_embd_gqa, n_embd_gqa, 0);
-+        layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), { n_embd_head_k * n_head, n_embd }, 0);
-+
-+        layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
-+        // per-head QK-norm (one head_dim vector)
-+        layer.attn_q_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_NORM, "weight", i), {n_embd_head_k}, 0);
-+        layer.attn_k_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_NORM, "weight", i), {n_embd_head_k}, 0);
-+
-+        // sparse-attn indexer (unused): GGML_OP_NONE -> loader skips; NOT_REQUIRED -> older GGUFs still load;
-+        // SKIP_IF_VIRTUAL -> no-file loader (test-llama-archs) skips them too
-+        const int64_t n_index_head = 4;    // sparse_num_index_heads
-+        const int64_t d_index      = 128;  // sparse_index_dim
-+        const int idx_flags = TENSOR_NOT_REQUIRED | TENSOR_SKIP_IF_VIRTUAL;
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q,      "weight", i), {n_embd, n_index_head * d_index}, idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K,      "weight", i), {n_embd, d_index},                idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_Q_NORM, "weight", i), {d_index},                        idx_flags);
-+        create_tensor(tn(LLM_TENSOR_ATTN_INDEX_K_NORM, "weight", i), {d_index},                        idx_flags);
-+
-+        layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
-+
-+        if (i < (int) hparams.n_layer_dense_lead) {
-+            layer.ffn_gate = create_tensor(tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd,   n_ff}, 0);
-+            layer.ffn_down = create_tensor(tn(LLM_TENSOR_FFN_DOWN, "weight", i), {  n_ff, n_embd}, 0);
-+            layer.ffn_up   = create_tensor(tn(LLM_TENSOR_FFN_UP,   "weight", i), {n_embd,   n_ff}, 0);
-+        } else {
-+            layer.ffn_gate_inp    = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP,    "weight", i), {n_embd, n_expert}, 0);
-+            layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias",   i), {n_expert}, 0);
-+            layer.ffn_gate_exps   = create_tensor(tn(LLM_TENSOR_FFN_GATE_EXPS,   "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+            layer.ffn_down_exps   = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS,   "weight", i), {n_ff_exp, n_embd, n_expert}, 0);
-+            layer.ffn_up_exps     = create_tensor(tn(LLM_TENSOR_FFN_UP_EXPS,     "weight", i), {n_embd, n_ff_exp, n_expert}, 0);
-+
-+            layer.ffn_gate_shexp = create_tensor(tn(LLM_TENSOR_FFN_GATE_SHEXP, "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+            layer.ffn_down_shexp = create_tensor(tn(LLM_TENSOR_FFN_DOWN_SHEXP, "weight", i), {        n_ff_exp * n_expert_shared, n_embd}, 0);
-+            layer.ffn_up_shexp   = create_tensor(tn(LLM_TENSOR_FFN_UP_SHEXP,   "weight", i), {n_embd, n_ff_exp * n_expert_shared}, 0);
-+        }
-+    }
-+}
-+
-+std::unique_ptr<llm_graph_context> llama_model_minimax_m3::build_arch_graph(const llm_graph_params & params) const {
-+    return std::make_unique<graph>(*this, params);
-+}
-+
-+llama_model_minimax_m3::graph::graph(const llama_model & model, const llm_graph_params & params) : llm_graph_context(params) {
-+    const int64_t n_embd_head = hparams.n_embd_head_v();
-+
-+    GGML_ASSERT(n_embd_head == hparams.n_embd_head_k());
-+    // partial rotary: head_dim != n_rot, so don't assert n_embd_head == n_rot
-+
-+    ggml_tensor * cur;
-+    ggml_tensor * inpL;
-+
-+    inpL = build_inp_embd(model.tok_embd);
-+
-+    ggml_tensor * inp_pos = build_inp_pos();
-+    auto inp_attn = build_attn_inp_kv();
-+    ggml_tensor * inp_out_ids = build_inp_out_ids();
-+
-+    for (int il = 0; il < n_layer; ++il) {
-+        ggml_tensor * inpSA = inpL;
-+
-+        // self-attention
-+        {
-+            cur = build_norm(inpL, model.layers[il].attn_norm, NULL, LLM_NORM_RMS, il);
-+            cb(cur, "attn_norm", il);
-+
-+            auto [Qcur, Kcur, Vcur] = build_qkv(model.layers[il], cur,
-+                    n_embd_head, n_head, n_head_kv, il);
-+
-+            // per-head QK RMSNorm (weights include Gemma +1)
-+            Qcur = build_norm(Qcur, model.layers[il].attn_q_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Qcur, "Qcur_normed", il);
-+            Kcur = build_norm(Kcur, model.layers[il].attn_k_norm, NULL, LLM_NORM_RMS, il);
-+            cb(Kcur, "Kcur_normed", il);
-+
-+            Qcur = ggml_rope_ext(
-+                ctx0, Qcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+            Kcur = ggml_rope_ext(
-+                ctx0, Kcur, inp_pos, nullptr,
-+                n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
-+                ext_factor, attn_factor, beta_fast, beta_slow
-+                );
-+
-+            cb(Qcur, "Qcur", il);
-+            cb(Kcur, "Kcur", il);
-+            cb(Vcur, "Vcur", il);
-+
-+            cur = build_attn(inp_attn,
-+                    model.layers[il].wo, NULL, model.layers[il].wo_s,
-+                    Qcur, Kcur, Vcur, nullptr, nullptr, nullptr, 1.0f/sqrtf(float(n_embd_head)), il);
-+        }
-+
-+        if (il == n_layer - 1 && inp_out_ids) {
-+            cur   = ggml_get_rows(ctx0,   cur, inp_out_ids);
-+            inpSA = ggml_get_rows(ctx0, inpSA, inp_out_ids);
-+        }
-+
-+        ggml_tensor * ffn_inp = ggml_add(ctx0, cur, inpSA);
-+        cb(ffn_inp, "ffn_inp", il);
-+
-+        cur = build_norm(ffn_inp, model.layers[il].ffn_norm, NULL, LLM_NORM_RMS, il);
-+        cb(cur, "ffn_norm", il);
-+
-+        if ((uint32_t) il < hparams.n_layer_dense_lead) {
-+            // leading dense
-+            cur = build_ffn(cur,
-+                    model.layers[il].ffn_up,   NULL, NULL,
-+                    model.layers[il].ffn_gate, NULL, NULL,
-+                    model.layers[il].ffn_down, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(cur, "ffn_out", il);
-+        } else {
-+            // routed experts
-+            ggml_tensor * moe_out = build_moe_ffn(cur,
-+                    model.layers[il].ffn_gate_inp,
-+                    model.layers[il].ffn_up_exps,
-+                    model.layers[il].ffn_gate_exps,
-+                    model.layers[il].ffn_down_exps,
-+                    model.layers[il].ffn_exp_probs_b,
-+                    n_expert, n_expert_used,
-+                    LLM_FFN_SWIGLU_OAI_MOE, hparams.expert_weights_norm,
-+                    hparams.expert_weights_scale,
-+                    (llama_expert_gating_func_type) hparams.expert_gating_func,
-+                    il);
-+            cb(moe_out, "ffn_moe_out", il);
-+
-+            // shared expert
-+            ggml_tensor * ffn_shexp = build_ffn(cur,
-+                    model.layers[il].ffn_up_shexp,   NULL, NULL,
-+                    model.layers[il].ffn_gate_shexp, NULL, NULL,
-+                    model.layers[il].ffn_down_shexp, NULL, NULL,
-+                    NULL,
-+                    LLM_FFN_SWIGLU_OAI, LLM_FFN_PAR, il);
-+            cb(ffn_shexp, "ffn_shexp", il);
-+
-+            cur = ggml_add(ctx0, moe_out, ffn_shexp);
-+            cb(cur, "ffn_out", il);
-+        }
-+
-+        cur = ggml_add(ctx0, cur, ffn_inp);
-+
-+        cur = build_cvec(cur, il);
-+        cb(cur, "l_out", il);
-+
-+        // input for next layer
-+        inpL = cur;
-+    }
-+
-+    cur = inpL;
-+
-+    cur = build_norm(cur, model.output_norm, NULL, LLM_NORM_RMS, -1);
-+    cb(cur, "result_norm", -1);
-+    res->t_embd = cur;
-+
-+    // lm_head
-+    cur = build_lora_mm(model.output, cur, model.output_s);
-+    cb(cur, "result_output", -1);
-+    res->t_logits = cur;
-+
-+    ggml_build_forward_expand(gf, cur);
-+}
-diff --git a/src/models/models.h b/src/models/models.h
-index 7a52e7b..5e2a826 100644
---- a/src/models/models.h
-+++ b/src/models/models.h
-@@ -1870,6 +1870,17 @@ struct llama_model_minimax_m2 : public llama_model_base {
-     std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
- };
- 
-+struct llama_model_minimax_m3 : public llama_model_base {
-+    llama_model_minimax_m3(const struct llama_model_params & params) : llama_model_base(params) {}
-+    void load_arch_hparams(llama_model_loader & ml) override;
-+    void load_arch_tensors(llama_model_loader & ml) override;
-+
-+    struct graph : public llm_graph_context {
-+        graph(const llama_model & model, const llm_graph_params & params);
-+    };
-+
-+    std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
-+};
- 
- struct llama_model_cogvlm : public llama_model_base {
-     llama_model_cogvlm(const struct llama_model_params & params) : llama_model_base(params) {}
-diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp
-index f39abe7..2085f43 100644
---- a/tests/test-llama-archs.cpp
-+++ b/tests/test-llama-archs.cpp
-@@ -352,6 +352,7 @@ static bool moe_mandatory(const llm_arch arch) {
-         case LLM_ARCH_LLADA_MOE:
-         case LLM_ARCH_GROVEMOE:
-         case LLM_ARCH_MINIMAX_M2:
-+        case LLM_ARCH_MINIMAX_M3:
-         case LLM_ARCH_RND1:
-         case LLM_ARCH_PADDLEOCR:
-         case LLM_ARCH_MIMO2:

backend/cpp/llama-cpp/prepare.sh

@@ -1,33 +1,23 @@
 #!/bin/bash
 
-set -e
-
 ## Patches
 
-## Apply patches from the `patches` directory. Runs under set -e so a
-## rejected patch aborts the build here, loudly, instead of surfacing later
-## as a confusing compile error. A missing or empty patches dir is a no-op.
+## Apply patches from the `patches` directory
 if [ -d "patches" ]; then
     for patch in $(ls patches); do
         echo "Applying patch $patch"
         patch -d llama.cpp/ -p1 < patches/$patch
-    done
+    done 
 fi
 
+set -e
+
 for file in $(ls llama.cpp/tools/server/); do
     cp -rfv llama.cpp/tools/server/$file llama.cpp/tools/grpc-server/
 done
 
 cp -r CMakeLists.txt llama.cpp/tools/grpc-server/
 cp -r grpc-server.cpp llama.cpp/tools/grpc-server/
-# Shared message-reconstruction helpers (included by grpc-server.cpp) and their
-# unit test (compiled only when -DLLAMA_GRPC_BUILD_TESTS=ON).
-cp -r message_content.h llama.cpp/tools/grpc-server/
-cp -r message_content_test.cpp llama.cpp/tools/grpc-server/
-# Parent-death watcher (included by grpc-server.cpp) and its standalone unit
-# test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest).
-cp -r parent_watch.h llama.cpp/tools/grpc-server/
-cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 

backend/cpp/llama-cpp/run.sh

@@ -2,7 +2,7 @@
 set -ex
 
 # Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath "$0")")
+CURDIR=$(dirname "$(realpath $0)")
 
 cd /
 
@@ -12,47 +12,55 @@ grep -e "flags" /proc/cpuinfo | head -1
 
 BINARY=llama-cpp-fallback
 
-# CPU images (x86, arm64, darwin) ship a single llama-cpp-cpu-all built with ggml
-# CPU_ALL_VARIANTS: ggml's backend registry dlopens the best libggml-cpu-*.so for this
-# host, so no shell-side AVX probing. GPU images (cublas/sycl/vulkan/hipblas) ship only
-# llama-cpp-fallback (the accelerator does the compute), so fall back to it when absent.
-if [ -e "$CURDIR"/llama-cpp-cpu-all ]; then
-	BINARY=llama-cpp-cpu-all
+if grep -q -e "\savx\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX    found OK"
+	if [ -e $CURDIR/llama-cpp-avx ]; then
+		BINARY=llama-cpp-avx
+	fi
+fi
+
+if grep -q -e "\savx2\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX2   found OK"
+	if [ -e $CURDIR/llama-cpp-avx2 ]; then
+		BINARY=llama-cpp-avx2
+	fi
+fi
+
+# Check avx 512
+if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX512F found OK"
+	if [ -e $CURDIR/llama-cpp-avx512 ]; then
+		BINARY=llama-cpp-avx512
+	fi
 fi
 
 if [ -n "$LLAMACPP_GRPC_SERVERS" ]; then
-	if [ -e "$CURDIR"/llama-cpp-grpc ]; then
+	if [ -e $CURDIR/llama-cpp-grpc ]; then
 		BINARY=llama-cpp-grpc
 	fi
 fi
  
 # Extend ld library path with the dir where this script is located/lib
 if [ "$(uname)" == "Darwin" ]; then
-	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
-	#export DYLD_FALLBACK_LIBRARY_PATH="$CURDIR"/lib:$DYLD_FALLBACK_LIBRARY_PATH
+	export DYLD_LIBRARY_PATH=$CURDIR/lib:$DYLD_LIBRARY_PATH
+	#export DYLD_FALLBACK_LIBRARY_PATH=$CURDIR/lib:$DYLD_FALLBACK_LIBRARY_PATH
 else
-	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+	export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
 	# Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files)
 	if [ -d "$CURDIR/lib/rocblas/library" ]; then
-		export ROCBLAS_TENSILE_LIBPATH="$CURDIR"/lib/rocblas/library
-	fi
-	# Same for hipBLASLt (rocblaslt): the bundled libhipblaslt.so resolves its
-	# TensileLibrary_lazy_gfx*.dat kernel data relative to itself, so point it at
-	# the bundled data or it falls back to slow generic kernels (issue #10660).
-	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
-		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
+		export ROCBLAS_TENSILE_LIBPATH=$CURDIR/lib/rocblas/library
 	fi
 fi
 
 # If there is a lib/ld.so, use it
-if [ -f "$CURDIR"/lib/ld.so ]; then
+if [ -f $CURDIR/lib/ld.so ]; then
 	echo "Using lib/ld.so"
 	echo "Using binary: $BINARY"
-	exec "$CURDIR"/lib/ld.so "$CURDIR"/$BINARY "$@"
+	exec $CURDIR/lib/ld.so $CURDIR/$BINARY "$@"
 fi
 
 echo "Using binary: $BINARY"
-exec "$CURDIR"/$BINARY "$@"
+exec $CURDIR/$BINARY "$@"
 
 # We should never reach this point, however just in case we do, run fallback
-exec "$CURDIR"/llama-cpp-fallback "$@"
+exec $CURDIR/llama-cpp-fallback "$@"

backend/cpp/privacy-filter/CMakeLists.txt

@@ -51,14 +51,6 @@ add_library(hw_grpc_proto STATIC
     ${HW_GRPC_SRCS} ${HW_GRPC_HDRS}
     ${HW_PROTO_SRCS} ${HW_PROTO_HDRS})
 target_include_directories(hw_grpc_proto PUBLIC ${CMAKE_CURRENT_BINARY_DIR})
-# The generated proto/grpc sources include protobuf and grpc++ headers, so this
-# library must see their include dirs. Linking the imported targets propagates
-# them. On Linux the apt headers live in /usr/include (default search path) so
-# this was a no-op; on macOS the Homebrew headers are under /opt/homebrew and
-# would otherwise be missed (runtime_version.h not found).
-target_link_libraries(hw_grpc_proto PUBLIC
-    protobuf::libprotobuf
-    gRPC::grpc++)
 
 # Build only the pf static lib (+ ggml) from the engine tree — no CLI/bench/tests.
 # PF_VULKAN is honored when passed on the cmake command line (it lands in the

backend/cpp/privacy-filter/Makefile

@@ -8,7 +8,7 @@
 # Local development: point at a working checkout instead of cloning, e.g.
 #   make PRIVACY_FILTER_SRC=$HOME/c/privacy-filter.cpp grpc-server
 
-PRIVACY_FILTER_VERSION?=735a6c28607ee82afc3a670383f41b55266a3b9a
+PRIVACY_FILTER_VERSION?=98f52c5ef2250f207cc6b9a6aef05393a120cb7c
 PRIVACY_FILTER_REPO?=https://github.com/localai-org/privacy-filter.cpp
 PRIVACY_FILTER_SRC?=
 

backend/cpp/privacy-filter/grpc-server.cpp

@@ -41,11 +41,6 @@ namespace {
 // per loaded model. g_mu guards (re)load against in-flight classification.
 std::mutex          g_mu;
 pf_ctx *            g_ctx = nullptr;
-// The ModelOptions.Model this process loaded, compared against
-// TokenClassifyRequest.ModelIdentity so a request that arrived through a stale
-// distributed route is rejected rather than answered from the wrong model
-// (#10952). Guarded by g_mu like the rest of the engine state.
-std::string         g_loaded_model_identity;
 std::atomic<Server *> g_server{nullptr};
 
 // Resolve the device string the engine expects ("cpu" / "gpu" / "cuda" /
@@ -118,48 +113,17 @@ public:
         }
 
         g_ctx = ctx;
-        // Record what we loaded so TokenClassify can reject a request meant
-        // for a different model. request->model(), not modelfile(): it is the
-        // value the controller also sends as ModelIdentity, and the two are
-        // read from the same ModelConfig.Model (#10952).
-        g_loaded_model_identity = request->model();
         result->set_success(true);
         result->set_message("privacy-filter loaded (" + device + ")");
         return GStatus::OK;
     }
 
-    // checkModelIdentity mirrors pkg/grpc/server.go,
-    // backend/python/common/model_identity.py and the llama-cpp server. In
-    // distributed mode a worker can recycle a stopped backend's gRPC port for
-    // another model's backend, and the controller's liveness-only probe cannot
-    // tell a stale cached route from a valid one, so the backend has to catch
-    // it. Either side empty means "skip": the request side is empty for a
-    // controller that predates the field, the loaded side when such a
-    // controller performed the load. A false rejection is worse than the miss.
-    // Callers must already hold g_mu.
-    GStatus checkModelIdentity(const backend::TokenClassifyRequest * request) {
-        if (request == nullptr || request->modelidentity().empty()) {
-            return GStatus::OK;
-        }
-        if (g_loaded_model_identity.empty() ||
-            g_loaded_model_identity == request->modelidentity()) {
-            return GStatus::OK;
-        }
-        // NOT_FOUND plus this exact sentinel is the cross-language contract
-        // the router matches on (grpcerrors.ModelMismatchSentinel).
-        return GStatus(StatusCode::NOT_FOUND,
-                       "privacy-filter: model identity mismatch: loaded \"" +
-                       g_loaded_model_identity + "\", requested \"" +
-                       request->modelidentity() + "\"");
-    }
-
     GStatus TokenClassify(ServerContext *, const backend::TokenClassifyRequest * request,
                           backend::TokenClassifyResponse * response) override {
         std::lock_guard<std::mutex> lock(g_mu);
         if (!g_ctx) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        if (GStatus id = checkModelIdentity(request); !id.ok()) return id;
 
         const std::string & text = request->text();
         if (text.empty()) {

backend/cpp/privacy-filter/run.sh

@@ -2,13 +2,7 @@
 # Entry point for the privacy-filter backend image / BACKEND_BINARY mode.
 set -e
 CURDIR=$(dirname "$(realpath "$0")")
-# macOS has no bundled ld.so; the darwin package ships only dylibs under lib/,
-# resolved via DYLD_LIBRARY_PATH (the ld.so branch below is skipped there).
-if [ "$(uname)" = "Darwin" ]; then
-    export DYLD_LIBRARY_PATH="$CURDIR/lib:$DYLD_LIBRARY_PATH"
-else
-    export LD_LIBRARY_PATH="$CURDIR/lib:$LD_LIBRARY_PATH"
-fi
+export LD_LIBRARY_PATH="$CURDIR/lib:$LD_LIBRARY_PATH"
 if [ -f "$CURDIR/lib/ld.so" ]; then
     exec "$CURDIR/lib/ld.so" "$CURDIR/grpc-server" "$@"
 fi

backend/cpp/run-unit-tests.sh

@@ -1,71 +0,0 @@
-#!/bin/bash
-#
-# Discovers and runs every standalone C++ unit test under backend/cpp/.
-#
-# A "standalone" unit test is a *_test.cpp that depends only on the C++ standard
-# library and nlohmann/json (single header) - i.e. it exercises pure helpers and
-# does not need the full llama.cpp + gRPC backend build. Tests that DO need the
-# backend build use the CMake/ctest path (e.g. -DLLAMA_GRPC_BUILD_TESTS=ON)
-# instead and are skipped here.
-#
-# This keeps CI generic: adding a new pure-C++ unit test file named *_test.cpp in
-# an active backend source dir is picked up automatically, with no CI edits.
-#
-# Env:
-#   NLOHMANN_INCLUDE  include dir that contains nlohmann/json.hpp. If unset, the
-#                     nlohmann/json single header is fetched to a temp dir.
-#   CXX               compiler (default: g++).
-#   JSON_VERSION      nlohmann/json tag to fetch when NLOHMANN_INCLUDE is unset
-#                     (default: v3.11.3).
-set -uo pipefail
-
-ROOT="$(cd "$(dirname "$0")" && pwd)"
-CXX="${CXX:-g++}"
-JSON_VERSION="${JSON_VERSION:-v3.11.3}"
-
-JSON_INC="${NLOHMANN_INCLUDE:-}"
-if [ -z "$JSON_INC" ]; then
-    JSON_INC="$(mktemp -d)"
-    mkdir -p "$JSON_INC/nlohmann"
-    echo "Fetching nlohmann/json ${JSON_VERSION} single header..."
-    if ! curl -L -sf \
-        "https://raw.githubusercontent.com/nlohmann/json/${JSON_VERSION}/single_include/nlohmann/json.hpp" \
-        -o "$JSON_INC/nlohmann/json.hpp"; then
-        echo "ERROR: failed to fetch nlohmann/json header" >&2
-        exit 1
-    fi
-fi
-
-# Active source dirs only - exclude per-variant build copies, dev snapshots and
-# the vendored upstream llama.cpp tree.
-mapfile -t tests < <(find "$ROOT" -name '*_test.cpp' \
-    -not -path '*/llama.cpp/*' \
-    -not -path '*-build/*' \
-    -not -path '*-dev/*' \
-    -not -path '*fallback*' | sort)
-
-if [ "${#tests[@]}" -eq 0 ]; then
-    echo "No standalone C++ unit tests found under $ROOT"
-    exit 0
-fi
-
-fail=0
-for test_src in "${tests[@]}"; do
-    name="$(basename "$test_src" .cpp)"
-    bin="$(mktemp -d)/$name"
-    echo "==> $test_src"
-    if ! "$CXX" -std=c++17 -Wall -Wextra -pthread \
-        -I"$JSON_INC" -I"$(dirname "$test_src")" \
-        "$test_src" -o "$bin"; then
-        echo "COMPILE FAILED: $test_src" >&2
-        fail=1
-        continue
-    fi
-    if ! "$bin"; then
-        echo "TEST FAILED: $test_src" >&2
-        fail=1
-    fi
-done
-
-echo "Ran ${#tests[@]} standalone C++ unit test file(s)"
-exit "$fail"

backend/cpp/turboquant/Makefile

@@ -37,10 +37,6 @@ PATCHES_DIR := $(CURRENT_MAKEFILE_DIR)/patches
 define turboquant-build
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build
 	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build
-	# Drop patches vendored for upstream llama.cpp: the fork tree diverges, so
-	# they reject there. Fork-specific patches live in backend/cpp/turboquant/patches/
-	# and are applied by apply-patches.sh below.
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build purge
 	# Augment the copied grpc-server.cpp's KV-cache allow-list with the
 	# fork's turbo2/turbo3/turbo4 types. We patch the *copy*, never the
@@ -69,33 +65,6 @@ turboquant-avx:
 turboquant-fallback:
 	$(call turboquant-build,fallback,-DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
 
-# Single-build CPU backend via ggml CPU_ALL_VARIANTS (mirrors llama-cpp-cpu-all).
-# turboquant reuses backend/cpp/llama-cpp's CMakeLists.txt (hw_grpc_proto STATIC) and
-# Makefile (SHARED_LIBS make-var + EXTRA_CMAKE_ARGS), so this passes the same overrides
-# through to the copied build: SHARED_LIBS=ON, the DL flags, and --target ggml (which
-# pulls in the per-microarch libggml-cpu-*.so via ggml's add_dependencies). The .so set
-# is collected for package.sh to bundle into package/lib.
-turboquant-cpu-all:
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build
-	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build
-	# Drop patches vendored for upstream llama.cpp: the fork tree diverges, so
-	# they reject there. Fork-specific patches live in backend/cpp/turboquant/patches/
-	# and are applied by apply-patches.sh below.
-	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/patches
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build purge
-	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
-	$(info $(GREEN)I turboquant build info:cpu-all-variants$(RESET))
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build llama.cpp
-	bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/llama.cpp $(PATCHES_DIR)
-	SHARED_LIBS=ON EXTRA_CMAKE_ARGS="-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON" TARGET="--target grpc-server --target ggml" \
-	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
-	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build grpc-server
-	cp -rfv $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server turboquant-cpu-all
-	rm -rf ggml-shared-libs && mkdir -p ggml-shared-libs
-	find $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/llama.cpp/build \( -name '*.so*' -o -name '*.dylib' \) -exec cp -av {} ggml-shared-libs/ \;
-	@echo "Collected ggml shared backends:" && ls -la ggml-shared-libs/
-
 turboquant-grpc:
 	$(call turboquant-build,grpc,-DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server --target rpc-server)
 

backend/cpp/turboquant/package.sh

@@ -14,15 +14,6 @@ mkdir -p $CURDIR/package/lib
 cp -avrf $CURDIR/turboquant-* $CURDIR/package/
 cp -rfv $CURDIR/run.sh $CURDIR/package/
 
-# Bundle the ggml shared backends from the CPU_ALL_VARIANTS build into package/lib. ggml
-# discovers the per-microarch libggml-cpu-*.so by scanning the executable directory, which
-# (via the bundled lib/ld.so that run.sh launches through) resolves to lib/. See the
-# matching comment in backend/cpp/llama-cpp/package.sh. No-op on the fallback/ROCm builds.
-if [ -d "$CURDIR/ggml-shared-libs" ]; then
-    echo "Bundling ggml shared backends (CPU_ALL_VARIANTS)..."
-    cp -avf $CURDIR/ggml-shared-libs/*.so* $CURDIR/package/lib/
-fi
-
 # Detect architecture and copy appropriate libraries
 if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
     # x86_64 architecture

backend/cpp/turboquant/run.sh

@@ -2,7 +2,7 @@
 set -ex
 
 # Get the absolute current dir where the script is located
-CURDIR=$(dirname "$(realpath "$0")")
+CURDIR=$(dirname "$(realpath $0)")
 
 cd /
 
@@ -12,45 +12,54 @@ grep -e "flags" /proc/cpuinfo | head -1
 
 BINARY=turboquant-fallback
 
-# x86/arm64 ship a single turboquant-cpu-all built with ggml CPU_ALL_VARIANTS: ggml's
-# backend registry dlopens the best libggml-cpu-*.so for this host, so no shell-side
-# probing. ROCm ships only turboquant-fallback, so fall back to it when cpu-all is absent.
-if [ -e "$CURDIR"/turboquant-cpu-all ]; then
-	BINARY=turboquant-cpu-all
+if grep -q -e "\savx\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX    found OK"
+	if [ -e $CURDIR/turboquant-avx ]; then
+		BINARY=turboquant-avx
+	fi
+fi
+
+if grep -q -e "\savx2\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX2   found OK"
+	if [ -e $CURDIR/turboquant-avx2 ]; then
+		BINARY=turboquant-avx2
+	fi
+fi
+
+# Check avx 512
+if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX512F found OK"
+	if [ -e $CURDIR/turboquant-avx512 ]; then
+		BINARY=turboquant-avx512
+	fi
 fi
 
 if [ -n "$LLAMACPP_GRPC_SERVERS" ]; then
-	if [ -e "$CURDIR"/turboquant-grpc ]; then
+	if [ -e $CURDIR/turboquant-grpc ]; then
 		BINARY=turboquant-grpc
 	fi
 fi
 
 # Extend ld library path with the dir where this script is located/lib
 if [ "$(uname)" == "Darwin" ]; then
-	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+	export DYLD_LIBRARY_PATH=$CURDIR/lib:$DYLD_LIBRARY_PATH
 else
-	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+	export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
 	# Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files)
 	if [ -d "$CURDIR/lib/rocblas/library" ]; then
-		export ROCBLAS_TENSILE_LIBPATH="$CURDIR"/lib/rocblas/library
-	fi
-	# Same for hipBLASLt (rocblaslt): the bundled libhipblaslt.so resolves its
-	# TensileLibrary_lazy_gfx*.dat kernel data relative to itself, so point it at
-	# the bundled data or it falls back to slow generic kernels (issue #10660).
-	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
-		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
+		export ROCBLAS_TENSILE_LIBPATH=$CURDIR/lib/rocblas/library
 	fi
 fi
 
 # If there is a lib/ld.so, use it
-if [ -f "$CURDIR"/lib/ld.so ]; then
+if [ -f $CURDIR/lib/ld.so ]; then
 	echo "Using lib/ld.so"
 	echo "Using binary: $BINARY"
-	exec "$CURDIR"/lib/ld.so "$CURDIR"/$BINARY "$@"
+	exec $CURDIR/lib/ld.so $CURDIR/$BINARY "$@"
 fi
 
 echo "Using binary: $BINARY"
-exec "$CURDIR"/$BINARY "$@"
+exec $CURDIR/$BINARY "$@"
 
 # We should never reach this point, however just in case we do, run fallback
-exec "$CURDIR"/turboquant-fallback "$@"
+exec $CURDIR/turboquant-fallback "$@"

backend/go/acestep-cpp/CMakeLists.txt

@@ -25,7 +25,7 @@ target_include_directories(goacestepcpp PRIVATE ${ACESTEP_DIR}/src ${ACESTEP_DIR
 target_include_directories(goacestepcpp SYSTEM PRIVATE ${ACESTEP_DIR}/ggml/include)
 
 # Link GPU backends if available (mirrors link_ggml_backends macro)
-foreach(backend blas cuda hip metal vulkan)
+foreach(backend blas cuda metal vulkan)
     if(TARGET ggml-${backend})
         target_link_libraries(goacestepcpp PRIVATE ggml-${backend})
         string(TOUPPER ${backend} BACKEND_UPPER)

backend/go/acestep-cpp/Makefile

@@ -24,14 +24,7 @@ else ifeq ($(BUILD_TYPE),openblas)
 else ifeq ($(BUILD_TYPE),clblas)
 	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
 else ifeq ($(BUILD_TYPE),hipblas)
-	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
-	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
-	ROCM_HOME ?= /opt/rocm
-	ROCM_PATH ?= /opt/rocm
-	export CXX=$(ROCM_HOME)/llvm/bin/clang++
-	export CC=$(ROCM_HOME)/llvm/bin/clang
-	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
-	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
+	CMAKE_ARGS+=-DGGML_HIPBLAS=ON
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DGGML_VULKAN=ON
 else ifeq ($(OS),Darwin)
@@ -124,8 +117,7 @@ libgoacestepcpp-custom: CMakeLists.txt cpp/goacestepcpp.cpp cpp/goacestepcpp.h
 	cmake .. $(CMAKE_ARGS) && \
 	cmake --build . --config Release -j$(JOBS) --target goacestepcpp && \
 	cd .. && \
-	(mv build-$(SO_TARGET)/libgoacestepcpp.so ./$(SO_TARGET) 2>/dev/null || \
-	 mv build-$(SO_TARGET)/libgoacestepcpp.dylib ./$(SO_TARGET) 2>/dev/null)
+	mv build-$(SO_TARGET)/libgoacestepcpp.so ./$(SO_TARGET)
 
 test: acestep-cpp
 	@echo "Running acestep-cpp tests..."