The turboquant and bonsai backends copy backend/cpp/llama-cpp/ wholesale
into their build directories and reuse its Makefile/prepare.sh against
their own llama.cpp forks. When PR #10837 added
backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch, the
copied patches/ directory was mis-applied to the fork checkouts: the
fork trees diverge from upstream, hunks rejected, and because the
patch-apply loop in prepare.sh ran before set -e took effect the build
kept going and died much later with a confusing compile error
("'LLM_ARCH_MINIMAX_M3' was not declared in this scope"). This broke
tests-turboquant-grpc on that PR.
Two hardening changes:
- turboquant/bonsai Makefiles: delete the copied patches/ directory
right after the cp -rf of backend/cpp/llama-cpp/. Patches vendored
for upstream llama.cpp must never be applied to the forks; each fork
carries its own patch series under backend/cpp/<backend>/patches/,
applied by its apply-patches.sh.
- llama-cpp prepare.sh: run the patch-apply loop under set -e so a
rejecting patch fails fast and loudly at apply time instead of
surfacing as a downstream compile error. A missing or empty patches/
directory remains a no-op success, so all existing callers (the
llama-cpp Makefile targets and the turboquant/bonsai copies) are
unaffected when no patches ship.
Exposed by PR #10837.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models
Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.
Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).
Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.
Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
