mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00

Files

LocalAI [bot] bbe018c1a0 feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834 )

feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models

Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.

Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).

Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.

Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

2026-07-16 10:09:14 +02:00

README.md

feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834 )

2026-07-16 10:09:14 +02:00

README.md

bonsai fork skew patches

The bonsai backend reuses backend/cpp/llama-cpp/grpc-server.cpp (written against LocalAI's pinned upstream llama.cpp) but compiles it against the PrismML prism fork, which branched from upstream some commits earlier. Any upstream API change that the shared gRPC server depends on, but that the fork does not yet carry, is back-ported here as a *.patch file and applied to the cloned fork checkout by ../apply-patches.sh.

Rules:

One upstream commit (or minimal hunk) per patch, named NNNN-short-description.patch.
Patches are applied with git apply from the fork's checkout root.
apply-patches.sh fails fast if a patch stops applying cleanly — that is the signal the fork has caught up (or diverged), so re-cut or drop the patch.
Keep this set as small as possible; the long-term fix is the fork rebasing onto a newer upstream (or Q1_0/Q2_0 landing in mainline llama.cpp, retiring this backend entirely).