feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834)

feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp (github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0 (1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants. Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models are served with the same OpenAI-compatible API. No grpc-server allow-list patch is needed (bonsai adds weight quants, transparent to the server, not KV-cache types), and the reused server compiles cleanly against the fork with no skew patches (validated locally via a CPU docker build; patches/ is present but empty for any future re-pin skew). Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh, backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml meta-backend + per-platform images, and a nightly bump_deps entry tracking the `prism` branch. Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64, +pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-07-30 01:48:06 -04:00 · 2026-07-16 10:09:14 +02:00
parent 3880812ed6
commit bbe018c1a0
13 changed files with 1039 additions and 3 deletions
--- a/.docker/bonsai-compile.sh
+++ b/.docker/bonsai-compile.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Shared compile logic for backend/Dockerfile.bonsai.
+# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
+
+set -euxo pipefail
+
+export CCACHE_DIR=/root/.ccache
+ccache --max-size=5G || true
+ccache -z || true
+
+export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/bonsai-*-build
+fi
+
+cd /LocalAI/backend/cpp/bonsai
+
+if [ -z "${BUILD_TYPE:-}" ]; then
+  # Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries.
+  # arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme).
+  if [ "${TARGETARCH}" = "arm64" ]; then
+    apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
+    export CC=gcc-14 CXX=g++-14
+  fi
+  make bonsai-cpu-all
+else
+  # GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator
+  # does the compute. Keeps the GPU compile from also building the CPU variant matrix and
+  # avoids the gcc-14 apt step on GPU base images such as nvidia l4t.
+  make bonsai-fallback
+fi
+make bonsai-grpc
+make bonsai-rpc-server
+
+ccache -s || true
--- a/.github/backend-matrix.yml
+++ b/.github/backend-matrix.yml
@@ -452,6 +452,22 @@ include:
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-12-amd64'
+    # bigger-runner: same rationale as -gpu-nvidia-cuda-12-llama-cpp above
+    # (observed 6h5m wall-clock on v4.2.1, just past the 6h job timeout).
+    runs-on: 'bigger-runner'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'cublas'
    cuda-major-version: "12"
    cuda-minor-version: "8"
@@ -1082,6 +1098,21 @@ include:
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64'
+    # bigger-runner: observed 6h5m wall-clock on v4.2.1 — at the GHA timeout.
+    runs-on: 'bigger-runner'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'cublas'
    cuda-major-version: "13"
    cuda-minor-version: "0"
@@ -1110,6 +1141,20 @@ include:
    backend: "turboquant"
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-arm64'
+    base-image: "ubuntu:24.04"
+    runs-on: 'ubuntu-24.04-arm'
+    ubuntu-version: '2404'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
  - build-type: 'cublas'
    cuda-major-version: "13"
    cuda-minor-version: "0"
@@ -1983,6 +2028,20 @@ include:
    dockerfile: "./backend/Dockerfile.llama-cpp"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-rocm-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'hipblas'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2247,6 +2306,20 @@ include:
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'sycl_f32'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f32-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'sycl_f16'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2275,6 +2348,20 @@ include:
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: 'sycl_f16'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f16-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: 'intel'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2727,6 +2814,21 @@ include:
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: ''
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2742,6 +2844,21 @@ include:
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-arm64'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
  - build-type: ''
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2884,6 +3001,20 @@ include:
    dockerfile: "./backend/Dockerfile.turboquant"
    context: "./"
    ubuntu-version: '2204'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-arm64-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-l4t-cuda-12-arm64'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2204'
  - build-type: 'vulkan'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2930,6 +3061,22 @@ include:
    context: "./"
    ubuntu-version: '2404'
  # Stablediffusion-ggml
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
+  # Stablediffusion-ggml
  - build-type: 'vulkan'
    cuda-major-version: ""
    cuda-minor-version: ""
@@ -2946,6 +3093,22 @@ include:
    context: "./"
    ubuntu-version: '2404'
  # Stablediffusion-ggml
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-bonsai'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-arm64'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "bonsai"
+    dockerfile: "./backend/Dockerfile.bonsai"
+    context: "./"
+    ubuntu-version: '2404'
+  # Stablediffusion-ggml
  - build-type: ''
    cuda-major-version: ""
    cuda-minor-version: ""
--- a/.github/workflows/bump_deps.yaml
+++ b/.github/workflows/bump_deps.yaml
@@ -22,6 +22,10 @@ jobs:
            variable: "TURBOQUANT_VERSION"
            branch: "feature/turboquant-kv-cache"
            file: "backend/cpp/turboquant/Makefile"
+          - repository: "PrismML-Eng/llama.cpp"
+            variable: "BONSAI_VERSION"
+            branch: "prism"
+            file: "backend/cpp/bonsai/Makefile"
          - repository: "antirez/ds4"
            variable: "DS4_VERSION"
            branch: "main"
--- a/19
+++ b/19
@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin

 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -700,6 +700,16 @@ test-extra-backend-turboquant: docker-build-turboquant
 	BACKEND_TEST_CACHE_TYPE_V=turbo3 \
 	$(MAKE) test-extra-backend

+## bonsai: exercises the llama.cpp-fork backend with a real Q1_0 (1-bit) model —
+## the PrismML Bonsai-8B GGUF, whose weight quant is *only* decodable by the fork's
+## Q1_0 kernels. Loading it is what makes this backend distinct from stock llama-cpp;
+## a standard-quant model would only test the upstream code path the llama-cpp backend
+## already covers.
+test-extra-backend-bonsai: docker-build-bonsai
+	BACKEND_IMAGE=local-ai-backend:bonsai \
+	BACKEND_TEST_MODEL_URL=https://huggingface.co/prism-ml/Bonsai-8B-gguf/resolve/main/Bonsai-8B-Q1_0.gguf \
+	$(MAKE) test-extra-backend
+
 ## Audio transcription wrapper for the llama-cpp backend.
 ## Drives the new AudioTranscription / AudioTranscriptionStream RPCs against
 ## ggml-org/Qwen3-ASR-0.6B-GGUF (a small ASR model that requires its mmproj
@@ -1211,6 +1221,10 @@ BACKEND_IK_LLAMA_CPP = ik-llama-cpp|ik-llama-cpp|.|false|false
 # turboquant is a llama.cpp fork with TurboQuant KV-cache quantization.
 # Reuses backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
 BACKEND_TURBOQUANT = turboquant|turboquant|.|false|false
+# bonsai is a llama.cpp fork (PrismML) adding the Q1_0 (1-bit) and Q2_0 (ternary)
+# weight-quant kernels the Bonsai / Ternary-Bonsai models ship in. Reuses
+# backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
+BACKEND_BONSAI = bonsai|bonsai|.|false|false
 # ds4 is antirez/ds4, a DeepSeek V4 Flash-specific inference engine.
 # Single-model; hardware-only validation lives at tests/e2e-backends/
 # (BACKEND_BINARY mode); see docs/superpowers/plans/2026-05-11-ds4-backend.md.
@@ -1314,6 +1328,7 @@ endef
 $(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_IK_LLAMA_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_TURBOQUANT)))
+$(eval $(call generate-docker-build-target,$(BACKEND_BONSAI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DS4)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PRIVACY_FILTER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PIPER)))
@@ -1379,7 +1394,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar

-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter

 ########################################################
 ### Mock Backend for E2E Tests
--- a/backend/Dockerfile.bonsai
+++ b/backend/Dockerfile.bonsai
@@ -0,0 +1,160 @@
+ARG BASE_IMAGE=ubuntu:24.04
+# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
+# when no prebuilt base is supplied. The builder-prebuilt stage is only
+# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
+# content here is harmless — BuildKit prunes the unreferenced builder.
+ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
+# BUILDER_TARGET selects which builder stage the final scratch image copies
+# package output from. Declared at global scope (before any FROM) so it's
+# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
+# `make backends/bonsai` on the from-source path.
+ARG BUILDER_TARGET=builder-fromsource
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
+
+
+# ============================================================================
+# Stage: builder-fromsource — self-contained build path.
+# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
+# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
+# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
+# default; local `make backends/bonsai`).
+#
+# The install script is the same one that backend/Dockerfile.base-grpc-builder
+# runs, so the result is bit-equivalent to the prebuilt-base path
+# (builder-prebuilt below).
+# ============================================================================
+FROM ${BASE_IMAGE} AS builder-fromsource
+ARG BUILD_TYPE
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
+ARG CMAKE_FROM_SOURCE=false
+# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
+ARG CMAKE_VERSION=3.31.10
+ARG GRPC_VERSION=v1.65.0
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+ARG SKIP_DRIVERS=false
+ARG TARGETARCH
+ARG TARGETVARIANT
+ARG GO_VERSION=1.25.4
+ARG UBUNTU_VERSION=2404
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
+ARG AMDGPU_TARGETS=""
+ARG BACKEND=rerankers
+# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ARG CMAKE_ARGS
+
+ENV BUILD_TYPE=${BUILD_TYPE} \
+    CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
+    CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
+    CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
+    CMAKE_VERSION=${CMAKE_VERSION} \
+    GRPC_VERSION=${GRPC_VERSION} \
+    GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
+    SKIP_DRIVERS=${SKIP_DRIVERS} \
+    TARGETARCH=${TARGETARCH} \
+    UBUNTU_VERSION=${UBUNTU_VERSION} \
+    APT_MIRROR=${APT_MIRROR} \
+    APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
+    AMDGPU_TARGETS=${AMDGPU_TARGETS} \
+    CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
+    CMAKE_ARGS=${CMAKE_ARGS} \
+    DEBIAN_FRONTEND=noninteractive
+
+# CUDA on PATH (no-op when CUDA isn't installed)
+ENV PATH=/usr/local/cuda/bin:${PATH}
+# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
+ENV PATH=/opt/rocm/bin:${PATH}
+
+WORKDIR /build
+
+# Install everything via the shared script — the same one that
+# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
+# this from-source path are bit-equivalent.
+RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
+    --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    bash /usr/local/sbin/install-base-deps
+
+# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
+# CMake's find_package finds it at the canonical prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
+
+COPY . /LocalAI
+
+# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
+# for rationale. bonsai is a llama.cpp fork that reuses
+# backend/cpp/llama-cpp source via a thin wrapper Makefile, so MOST TUs
+# are content-identical to the upstream llama-cpp build. Sharing a cache
+# id with llama-cpp could give cross-fork hits — but for now keep them
+# separate so a regression in one doesn't poison the other. Revisit
+# sharing after measuring the actual hit rate.
+#
+# The compile body is shared with builder-prebuilt via .docker/bonsai-compile.sh.
+RUN --mount=type=bind,source=.docker/bonsai-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=bonsai-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
+
+
+# Copy libraries using a script to handle architecture differences
+RUN make -BC /LocalAI/backend/cpp/bonsai package
+
+
+# ============================================================================
+# Stage: builder-prebuilt — uses the pre-built base from
+# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
+# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
+# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
+# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
+# builder-base-image).
+# ============================================================================
+FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
+
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+# AMDGPU_TARGETS must be forwarded into the env here too — backend/cpp/llama-cpp/Makefile
+# (which the bonsai Makefile reuses via a sibling build dir) errors out when the var
+# is empty on a hipblas build, and the prebuilt path is what CI exercises most of the
+# time. The builder-fromsource stage above already does this; mirror it here.
+ARG AMDGPU_TARGETS
+ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
+ARG TARGETARCH
+ARG TARGETVARIANT
+
+# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
+# /usr/local. Mirror what the from-source path does so the compile step
+# can find gRPC at the canonical prefix the Makefile expects.
+RUN cp -a /opt/grpc/. /usr/local/
+
+COPY . /LocalAI
+
+RUN --mount=type=bind,source=.docker/bonsai-compile.sh,target=/usr/local/sbin/compile.sh \
+    --mount=type=cache,target=/root/.ccache,id=bonsai-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
+    bash /usr/local/sbin/compile.sh
+
+RUN make -BC /LocalAI/backend/cpp/bonsai package
+
+
+# ============================================================================
+# Final stage — copies package output from one of the two builders.
+# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
+#
+# BuildKit doesn't support variable expansion in `COPY --from=` directly,
+# so we resolve the ARG by aliasing the chosen builder to a fixed stage
+# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
+# BUILDER_TARGET itself is declared as a global ARG at the top of this
+# file (required for use in FROM), so we just re-import it into this
+# stage's scope before the FROM directive.
+# ============================================================================
+FROM ${BUILDER_TARGET} AS builder
+
+FROM scratch
+
+
+# Copy all available binaries (the build process only creates the appropriate ones for the target architecture)
+COPY --from=builder /LocalAI/backend/cpp/bonsai/package/. ./
--- a/backend/cpp/bonsai/Makefile
+++ b/backend/cpp/bonsai/Makefile
@@ -0,0 +1,97 @@
+
+# Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp.
+# Auto-bumped nightly by .github/workflows/bump_deps.yaml.
+BONSAI_VERSION?=62061f910882
+LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp
+
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+ONEAPI_VARS?=/opt/intel/oneapi/setvars.sh
+TARGET?=--target grpc-server
+JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
+ARCH?=$(shell uname -m)
+
+CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
+LLAMA_CPP_DIR := $(CURRENT_MAKEFILE_DIR)/../llama-cpp
+
+GREEN := \033[0;32m
+RESET := \033[0m
+
+# bonsai is a llama.cpp fork (PrismML) adding the Q1_0 (1-bit) and Q2_0 (ternary)
+# weight-quantization kernels that the Bonsai / Ternary-Bonsai models ship in. Rather
+# than duplicating grpc-server.cpp / CMakeLists.txt / prepare.sh we reuse the ones in
+# backend/cpp/llama-cpp, and only swap which repo+sha the fetch step pulls. Each flavor
+# target copies ../llama-cpp into a sibling ../bonsai-<flavor>-build directory, then
+# invokes llama-cpp's own build with LLAMA_REPO/LLAMA_VERSION overridden to point at the
+# fork.
+#
+# The Q1_0/Q2_0 additions are model *weight* types decoded inside libllama, transparent
+# to the reused gRPC server, so (unlike turboquant's KV-cache types) no grpc-server.cpp
+# allow-list patch is needed. The fork branched from upstream before a few API changes
+# the shared grpc-server.cpp depends on; those are carried as patch files under
+# backend/cpp/bonsai/patches/ and applied to the cloned fork by apply-patches.sh.
+PATCHES_DIR := $(CURRENT_MAKEFILE_DIR)/patches
+
+define bonsai-build
+	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build
+	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
+	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
+	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build llama.cpp
+	bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/llama.cpp $(PATCHES_DIR)
+	CMAKE_ARGS="$(CMAKE_ARGS) $(2)" TARGET="$(3)" \
+	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build grpc-server
+	cp -rfv $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server bonsai-$(1)
+endef
+
+bonsai-avx2:
+	$(call bonsai-build,avx2,-DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
+
+bonsai-avx512:
+	$(call bonsai-build,avx512,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
+
+bonsai-avx:
+	$(call bonsai-build,avx,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
+
+bonsai-fallback:
+	$(call bonsai-build,fallback,-DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
+
+# Single-build CPU backend via ggml CPU_ALL_VARIANTS (mirrors llama-cpp-cpu-all).
+# bonsai reuses backend/cpp/llama-cpp's CMakeLists.txt (hw_grpc_proto STATIC) and
+# Makefile (SHARED_LIBS make-var + EXTRA_CMAKE_ARGS), so this passes the same overrides
+# through to the copied build: SHARED_LIBS=ON, the DL flags, and --target ggml (which
+# pulls in the per-microarch libggml-cpu-*.so via ggml's add_dependencies). The .so set
+# is collected for package.sh to bundle into package/lib.
+bonsai-cpu-all:
+	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build
+	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
+	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
+	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build llama.cpp
+	bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/llama.cpp $(PATCHES_DIR)
+	SHARED_LIBS=ON EXTRA_CMAKE_ARGS="-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON" TARGET="--target grpc-server --target ggml" \
+	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build grpc-server
+	cp -rfv $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server bonsai-cpu-all
+	rm -rf ggml-shared-libs && mkdir -p ggml-shared-libs
+	find $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/llama.cpp/build \( -name '*.so*' -o -name '*.dylib' \) -exec cp -av {} ggml-shared-libs/ \;
+	@echo "Collected ggml shared backends:" && ls -la ggml-shared-libs/
+
+bonsai-grpc:
+	$(call bonsai-build,grpc,-DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server --target rpc-server)
+
+bonsai-rpc-server: bonsai-grpc
+	cp -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-grpc-build/llama.cpp/build/bin/rpc-server bonsai-rpc-server
+
+package:
+	bash package.sh
+
+purge:
+	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-*-build
+	rm -rf bonsai-* package
+
+clean: purge
--- a/backend/cpp/bonsai/apply-patches.sh
+++ b/backend/cpp/bonsai/apply-patches.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+# Apply the bonsai patch series to a cloned PrismML llama.cpp (prism branch) checkout.
+#
+# The prism fork branched from upstream llama.cpp before a number of API changes that the
+# shared backend/cpp/llama-cpp/grpc-server.cpp depends on. We carry those upstream commits
+# as patch files under backend/cpp/bonsai/patches/ and apply them here so the reused
+# grpc-server source compiles against the fork unmodified.
+#
+# Drop the corresponding patch from patches/ whenever the fork catches up with upstream —
+# the build will fail fast if a patch stops applying, which is the signal to retire it.
+
+set -euo pipefail
+
+if [[ $# -ne 2 ]]; then
+    echo "usage: $0 <llama.cpp-src-dir> <patches-dir>" >&2
+    exit 2
+fi
+
+SRC_DIR=$1
+PATCHES_DIR=$2
+
+if [[ ! -d "$SRC_DIR" ]]; then
+    echo "source dir does not exist: $SRC_DIR" >&2
+    exit 2
+fi
+
+if [[ ! -d "$PATCHES_DIR" ]]; then
+    echo "no patches dir at $PATCHES_DIR, nothing to apply"
+    exit 0
+fi
+
+shopt -s nullglob
+patches=("$PATCHES_DIR"/*.patch)
+shopt -u nullglob
+
+if [[ ${#patches[@]} -eq 0 ]]; then
+    echo "no .patch files in $PATCHES_DIR, nothing to apply"
+    exit 0
+fi
+
+cd "$SRC_DIR"
+
+for patch in "${patches[@]}"; do
+    echo "==> applying $patch"
+    git apply --verbose "$patch"
+done
+
+echo "all bonsai patches applied successfully"
--- a/backend/cpp/bonsai/package.sh
+++ b/backend/cpp/bonsai/package.sh
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avrf $CURDIR/bonsai-* $CURDIR/package/
+cp -rfv $CURDIR/run.sh $CURDIR/package/
+
+# Bundle the ggml shared backends from the CPU_ALL_VARIANTS build into package/lib. ggml
+# discovers the per-microarch libggml-cpu-*.so by scanning the executable directory, which
+# (via the bundled lib/ld.so that run.sh launches through) resolves to lib/. See the
+# matching comment in backend/cpp/llama-cpp/package.sh. No-op on the fallback/ROCm builds.
+if [ -d "$CURDIR/ggml-shared-libs" ]; then
+    echo "Bundling ggml shared backends (CPU_ALL_VARIANTS)..."
+    cp -avf $CURDIR/ggml-shared-libs/*.so* $CURDIR/package/lib/
+fi
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/
--- a/backend/cpp/bonsai/patches/README.md
+++ b/backend/cpp/bonsai/patches/README.md
@@ -0,0 +1,16 @@
+# bonsai fork skew patches
+
+The `bonsai` backend reuses `backend/cpp/llama-cpp/grpc-server.cpp` (written against
+LocalAI's pinned *upstream* llama.cpp) but compiles it against the PrismML `prism` fork,
+which branched from upstream some commits earlier. Any upstream API change that the shared
+gRPC server depends on, but that the fork does not yet carry, is back-ported here as a
+`*.patch` file and applied to the cloned fork checkout by `../apply-patches.sh`.
+
+Rules:
+
+- One upstream commit (or minimal hunk) per patch, named `NNNN-short-description.patch`.
+- Patches are applied with `git apply` from the fork's checkout root.
+- `apply-patches.sh` fails fast if a patch stops applying cleanly — that is the signal the
+  fork has caught up (or diverged), so re-cut or drop the patch.
+- Keep this set as small as possible; the long-term fix is the fork rebasing onto a newer
+  upstream (or Q1_0/Q2_0 landing in mainline llama.cpp, retiring this backend entirely).
--- a/backend/cpp/bonsai/run.sh
+++ b/backend/cpp/bonsai/run.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+echo "CPU info:"
+grep -e "model\sname" /proc/cpuinfo | head -1
+grep -e "flags" /proc/cpuinfo | head -1
+
+BINARY=bonsai-fallback
+
+# x86/arm64 ship a single bonsai-cpu-all built with ggml CPU_ALL_VARIANTS: ggml's
+# backend registry dlopens the best libggml-cpu-*.so for this host, so no shell-side
+# probing. ROCm ships only bonsai-fallback, so fall back to it when cpu-all is absent.
+if [ -e "$CURDIR"/bonsai-cpu-all ]; then
+	BINARY=bonsai-cpu-all
+fi
+
+if [ -n "$LLAMACPP_GRPC_SERVERS" ]; then
+	if [ -e "$CURDIR"/bonsai-grpc ]; then
+		BINARY=bonsai-grpc
+	fi
+fi
+
+# Extend ld library path with the dir where this script is located/lib
+if [ "$(uname)" == "Darwin" ]; then
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+	# Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files)
+	if [ -d "$CURDIR/lib/rocblas/library" ]; then
+		export ROCBLAS_TENSILE_LIBPATH="$CURDIR"/lib/rocblas/library
+	fi
+	# Same for hipBLASLt (rocblaslt): the bundled libhipblaslt.so resolves its
+	# TensileLibrary_lazy_gfx*.dat kernel data relative to itself, so point it at
+	# the bundled data or it falls back to slow generic kernels (issue #10660).
+	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
+		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
+	fi
+fi
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using binary: $BINARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/$BINARY "$@"
+fi
+
+echo "Using binary: $BINARY"
+exec "$CURDIR"/$BINARY "$@"
+
+# We should never reach this point, however just in case we do, run fallback
+exec "$CURDIR"/bonsai-fallback "$@"
--- a/backend/index.yaml
+++ b/backend/index.yaml
@@ -72,6 +72,40 @@
    nvidia-cuda-12: "cuda12-turboquant"
    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-turboquant"
    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-turboquant"
+- &bonsai
+  name: "bonsai"
+  alias: "bonsai"
+  license: apache-2.0
+  description: |
+    Fork of llama.cpp (PrismML) adding the Q1_0 (1-bit) and Q2_0 (ternary, 1.58-bit)
+    weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Reuses
+    the LocalAI llama.cpp gRPC server sources against the fork's libllama, so it drives
+    these sub-2-bit models with the same OpenAI-compatible API as the stock llama.cpp
+    backend.
+  urls:
+    - https://github.com/PrismML-Eng/llama.cpp
+    - https://huggingface.co/prism-ml
+  tags:
+    - text-to-text
+    - LLM
+    - CPU
+    - GPU
+    - CUDA
+    - HIP
+    - bonsai
+    - 1-bit
+    - ternary
+  capabilities:
+    default: "cpu-bonsai"
+    nvidia: "cuda12-bonsai"
+    intel: "intel-sycl-f16-bonsai"
+    amd: "rocm-bonsai"
+    vulkan: "vulkan-bonsai"
+    nvidia-l4t: "nvidia-l4t-arm64-bonsai"
+    nvidia-cuda-13: "cuda13-bonsai"
+    nvidia-cuda-12: "cuda12-bonsai"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-bonsai"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-bonsai"
 - &ds4
  name: "ds4"
  alias: "ds4"
@@ -1779,6 +1813,19 @@
    nvidia-cuda-12: "cuda12-turboquant-development"
    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-turboquant-development"
    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-turboquant-development"
+- !!merge <<: *bonsai
+  name: "bonsai-development"
+  capabilities:
+    default: "cpu-bonsai-development"
+    nvidia: "cuda12-bonsai-development"
+    intel: "intel-sycl-f16-bonsai-development"
+    amd: "rocm-bonsai-development"
+    vulkan: "vulkan-bonsai-development"
+    nvidia-l4t: "nvidia-l4t-arm64-bonsai-development"
+    nvidia-cuda-13: "cuda13-bonsai-development"
+    nvidia-cuda-12: "cuda12-bonsai-development"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-bonsai-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-bonsai-development"
 - !!merge <<: *ds4
  name: "ds4-development"
  capabilities:
@@ -2472,6 +2519,97 @@
  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-turboquant"
  mirrors:
    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-turboquant
+## bonsai
+- !!merge <<: *bonsai
+  name: "cpu-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-cpu-bonsai
+- !!merge <<: *bonsai
+  name: "cpu-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-bonsai"
+  mirrors:
+    - localai/localai-backends:master-cpu-bonsai
+- !!merge <<: *bonsai
+  name: "cuda12-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-bonsai
+- !!merge <<: *bonsai
+  name: "cuda12-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-bonsai"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-bonsai
+- !!merge <<: *bonsai
+  name: "cuda13-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-bonsai
+- !!merge <<: *bonsai
+  name: "cuda13-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-bonsai"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-bonsai
+- !!merge <<: *bonsai
+  name: "rocm-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-bonsai
+- !!merge <<: *bonsai
+  name: "rocm-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-bonsai"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-bonsai
+- !!merge <<: *bonsai
+  name: "intel-sycl-f32-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f32-bonsai
+- !!merge <<: *bonsai
+  name: "intel-sycl-f32-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-bonsai"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f32-bonsai
+- !!merge <<: *bonsai
+  name: "intel-sycl-f16-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f16-bonsai
+- !!merge <<: *bonsai
+  name: "intel-sycl-f16-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-bonsai"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f16-bonsai
+- !!merge <<: *bonsai
+  name: "vulkan-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-bonsai
+- !!merge <<: *bonsai
+  name: "vulkan-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-bonsai"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-bonsai
+- !!merge <<: *bonsai
+  name: "nvidia-l4t-arm64-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-arm64-bonsai
+- !!merge <<: *bonsai
+  name: "nvidia-l4t-arm64-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-bonsai"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-arm64-bonsai
+- !!merge <<: *bonsai
+  name: "cuda13-nvidia-l4t-arm64-bonsai"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-bonsai"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-bonsai
+- !!merge <<: *bonsai
+  name: "cuda13-nvidia-l4t-arm64-bonsai-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-bonsai"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-bonsai
 ## ds4
 - !!merge <<: *ds4
  name: "cpu-ds4"
--- a/core/application/startup.go
+++ b/core/application/startup.go
@@ -688,7 +688,7 @@ func loadRuntimeSettingsFromFile(options *config.ApplicationConfig) {
 	// physical-core fallback. Detect the env/CLI explicitly so
 	// LOCALAI_THREADS/THREADS still win over the persisted file value.
 	if settings.Threads != nil {
-		if os.Getenv("LOCALAI_THREADS") == "" && os.Getenv("THREADS") == "" {
+		if os.Getenv("LOCALAI_THREADS") == "" && os.Getenv("THREADS") == "" { //nolint:forbidigo // deliberate env probe, see comment above
 			options.Threads = *settings.Threads
 		}
 	}
--- a/gallery/index.yaml
+++ b/gallery/index.yaml
@@ -1,4 +1,238 @@
 ---
+- &bonsai-8b
+  name: "bonsai-8b-1bit"
+  url: "github:mudler/LocalAI/gallery/qwen3.yaml@master"
+  license: apache-2.0
+  icon: https://huggingface.co/prism-ml/Bonsai-8B-gguf/resolve/main/assets/bonsai-logo.svg
+  urls:
+    - https://huggingface.co/prism-ml/Bonsai-8B-gguf
+    - https://github.com/PrismML-Eng/llama.cpp
+  description: |
+    Bonsai 8B (PrismML) is an end-to-end 1-bit language model built on the Qwen3-8B dense
+    architecture (GQA, SwiGLU, RoPE, RMSNorm, 36 layers, 65,536 context). Every weight is a
+    single sign bit (`-scale` / `+scale`) with one FP16 scale per group of 128 weights, for
+    an effective 1.125 bits/weight and a ~1.15 GB footprint (14.2x smaller than FP16) while
+    matching full-precision 8B instruct models at ~70.5 average across 6 benchmark
+    categories.
+
+    The Q1_0 quantization is only decodable by the PrismML llama.cpp fork, so this entry
+    runs on LocalAI's `bonsai` backend (that fork), not the stock `llama-cpp` backend.
+    License: Apache 2.0.
+  tags:
+    - llm
+    - gguf
+    - gpu
+    - cpu
+    - qwen
+    - bonsai
+    - 1-bit
+  overrides:
+    backend: bonsai
+    parameters:
+      model: bonsai/models/Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf
+      temperature: 0.5
+      top_p: 0.9
+      top_k: 20
+  files:
+    - filename: bonsai/models/Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf
+      sha256: 284a335aa3fb2ced3b1b01fcb40b08aa783e3b70832767f0dd2e3fdfa134bd54
+      uri: huggingface://prism-ml/Bonsai-8B-gguf/Bonsai-8B-Q1_0.gguf
+- &ternary-bonsai-8b
+  name: "ternary-bonsai-8b"
+  url: "github:mudler/LocalAI/gallery/qwen3.yaml@master"
+  license: apache-2.0
+  icon: https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf/resolve/main/assets/bonsai-logo.svg
+  urls:
+    - https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf
+    - https://github.com/PrismML-Eng/llama.cpp
+  description: |
+    Ternary Bonsai 8B (PrismML) is a 1.58-bit ternary language model on the Qwen3-8B dense
+    architecture. Each weight takes a value from {-1, 0, +1} with one shared FP16 scale per
+    group of 128 weights (GGUF Q2_0, ~2.18 GB deployed, 7.5x smaller than FP16). The extra
+    zero state recovers more of the full-precision model than the 1-bit build: it ranks 2nd
+    among compared 6-9B models at 75.5 average despite being ~1/8th their size.
+
+    Q2_0 is the recommended, ternary-lossless variant. The Q2_0 kernels are only in the
+    PrismML llama.cpp fork, so this runs on LocalAI's `bonsai` backend. License: Apache 2.0.
+  tags:
+    - llm
+    - gguf
+    - gpu
+    - cpu
+    - qwen
+    - bonsai
+    - ternary
+  overrides:
+    backend: bonsai
+    parameters:
+      model: bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0.gguf
+      temperature: 0.5
+      top_p: 0.9
+      top_k: 20
+  files:
+    - filename: bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0.gguf
+      sha256: 3c8d70470a5d97e5a2b9410ddd899cb740116591462626c60cb2fead6448f60b
+      uri: huggingface://prism-ml/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0.gguf
+- !!merge <<: *ternary-bonsai-8b
+  name: "ternary-bonsai-8b-q2-g64"
+  description: |
+    Ternary Bonsai 8B (PrismML), GGUF Q2_0 with group-64 packing (each FP16 scale shared
+    across 64 weights instead of 128). Slightly larger (~2.31 GB) but matches llama.cpp's
+    native 64-value Q2_0 block layout. Runs on LocalAI's `bonsai` backend. License: Apache 2.0.
+  overrides:
+    backend: bonsai
+    parameters:
+      model: bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0_g64.gguf
+      temperature: 0.5
+      top_p: 0.9
+      top_k: 20
+  files:
+    - filename: bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0_g64.gguf
+      sha256: e17b298d84ee78797916ae5c2ecc8211469cc65cccfe3080cd9a9bb503fbc55e
+      uri: huggingface://prism-ml/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0_g64.gguf
+- !!merge <<: *ternary-bonsai-8b
+  name: "ternary-bonsai-8b-pq2"
+  description: |
+    Ternary Bonsai 8B (PrismML), GGUF PQ2_0 (packed Q2_0) ternary variant (~2.18 GB).
+    Same {-1, 0, +1} weight alphabet as Q2_0. Runs on LocalAI's `bonsai` backend.
+    License: Apache 2.0.
+  overrides:
+    backend: bonsai
+    parameters:
+      model: bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-PQ2_0.gguf
+      temperature: 0.5
+      top_p: 0.9
+      top_k: 20
+  files:
+    - filename: bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-PQ2_0.gguf
+      sha256: 1376f942aa90e60f7b570c1d81b3916fea1315ff85aa1c4d19006af68fb4b922
+      uri: huggingface://prism-ml/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-PQ2_0.gguf
+- &bonsai-27b
+  name: "bonsai-27b-1bit"
+  url: "github:mudler/LocalAI/gallery/qwen3.yaml@master"
+  license: apache-2.0
+  icon: https://huggingface.co/prism-ml/Bonsai-27B-gguf/resolve/main/assets/bonsai-logo.svg
+  urls:
+    - https://huggingface.co/prism-ml/Bonsai-27B-gguf
+    - https://github.com/PrismML-Eng/llama.cpp
+  description: |
+    Bonsai 27B (PrismML) is a full 27B-class reasoning model in end-to-end 1-bit weights,
+    derived from the Qwen3.6-27B hybrid-attention backbone (~75% linear attention, 262K
+    context). At a true 1.125 bits/weight it deploys in ~3.9 GB (~14.2x smaller than FP16)
+    while retaining 89.5% of FP16 intelligence across 15 thinking-mode benchmarks (math
+    91.66, coding 81.88). Ships an optional 4-bit vision tower (mmproj) for image input,
+    included here.
+
+    The Q1_0_g128 weights and hybrid-attention kernels are only in the PrismML llama.cpp
+    fork, so this runs on LocalAI's `bonsai` backend. A GPU is recommended. License: Apache 2.0.
+  tags:
+    - llm
+    - gguf
+    - gpu
+    - qwen
+    - bonsai
+    - 1-bit
+    - vision
+    - multimodal
+  overrides:
+    backend: bonsai
+    mmproj: bonsai/mmproj/Bonsai-27B-gguf/Bonsai-27B-mmproj-Q8_0.gguf
+    parameters:
+      model: bonsai/models/Bonsai-27B-gguf/Bonsai-27B-Q1_0.gguf
+      temperature: 0.7
+      top_p: 0.95
+      top_k: 20
+  files:
+    - filename: bonsai/models/Bonsai-27B-gguf/Bonsai-27B-Q1_0.gguf
+      sha256: 17ef842e47450caeb8eaa3ebfbbab5d2f2278b62b79be107985fb69a2f819aa0
+      uri: huggingface://prism-ml/Bonsai-27B-gguf/Bonsai-27B-Q1_0.gguf
+    - filename: bonsai/mmproj/Bonsai-27B-gguf/Bonsai-27B-mmproj-Q8_0.gguf
+      sha256: eb561d41a7bbeb0fcf04883c8af11078ef6cae0a66862a0b68443cfca495269d
+      uri: huggingface://prism-ml/Bonsai-27B-gguf/Bonsai-27B-mmproj-Q8_0.gguf
+- &ternary-bonsai-27b
+  name: "ternary-bonsai-27b"
+  url: "github:mudler/LocalAI/gallery/qwen3.yaml@master"
+  license: apache-2.0
+  icon: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/resolve/main/assets/bonsai-logo.svg
+  urls:
+    - https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
+    - https://github.com/PrismML-Eng/llama.cpp
+  description: |
+    Ternary Bonsai 27B (PrismML) is the quality-oriented operating point of the Bonsai 27B
+    family: full 27B-class reasoning in ternary {-1, 0, +1} weights on the Qwen3.6-27B
+    hybrid-attention backbone (262K context). At a true 1.71 bits/weight it deploys in
+    ~7.2 GB (GGUF Q2_0_g128) and retains 95% of FP16 intelligence (80.49 average across 15
+    thinking-mode benchmarks) - a higher score than a conventional IQ2_XXS build at less
+    than two-thirds its footprint. Ships an optional 4-bit vision tower (mmproj), included.
+
+    The Q2_0 weights and hybrid-attention kernels are only in the PrismML llama.cpp fork,
+    so this runs on LocalAI's `bonsai` backend. A GPU is recommended. License: Apache 2.0.
+  tags:
+    - llm
+    - gguf
+    - gpu
+    - qwen
+    - bonsai
+    - ternary
+    - vision
+    - multimodal
+  overrides:
+    backend: bonsai
+    mmproj: bonsai/mmproj/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+    parameters:
+      model: bonsai/models/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf
+      temperature: 0.7
+      top_p: 0.95
+      top_k: 20
+  files:
+    - filename: bonsai/models/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf
+      sha256: 868c11714cf8fe47f5ec9eeb2be0ab1a337112886f92ee0ede6b855c4fa31757
+      uri: huggingface://prism-ml/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf
+    - filename: bonsai/mmproj/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+      sha256: eb561d41a7bbeb0fcf04883c8af11078ef6cae0a66862a0b68443cfca495269d
+      uri: huggingface://prism-ml/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+- !!merge <<: *ternary-bonsai-27b
+  name: "ternary-bonsai-27b-pq2"
+  description: |
+    Ternary Bonsai 27B (PrismML), GGUF PQ2_0 (packed Q2_0) ternary variant (~7.17 GB) with
+    the 4-bit vision tower (mmproj) included. Runs on LocalAI's `bonsai` backend.
+    License: Apache 2.0.
+  overrides:
+    backend: bonsai
+    mmproj: bonsai/mmproj/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+    parameters:
+      model: bonsai/models/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-PQ2_0.gguf
+      temperature: 0.7
+      top_p: 0.95
+      top_k: 20
+  files:
+    - filename: bonsai/models/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-PQ2_0.gguf
+      sha256: e4781999f1997ef97ce0c58d05750835acc999d18d83ee6489ba7ac7b14cb5f6
+      uri: huggingface://prism-ml/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-PQ2_0.gguf
+    - filename: bonsai/mmproj/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+      sha256: eb561d41a7bbeb0fcf04883c8af11078ef6cae0a66862a0b68443cfca495269d
+      uri: huggingface://prism-ml/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+- !!merge <<: *ternary-bonsai-27b
+  name: "ternary-bonsai-27b-q2-g64"
+  description: |
+    Ternary Bonsai 27B (PrismML), GGUF Q2_0 with group-64 packing (~7.59 GB), matching
+    llama.cpp's native 64-value Q2_0 block layout, with the 4-bit vision tower (mmproj)
+    included. Runs on LocalAI's `bonsai` backend. License: Apache 2.0.
+  overrides:
+    backend: bonsai
+    mmproj: bonsai/mmproj/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+    parameters:
+      model: bonsai/models/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_g64.gguf
+      temperature: 0.7
+      top_p: 0.95
+      top_k: 20
+  files:
+    - filename: bonsai/models/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_g64.gguf
+      sha256: 59a45d1ecef702b14531b06d22949f33b25c1897da31a8c0b298e01e4d9138eb
+      uri: huggingface://prism-ml/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_g64.gguf
+    - filename: bonsai/mmproj/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
+      sha256: eb561d41a7bbeb0fcf04883c8af11078ef6cae0a66862a0b68443cfca495269d
+      uri: huggingface://prism-ml/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-mmproj-Q8_0.gguf
 - name: "minicpm5-1b-claude-opus-fable5-thinking"
  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
  urls: