mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834)

Browse Source 
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models

Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.

Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).

Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.

Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
LocalAI [bot]
2026-07-16 10:09:14 +02:00
committed by GitHub
parent 3880812ed6
commit bbe018c1a0
13 changed files with 1039 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

163
.github/backend-matrix.yml vendored
View File

@@ -452,6 +452,22 @@ include:
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
ubuntu-version: '2404'
- build-type: 'cublas'
cuda-major-version: "12"
cuda-minor-version: "8"
platforms: 'linux/amd64'
tag-latest: 'auto'
tag-suffix: '-gpu-nvidia-cuda-12-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-12-amd64'
# bigger-runner: same rationale as -gpu-nvidia-cuda-12-llama-cpp above
# (observed 6h5m wall-clock on v4.2.1, just past the 6h job timeout).
runs-on: 'bigger-runner'
base-image: "ubuntu:24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
- build-type: 'cublas'
cuda-major-version: "12"
cuda-minor-version: "8"
@@ -1082,6 +1098,21 @@ include:
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
ubuntu-version: '2404'
- build-type: 'cublas'
cuda-major-version: "13"
cuda-minor-version: "0"
platforms: 'linux/amd64'
tag-latest: 'auto'
tag-suffix: '-gpu-nvidia-cuda-13-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64'
# bigger-runner: observed 6h5m wall-clock on v4.2.1 — at the GHA timeout.
runs-on: 'bigger-runner'
base-image: "ubuntu:24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
- build-type: 'cublas'
cuda-major-version: "13"
cuda-minor-version: "0"
@@ -1110,6 +1141,20 @@ include:
backend: "turboquant"
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
- build-type: 'cublas'
cuda-major-version: "13"
cuda-minor-version: "0"
platforms: 'linux/arm64'
skip-drivers: 'false'
tag-latest: 'auto'
tag-suffix: '-nvidia-l4t-cuda-13-arm64-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-arm64'
base-image: "ubuntu:24.04"
runs-on: 'ubuntu-24.04-arm'
ubuntu-version: '2404'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
- build-type: 'cublas'
cuda-major-version: "13"
cuda-minor-version: "0"
@@ -1983,6 +2028,20 @@ include:
dockerfile: "./backend/Dockerfile.llama-cpp"
context: "./"
ubuntu-version: '2404'
- build-type: 'hipblas'
cuda-major-version: ""
cuda-minor-version: ""
platforms: 'linux/amd64'
tag-latest: 'auto'
tag-suffix: '-gpu-rocm-hipblas-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-rocm-amd64'
runs-on: 'ubuntu-latest'
base-image: "rocm/dev-ubuntu-24.04:7.2.1"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
- build-type: 'hipblas'
cuda-major-version: ""
cuda-minor-version: ""
@@ -2247,6 +2306,20 @@ include:
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
ubuntu-version: '2404'
- build-type: 'sycl_f32'
cuda-major-version: ""
cuda-minor-version: ""
platforms: 'linux/amd64'
tag-latest: 'auto'
tag-suffix: '-gpu-intel-sycl-f32-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
runs-on: 'ubuntu-latest'
base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
- build-type: 'sycl_f16'
cuda-major-version: ""
cuda-minor-version: ""
@@ -2275,6 +2348,20 @@ include:
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
ubuntu-version: '2404'
- build-type: 'sycl_f16'
cuda-major-version: ""
cuda-minor-version: ""
platforms: 'linux/amd64'
tag-latest: 'auto'
tag-suffix: '-gpu-intel-sycl-f16-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
runs-on: 'ubuntu-latest'
base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
- build-type: 'intel'
cuda-major-version: ""
cuda-minor-version: ""
@@ -2727,6 +2814,21 @@ include:
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
ubuntu-version: '2404'
- build-type: ''
cuda-major-version: ""
cuda-minor-version: ""
platforms: 'linux/amd64'
platform-tag: 'amd64'
tag-latest: 'auto'
tag-suffix: '-cpu-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-amd64'
runs-on: 'ubuntu-latest'
base-image: "ubuntu:24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
- build-type: ''
cuda-major-version: ""
cuda-minor-version: ""
@@ -2742,6 +2844,21 @@ include:
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
ubuntu-version: '2404'
- build-type: ''
cuda-major-version: ""
cuda-minor-version: ""
platforms: 'linux/arm64'
platform-tag: 'arm64'
tag-latest: 'auto'
tag-suffix: '-cpu-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-arm64'
runs-on: 'ubuntu-24.04-arm'
base-image: "ubuntu:24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
- build-type: ''
cuda-major-version: ""
cuda-minor-version: ""
@@ -2884,6 +3001,20 @@ include:
dockerfile: "./backend/Dockerfile.turboquant"
context: "./"
ubuntu-version: '2204'
- build-type: 'cublas'
cuda-major-version: "12"
cuda-minor-version: "0"
platforms: 'linux/arm64'
skip-drivers: 'false'
tag-latest: 'auto'
tag-suffix: '-nvidia-l4t-arm64-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-l4t-cuda-12-arm64'
base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
runs-on: 'ubuntu-24.04-arm'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2204'
- build-type: 'vulkan'
cuda-major-version: ""
cuda-minor-version: ""
@@ -2930,6 +3061,22 @@ include:
context: "./"
ubuntu-version: '2404'
# Stablediffusion-ggml
- build-type: 'vulkan'
cuda-major-version: ""
cuda-minor-version: ""
platforms: 'linux/amd64'
platform-tag: 'amd64'
tag-latest: 'auto'
tag-suffix: '-gpu-vulkan-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-amd64'
runs-on: 'ubuntu-latest'
base-image: "ubuntu:24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
# Stablediffusion-ggml
- build-type: 'vulkan'
cuda-major-version: ""
cuda-minor-version: ""
@@ -2946,6 +3093,22 @@ include:
context: "./"
ubuntu-version: '2404'
# Stablediffusion-ggml
- build-type: 'vulkan'
cuda-major-version: ""
cuda-minor-version: ""
platforms: 'linux/arm64'
platform-tag: 'arm64'
tag-latest: 'auto'
tag-suffix: '-gpu-vulkan-bonsai'
builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-arm64'
runs-on: 'ubuntu-24.04-arm'
base-image: "ubuntu:24.04"
skip-drivers: 'false'
backend: "bonsai"
dockerfile: "./backend/Dockerfile.bonsai"
context: "./"
ubuntu-version: '2404'
# Stablediffusion-ggml
- build-type: ''
cuda-major-version: ""
cuda-minor-version: ""

4
.github/workflows/bump_deps.yaml vendored
View File

@@ -22,6 +22,10 @@ jobs:
variable: "TURBOQUANT_VERSION"
branch: "feature/turboquant-kv-cache"
file: "backend/cpp/turboquant/Makefile"
- repository: "PrismML-Eng/llama.cpp"
variable: "BONSAI_VERSION"
branch: "prism"
file: "backend/cpp/bonsai/Makefile"
- repository: "antirez/ds4"
variable: "DS4_VERSION"
branch: "main"