mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-02 11:30:44 -04:00
bbe018c1a0
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp (github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0 (1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants. Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models are served with the same OpenAI-compatible API. No grpc-server allow-list patch is needed (bonsai adds weight quants, transparent to the server, not KV-cache types), and the reused server compiles cleanly against the fork with no skew patches (validated locally via a CPU docker build; patches/ is present but empty for any future re-pin skew). Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh, backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml meta-backend + per-platform images, and a nightly bump_deps entry tracking the `prism` branch. Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64, +pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
40 lines
1.4 KiB
Bash
Executable File
40 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Shared compile logic for backend/Dockerfile.bonsai.
|
|
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
|
|
|
set -euxo pipefail
|
|
|
|
export CCACHE_DIR=/root/.ccache
|
|
ccache --max-size=5G || true
|
|
ccache -z || true
|
|
|
|
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
|
|
|
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
|
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
|
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
|
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
|
rm -rf /LocalAI/backend/cpp/bonsai-*-build
|
|
fi
|
|
|
|
cd /LocalAI/backend/cpp/bonsai
|
|
|
|
if [ -z "${BUILD_TYPE:-}" ]; then
|
|
# Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries.
|
|
# arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme).
|
|
if [ "${TARGETARCH}" = "arm64" ]; then
|
|
apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
|
|
export CC=gcc-14 CXX=g++-14
|
|
fi
|
|
make bonsai-cpu-all
|
|
else
|
|
# GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator
|
|
# does the compute. Keeps the GPU compile from also building the CPU variant matrix and
|
|
# avoids the gcc-14 apt step on GPU base images such as nvidia l4t.
|
|
make bonsai-fallback
|
|
fi
|
|
make bonsai-grpc
|
|
make bonsai-rpc-server
|
|
|
|
ccache -s || true
|