From cd6079b2f31a323418d851b3638bbccad840e282 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 24 Apr 2026 08:05:21 +0000 Subject: [PATCH] feat(backend): add buun-llama-cpp fork (DFlash + TCQ KV-cache) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spiritbuun/buun-llama-cpp is a fork of TheTom/llama-cpp-turboquant that adds two independent features on top: DFlash block-diffusion speculative decoding (via a dedicated DFlashDraftModel GGUF arch) and two extra TCQ KV-cache variants (turbo2_tcq, turbo3_tcq) on top of TurboQuant's turbo2/turbo3/turbo4. Follows the turboquant thin-wrapper pattern — reuses backend/cpp/llama-cpp grpc-server sources verbatim, patches only the build copy to extend the KV allow-list and wire up buun-exclusive tree_budget / draft_topk options. DraftModel is already wired end-to-end (proto field 39 → params.speculative), so DFlash activation only needs the existing options passthrough (spec_type:dflash) plus the drafter path in draft_model. CacheTypeOptions now surfaces the five turbo* values so the React UI dropdown shows them — benefits turboquant too (previously users had to type them in YAML manually). Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash] [WebFetch] Signed-off-by: Ettore Di Giacinto --- Makefile | 23 +- backend/Dockerfile.buun-llama-cpp | 290 ++++++++++++++++++ backend/cpp/buun-llama-cpp/Makefile | 85 +++++ backend/cpp/buun-llama-cpp/apply-patches.sh | 50 +++ backend/cpp/buun-llama-cpp/package.sh | 57 ++++ .../cpp/buun-llama-cpp/patch-grpc-server.sh | 141 +++++++++ backend/cpp/buun-llama-cpp/run.sh | 65 ++++ core/config/meta/constants.go | 8 + core/gallery/importers/llama-cpp.go | 3 +- docs/content/features/text-generation.md | 77 +++++ docs/content/reference/compatibility-table.md | 1 + 11 files changed, 797 insertions(+), 3 deletions(-) create mode 100644 backend/Dockerfile.buun-llama-cpp create mode 100644 backend/cpp/buun-llama-cpp/Makefile create mode 100755 backend/cpp/buun-llama-cpp/apply-patches.sh create mode 100755 backend/cpp/buun-llama-cpp/package.sh create mode 100755 backend/cpp/buun-llama-cpp/patch-grpc-server.sh create mode 100755 backend/cpp/buun-llama-cpp/run.sh diff --git a/Makefile b/Makefile index 3f59e1d13..65fb70b48 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ # Disable parallel execution for backend builds -.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/tinygrad backends/sherpa-onnx +.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/buun-llama-cpp backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/tinygrad backends/sherpa-onnx GOCMD=go GOTEST=$(GOCMD) test @@ -545,6 +545,19 @@ test-extra-backend-turboquant: docker-build-turboquant BACKEND_TEST_CACHE_TYPE_V=turbo3 \ $(MAKE) test-extra-backend +## buun-llama-cpp: exercises the fork-of-a-fork backend (spiritbuun/buun-llama-cpp) +## with the *TurboQuant/TCQ-specific* KV-cache types (turbo3 for V). Same rationale +## as turboquant above: picking a standard llama.cpp type would only re-test the +## shared code path. buun inherits turboquant's turbo2/turbo3/turbo4 and adds +## turbo2_tcq / turbo3_tcq on top. DFlash speculative decoding is not exercised +## here because no small DFlash drafter model exists (the known public pair is +## Qwen3.5-27B, ~54 GB). +test-extra-backend-buun-llama-cpp: docker-build-buun-llama-cpp + BACKEND_IMAGE=local-ai-backend:buun-llama-cpp \ + BACKEND_TEST_CACHE_TYPE_K=q8_0 \ + BACKEND_TEST_CACHE_TYPE_V=turbo3 \ + $(MAKE) test-extra-backend + ## Audio transcription wrapper for the llama-cpp backend. ## Drives the new AudioTranscription / AudioTranscriptionStream RPCs against ## ggml-org/Qwen3-ASR-0.6B-GGUF (a small ASR model that requires its mmproj @@ -949,6 +962,11 @@ BACKEND_IK_LLAMA_CPP = ik-llama-cpp|ik-llama-cpp|.|false|false # turboquant is a llama.cpp fork with TurboQuant KV-cache quantization. # Reuses backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile. BACKEND_TURBOQUANT = turboquant|turboquant|.|false|false +# buun-llama-cpp is a fork-of-a-fork (spiritbuun/buun-llama-cpp forks +# TheTom/llama-cpp-turboquant) that adds DFlash block-diffusion speculative +# decoding and extra TCQ KV-cache variants on top of TurboQuant. Same thin +# wrapper pattern as turboquant — reuses backend/cpp/llama-cpp grpc-server. +BACKEND_BUUN_LLAMA_CPP = buun-llama-cpp|buun-llama-cpp|.|false|false # Golang backends BACKEND_PIPER = piper|golang|.|false|true @@ -1029,6 +1047,7 @@ endef $(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP))) $(eval $(call generate-docker-build-target,$(BACKEND_IK_LLAMA_CPP))) $(eval $(call generate-docker-build-target,$(BACKEND_TURBOQUANT))) +$(eval $(call generate-docker-build-target,$(BACKEND_BUUN_LLAMA_CPP))) $(eval $(call generate-docker-build-target,$(BACKEND_PIPER))) $(eval $(call generate-docker-build-target,$(BACKEND_LOCAL_STORE))) $(eval $(call generate-docker-build-target,$(BACKEND_HUGGINGFACE))) @@ -1080,7 +1099,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX))) docker-save-%: backend-images docker save local-ai-backend:$* -o backend-images/$*.tar -docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx +docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-buun-llama-cpp docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx ######################################################## ### Mock Backend for E2E Tests diff --git a/backend/Dockerfile.buun-llama-cpp b/backend/Dockerfile.buun-llama-cpp new file mode 100644 index 000000000..2e5d95c02 --- /dev/null +++ b/backend/Dockerfile.buun-llama-cpp @@ -0,0 +1,290 @@ +ARG BASE_IMAGE=ubuntu:24.04 +ARG GRPC_BASE_IMAGE=${BASE_IMAGE} + + +# The grpc target does one thing, it builds and installs GRPC. This is in it's own layer so that it can be effectively cached by CI. +# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work. +FROM ${GRPC_BASE_IMAGE} AS grpc + +# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI +ARG GRPC_MAKEFLAGS="-j4 -Otarget" +ARG GRPC_VERSION=v1.65.0 +ARG CMAKE_FROM_SOURCE=false +# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues +ARG CMAKE_VERSION=3.31.10 + +ENV MAKEFLAGS=${GRPC_MAKEFLAGS} + +WORKDIR /build + +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + ca-certificates \ + build-essential curl libssl-dev \ + git wget && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +# Install CMake (the version in 22.04 is too old) +RUN </dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \ + echo "WARNING: No rocBLAS kernel data found" \ + ; fi + +RUN echo "TARGETARCH: $TARGETARCH" + +# We need protoc installed, and the version in 22.04 is too old. We will create one as part installing the GRPC build below +# but that will also being in a newer version of absl which stablediffusion cannot compile with. This version of protoc is only +# here so that we can generate the grpc code for the stablediffusion build +RUN </dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1) +ARCH?=$(shell uname -m) + +CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) +LLAMA_CPP_DIR := $(CURRENT_MAKEFILE_DIR)/../llama-cpp + +GREEN := \033[0;32m +RESET := \033[0m + +# buun-llama-cpp is a llama.cpp fork-of-a-fork (spiritbuun/buun-llama-cpp forked +# TheTom/llama-cpp-turboquant, which itself forked ggml-org/llama.cpp). Rather +# than duplicating grpc-server.cpp / CMakeLists.txt / prepare.sh we reuse the +# ones in backend/cpp/llama-cpp, and only swap which repo+sha the fetch step +# pulls. Each flavor target copies ../llama-cpp into a sibling +# ../buun-llama-cpp--build directory, then invokes llama-cpp's own +# build-llama-cpp-grpc-server with LLAMA_REPO/LLAMA_VERSION overridden to point +# at the fork. +PATCHES_DIR := $(CURRENT_MAKEFILE_DIR)/patches + +# Each flavor target: +# 1. copies backend/cpp/llama-cpp/ (grpc-server.cpp + prepare.sh + CMakeLists.txt + Makefile) +# into a sibling buun-llama-cpp--build directory; +# 2. clones the buun fork into buun-llama-cpp--build/llama.cpp via the +# copy's own `llama.cpp` target, overriding LLAMA_REPO/LLAMA_VERSION; +# 3. applies patches from backend/cpp/buun-llama-cpp/patches/ to the cloned +# fork sources (for backporting upstream commits the fork hasn't pulled); +# 4. runs the copy's `grpc-server` target, which produces the binary we copy +# up as buun-llama-cpp-. +define buun-llama-cpp-build + rm -rf $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build + cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build + $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build purge + # Augment the copied grpc-server.cpp's KV-cache allow-list with the + # fork's turbo2/turbo3/turbo4/turbo2_tcq/turbo3_tcq types and wire up the + # DFlash-specific option handlers (tree_budget / draft_topk). We patch the + # *copy*, never the original under backend/cpp/llama-cpp/, so the stock + # llama-cpp build stays compiling against vanilla upstream. + bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/grpc-server.cpp + $(info $(GREEN)I buun-llama-cpp build info:$(1)$(RESET)) + LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BUUN_LLAMA_VERSION) \ + $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build llama.cpp + bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/llama.cpp $(PATCHES_DIR) + CMAKE_ARGS="$(CMAKE_ARGS) $(2)" TARGET="$(3)" \ + LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BUUN_LLAMA_VERSION) \ + $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build grpc-server + cp -rfv $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/grpc-server buun-llama-cpp-$(1) +endef + +buun-llama-cpp-avx2: + $(call buun-llama-cpp-build,avx2,-DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server) + +buun-llama-cpp-avx512: + $(call buun-llama-cpp-build,avx512,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server) + +buun-llama-cpp-avx: + $(call buun-llama-cpp-build,avx,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server) + +buun-llama-cpp-fallback: + $(call buun-llama-cpp-build,fallback,-DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server) + +buun-llama-cpp-grpc: + $(call buun-llama-cpp-build,grpc,-DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server --target rpc-server) + +buun-llama-cpp-rpc-server: buun-llama-cpp-grpc + cp -rf $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-grpc-build/llama.cpp/build/bin/rpc-server buun-llama-cpp-rpc-server + +package: + bash package.sh + +purge: + rm -rf $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-*-build + rm -rf buun-llama-cpp-* package + +clean: purge diff --git a/backend/cpp/buun-llama-cpp/apply-patches.sh b/backend/cpp/buun-llama-cpp/apply-patches.sh new file mode 100755 index 000000000..7ad020514 --- /dev/null +++ b/backend/cpp/buun-llama-cpp/apply-patches.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# Apply the buun-llama-cpp patch series to a cloned buun-llama-cpp checkout. +# +# buun-llama-cpp is a fork-of-a-fork that branched off upstream llama.cpp +# before some API changes the shared backend/cpp/llama-cpp/grpc-server.cpp +# depends on. We carry those upstream commits as patch files under +# backend/cpp/buun-llama-cpp/patches/ and apply them here so the reused +# grpc-server source compiles against the fork unmodified. +# +# Drop the corresponding patch from patches/ whenever the fork catches up with +# upstream — the build will fail fast if a patch stops applying, which is the +# signal to retire it. + +set -euo pipefail + +if [[ $# -ne 2 ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +SRC_DIR=$1 +PATCHES_DIR=$2 + +if [[ ! -d "$SRC_DIR" ]]; then + echo "source dir does not exist: $SRC_DIR" >&2 + exit 2 +fi + +if [[ ! -d "$PATCHES_DIR" ]]; then + echo "no patches dir at $PATCHES_DIR, nothing to apply" + exit 0 +fi + +shopt -s nullglob +patches=("$PATCHES_DIR"/*.patch) +shopt -u nullglob + +if [[ ${#patches[@]} -eq 0 ]]; then + echo "no .patch files in $PATCHES_DIR, nothing to apply" + exit 0 +fi + +cd "$SRC_DIR" + +for patch in "${patches[@]}"; do + echo "==> applying $patch" + git apply --verbose "$patch" +done + +echo "all buun-llama-cpp patches applied successfully" diff --git a/backend/cpp/buun-llama-cpp/package.sh b/backend/cpp/buun-llama-cpp/package.sh new file mode 100755 index 000000000..046149dfd --- /dev/null +++ b/backend/cpp/buun-llama-cpp/package.sh @@ -0,0 +1,57 @@ +#!/bin/bash + +# Script to copy the appropriate libraries based on architecture +# This script is used in the final stage of the Dockerfile + +set -e + +CURDIR=$(dirname "$(realpath $0)") +REPO_ROOT="${CURDIR}/../../.." + +# Create lib directory +mkdir -p $CURDIR/package/lib + +cp -avrf $CURDIR/buun-llama-cpp-* $CURDIR/package/ +cp -rfv $CURDIR/run.sh $CURDIR/package/ + +# Detect architecture and copy appropriate libraries +if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then + # x86_64 architecture + echo "Detected x86_64 architecture, copying x86_64 libraries..." + cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so + cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6 + cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1 + cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6 + cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6 + cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1 + cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2 + cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1 + cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0 +elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then + # ARM64 architecture + echo "Detected ARM64 architecture, copying ARM64 libraries..." + cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so + cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6 + cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1 + cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6 + cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6 + cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1 + cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2 + cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1 + cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0 +else + echo "Error: Could not detect architecture" + exit 1 +fi + +# Package GPU libraries based on BUILD_TYPE +GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh" +if [ -f "$GPU_LIB_SCRIPT" ]; then + echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..." + source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib" + package_gpu_libs +fi + +echo "Packaging completed successfully" +ls -liah $CURDIR/package/ +ls -liah $CURDIR/package/lib/ diff --git a/backend/cpp/buun-llama-cpp/patch-grpc-server.sh b/backend/cpp/buun-llama-cpp/patch-grpc-server.sh new file mode 100755 index 000000000..de3e45f93 --- /dev/null +++ b/backend/cpp/buun-llama-cpp/patch-grpc-server.sh @@ -0,0 +1,141 @@ +#!/bin/bash +# Patch the shared backend/cpp/llama-cpp/grpc-server.cpp *copy* used by the +# buun-llama-cpp build to account for three gaps between upstream and the fork: +# +# 1. Augment the kv_cache_types[] allow-list so `LoadModel` accepts the +# fork-specific `turbo2` / `turbo3` / `turbo4` cache types plus the buun +# additions `turbo2_tcq` / `turbo3_tcq`. +# +# 2. Wire up buun-exclusive speculative-decoding option handlers +# (tree_budget / draft_topk) alongside the existing spec_* handlers. +# These reference struct fields (common_params.speculative.tree_budget +# and .draft_topk) that only exist in buun's common/common.h — adding +# them to the shared backend/cpp/llama-cpp/grpc-server.cpp would break +# the stock llama-cpp build, so we inject them only into the buun copy. +# +# 3. Replace `get_media_marker()` (added upstream in ggml-org/llama.cpp#21962, +# server-side random per-instance marker) with the legacy "<__media__>" +# literal. The fork branched before that PR, so server-common.cpp has no +# get_media_marker symbol. The fork's mtmd_default_marker() still returns +# "<__media__>", and Go-side tooling falls back to that sentinel when the +# backend does not expose media_marker, so substituting the literal keeps +# behavior identical on the buun path. +# +# We patch the *copy* sitting in buun-llama-cpp--build/, never the +# original under backend/cpp/llama-cpp/, so the stock llama-cpp build keeps +# compiling against vanilla upstream. +# +# Idempotent: skips each insertion if its marker is already present (so re-runs +# of the same build dir don't double-insert). + +set -euo pipefail + +if [[ $# -ne 1 ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +SRC=$1 + +if [[ ! -f "$SRC" ]]; then + echo "grpc-server.cpp not found at $SRC" >&2 + exit 2 +fi + +if grep -q 'GGML_TYPE_TURBO2_TCQ' "$SRC"; then + echo "==> $SRC already has buun cache types, skipping KV allow-list patch" +else + echo "==> patching $SRC to allow turbo2/turbo3/turbo4/turbo2_tcq/turbo3_tcq KV-cache types" + + # Insert the five TURBO entries right after the first ` GGML_TYPE_Q5_1,` + # line (the kv_cache_types[] allow-list). Using awk because the builder + # image does not ship python3, and GNU sed's multi-line `a\` quoting is + # awkward. + awk ' + /^ GGML_TYPE_Q5_1,$/ && !done { + print + print " // buun-llama-cpp fork extras — added by patch-grpc-server.sh" + print " GGML_TYPE_TURBO2_0," + print " GGML_TYPE_TURBO3_0," + print " GGML_TYPE_TURBO4_0," + print " GGML_TYPE_TURBO2_TCQ," + print " GGML_TYPE_TURBO3_TCQ," + done = 1 + next + } + { print } + END { + if (!done) { + print "patch-grpc-server.sh: anchor ` GGML_TYPE_Q5_1,` not found" > "/dev/stderr" + exit 1 + } + } + ' "$SRC" > "$SRC.tmp" + mv "$SRC.tmp" "$SRC" + + echo "==> KV allow-list patch OK" +fi + +if grep -q 'optname, "tree_budget"' "$SRC"; then + echo "==> $SRC already has DFlash option handlers, skipping" +else + echo "==> patching $SRC to add tree_budget / draft_topk option handlers" + + # Insert two new `else if` handlers between the inner close-brace of the + # `spec_p_split` block and the next `} else if (…spec_ngram_size_n…)` line. + # Upstream writes each `} else if` as a single physical line, so we don't + # emit an outer `}` ourselves — the existing next line provides both the + # close of our `draft_topk` block and the open of `spec_ngram_size_n`. + # Anchor on the exact 3-line body of spec_p_split so we can't drift. + awk ' + prev2 == " } else if (!strcmp(optname, \"spec_p_split\")) {" && + prev1 ~ /^ +if \(optval != NULL\) \{$/ && + $0 ~ /^ +try \{ params\.speculative\.p_split = std::stof\(optval_str\); \} catch \(\.\.\.\) \{\}$/ && + !done { + print # print the try-line itself + getline inner_close # read " }" closing the inner if + print inner_close # print it — this closes spec_p_split body + print " // buun-llama-cpp DFlash options — added by patch-grpc-server.sh" + print " } else if (!strcmp(optname, \"tree_budget\")) {" + print " if (optval != NULL) {" + print " try { params.speculative.tree_budget = std::stoi(optval_str); } catch (...) {}" + print " }" + print " } else if (!strcmp(optname, \"draft_topk\")) {" + print " if (optval != NULL) {" + print " try { params.speculative.draft_topk = std::stoi(optval_str); } catch (...) {}" + print " }" + # The next source line (`} else if (…spec_ngram_size_n…) {`) closes + # our draft_topk block and continues the chain naturally; fall back + # into the main loop to emit it and everything after. + done = 1 + prev2 = prev1 + prev1 = inner_close + next + } + { print; prev2 = prev1; prev1 = $0 } + END { + if (!done) { + print "patch-grpc-server.sh: spec_p_split anchor not found" > "/dev/stderr" + exit 1 + } + } + ' "$SRC" > "$SRC.tmp" + mv "$SRC.tmp" "$SRC" + + echo "==> DFlash option-handler patch OK" +fi + +if grep -q 'get_media_marker()' "$SRC"; then + echo "==> patching $SRC to replace get_media_marker() with legacy \"<__media__>\" literal" + # Only one call site today (ModelMetadata), but replace all occurrences to + # stay robust if upstream adds more. Use a temp file to avoid relying on + # sed -i portability (the builder image uses GNU sed, but keeping this + # consistent with the awk block above). + sed 's/get_media_marker()/"<__media__>"/g' "$SRC" > "$SRC.tmp" + mv "$SRC.tmp" "$SRC" + echo "==> get_media_marker() substitution OK" +else + echo "==> $SRC has no get_media_marker() call, skipping media-marker patch" +fi + +echo "==> all patches applied" diff --git a/backend/cpp/buun-llama-cpp/run.sh b/backend/cpp/buun-llama-cpp/run.sh new file mode 100755 index 000000000..7eb48f389 --- /dev/null +++ b/backend/cpp/buun-llama-cpp/run.sh @@ -0,0 +1,65 @@ +#!/bin/bash +set -ex + +# Get the absolute current dir where the script is located +CURDIR=$(dirname "$(realpath $0)") + +cd / + +echo "CPU info:" +grep -e "model\sname" /proc/cpuinfo | head -1 +grep -e "flags" /proc/cpuinfo | head -1 + +BINARY=buun-llama-cpp-fallback + +if grep -q -e "\savx\s" /proc/cpuinfo ; then + echo "CPU: AVX found OK" + if [ -e $CURDIR/buun-llama-cpp-avx ]; then + BINARY=buun-llama-cpp-avx + fi +fi + +if grep -q -e "\savx2\s" /proc/cpuinfo ; then + echo "CPU: AVX2 found OK" + if [ -e $CURDIR/buun-llama-cpp-avx2 ]; then + BINARY=buun-llama-cpp-avx2 + fi +fi + +# Check avx 512 +if grep -q -e "\savx512f\s" /proc/cpuinfo ; then + echo "CPU: AVX512F found OK" + if [ -e $CURDIR/buun-llama-cpp-avx512 ]; then + BINARY=buun-llama-cpp-avx512 + fi +fi + +if [ -n "$LLAMACPP_GRPC_SERVERS" ]; then + if [ -e $CURDIR/buun-llama-cpp-grpc ]; then + BINARY=buun-llama-cpp-grpc + fi +fi + +# Extend ld library path with the dir where this script is located/lib +if [ "$(uname)" == "Darwin" ]; then + export DYLD_LIBRARY_PATH=$CURDIR/lib:$DYLD_LIBRARY_PATH +else + export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH + # Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files) + if [ -d "$CURDIR/lib/rocblas/library" ]; then + export ROCBLAS_TENSILE_LIBPATH=$CURDIR/lib/rocblas/library + fi +fi + +# If there is a lib/ld.so, use it +if [ -f $CURDIR/lib/ld.so ]; then + echo "Using lib/ld.so" + echo "Using binary: $BINARY" + exec $CURDIR/lib/ld.so $CURDIR/$BINARY "$@" +fi + +echo "Using binary: $BINARY" +exec $CURDIR/$BINARY "$@" + +# We should never reach this point, however just in case we do, run fallback +exec $CURDIR/buun-llama-cpp-fallback "$@" diff --git a/core/config/meta/constants.go b/core/config/meta/constants.go index b0633c22d..cefa6e41f 100644 --- a/core/config/meta/constants.go +++ b/core/config/meta/constants.go @@ -37,6 +37,14 @@ var CacheTypeOptions = []FieldOption{ {Value: "q4_1", Label: "Q4_1"}, {Value: "q5_0", Label: "Q5_0"}, {Value: "q5_1", Label: "Q5_1"}, + // TurboQuant KV-cache types — accepted by the turboquant and + // buun-llama-cpp fork backends; stock llama-cpp will reject them at load. + {Value: "turbo2", Label: "Turbo2 (TurboQuant)"}, + {Value: "turbo3", Label: "Turbo3 (TurboQuant)"}, + {Value: "turbo4", Label: "Turbo4 (TurboQuant)"}, + // Trellis-Coded Quantization variants — buun-llama-cpp only. + {Value: "turbo2_tcq", Label: "Turbo2 TCQ (buun-llama-cpp)"}, + {Value: "turbo3_tcq", Label: "Turbo3 TCQ (buun-llama-cpp)"}, } var DiffusersPipelineOptions = []FieldOption{ diff --git a/core/gallery/importers/llama-cpp.go b/core/gallery/importers/llama-cpp.go index 92196ed8a..05bc9ecf3 100644 --- a/core/gallery/importers/llama-cpp.go +++ b/core/gallery/importers/llama-cpp.go @@ -34,6 +34,7 @@ func (i *LlamaCPPImporter) AdditionalBackends() []KnownBackendEntry { return []KnownBackendEntry{ {Name: "ik-llama-cpp", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with ik-quants"}, {Name: "turboquant", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with TurboQuant optimizations"}, + {Name: "buun-llama-cpp", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with DFlash speculative decoding and TurboQuant/TCQ KV-cache quantization"}, } } @@ -127,7 +128,7 @@ func (i *LlamaCPPImporter) Import(details Details) (gallery.ModelConfig, error) backend := "llama-cpp" if b, ok := preferencesMap["backend"].(string); ok { switch b { - case "ik-llama-cpp", "turboquant": + case "ik-llama-cpp", "turboquant", "buun-llama-cpp": backend = b } } diff --git a/docs/content/features/text-generation.md b/docs/content/features/text-generation.md index 05efcc8fe..2f099e5cc 100644 --- a/docs/content/features/text-generation.md +++ b/docs/content/features/text-generation.md @@ -631,6 +631,83 @@ The `cache_type_k` / `cache_type_v` fields map to llama.cpp's `-ctk` / `-ctv` fl - [Tracked branch: `feature/turboquant-kv-cache`](https://github.com/TheTom/llama-cpp-turboquant/tree/feature/turboquant-kv-cache) +### buun-llama-cpp (DFlash speculative decoding + TurboQuant/TCQ KV-cache) + +[buun-llama-cpp](https://github.com/spiritbuun/buun-llama-cpp) is a fork-of-a-fork: spiritbuun forked `TheTom/llama-cpp-turboquant` (the `turboquant` backend above) and added two independent features on top: + +1. **DFlash** — a block-diffusion speculative decoding scheme that uses a dedicated drafter model (new `DFlashDraftModel` GGUF architecture). On a target/drafter pair it emits a block of tokens per speculation step and can be combined with tree-structured verification ("DDTree") for multi-branch draft expansion. +2. **TCQ (Trellis-Coded Quantization)** — two additional KV-cache types (`turbo2_tcq`, `turbo3_tcq`) on top of the TurboQuant `turbo2` / `turbo3` / `turbo4` already shipped by the parent fork, delivering 10–44% KL reduction over scalar quantization at 2–3 bits per value. + +Like `turboquant`, this backend shares LocalAI's stock `llama-cpp` gRPC server sources — so any GGUF model that runs on `llama-cpp` also runs on `buun-llama-cpp`. Pick it over `turboquant` specifically when you want DFlash speculative decoding or the newer TCQ KV-cache variants. + +#### Features + +- Drop-in GGUF compatibility with upstream `llama.cpp`. +- DFlash block-diffusion speculative decoding (CUDA/Metal; no CPU fallback). +- TurboQuant KV-cache types (`turbo2`, `turbo3`, `turbo4`) inherited from the parent `turboquant` fork, plus buun-exclusive `turbo2_tcq` and `turbo3_tcq` variants. +- Same feature surface as `llama-cpp`: text generation, embeddings, tool calls, multimodal via mmproj. +- Available on CPU (AVX/AVX2/AVX512/fallback), NVIDIA CUDA 12/13, AMD ROCm/HIP, Intel SYCL f32/f16, Vulkan, and NVIDIA L4T — but note that DFlash and `turbo*` KV types have no CPU fallback and error at model-load on CPU-only builds. + +#### Setup + +`buun-llama-cpp` ships as a separate container image in the LocalAI backend gallery. Install it like any other backend: + +```bash +local-ai backends install buun-llama-cpp +``` + +Or pick a specific flavor for your hardware (example tags: `cpu-buun-llama-cpp`, `cuda12-buun-llama-cpp`, `cuda13-buun-llama-cpp`, `rocm-buun-llama-cpp`, `intel-sycl-f16-buun-llama-cpp`, `vulkan-buun-llama-cpp`). + +#### YAML configuration — TCQ KV-cache + +To run a model with TurboQuant/TCQ quantized KV-cache, set the backend and pick a `turbo*` cache type: + +```yaml +name: my-model +backend: buun-llama-cpp +parameters: + model: file.gguf +# Accepted values for the two fork-aware backends include the stock llama.cpp +# types (f16, f32, q8_0, q4_0, q4_1, q5_0, q5_1), the TurboQuant types +# (turbo2, turbo3, turbo4), and the buun-only TCQ variants (turbo2_tcq, +# turbo3_tcq). turbo3 / turbo4 / turbo*_tcq auto-enable flash_attention. +cache_type_k: turbo3 +cache_type_v: turbo3_tcq +context_size: 8192 +``` + +#### YAML configuration — DFlash speculative decoding + +DFlash requires a **dedicated drafter model** in the new `DFlashDraftModel` GGUF architecture. At time of writing the only known public target/drafter pair is [`z-lab/Qwen3.5-27B`](https://huggingface.co/z-lab/Qwen3.5-27B) + [`z-lab/Qwen3.5-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-27B-DFlash). + +```yaml +name: qwen3-dflash +backend: buun-llama-cpp +parameters: + # Target model (quantized as usual) + model: Qwen3.5-27B-Q4_K_M.gguf +# Drafter model produced by buun's convert_hf_to_gguf.py from the +# DFlashDraftModel checkpoint. Resolved relative to the models path. +draft_model: Qwen3.5-27B-DFlash.gguf +options: + # Switches the speculative pipeline from the default draft-model mode to + # DFlash (block-diffusion). Required to activate the DFlash code path. + - spec_type:dflash + # Optional tuning: + # - tree_budget:0 # 0 = flat DFlash; >0 = DDTree verification budget + # - draft_topk:1 # drafter top-K per position (1 = argmax) + # - spec_n_max:16 # cap on draft tokens per speculation step +``` + +Under the hood LocalAI wires `draft_model` through to the grpc-server's `params.speculative.mparams_dft.path`, and `spec_type:dflash` is forwarded through the options passthrough to buun's `common_speculative_type_from_name("dflash")`. The `tree_budget` and `draft_topk` options are buun-exclusive; they reference struct fields that only exist in buun's fork, so they're surfaced on this backend only (passing them to stock `llama-cpp` is a no-op). + +#### Reference + +- [spiritbuun/buun-llama-cpp](https://github.com/spiritbuun/buun-llama-cpp) +- [TCQ paper / dataset](https://huggingface.co/datasets/spiritbuun/turboquant-tcq-kv-cache) — *"Closing the Gap: Trellis-Coded Quantization for KV Cache at 2-3 Bits"* +- DFlash target/drafter pair: [`z-lab/Qwen3.5-27B`](https://huggingface.co/z-lab/Qwen3.5-27B) + [`z-lab/Qwen3.5-27B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-27B-DFlash) + + ### vLLM [vLLM](https://github.com/vllm-project/vllm) is a fast and easy-to-use library for LLM inference. diff --git a/docs/content/reference/compatibility-table.md b/docs/content/reference/compatibility-table.md index e952deba7..99f654839 100644 --- a/docs/content/reference/compatibility-table.md +++ b/docs/content/reference/compatibility-table.md @@ -20,6 +20,7 @@ LocalAI will attempt to automatically load models which are not explicitly confi |---------|-------------|------------|------------|-----------|-------------| | [llama.cpp](https://github.com/ggerganov/llama.cpp) | LLM inference in C/C++. Supports LLaMA, Mamba, RWKV, Falcon, Starcoder, GPT-2, [and many others](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#description) | GPT, Functions | yes | yes | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T | | [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) | Hard fork of llama.cpp optimized for CPU/hybrid CPU+GPU with IQK quants, custom quant mixes, and MLA for DeepSeek | GPT | yes | yes | CPU (AVX2+) | +| [buun-llama-cpp](https://github.com/spiritbuun/buun-llama-cpp) | llama.cpp fork with DFlash block-diffusion speculative decoding and TurboQuant/TCQ KV-cache quantization (2–3 bits per value). Accelerated paths are CUDA/Metal only. | GPT, Functions | yes | yes | CUDA, Metal (CPU fallback for non-turbo/non-DFlash only) | | [vLLM](https://github.com/vllm-project/vllm) | Fast LLM serving with PagedAttention | GPT, Functions | no | yes | CPU, CUDA 12, ROCm, Intel | | [vLLM Omni](https://github.com/vllm-project/vllm) | Unified multimodal generation (text, image, video, audio) | Multimodal GPT, Functions | no | yes | CUDA 12, ROCm | | [transformers](https://github.com/huggingface/transformers) | HuggingFace Transformers framework | GPT, Embeddings, Multimodal | yes | yes* | CPU, CUDA 12/13, ROCm, Intel, Metal |