mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 18:38:23 -04:00
5945b0eb11
The vllm-cpp backend could configure four of the engine's knobs - block size,
KV block count, max sequence length and max concurrent sequences - out of a
config surface that is considerably larger. Speculative decoding, prefix
caching, the chunked-prefill token budget, the scheduling policy and the
external KV connector were all reachable from vllm.cpp's own HTTP server and
from nothing LocalAI could write in a model config.
Part of that gap was the C ABI itself, which carried strictly less than
EngineParams does; that is fixed upstream in vllm.cpp ABI v9 (this bumps the pin
to it). The rest was here: the backend parsed a flat `options:` list with five
recognised keys and had no way to express a nested JSON document at all.
Configuration now goes through `engine_args:`, the same map the vLLM and SGLang
backends already take, with keys spelled as vLLM's own CLI flags - so a
`speculative_config` or `kv_transfer_config` block written for vLLM works
verbatim:
engine_args:
max_num_batched_tokens: 8192
enable_prefix_caching: true
scheduling_policy: lpm
speculative_config:
method: dflash
model: z-lab/Qwen3.6-27B-DFlash
num_speculative_tokens: 4
kv_transfer_config:
kv_connector: LMCacheConnector
kv_role: kv_both
kv_connector_extra_config: {host: 127.0.0.1, port: 65432}
The `options:` list keeps working, and now reads every key too, so no existing
config breaks; engine_args wins where both set the same key.
Two details worth calling out. `enable_prefix_caching: false` maps to the ABI's
force-OFF state (2), not the 0 that means "let the model capability decide" -
collapsing them would silently turn the cache ON for the dense architectures
that default it on. And cSamplingParams grows the ABI v8 logits-processor tail:
LocalAI installs no processor, but the C side reads those fields off the pointer
we hand it, so a Go struct that stopped short would have had the engine read 16
bytes past our allocation and call whatever sat there.
The importer gets the safetensors counterpart of the llama-cpp MTP hook: a
`vllm-cpp` import of a HuggingFace repo probes config.json and, on a checkpoint
that declares an MTP head, writes speculative_config {method: mtp} into the
generated engine_args. DFlash draft repositories are detected and refused with a
warning rather than configured as standalone models, since a drafter cannot
serve alone. The llama-cpp importer stops applying its own `spec_type:draft-mtp`
options when the chosen backend is vllm-cpp: those are llama.cpp option keys
vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source anyway
because the `mtp.*` draft tensors only exist in the safetensors checkpoint.
docs/content/features/text-generation.md gains a vllm.cpp section - the backend
had no documentation page at all - covering the engine_args table, all three
speculative methods, LMCache, and the legacy options list.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
103 lines
3.5 KiB
Makefile
103 lines
3.5 KiB
Makefile
CMAKE_ARGS?=
|
|
BUILD_TYPE?=
|
|
NATIVE?=false
|
|
|
|
GOCMD?=go
|
|
GO_TAGS?=
|
|
# nproc doesn't exist on the macOS runners: an empty JOBS turns `-j$(JOBS)`
|
|
# into bare `-j` (unlimited clang jobs), which swap-thrashes the 3-core Mac
|
|
# until the 6h GHA timeout. Fall back to sysctl there, then to a constant.
|
|
JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
|
|
|
|
# vllm.cpp version
|
|
VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
|
|
VLLM_CPP_VERSION?=974d9d72c2365d20dc76dbce4a98fddac32525c0
|
|
|
|
# The backend consumes only the stable C ABI (libvllm + include/vllm.h), so the
|
|
# server, examples and tests of the engine are never built here.
|
|
CMAKE_ARGS+=-DVLLM_CPP_SERVER=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_BUILD_EXAMPLES=OFF
|
|
CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
|
|
|
|
# vllm.cpp sets no global -march: SIMD tiers are per-file with runtime dispatch,
|
|
# so ONE portable library serves every CPU of the target arch (unlike the
|
|
# ggml-based backends and their avx/avx2/avx512 variant builds).
|
|
UNAME_M := $(shell uname -m)
|
|
|
|
ifeq ($(BUILD_TYPE),cublas)
|
|
# Blackwell-family targets only: other CUDA arches are build-supported
|
|
# upstream but have no runtime-proven fast path. amd64 gets the consumer
|
|
# (120a) + GB10 (121a) fat binary; arm64 CUDA (l4t-style images, DGX
|
|
# Spark) is GB10 only. Triton-AOT GDN cubins are vendored per-arch, no
|
|
# Python needed to consume them.
|
|
ifeq ($(UNAME_M),x86_64)
|
|
# NO -DVLLM_CPP_TRITON on fat builds: the vendored Triton-AOT cubin
|
|
# trees are per-arch and the engine refuses a multi-arch build unless
|
|
# pinned to one tree (unsound for the other arch). The non-AOT GDN
|
|
# path serves the fat binary; single-arch builds keep the cubins.
|
|
#
|
|
# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks
|
|
# compute_121a (GB10) and its ptxas rejects the sm_120a NVFP4 MMA
|
|
# kernels ("Vector type too large"), so no cuda-12 variant is shipped.
|
|
ifeq ($(CUDA_MAJOR_VERSION),12)
|
|
$(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
|
|
endif
|
|
CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=120a;121a"
|
|
else
|
|
CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
|
|
endif
|
|
else ifeq ($(BUILD_TYPE),vulkan)
|
|
CMAKE_ARGS+=-DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF
|
|
else ifeq ($(BUILD_TYPE),metal)
|
|
CMAKE_ARGS+=-DVLLM_CPP_METAL=ON
|
|
else
|
|
CMAKE_ARGS+=-DVLLM_CPP_CUDA=OFF
|
|
endif
|
|
|
|
UNAME_S := $(shell uname -s)
|
|
ifeq ($(UNAME_S),Darwin)
|
|
LIB=libvllm.dylib
|
|
else
|
|
LIB=libvllm.so
|
|
endif
|
|
|
|
sources/vllm.cpp:
|
|
mkdir -p sources/vllm.cpp
|
|
cd sources/vllm.cpp && \
|
|
git init && \
|
|
git remote add origin $(VLLM_CPP_REPO) && \
|
|
git fetch --depth 1 origin $(VLLM_CPP_VERSION) && \
|
|
git checkout FETCH_HEAD
|
|
|
|
$(LIB): sources/vllm.cpp
|
|
mkdir -p build && \
|
|
cd build && \
|
|
cmake ../sources/vllm.cpp $(CMAKE_ARGS) && \
|
|
cmake --build . --config Release -j$(JOBS) --target vllm_shared
|
|
cp -fL build/$(LIB) ./$(LIB)
|
|
|
|
vllm-cpp: main.go govllmcpp.go backend.go options.go $(LIB)
|
|
CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vllm-cpp ./
|
|
|
|
package: vllm-cpp
|
|
bash package.sh
|
|
|
|
build: package
|
|
|
|
clean: purge
|
|
rm -rf libvllm.so libvllm.dylib package sources/vllm.cpp vllm-cpp
|
|
|
|
purge:
|
|
rm -rf build
|
|
|
|
.NOTPARALLEL:
|
|
|
|
# The unit specs are pure Go (struct mirrors, option mapping, load
|
|
# validation): no libvllm build is needed. The e2e specs skip unless
|
|
# VLLM_CPP_MODEL points at a real model (then build the lib first).
|
|
test:
|
|
@echo "Running vllm-cpp tests..."
|
|
bash test.sh
|
|
@echo "vllm-cpp tests completed."
|
|
|
|
all: vllm-cpp package
|