mudler's LocalAI [bot]andEttore Di Giacinto
ea438cdeaf
The backend could configure four of the engine's knobs (block size, KV block
count, max sequence length, max concurrent sequences) out of a config surface
that is considerably larger. Speculative decoding, prefix caching, the
chunked-prefill token budget, the scheduling policy and the external KV
connector were reachable from vllm.cpp's own HTTP server and from nothing
LocalAI could write in a model config.
Config now goes through `engine_args:`, the same map the vLLM and SGLang
backends take, with keys spelled as vLLM's own CLI flags so a speculative_config
or kv_transfer_config block written for vLLM works verbatim. The legacy
`options:` list keeps working and reads every key too; engine_args wins where
both set one. Unknown keys are logged and ignored rather than fatal: the field
is shared with the other engines, so a config carrying their knobs must not take
the model down.
Two details worth knowing:
`enable_prefix_caching: false` maps to the ABI tri-state force-OFF (2), not 0.
0 means "let the model capability decide" and dense architectures default the
cache on, so collapsing the two would silently enable it against an explicit
false. enable_jump_forward (ABI v10) shares the encoding, deferring to
VT_ENABLE_JUMP_FORWARD instead of to the model.
The importer probes config.json on a vllm-cpp import and writes
speculative_config: {method: mtp} when the checkpoint declares an MTP head, the
safetensors analogue of the llama-cpp importer's GGUF probe. DFlash draft repos
are refused with a warning instead, since a drafter cannot serve alone and the
pairing is not derivable from either repo. The draft path is resolved against
LocalAI's model directory, because the engine only looks in a directory holding
config.json or in the HF cache and never downloads: the repo-id spelling the
vLLM docs teach used to die deep in the load with "draft checkpoint not found".
docs/content/features/text-generation.md gains a vllm.cpp section covering the
engine_args table, all three speculative methods, LMCache and the legacy list.
The backend had no documentation page before.
This replaces a branch that had gone stale behind master and carried its own
route to ABI v10, which #11386 has since landed in minimal form. Rebased onto
that as a single commit rather than replaying the intermediate steps, whose
ABI v9 mirrors no longer make sense against master's pin. The Darwin build
fixes for Apple Clang's gnu-folding-constant diagnostic on C++, Objective-C and
Objective-C++, originally authored by localai-org-maint-bot, are folded in here.
Verified: `make abi-check` agrees at v10; unit specs, core/config and
core/gallery/importers green; and the full e2e passes in 1330s against a CPU
libvllm.so reporting ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, blocking
completion, streaming, chat and tool calls).
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
vllm-cpp backend
LocalAI text-generation backend for vllm.cpp, the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching, safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at inference time.
The backend dlopens the engine's stable C ABI (libvllm, include/vllm.h,
ABI v10) through purego:
Load->vllm_engine_load: accepts a.gguffile or a HF-style model directory (config.json+ safetensors).context_sizemaps tomax_model_len;options: ["block_size:<n>", "num_blocks:<n>", "max_num_seqs:<n>"]size the KV cache and scheduler admission.Predict->vllm_complete(blocking).PredictStream->vllm_complete_stream; concurrent gRPC requests batch continuously in the engine's shared AsyncLLM scheduler.- Chat / tool calling rides the SAME code path as the llama.cpp autoparser:
with
use_tokenizer_template: truethe backend implementsPredictRich/PredictStreamRichover the ABI v3 chat entry points (vllm_chat/vllm_chat_stream). The ENGINE applies the model's chat template (GGUFtokenizer.chat_templateortokenizer_config.json), decides when a tool call engages (tool_choice: autolowers to a LAZY structural-tag decode constraint;required/named force one), parses tool calls with its streaming Hermes-style parser, and the backend maps eachchat.completion.chunkontoChatDelta/ToolCallDeltaprotos. - Without structured messages the plain path applies:
PredictOptions.Grammar-> the ABI'sstructured_grammar(GBNF) for LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex / choice constraints are also exposed by the ABI.
The struct mirrors in govllmcpp.go are hand-written against one ABI version,
and the engine refuses to load against any other. Moving VLLM_CPP_VERSION in
the Makefile therefore means updating abiVersion plus the mirrors (and their
offsets in vllmcpp_test.go) in the same change; make abi-check compares the
pinned header against the bindings and the library build runs it first.
Model config example:
name: qwen3-vllm
backend: vllm-cpp
context_size: 8192
parameters:
model: Qwen3-4B # model dir (safetensors) or .gguf file
options:
- max_num_seqs:16
Apple Silicon: the MLX GEMM provider (ON by default, gated to prefill)
BUILD_TYPE=metal builds vllm.cpp's MLX provider for the dense GEMM
(VLLM_CPP_MLX=on, the default here). It is on because upstream now SHAPE-GATES
it to prefill; it was briefly off in this branch's history, and that was correct
at the time for an ungated provider.
The gate matters more than the flag. MLX's steel GEMM wins prefill but loses
decode, because the provider pays an mx::eval synchronisation plus an output
memcpy on every call and decode makes ~112 calls per token. Measured on an
Apple M4, Qwen3-1.7B-bf16 warm at p=512 g=128:
| configuration | prefill TTFT | warm throughput |
|---|---|---|
| MLX gated to prefill (pin >= 89c46aeb) | 524.5 ms | 24.37 tok/s, 97.6% of MLX-LM |
| MLX ungated (older pins) | 537 ms | 12.7 tok/s |
| MLX off | 602 ms | 23.9 tok/s, 95.9% |
Ratios are against an MLX-LM baseline measured INTERLEAVED with ours over four ABBA blocks (its spread 0.34%, ours 0.12%). An earlier revision of this file claimed 99.1%; that used a two-run MLX-LM baseline containing an outlier and overstated us by about 1.5 points.
VLLM_CPP_VERSION and this flag are coupled. Moving the pin back before
89c46aeb while leaving VLLM_CPP_MLX=on would take the middle row — roughly
half throughput. If you roll the pin back, roll the default back with it.
One caveat: MLX's GEMM is not bit-identical to the native kernel, so an MLX build
produces a different greedy sequence than a non-MLX one. That is a property of the
provider, not of the gate, and it predates this packaging. Full disposition in
vllm.cpp docs/BENCHMARKS.md.
Build knobs:
VLLM_CPP_MLX=offbuilds Metal without the provider: ~124 MB smaller, and 96.4% of MLX-LM instead of 99.1%.MLX_VERSIONpins the wheel (default0.29.4). MLX is consumed as the prebuilt pip wheel because building it from source needsxcrun metal, i.e. a full Xcode the macOS runners do not have.
Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into
package/lib/, and rewrites libvllm.dylib's rpath to @loader_path/lib
(re-signing it, since install_name_tool invalidates the signature). The
metallib must stay beside libmlx.dylib: MLX looks for it there.
Testing: make test runs the unit specs; export VLLM_CPP_MODEL=<model> (and
optionally VLLM_CPP_LIBRARY=<libvllm path>) to enable the e2e specs.