LocalAI/backend/go/vllm-cpp

vllm-cpp backend

LocalAI text-generation backend for vllm.cpp, the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching, safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at inference time.

The backend dlopens the engine's stable C ABI (libvllm, include/vllm.h, ABI v2) through purego:

• Load -> vllm_engine_load: accepts a .gguf file or a HF-style model directory (config.json + safetensors). context_size maps to max_model_len; options: ["block_size:<n>", "num_blocks:<n>", "max_num_seqs:<n>"] size the KV cache and scheduler admission.
• Predict -> vllm_complete (blocking).
• PredictStream -> vllm_complete_stream; concurrent gRPC requests batch continuously in the engine's shared AsyncLLM scheduler.
• Chat / tool calling rides the SAME code path as the llama.cpp autoparser: with use_tokenizer_template: true the backend implements PredictRich/PredictStreamRich over the ABI v3 chat entry points (vllm_chat / vllm_chat_stream). The ENGINE applies the model's chat template (GGUF tokenizer.chat_template or tokenizer_config.json), decides when a tool call engages (tool_choice: auto lowers to a LAZY structural-tag decode constraint; required/named force one), parses tool calls with its streaming Hermes-style parser, and the backend maps each chat.completion.chunk onto ChatDelta/ToolCallDelta protos.
• Without structured messages the plain path applies: PredictOptions.Grammar -> the ABI's structured_grammar (GBNF) for LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex / choice constraints are also exposed by the ABI.

Model config example:

name: qwen3-vllm
backend: vllm-cpp
context_size: 8192
parameters:
  model: Qwen3-4B   # model dir (safetensors) or .gguf file
options:
- max_num_seqs:16

Testing: make test runs the unit specs; export VLLM_CPP_MODEL=<model> (and optionally VLLM_CPP_LIBRARY=<libvllm path>) to enable the e2e specs.