LocalAI/backend/go/vllm-cpp

vllm-cpp backend

LocalAI backend for vllm.cpp, the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching, safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at inference time.

It serves two things: text generation, and MiniMax-H3 joint video+audio generation.

The backend dlopens the engine's stable C ABI (libvllm, include/vllm.h, ABI v30) through purego:

• Load -> vllm_engine_load: accepts a .gguf file or a HF-style model directory (config.json + safetensors). context_size maps to max_model_len; options: ["block_size:<n>", "num_blocks:<n>", "max_num_seqs:<n>"] size the KV cache and scheduler admission.
• Predict -> vllm_complete (blocking).
• PredictStream -> vllm_complete_stream; concurrent gRPC requests batch continuously in the engine's shared AsyncLLM scheduler.
• Chat / tool calling rides the SAME code path as the llama.cpp autoparser: with use_tokenizer_template: true the backend implements PredictRich/PredictStreamRich over the ABI v3 chat entry points (vllm_chat / vllm_chat_stream). The ENGINE applies the model's chat template (GGUF tokenizer.chat_template or tokenizer_config.json), decides when a tool call engages (tool_choice: auto lowers to a LAZY structural-tag decode constraint; required/named force one), parses tool calls with its streaming Hermes-style parser, and the backend maps each chat.completion.chunk onto ChatDelta/ToolCallDelta protos.
• Without structured messages the plain path applies: PredictOptions.Grammar -> the ABI's structured_grammar (GBNF) for LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex / choice constraints are also exposed by the ABI.

patches/ carries fixes the pinned engine SHA does not have yet, applied to the clone the same way longcat-video patches its upstream. git apply is unguarded on purpose: a patch that stops applying must fail the clone loudly rather than leave a pin silently missing a fix it is documented to carry. Each patch header says what retires it.

The struct mirrors in govllmcpp.go are hand-written against one ABI version, and the engine refuses to load against any other. Moving VLLM_CPP_VERSION in the Makefile therefore means updating abiVersion plus the mirrors (and their offsets in vllmcpp_test.go) in the same change; make abi-check compares the pinned header against the bindings and the library build runs it first.

The Makefile builds libvllm with -DVLLM_CPP_WITH_DIARIZATION=OFF. vllm.cpp turns that option ON by default since ABI v30, and ON fetches a pinned parakeet.cpp (with its own ggml) at configure time. This backend does not bind the diarization entry points, so OFF adds no dependency and changes nothing it serves: those calls exist in libvllm but refuse with "not compiled in". If a future change binds them, pin the parakeet.cpp source with -DVLLM_CPP_PARAKEET_CPP_DIR and make package.sh bundle what it links.

Model config example:

name: qwen3-vllm
backend: vllm-cpp
context_size: 8192
parameters:
  model: Qwen3-4B   # model dir (safetensors) or .gguf file
options:
- max_num_seqs:16

hf_overrides

engine_args.hf_overrides (vLLM parity) is a JSON object of top-level config.json keys merged over the model directory's config.json. The C ABI has no override input and the engine reads config.json from the directory it is given, so Load builds an overlay (hfoverrides.go): a temp dir with the merged config.json plus a symlink to every other entry of the model dir, and passes the overlay as model_path. validModelPath and the DFlash draft resolution still run against the real model dir. Free (and a failed load, or the next Load) removes the overlay. A value that is not an object, a .gguf model, or a dir without config.json fails the load instead of being ignored, because loading the unmodified config would serve another architecture.

engine_args:
  hf_overrides:
    architectures: ["Tev1Model"]   # opt a Qwen3.5-declared Tev1 snapshot into the Tev1 adapter

MiniMax-H3 video+audio generation

GenerateVideo -> vllm_video_generate (ABI v12). H3 renders picture and sound together, so the output MP4 carries a real AAC track.

The video engine is a SECOND handle (vllm_video_engine), not a mode of the text one, because H3 is a checkpoint SET rather than a model directory: the DiT, the text encoder and two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse each other's checkpoints. Load takes the video branch when the model config carries any of the video options below; parameters.model is the DiT and everything else is named in options:.

name: minimax-h3-fl2va-q4
backend: vllm-cpp
cuda: true
known_usecases: [video]
parameters:
  model: minimax-h3/MiniMax-H3-FL2VA-Q4_K_M.gguf
options:
- video_encoder:minimax-h3/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf
- video_tokenizer:minimax-h3/tokenizer.json
- video_vae:minimax-h3/video_vae.safetensors
- video_vae_config:minimax-h3/video_vae_config.json
- audio_vae:minimax-h3/audio_vae.safetensors
- audio_vae_config:minimax-h3/audio_vae_config.json
- video_partition:fl2va
- video_device:cuda
- video_dequant_bf16:true
- video_width:1344
- video_height:768
- video_num_frames:124

Three things are worth knowing before touching this path.

The partition is declared, not detected, and a mismatch does not fail cleanly. The FL2VA DiT serves t2va and fl2va; ref2va is a different checkpoint. The community GGUF/NVFP4 quantisations strip the release metadata and the two DiTs are byte-structurally identical, so the engine refuses every generate until video_partition says which one it has. Handing reference conditioning to an FL2VA DiT renders for hours and returns a coloured lattice over the frame, so checkPartitionConditioning refuses that combination here, before the engine is called.

ffmpeg comes from the host. libvllm writes the frames and the WAV and COMPOSES the mux argv, then spawns nothing — that process boundary is upstream's decision. muxVideo takes the composed argv, substitutes argv[0] with the resolved binary and execs it; the backend image is FROM scratch and carries no ffmpeg, the same arrangement vibevoice-cpp uses for transcoding. ffmpeg also converts a start_image/end_image upload into the binary PPM at the exact output canvas the engine requires, since libvllm vendors neither an image codec nor a resampler.

It is slow. Roughly 176 s per denoise step at 1344x768 on a 20-SM device, so the 50-step default is hours. Nothing here imposes a deadline.

Geometry mirrors the engine so the two agree: the canvas is truncated onto a 32-pixel grid, the frame count sits on the 17n+5 grid, and an unspecified canvas with a keyframe is derived from that image's aspect on a 768-pixel short edge (MiniMaxH3ResolveShape, minimax_h3_planner.cpp).

Apple Silicon: the MLX GEMM provider (ON by default, gated to prefill)

BUILD_TYPE=metal builds vllm.cpp's MLX provider for the dense GEMM (VLLM_CPP_MLX=on, the default here). It is on because upstream now SHAPE-GATES it to prefill; it was briefly off in this branch's history, and that was correct at the time for an ungated provider.

The gate matters more than the flag. MLX's steel GEMM wins prefill but loses decode, because the provider pays an mx::eval synchronisation plus an output memcpy on every call and decode makes ~112 calls per token. Measured on an Apple M4, Qwen3-1.7B-bf16 warm at p=512 g=128:

configuration	prefill TTFT	warm throughput
MLX gated to prefill (pin >= 89c46aeb)	524.5 ms	24.37 tok/s, 97.6% of MLX-LM
MLX ungated (older pins)	537 ms	12.7 tok/s
MLX off	602 ms	23.9 tok/s, 95.9%

Ratios are against an MLX-LM baseline measured INTERLEAVED with ours over four ABBA blocks (its spread 0.34%, ours 0.12%). An earlier revision of this file claimed 99.1%; that used a two-run MLX-LM baseline containing an outlier and overstated us by about 1.5 points.

VLLM_CPP_VERSION and this flag are coupled. Moving the pin back before 89c46aeb while leaving VLLM_CPP_MLX=on would take the middle row — roughly half throughput. If you roll the pin back, roll the default back with it.

One caveat: MLX's GEMM is not bit-identical to the native kernel, so an MLX build produces a different greedy sequence than a non-MLX one. That is a property of the provider, not of the gate, and it predates this packaging. Full disposition in vllm.cpp docs/BENCHMARKS.md.

Build knobs:

• VLLM_CPP_MLX=off builds Metal without the provider: ~124 MB smaller, and 96.4% of MLX-LM instead of 99.1%.
• MLX_VERSION pins the wheel (default 0.29.4). MLX is consumed as the prebuilt pip wheel because building it from source needs xcrun metal, i.e. a full Xcode the macOS runners do not have.

Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into package/lib/, and rewrites libvllm.dylib's rpath to @loader_path/lib (re-signing it, since install_name_tool invalidates the signature). The metallib must stay beside libmlx.dylib: MLX looks for it there.

Testing: make test runs the unit specs; export VLLM_CPP_MODEL=<model> (and optionally VLLM_CPP_LIBRARY=<libvllm path>) to enable the e2e specs.