Adds eight curated vllm-cpp entries to the model gallery. Until now the
backend had gallery coverage only for MiniMax-H3 video, so serving text on
it meant hand-writing engine_args.
The flagship tier is what vllm.cpp gates its correctness and speed claims
on: Qwen3.6-27B and Qwen3.6-35B-A3B in NVFP4, each with a speculative
sibling (MTP on both, DFlash on the 27B). Qwen3-Coder-30B-A3B covers
agentic tool use, and Qwen3-4B / Qwen3-0.6B in bf16 are the entries that
run where NVFP4 cannot, CPU included.
Three details are load-bearing rather than incidental:
- The 27B entries pin revision 890bdef7. That repository was later
re-quantized in place from NVFP4 to FP8 W8A8 under the same name, so an
unpinned entry resolves to different weights and reports nothing.
- Qwen3-Coder names tool_parser: qwen3_coder explicitly. Its dialect is
byte-identical on the wire to step3p5's, so chat-template sniffing
cannot separate them and auto-detection picks wrong.
- enable_prefix_caching is deliberately left unset everywhere. It defaults
on for dense models and off for the GDN hybrids, and that per-model
default is the right answer.
num_blocks is sized per model from its real KV footprint rather than
copied between entries, which ranges from 20 KiB/token on the 35B to
144 KiB/token on the 4B.
Docs: adds features/vllm-cpp.md covering installation, the model table,
the pinning rationale and how to choose between the speculative variants,
and cross-links it from the existing engine_args reference. It also
records that the CUDA images are built for Blackwell only, which is
narrower than vllm.cpp's own ten-architecture release and makes an
otherwise cryptic "no kernel image is available" failure legible.
Verified: gallery suite green; all eight decode and validate as a
ModelConfig. qwen3-0.6b-vllm-cpp confirmed end to end on a real cluster,
chat plus engine-parsed tool_calls. The NVFP4 entries are not yet
runtime-verified: no available node has kernels for them.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
