LocalAI/docs/content/features/vllm-cpp.md

+++ disableToc = false title = "vllm.cpp backend" weight = 16 url = "/features/vllm-cpp/" +++

vllm.cpp is the LocalAI team's C++20 port of vLLM: the same continuous-batching scheduler, paged KV cache and automatic prefix caching, with no Python at inference time. LocalAI serves it through the native vllm-cpp backend, which loads either a HuggingFace safetensors directory or a .gguf file and applies the chat template, tool-call parsing and reasoning split inside the engine.

This page covers installing the backend and the models LocalAI ships ready to run on it. For the full engine_args reference (KV sizing, scheduling policy, speculative decoding, LMCache), see the [vllm.cpp section of the text generation guide]({{% relref "features/text-generation" %}}#vllmcpp).

Installing

local-ai backends install vllm-cpp

Or install it from the Backends page in the web UI. Images are published for CPU, CUDA 13, Vulkan, Metal and Jetson L4T.

Which GPUs the CUDA images cover

The CUDA images are currently built for Blackwell-family architectures only: sm_120a (RTX 50 series, RTX PRO 6000 Blackwell) and sm_121a (GB10 / DGX Spark) on x86-64, and sm_121a alone on arm64. CUDA 13 is required, so there is no CUDA 12 variant.

That is narrower than vllm.cpp itself, which builds ten architectures. On an Ampere, Ada, Hopper, Jetson Orin or Jetson Thor GPU the CUDA backend installs successfully and then fails at the first request with no kernel image is available for execution on the device. Until the build widens, use the Vulkan or CPU image on those cards: vulkan-vllm-cpp builds with CUDA off entirely and gives them a GPU path, without the NVFP4 and Marlin kernels.

Ready-made models

The model gallery carries a curated set of vllm.cpp configurations. Each one arrives with the engine settings already applied, so tool calling, the reasoning split and speculative decoding work without hand-editing YAML.

Gallery entry	Model	Size	Needs
qwen3.6-27b-nvfp4-vllm-cpp	Qwen3.6-27B, NVFP4	25 GB	Blackwell GPU
qwen3.6-27b-nvfp4-mtp-vllm-cpp	the same, with MTP speculative decoding	25 GB	Blackwell GPU
qwen3.6-27b-nvfp4-dflash-vllm-cpp	the same, with DFlash speculative decoding	28 GB	Blackwell GPU
qwen3.6-35b-a3b-nvfp4-vllm-cpp	Qwen3.6-35B-A3B MoE, NVFP4	23 GB	Blackwell GPU
qwen3.6-35b-a3b-nvfp4-mtp-vllm-cpp	the same, with MTP speculative decoding	23 GB	Blackwell GPU
qwen3-coder-30b-a3b-vllm-cpp	Qwen3-Coder-30B-A3B, bf16	57 GB	Blackwell GPU, or CPU
qwen3-4b-vllm-cpp	Qwen3-4B, bf16	8 GB	CPU, Metal, Vulkan, Blackwell GPU
qwen3-0.6b-vllm-cpp	Qwen3-0.6B, bf16	1.4 GB	CPU, Metal, Vulkan, Blackwell GPU

local-ai models install qwen3-0.6b-vllm-cpp

The two small bf16 entries are the ones that run anywhere the backend does, including CPU. The NVFP4 entries need a Blackwell-class NVIDIA GPU on two counts: NVFP4 has no kernel on older architectures, and the CUDA images are built only for Blackwell in any case.

Sizing note: every entry sets num_blocks to give roughly one to four full contexts of KV cache, which is a starting point rather than a tuned value. KV is not free; the 4B entry, for instance, spends 144 KiB per token, so its 1024 blocks are about 4.5 GB on top of the weights. Raise num_blocks for more concurrency, lower it on a small box.

Why the 27B entries pin a revision

The Qwen3.6-27B entries pin their weights to a specific HuggingFace commit rather than tracking the repository's default branch. This is deliberate and worth understanding before you copy one of these configs.

The upstream repository was later re-quantized in place, under the same name, from NVFP4 to FP8 W8A8. A config that names the repository without a revision therefore resolves to entirely different weights, with different numerics and different performance, and nothing about the load reports that anything changed. Pinning is what makes the entry reproducible:

artifacts:
  - name: model
    target: model
    source:
      type: huggingface
      repo: unsloth/Qwen3.6-27B-NVFP4
      revision: 890bdef7a42feba6d83b6e17a03315c694112f2a

The same reasoning applies to any quantized community repository you depend on.

Choosing between the speculative variants

Speculative decoding trades memory for decode throughput. All three Qwen3.6-27B entries serve the same weights and produce the same quality; they differ only in how tokens are proposed.

Entry	Method	Extra weights	Extra memory
qwen3.6-27b-nvfp4-vllm-cpp	none	none	none
qwen3.6-27b-nvfp4-mtp-vllm-cpp	MTP, depth 1	none, the draft head ships inside the checkpoint	about 3.6 GB
qwen3.6-27b-nvfp4-dflash-vllm-cpp	DFlash, 16-token blocks	a separate 3.5 GB drafter	drafter plus draft cache

MTP drafts one token per step from a head that already lives in the target checkpoint's own mtp.* tensors, so it costs no extra download. DFlash drafts a whole 16-token block in one non-autoregressive pass from a separate drafter, which is the larger win at the cost of a second checkpoint on disk.

Start with the plain entry if you are short on memory, and with the DFlash entry if you are not.

Tool calling

Every entry above sets use_tokenizer_template: true and disables LocalAI's Go-side grammar path, so tool calls are detected and parsed by the engine's own streaming parsers and arrive as real tool_calls on the OpenAI response.

The parser is normally auto-detected from the chat template, but one case cannot be: Qwen3-Coder's tool dialect is byte-identical on the wire to another family's, so template sniffing would pick the wrong parser. The qwen3-coder-30b-a3b-vllm-cpp entry therefore names it explicitly, and any Qwen3-Coder config you write yourself should do the same:

engine_args:
  tool_parser: qwen3_coder

Overriding config.json keys (hf_overrides)

engine_args.hf_overrides is a JSON object of top-level config.json keys that the backend merges over the model directory's own config.json before the engine loads it, like vLLM's --hf-overrides. The main use is to opt a published checkpoint into an engine adapter that its config does not name. For example, the Tev1 repositories declare Qwen3_5ForConditionalGeneration, and vllm.cpp serves them as decision models only when the architecture is Tev1Model:

engine_args:
  hf_overrides:
    architectures: ["Tev1Model"]

The downloaded model files do not change. At load the backend creates a private temporary directory. It writes the merged config.json there and adds a symlink for each other entry of the model directory (weights, tokenizer files, a tokenizer/ subdirectory). Then it gives that directory to the engine. When the model unloads, the backend removes the directory.

Rules:

• The merge is top-level only. An override key replaces the whole value of that key, including a nested object such as text_config.
• The model must be a directory that contains a config.json. A .gguf file or a directory without config.json fails the load.
• A value that is not a JSON object (an array, a scalar, or JSON that does not parse) fails the load. The backend does not ignore it, because loading the unchanged config would serve a different architecture than the one you configured.
• An empty object ({}) does nothing.

Named entity recognition (GLiNER2.5)

The vllm-cpp backend serves GLiNER2.5, a zero-shot NER and structured-extraction model. Point the backend at the safetensors directory and the backend exposes the TokenClassify gRPC method, which LocalAI maps to its standard NER API surface.

Labels are supplied at inference time, not baked into the model config. Set them in engine_args:

engine_args:
  ner_labels: "person,organization,location,date,time,money,quantity"
  ner_threshold: 0.5
  ner_max_width: 12

ner_labels is a comma-separated list. When omitted, the backend falls back to a built-in default set (person, organization, location, date, time, money, quantity). ner_threshold is the sigmoid cutoff (default 0.5); ner_max_width is the maximum span length in tokens (default 12).

The model runs the DeBERTa v2 encoder with disentangled attention on the host forward, which is the required contract for pooling models in vllm.cpp. A device-resident forward is tracked as a performance optimization, not a correctness gap.

Decisions API

The vllm-cpp backend serves the kev-compatible SystemOne endpoints (the Decisions API): typed choice, noul and score questions over a state text, answered by a non-generative decision model in one pass. A decision model declares known_usecases: [decisions]. See [Decisions API]({{% relref "features/decisions" %}}) for the request shape, the models you can install and the access rules.

Endpoint	Method	Description
/v1/systemone	POST	Answer all questions in one pass
/v1/systemone/permute	POST	Re-run one choice question under n_perm option orders
/v1/systemone/separate	POST	Answer each question in its own pass (N passes)

The GLiNER2.5 zero-shot NER model (token_classify) also serves /v1/systemone, through the NER path, and it is the model to use for /v1/systemone/permute and /v1/systemone/separate, which decision models refuse with a 400. It derives its NER labels from the question definitions, so no ner_labels configuration is needed.

Beyond text generation

The vllm-cpp backend also serves MiniMax-H3, which generates video and audio jointly. See [Video generation]({{% relref "features/video-generation" %}}#minimax-h3-vllmcpp).