mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 18:09:05 -04:00

Files

Ettore Di Giacinto 8ea2b9d912 fix(vllm-cpp): resolve the DFlash draft path instead of missing the HF cache

The engine resolves speculative_config.model against a directory containing
config.json, or against ~/.cache/huggingface/hub/models--<org>--<repo>/
snapshots/*, and it never downloads. LocalAI keeps models in its own directory,
so the repo-id spelling the vLLM docs teach - "z-lab/Qwen3.6-27B-DFlash" - misses
the HF cache and dies deep inside the load with "draft checkpoint not found",
which reads like a broken checkpoint rather than a model nobody fetched.

Resolve it before the load call: the reference as given, then its last path
segment under LocalAI's models dir (what LocalAI's own downloader produces),
then the whole reference under the models dir. When none resolve, fail there
naming both what was asked for and every location tried, so the message says
what to do about it.

mtp and ngram pass through untouched - neither has a separate draft checkpoint.
A speculative_config that does not parse also passes through, because the engine
owns config validation and produces the better error.

Docs also gain the two limits that were missing and are easy to lose an
afternoon to: speculation is Qwen3.5/3.6-only at this engine pin regardless of
format, and mtp/dflash need a safetensors target. The latter is a gap in the
engine's GGUF loader rather than a property of GGUF - the format carries MTP
weights fine, llama.cpp reads them as nextn.* tensors plus a
<arch>.nextn_predict_layers key - so the docs say that rather than implying GGUF
cannot express it.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

2026-07-29 09:06:37 +00:00

.gitignore

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

backend.go

fix(vllm-cpp): resolve the DFlash draft path instead of missing the HF cache

2026-07-29 09:06:37 +00:00

chat_test.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

chat.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

e2e_test.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

govllmcpp.go

feat(vllm-cpp): wire the full engine config surface through engine_args

2026-07-29 09:06:36 +00:00

main.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

Makefile

chore(vllm-cpp): pin vllm.cpp to the merged ABI v9 tip

2026-07-29 09:06:37 +00:00

options.go

fix(vllm-cpp): resolve the DFlash draft path instead of missing the HF cache

2026-07-29 09:06:37 +00:00

package.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

README.md

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

run.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

test.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

vllmcpp_test.go

fix(vllm-cpp): resolve the DFlash draft path instead of missing the HF cache

2026-07-29 09:06:37 +00:00

README.md

vllm-cpp backend

LocalAI text-generation backend for vllm.cpp, the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching, safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at inference time.

The backend dlopens the engine's stable C ABI (libvllm, include/vllm.h, ABI v2) through purego:

Load -> vllm_engine_load: accepts a .gguf file or a HF-style model directory (config.json + safetensors). context_size maps to max_model_len; options: ["block_size:<n>", "num_blocks:<n>", "max_num_seqs:<n>"] size the KV cache and scheduler admission.
Predict -> vllm_complete (blocking).
PredictStream -> vllm_complete_stream; concurrent gRPC requests batch continuously in the engine's shared AsyncLLM scheduler.
Chat / tool calling rides the SAME code path as the llama.cpp autoparser: with use_tokenizer_template: true the backend implements PredictRich/PredictStreamRich over the ABI v3 chat entry points (vllm_chat / vllm_chat_stream). The ENGINE applies the model's chat template (GGUF tokenizer.chat_template or tokenizer_config.json), decides when a tool call engages (tool_choice: auto lowers to a LAZY structural-tag decode constraint; required/named force one), parses tool calls with its streaming Hermes-style parser, and the backend maps each chat.completion.chunk onto ChatDelta/ToolCallDelta protos.
Without structured messages the plain path applies: PredictOptions.Grammar -> the ABI's structured_grammar (GBNF) for LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex / choice constraints are also exposed by the ABI.

Model config example:

name: qwen3-vllm
backend: vllm-cpp
context_size: 8192
parameters:
  model: Qwen3-4B   # model dir (safetensors) or .gguf file
options:
- max_num_seqs:16

Testing: make test runs the unit specs; export VLLM_CPP_MODEL=<model> (and optionally VLLM_CPP_LIBRARY=<libvllm path>) to enable the e2e specs.