mirror of https://github.com/mudler/LocalAI.git synced 2026-07-31 02:18:50 -04:00

Files

Ettore Di Giacinto 40e7c76ec2 chore(vllm-cpp): bump the vllm.cpp pin to main; GGUF speculative decoding is real now

The pin sat at f384edcd while vllm.cpp main moved a long way. The ABI is
unchanged at v10 and both POD structs are field-identical to the pinned commit
(verified by diffing vllm_model_params and vllm_sampling_params across the
range), so the Go mirror needs no edit and this is a clean bump.

What it picks up matters for this backend:

- MTP speculative decoding from a GGUF target, gated end to end on GPU.
- DFlash speculative decoding with a GGUF draft AND a GGUF target.
- NVFP4 GGUF: dequant, plus a native fp4 compute path for dense and
  full-attention projections. On the 27B that closed a cross-container
  divergence entirely (the GGUF and safetensors builds of the same
  quantization run now emit identical tokens) and halved peak RSS.
- A real engine fix: the GDN speculative state gather/scatter was mis-striding
  the widened conv row, so speculation silently corrupted the target's own
  recurrent state on CPU.

Docs corrected accordingly. The section previously told users that mtp and
dflash are rejected on a .gguf target and called it a gap in the engine's GGUF
loader. That is no longer true, and leaving it would send people to safetensors
for no reason. A head-less GGUF is still refused, and the text now says so with
the actual cause.

The real-library ABI handshake was re-run against a libvllm.so built at the
exact pinned commit rather than a stale one: 43 specs pass, reported ABI 10.
make lint clean.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

2026-07-29 16:21:04 +00:00

.gitignore

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

backend.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

chat_test.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

chat.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

e2e_test.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

govllmcpp.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

main.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

Makefile

chore(vllm-cpp): bump the vllm.cpp pin to main; GGUF speculative decoding is real now

2026-07-29 16:21:04 +00:00

options.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

package.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

README.md

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

run.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

test.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

vllmcpp_test.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

README.md

vllm-cpp backend

LocalAI text-generation backend for vllm.cpp, the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching, safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at inference time.

The backend dlopens the engine's stable C ABI (libvllm, include/vllm.h, ABI v2) through purego:

Load -> vllm_engine_load: accepts a .gguf file or a HF-style model directory (config.json + safetensors). context_size maps to max_model_len; options: ["block_size:<n>", "num_blocks:<n>", "max_num_seqs:<n>"] size the KV cache and scheduler admission.
Predict -> vllm_complete (blocking).
PredictStream -> vllm_complete_stream; concurrent gRPC requests batch continuously in the engine's shared AsyncLLM scheduler.
Chat / tool calling rides the SAME code path as the llama.cpp autoparser: with use_tokenizer_template: true the backend implements PredictRich/PredictStreamRich over the ABI v3 chat entry points (vllm_chat / vllm_chat_stream). The ENGINE applies the model's chat template (GGUF tokenizer.chat_template or tokenizer_config.json), decides when a tool call engages (tool_choice: auto lowers to a LAZY structural-tag decode constraint; required/named force one), parses tool calls with its streaming Hermes-style parser, and the backend maps each chat.completion.chunk onto ChatDelta/ToolCallDelta protos.
Without structured messages the plain path applies: PredictOptions.Grammar -> the ABI's structured_grammar (GBNF) for LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex / choice constraints are also exposed by the ABI.

Model config example:

name: qwen3-vllm
backend: vllm-cpp
context_size: 8192
parameters:
  model: Qwen3-4B   # model dir (safetensors) or .gguf file
options:
- max_num_seqs:16

Testing: make test runs the unit specs; export VLLM_CPP_MODEL=<model> (and optionally VLLM_CPP_LIBRARY=<libvllm path>) to enable the e2e specs.