mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00

Files

Ettore Di Giacinto df5e2a6d27 feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

vllm.cpp landed ABI v10 on main while this branch was open. The backend's
runtime handshake refuses any library whose reported ABI differs from the
mirrors', so the pin and the Go PODs move together or not at all.

v10 appends one int32, `enable_jump_forward`, AFTER the v9 fields. Nothing else
in the config surface changed: the SGLang reconciliation that carried it
explicitly dropped its own duplicate scheduler_policy int in favour of the v9
`scheduling_policy` string this branch already wires, and a diff of EngineParams
and the server flags across the window turns up jump forward and nothing else.

So the exposure is one new knob, `engine_args.enable_jump_forward` - SGLang's
grammar-speed subset, which emits grammar-forced tokens without spending a model
step and therefore only affects constrained requests.

It is the SECOND tri-state on this struct, and it repeats the trap the first one
had: 0 is not "off", it is "defer" (to the environment here, to the model
capability for prefix caching), so an explicit `false` has to reach the engine as
2. The bool->tri-state helper and the log renderer are now shared rather than
duplicated per field, and named for the encoding instead of for prefix caching,
since the next tri-state will want them too. The docs say this outright, because
"omitting the key" and "setting it false" being different is not guessable.

The Go mirror grows the field plus an EXPLICIT trailing pad: the struct is
8-aligned and now ends on a lone int32, so it is 88 bytes rather than 84. The
offset assertions cover it, and the real-library handshake spec (VLLM_CPP_LIBRARY
against a CPU libvllm.so built at the new pin) confirms the version agrees:
43 specs pass, ABI reported 10.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

2026-07-29 09:06:37 +00:00

.gitignore

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

backend.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

chat_test.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

chat.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

e2e_test.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

govllmcpp.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

main.go

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

Makefile

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

options.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

package.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

README.md

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

run.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

test.sh

feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100 )

2026-07-26 23:04:48 +02:00

vllmcpp_test.go

feat(vllm-cpp): move to vllm.cpp ABI v10 and expose jump-forward decoding

2026-07-29 09:06:37 +00:00

README.md

vllm-cpp backend

LocalAI text-generation backend for vllm.cpp, the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching, safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at inference time.

The backend dlopens the engine's stable C ABI (libvllm, include/vllm.h, ABI v2) through purego:

Load -> vllm_engine_load: accepts a .gguf file or a HF-style model directory (config.json + safetensors). context_size maps to max_model_len; options: ["block_size:<n>", "num_blocks:<n>", "max_num_seqs:<n>"] size the KV cache and scheduler admission.
Predict -> vllm_complete (blocking).
PredictStream -> vllm_complete_stream; concurrent gRPC requests batch continuously in the engine's shared AsyncLLM scheduler.
Chat / tool calling rides the SAME code path as the llama.cpp autoparser: with use_tokenizer_template: true the backend implements PredictRich/PredictStreamRich over the ABI v3 chat entry points (vllm_chat / vllm_chat_stream). The ENGINE applies the model's chat template (GGUF tokenizer.chat_template or tokenizer_config.json), decides when a tool call engages (tool_choice: auto lowers to a LAZY structural-tag decode constraint; required/named force one), parses tool calls with its streaming Hermes-style parser, and the backend maps each chat.completion.chunk onto ChatDelta/ToolCallDelta protos.
Without structured messages the plain path applies: PredictOptions.Grammar -> the ABI's structured_grammar (GBNF) for LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex / choice constraints are also exposed by the ABI.

Model config example:

name: qwen3-vllm
backend: vllm-cpp
context_size: 8192
parameters:
  model: Qwen3-4B   # model dir (safetensors) or .gguf file
options:
- max_num_seqs:16

Testing: make test runs the unit specs; export VLLM_CPP_MODEL=<model> (and optionally VLLM_CPP_LIBRARY=<libvllm path>) to enable the e2e specs.