feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100)

* feat(backend): add vllm-cpp text-generation backend (vllm.cpp)

Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20
port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors
+ GGUF loading, no Python at inference) - as a Go gRPC backend over its stable
C ABI (ABI v2) via purego.

Backend (backend/go/vllm-cpp):
- Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir
  (anything else is refused, satisfying the greedy-probe rule); context_size
  maps to max_model_len, options block_size/num_blocks/max_num_seqs size the
  KV cache and scheduler admission.
- Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream
  with the per-delta C callback bridged into the gRPC stream. The backend
  embeds base.Base (not SingleThread): concurrent requests batch continuously
  in the engine's shared AsyncLLM scheduler.
- PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving
  grammar-constrained tool calling at parity with llama-cpp; the ABI also
  exposes JSON-schema/regex/choice constraints.
- Hand-mirrored POD structs with layout locked by unit tests
  (unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate.
- One portable library per platform (vllm.cpp uses per-file SIMD tiers with
  runtime dispatch), so no avx/avx2/avx512 variant builds.

Wiring:
- backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13
  amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark -
  the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal.
- backend/index.yaml meta + 12 image entries (latest/development x cpu,
  cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the
  VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit
  specs (pure Go, no engine build).
- Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors)
  advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp
  without tokenizer templating (the C ABI takes the FINAL prompt; templating
  and tool parsing stay LocalAI-side). No auto-detect importer.
- Docs: backends list, top-level README maintained-engines table,
  compatibility table.

Verified: 20/20 Ginkgo specs against the real pinned engine and
Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy
determinism, stop words, GBNF-constrained generation, and 4 concurrent
streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary.
Upstream ABI v2 + production structured-output wiring landed as
mudler/vllm.cpp@86013f3.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3)

The backend now implements AIModelRich (PredictRich / PredictStreamRich) over
vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME
code path as the llama.cpp autoparser: the ENGINE renders the model's chat
template, decides when a tool call engages, and parses it - LocalAI receives
pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp.

- With use_tokenizer_template + structured Messages, PredictOptions lowers to
  ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling,
  stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice
  auto lowers engine-side to a LAZY structural-tag decode constraint - free
  text until the model emits the tool trigger, then the call is
  grammar-constrained; required/named force a call. Tool output is parsed by
  the engine's streaming Hermes-style parser; each chat.completion.chunk maps
  onto ChatDeltas (content / reasoning_content / tool_calls) which the host
  already prefers over Go-side tag extraction. Without structured messages the
  plain path (LocalAI templating + optional GBNF grammar) applies unchanged.
- The engine resolves the chat template from the GGUF tokenizer.chat_template
  metadata (or tokenizer_config.json); templates beyond its minja subset -
  e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a
  Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with
  a stderr witness, so structural-tag engagement keeps working.
- Importers now emit the same config shape as llama-cpp for vllm-cpp
  (use_tokenizer_template: true, no-grammar autoparser flow); only the
  llama-cpp-specific use_jinja option and the vllm-python parser options are
  dropped.
- Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution).

Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full
suite green - blocking chat, streaming deltas concatenating byte-equal to the
blocking answer, a REQUIRED tool call returning schema-valid arguments JSON,
and an AUTO run where the engine itself engages get_weather and streams parsed
tool deltas; plus unit specs for the request lowering, chunk->ChatDelta
mapping, and the C struct mirrors (ABI gate now v3).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning

Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects
(every pure-text parser in the pinned vLLM registry, each ported 1:1 with its
upstream tests), 7 reasoning parsers, google/minja as the template renderer
(the full Qwen3.5 template now renders engine-side), per-family structural
tags (tool_choice required/named compiles the model's NATIVE syntax where
expressible), and template auto-detection for both parser axes.

Backend changes:
- cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields,
  layout-locked by the offset tests; ABI gate now v5).
- New model options tool_parser:<name> / reasoning_parser:<name> pass through
  to the engine; unset means template auto-detection (18-row tool marker
  table; [THINK]->mistral, <think>->think_auto for reasoning); "none"
  disables the reasoning split; unknown names fail the first chat call.
- Chat chunks parse the `reasoning` field (the pin renamed
  reasoning_content), flowing into ChatDelta.ReasoningContent which the host
  already prefers.

Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the
real chat template renders (no more fallback), reasoning auto-detection picks
think_auto so markerless answers stay pure content (the live run caught the
deepseek_r1 content-swallow upstream and drove the think_auto fix), required
tool_choice returns schema-valid arguments, auto tool_choice engages
engine-side and streams parsed deltas, and blocking/streaming stay
byte-identical. Turn latency also dropped (proper template EOS behavior).

Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5,
minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry,
think_auto).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(vllm-cpp): bump the engine pin to the ENG-wave close-out

mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families
(qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss)
text-reimplemented from their wire formats and held to the upstream test
suites - 39 registered dialects; the pinned vLLM registry is now covered
except the three Rust/Harmony-backed families, descoped by decision. kimi_k2
also gains a full native structural-tag builder; four new template
auto-detection rows land with test-pinned ordering.

Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): add the vllm-cpp-development gallery meta

The gallery grew the twelve latest/development image entries but was missing
the separate vllm-cpp-development meta (own capabilities map targeting the
-development image names), which every backend ships so the development
gallery resolves per-platform. Validated: all capability targets in both
metas resolve to existing entries, and every image URI's tag suffix matches
a backend-matrix build.

Also full-stack verified in this change's context (single-node local-ai from
this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF):
/v1/chat/completions non-stream (clean content + usage), streaming (SSE
deltas), tool_choice auto engaging get_weather engine-side with schema-valid
arguments and finish_reason=tool_calls, and streamed tool-call deltas in the
standard name-first cadence.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton

Two distinct failures took down all five vllm-cpp backend builds on the PR:

1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the
   engine build with -Werror=maybe-uninitialized in InputBatch::condense - a
   false positive through a staging std::optional's raw storage. Fixed
   upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly;
   verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two
   diagnostics exactly, fixed compiles clean) with the engine's behavior
   suites green. Pin bumped to that sha.

2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT
   cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a
   multi-arch (120a;121a) fat build unless pinned to one tree, which would be
   unsound for the other arch. Triton is now enabled only on the single-arch
   arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the
   engine's non-AOT GDN path.

Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only

The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8)
image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13),
killing the amd64 cuda-12 build at nvcc. Gate the architecture list on
CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer
Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t
(cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so
the amd64 cuda-12 image never served it - no capability change.

Verified by Makefile dry-run variable dumps for all three combinations
(cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain

Third CI round, third layer: with the arch list already narrowed to 120a,
the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA
kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4
path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports
Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels
would be a crippled build of an engine whose whole GPU story is fp4, so the
variant is dropped instead:

- backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64,
  cpu, vulkan, metal remain).
- gallery: cuda12 image entries removed; the nvidia capability now resolves
  to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so
  older-driver hosts fall back to the CPU image instead of an unrunnable one.
- backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails
  fast with a clear message; cuda-13 keeps the 120a;121a fat binary and
  arm64/l4t keeps 121a with the Triton cubins.

Verified: Makefile branch dumps for all four combinations (cuda12 loud
error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter
tests green, gallery capability targets all resolve.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine

chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool"
replies and Message.ReasoningContent on assistant history, so a second
turn after tool execution reached the engine's chat template without the
fields that bind a tool result to the call it answers. Forward all three
(present-only, matching the OpenAI wire shape) and pin vllm.cpp to
6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id,
name and reasoning and the minja adapter exposes them to the template
context.

Adds the round-trip request-lowering spec (user -> assistant tool_call ->
tool reply -> lowered request) and re-ran the gated e2e suite against the
new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming
parity, required-tool and auto-tool cases all green.

Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix

The darwin-metal CI job was the first build to compile the engine's arm
CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> /
<sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on
Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite
re-run green against the new pin with a real Qwen3.5 GGUF.

Assisted-by: Claude Code:claude-fable-5 [Bash] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent

The macOS runners have no nproc, so JOBS evaluated empty and
`cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a
3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log
shows "nproc: Command not found" and 7+ concurrent clang processes being
reaped at the cutoff). Use the same portable fallback chain as the other
darwin backends: nproc, then sysctl hw.ncpu, then 4.

Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

.github/backend-matrix.yml

@@ -1961,6 +1961,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2026,6 +2039,19 @@ include:
     backend: "moss-tts-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -3002,6 +3028,19 @@ include:
     dockerfile: "./backend/Dockerfile.privacy-filter"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # Vulkan: base-grpc-vulkan-amd64 carries the SDK. arm64 vulkan is a one-line
   # add once amd64 is proven in CI.
   - build-type: 'vulkan'
@@ -4650,6 +4689,35 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # vllm-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-vllm-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "vllm-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # omnivoice-cpp
   - build-type: ''
     cuda-major-version: ""
@@ -5925,6 +5993,10 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-magpie-tts-cpp"
     build-type: "metal"
     lang: "go"
+  - backend: "vllm-cpp"
+    tag-suffix: "-metal-darwin-arm64-vllm-cpp"
+    build-type: "metal"
+    lang: "go"
   - backend: "omnivoice-cpp"
     tag-suffix: "-metal-darwin-arm64-omnivoice-cpp"
     build-type: "metal"

.github/workflows/bump_deps.yaml

@@ -50,6 +50,10 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
+          - repository: "mudler/vllm.cpp"
+            variable: "VLLM_CPP_VERSION"
+            branch: "main"
+            file: "backend/go/vllm-cpp/Makefile"
           - repository: "localai-org/moss-transcribe.cpp"
             variable: "MOSS_VERSION"
             branch: "master"

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -625,6 +625,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/go/locate-anything-cpp test
 	$(MAKE) -C backend/go/depth-anything-cpp test
 	$(MAKE) -C backend/go/supertonic test
+	$(MAKE) -C backend/go/vllm-cpp test
 
 ##
 ## End-to-end gRPC tests that exercise a built backend container image.
@@ -1258,6 +1259,7 @@ BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
 BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
 BACKEND_MOSS_TTS_CPP = moss-tts-cpp|golang|.|false|true
 BACKEND_MAGPIE_TTS_CPP = magpie-tts-cpp|golang|.|false|true
+BACKEND_VLLM_CPP = vllm-cpp|golang|.|false|true
 BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
 BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
 BACKEND_LOCALVQE = localvqe|golang|.|false|true
@@ -1386,6 +1388,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TTS_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_MAGPIE_TTS_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
@@ -1405,7 +1408,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
 
 ########################################################
 ### Mock Backend for E2E Tests

README.md

@@ -231,6 +231,7 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 
 | Backend | What it does |
 |---------|-------------|
+| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
 | [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
 | [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |

backend/go/vllm-cpp/.gitignore

@@ -0,0 +1,6 @@
+sources/
+build/
+package/
+vllm-cpp
+libvllm.so
+libvllm.dylib

backend/go/vllm-cpp/Makefile

@@ -0,0 +1,102 @@
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+
+GOCMD?=go
+GO_TAGS?=
+# nproc doesn't exist on the macOS runners: an empty JOBS turns `-j$(JOBS)`
+# into bare `-j` (unlimited clang jobs), which swap-thrashes the 3-core Mac
+# until the 6h GHA timeout. Fall back to sysctl there, then to a constant.
+JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+# vllm.cpp version
+VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
+VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845
+
+# The backend consumes only the stable C ABI (libvllm + include/vllm.h), so the
+# server, examples and tests of the engine are never built here.
+CMAKE_ARGS+=-DVLLM_CPP_SERVER=OFF -DVLLM_CPP_BUILD_TESTS=OFF -DVLLM_CPP_BUILD_EXAMPLES=OFF
+CMAKE_ARGS+=-DCMAKE_BUILD_TYPE=Release
+
+# vllm.cpp sets no global -march: SIMD tiers are per-file with runtime dispatch,
+# so ONE portable library serves every CPU of the target arch (unlike the
+# ggml-based backends and their avx/avx2/avx512 variant builds).
+UNAME_M := $(shell uname -m)
+
+ifeq ($(BUILD_TYPE),cublas)
+	# Blackwell-family targets only: other CUDA arches are build-supported
+	# upstream but have no runtime-proven fast path. amd64 gets the consumer
+	# (120a) + GB10 (121a) fat binary; arm64 CUDA (l4t-style images, DGX
+	# Spark) is GB10 only. Triton-AOT GDN cubins are vendored per-arch, no
+	# Python needed to consume them.
+	ifeq ($(UNAME_M),x86_64)
+		# NO -DVLLM_CPP_TRITON on fat builds: the vendored Triton-AOT cubin
+		# trees are per-arch and the engine refuses a multi-arch build unless
+		# pinned to one tree (unsound for the other arch). The non-AOT GDN
+		# path serves the fat binary; single-arch builds keep the cubins.
+		#
+		# CUDA builds REQUIRE the CUDA 13 toolchain: 12.x nvcc lacks
+		# compute_121a (GB10) and its ptxas rejects the sm_120a NVFP4 MMA
+		# kernels ("Vector type too large"), so no cuda-12 variant is shipped.
+		ifeq ($(CUDA_MAJOR_VERSION),12)
+      $(error vllm.cpp needs the CUDA 13 toolchain: CUDA 12.x cannot compile the Blackwell fp4 kernels)
+		endif
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON "-DVLLM_CPP_CUDA_ARCHITECTURES=120a;121a"
+	else
+		CMAKE_ARGS+=-DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_TRITON=ON
+	endif
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF
+else ifeq ($(BUILD_TYPE),metal)
+	CMAKE_ARGS+=-DVLLM_CPP_METAL=ON
+else
+	CMAKE_ARGS+=-DVLLM_CPP_CUDA=OFF
+endif
+
+UNAME_S := $(shell uname -s)
+ifeq ($(UNAME_S),Darwin)
+	LIB=libvllm.dylib
+else
+	LIB=libvllm.so
+endif
+
+sources/vllm.cpp:
+	mkdir -p sources/vllm.cpp
+	cd sources/vllm.cpp && \
+	git init && \
+	git remote add origin $(VLLM_CPP_REPO) && \
+	git fetch --depth 1 origin $(VLLM_CPP_VERSION) && \
+	git checkout FETCH_HEAD
+
+$(LIB): sources/vllm.cpp
+	mkdir -p build && \
+	cd build && \
+	cmake ../sources/vllm.cpp $(CMAKE_ARGS) && \
+	cmake --build . --config Release -j$(JOBS) --target vllm_shared
+	cp -fL build/$(LIB) ./$(LIB)
+
+vllm-cpp: main.go govllmcpp.go backend.go options.go $(LIB)
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vllm-cpp ./
+
+package: vllm-cpp
+	bash package.sh
+
+build: package
+
+clean: purge
+	rm -rf libvllm.so libvllm.dylib package sources/vllm.cpp vllm-cpp
+
+purge:
+	rm -rf build
+
+.NOTPARALLEL:
+
+# The unit specs are pure Go (struct mirrors, option mapping, load
+# validation): no libvllm build is needed. The e2e specs skip unless
+# VLLM_CPP_MODEL points at a real model (then build the lib first).
+test:
+	@echo "Running vllm-cpp tests..."
+	bash test.sh
+	@echo "vllm-cpp tests completed."
+
+all: vllm-cpp package

backend/go/vllm-cpp/README.md

@@ -0,0 +1,45 @@
+# vllm-cpp backend
+
+LocalAI text-generation backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
+the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching,
+safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
+inference time.
+
+The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
+ABI v2) through purego:
+
+- `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
+  directory (`config.json` + safetensors). `context_size` maps to
+  `max_model_len`; `options: ["block_size:<n>", "num_blocks:<n>",
+  "max_num_seqs:<n>"]` size the KV cache and scheduler admission.
+- `Predict` -> `vllm_complete` (blocking).
+- `PredictStream` -> `vllm_complete_stream`; concurrent gRPC requests batch
+  continuously in the engine's shared AsyncLLM scheduler.
+- Chat / tool calling rides the SAME code path as the llama.cpp autoparser:
+  with `use_tokenizer_template: true` the backend implements
+  `PredictRich`/`PredictStreamRich` over the ABI v3 chat entry points
+  (`vllm_chat` / `vllm_chat_stream`). The ENGINE applies the model's chat
+  template (GGUF `tokenizer.chat_template` or `tokenizer_config.json`),
+  decides when a tool call engages (`tool_choice: auto` lowers to a LAZY
+  structural-tag decode constraint; `required`/named force one), parses tool
+  calls with its streaming Hermes-style parser, and the backend maps each
+  `chat.completion.chunk` onto `ChatDelta`/`ToolCallDelta` protos.
+- Without structured messages the plain path applies:
+  `PredictOptions.Grammar` -> the ABI's `structured_grammar` (GBNF) for
+  LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
+  choice constraints are also exposed by the ABI.
+
+Model config example:
+
+```yaml
+name: qwen3-vllm
+backend: vllm-cpp
+context_size: 8192
+parameters:
+  model: Qwen3-4B   # model dir (safetensors) or .gguf file
+options:
+- max_num_seqs:16
+```
+
+Testing: `make test` runs the unit specs; export `VLLM_CPP_MODEL=<model>` (and
+optionally `VLLM_CPP_LIBRARY=<libvllm path>`) to enable the e2e specs.

backend/go/vllm-cpp/backend.go

@@ -0,0 +1,245 @@
+package main
+
+// LocalAI gRPC backend over the vllm.cpp C ABI.
+//
+// Predict maps to the blocking vllm_complete; PredictStream maps to
+// vllm_complete_stream, whose per-delta C callback bridges into the gRPC
+// stream channel. Concurrent calls are intentional: every completion entry
+// point submits into the engine's shared AsyncLLM scheduler, so parallel
+// LocalAI requests batch continuously inside the engine (the reason this
+// backend embeds base.Base and not base.SingleThread).
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+type VllmCpp struct {
+	base.Base
+
+	engine uintptr
+	opts   loadOptions
+}
+
+// Stream registry: the per-request bridge between the C token callback and
+// the gRPC stream channel, keyed by an integer handle round-tripped through
+// the C user_data pointer (never a Go pointer across the ABI). The host gRPC
+// server drains the channel even after a client disconnect, so sends here
+// cannot wedge the engine's delivery loop.
+var (
+	streamsMu   sync.Mutex
+	streams     = map[uintptr]chan string{}
+	streamNext  uintptr
+	tokenCbOnce sync.Once
+	tokenCbPtr  uintptr
+)
+
+// tokenCallback is the single C-shared callback for every stream; it
+// dispatches on the user_data handle. Returning 0 aborts the in-flight
+// request (vllm_token_callback contract).
+func tokenCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	streamsMu.Lock()
+	results := streams[userData]
+	streamsMu.Unlock()
+	if results == nil {
+		return 0 // unknown request: stop generation.
+	}
+	if text := goString(delta); text != "" {
+		results <- text
+	}
+	return 1
+}
+
+func registerStream(results chan string) uintptr {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	streamNext++
+	streams[streamNext] = results
+	return streamNext
+}
+
+func unregisterStream(h uintptr) {
+	streamsMu.Lock()
+	defer streamsMu.Unlock()
+	delete(streams, h)
+}
+
+// validModelPath enforces the greedy-probe rule: when a model config has no
+// explicit backend, the loader probes every backend with the model name, so
+// Load must refuse anything vllm.cpp cannot serve (a GGUF file, or a HF-style
+// directory with config.json + safetensors).
+func validModelPath(model string) error {
+	info, err := os.Stat(model)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: model path %q not found: %w", model, err)
+	}
+	if info.IsDir() {
+		if _, err := os.Stat(filepath.Join(model, "config.json")); err != nil {
+			return fmt.Errorf("vllm-cpp: model dir %q has no config.json", model)
+		}
+		return nil
+	}
+	if strings.EqualFold(filepath.Ext(model), ".gguf") {
+		return nil
+	}
+	return fmt.Errorf("vllm-cpp: model %q is neither a .gguf file nor a config.json model dir", model)
+}
+
+func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
+	model := opts.ModelFile
+	if model == "" {
+		model = opts.ModelPath
+	}
+	if !filepath.IsAbs(model) && opts.ModelPath != "" {
+		model = filepath.Join(opts.ModelPath, model)
+	}
+	if err := validModelPath(model); err != nil {
+		return err
+	}
+
+	v.opts = parseOptions(opts)
+
+	mp := defaultModelParams()
+	if v.opts.blockSize > 0 {
+		mp.BlockSize = v.opts.blockSize
+	}
+	if v.opts.numBlocks > 0 {
+		mp.NumBlocks = v.opts.numBlocks
+	}
+	if opts.ContextSize > 0 {
+		mp.MaxModelLen = opts.ContextSize
+	}
+	if v.opts.maxNumSeqs > 0 {
+		mp.MaxNumSeqs = v.opts.maxNumSeqs
+	}
+
+	modelC := cString(model)
+	mp.ModelPath = uintptr(unsafe.Pointer(&modelC[0])) // #nosec G103 -- borrowed by C for the load call only
+	var toolParserC, reasoningParserC []byte
+	if v.opts.toolParser != "" {
+		toolParserC = cString(v.opts.toolParser)
+		mp.ToolParser = uintptr(unsafe.Pointer(&toolParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+	if v.opts.reasoningParser != "" {
+		reasoningParserC = cString(v.opts.reasoningParser)
+		mp.ReasoningParser = uintptr(unsafe.Pointer(&reasoningParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	}
+
+	xlog.Info("[vllm-cpp] Load", "model", model, "engine", vllmVersion(),
+		"blockSize", mp.BlockSize, "numBlocks", mp.NumBlocks,
+		"maxModelLen", mp.MaxModelLen, "maxNumSeqs", mp.MaxNumSeqs)
+
+	var engine uintptr
+	rc := vllmEngineLoad(unsafe.Pointer(&mp), unsafe.Pointer(&engine)) // #nosec G103 -- POD out-params
+	runtime.KeepAlive(modelC)
+	runtime.KeepAlive(toolParserC)
+	runtime.KeepAlive(reasoningParserC)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: engine load failed: %s", vllmLastError())
+	}
+	v.engine = engine
+	return nil
+}
+
+func (v *VllmCpp) Free() error {
+	if v.engine != 0 {
+		vllmEngineFree(v.engine)
+		v.engine = 0
+	}
+	return nil
+}
+
+// samplingFromPredict lowers PredictOptions into the C sampling POD plus the
+// backing buffers that must stay alive for the duration of the C call.
+func samplingFromPredict(opts *pb.PredictOptions) (sp cSamplingParams, keep []any) {
+	sp = defaultSamplingParams()
+	sp.Temperature = opts.Temperature
+	if opts.TopP > 0 {
+		sp.TopP = opts.TopP
+	}
+	if opts.TopK > 0 {
+		sp.TopK = opts.TopK
+	}
+	if opts.MinP > 0 {
+		sp.MinP = opts.MinP
+	}
+	if opts.Tokens > 0 {
+		sp.MaxTokens = opts.Tokens
+	} else {
+		sp.MaxTokens = 0 // unbounded; the engine caps at max_model_len.
+	}
+	if opts.Seed > 0 {
+		sp.Seed = uint64(opts.Seed)
+		sp.HasSeed = 1
+	}
+	sp.PresencePenalty = opts.PresencePenalty
+	sp.FrequencyPenalty = opts.FrequencyPenalty
+	if opts.Penalty > 0 {
+		sp.RepetitionPenalty = opts.Penalty
+	}
+	if opts.IgnoreEOS {
+		sp.IgnoreEOS = 1
+	}
+	if len(opts.StopPrompts) > 0 {
+		ptrs, backing := cStringArray(opts.StopPrompts)
+		sp.Stop = uintptr(unsafe.Pointer(&ptrs[0])) // #nosec G103 -- borrowed by C for the call only
+		sp.NStop = int32(len(ptrs))
+		keep = append(keep, ptrs, backing)
+	}
+	if opts.Grammar != "" {
+		g := cString(opts.Grammar)
+		sp.StructuredGrammar = uintptr(unsafe.Pointer(&g[0])) // #nosec G103 -- borrowed by C for the call only
+		keep = append(keep, g)
+	}
+	return sp, keep
+}
+
+func (v *VllmCpp) Predict(opts *pb.PredictOptions) (string, error) {
+	if v.engine == 0 {
+		return "", fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	sp, keep := samplingFromPredict(opts)
+	var out cCompletion
+	rc := vllmComplete(v.engine, opts.Prompt, unsafe.Pointer(&sp), unsafe.Pointer(&out)) // #nosec G103 -- POD in/out params
+	runtime.KeepAlive(keep)
+	if rc != vllmOK {
+		return "", fmt.Errorf("vllm-cpp: completion failed: %s", vllmLastError())
+	}
+	text := goString(out.Text)
+	vllmCompletionFree(unsafe.Pointer(&out)) // #nosec G103 -- frees out.Text
+	return text, nil
+}
+
+func (v *VllmCpp) PredictStream(opts *pb.PredictOptions, results chan string) error {
+	if v.engine == 0 {
+		close(results)
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	tokenCbOnce.Do(func() {
+		tokenCbPtr = purego.NewCallback(tokenCallback)
+	})
+
+	sp, keep := samplingFromPredict(opts)
+	handle := registerStream(results)
+
+	go func() {
+		defer close(results)
+		defer unregisterStream(handle)
+		rc := vllmCompleteStream(v.engine, opts.Prompt, unsafe.Pointer(&sp), tokenCbPtr, handle) // #nosec G103 -- POD in-params
+		runtime.KeepAlive(keep)
+		if rc != vllmOK {
+			xlog.Error("[vllm-cpp] stream failed", "error", vllmLastError())
+		}
+	}()
+	return nil
+}

backend/go/vllm-cpp/chat.go

@@ -0,0 +1,289 @@
+package main
+
+// The rich chat path (AIModelRich): rides the ENGINE's serving pipeline via
+// the ABI v3 chat entry points, exactly like the llama-cpp autoparser flow.
+// The engine applies the model's chat template, decides when a tool call
+// engages (tool_choice auto lowers to a LAZY structural-tag decode
+// constraint), parses tool calls with its streaming-stateful Hermes-style
+// parser, and hands back chat.completion.chunk JSON that this file maps 1:1
+// onto pb.Reply ChatDelta / ToolCallDelta.
+
+import (
+	"encoding/json"
+	"fmt"
+	"sync"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+)
+
+// useChatPath reports whether the request should go through the engine-side
+// chat pipeline: the model config asked for backend-side templating and the
+// host handed us structured messages.
+func useChatPath(opts *pb.PredictOptions) bool {
+	return opts.UseTokenizerTemplate && len(opts.Messages) > 0
+}
+
+// chatRequestJSON lowers PredictOptions into one OpenAI chat-completions
+// request object for the ABI (the engine ignores `model`/`stream`).
+func chatRequestJSON(opts *pb.PredictOptions, stream bool) (string, error) {
+	messages := make([]map[string]any, 0, len(opts.Messages))
+	for _, m := range opts.Messages {
+		msg := map[string]any{"role": m.Role, "content": m.Content}
+		if m.ToolCalls != "" {
+			var toolCalls any
+			if err := json.Unmarshal([]byte(m.ToolCalls), &toolCalls); err == nil {
+				msg["tool_calls"] = toolCalls
+			}
+		}
+		// Multi-turn tool identity + prior reasoning: a role="tool" reply
+		// carries the id (and optionally the name) of the assistant call it
+		// answers, and assistant history may carry its reasoning span. The
+		// engine's template context needs all three or a second turn after
+		// tool execution is malformed.
+		if m.ToolCallId != "" {
+			msg["tool_call_id"] = m.ToolCallId
+		}
+		if m.Name != "" {
+			msg["name"] = m.Name
+		}
+		if m.ReasoningContent != "" {
+			msg["reasoning"] = m.ReasoningContent
+		}
+		messages = append(messages, msg)
+	}
+	req := map[string]any{"messages": messages}
+
+	if opts.Tools != "" {
+		var tools any
+		if err := json.Unmarshal([]byte(opts.Tools), &tools); err != nil {
+			return "", fmt.Errorf("vllm-cpp: tools is not valid JSON: %w", err)
+		}
+		req["tools"] = tools
+	}
+	if opts.ToolChoice != "" {
+		var choice any
+		// ToolChoice arrives either as a bare string ("auto"/"required"/"none")
+		// or as the OpenAI named-function JSON object.
+		if err := json.Unmarshal([]byte(opts.ToolChoice), &choice); err == nil {
+			req["tool_choice"] = choice
+		} else {
+			req["tool_choice"] = opts.ToolChoice
+		}
+	}
+
+	req["temperature"] = opts.Temperature
+	if opts.TopP > 0 {
+		req["top_p"] = opts.TopP
+	}
+	if opts.TopK > 0 {
+		req["top_k"] = opts.TopK
+	}
+	if opts.Tokens > 0 {
+		req["max_tokens"] = opts.Tokens
+	}
+	if opts.Seed > 0 {
+		req["seed"] = opts.Seed
+	}
+	if len(opts.StopPrompts) > 0 {
+		req["stop"] = opts.StopPrompts
+	}
+	if opts.PresencePenalty != 0 {
+		req["presence_penalty"] = opts.PresencePenalty
+	}
+	if opts.FrequencyPenalty != 0 {
+		req["frequency_penalty"] = opts.FrequencyPenalty
+	}
+	if stream {
+		// The engine's request parser validates stream_options against the
+		// stream flag at parse time (before the ABI entry point forces it),
+		// so state the intent explicitly.
+		req["stream"] = true
+		req["stream_options"] = map[string]any{"include_usage": true}
+	}
+
+	b, err := json.Marshal(req)
+	if err != nil {
+		return "", err
+	}
+	return string(b), nil
+}
+
+// chatChunk is the subset of an OpenAI chat.completion(.chunk) object the
+// backend consumes.
+type chatChunk struct {
+	Object  string `json:"object"`
+	Choices []struct {
+		Delta        *chatDelta `json:"delta"`   // streaming chunks
+		Message      *chatDelta `json:"message"` // non-stream response
+		FinishReason string     `json:"finish_reason"`
+	} `json:"choices"`
+	Usage *struct {
+		PromptTokens     int32 `json:"prompt_tokens"`
+		CompletionTokens int32 `json:"completion_tokens"`
+	} `json:"usage"`
+}
+
+type chatDelta struct {
+	Content          string `json:"content"`
+	ReasoningContent string `json:"reasoning"`
+	ToolCalls        []struct {
+		Index    int32  `json:"index"`
+		ID       string `json:"id"`
+		Function struct {
+			Name      string `json:"name"`
+			Arguments string `json:"arguments"`
+		} `json:"function"`
+	} `json:"tool_calls"`
+}
+
+// toReply maps one parsed chunk onto a pb.Reply carrying the content bytes
+// plus the structured ChatDelta (the host prefers ChatDeltas when present).
+func (c *chatChunk) toReply() *pb.Reply {
+	reply := &pb.Reply{}
+	if c.Usage != nil {
+		reply.PromptTokens = c.Usage.PromptTokens
+		reply.Tokens = c.Usage.CompletionTokens
+	}
+	if len(c.Choices) == 0 {
+		return reply
+	}
+	d := c.Choices[0].Delta
+	if d == nil {
+		d = c.Choices[0].Message
+	}
+	if d == nil {
+		return reply
+	}
+	delta := &pb.ChatDelta{
+		Content:          d.Content,
+		ReasoningContent: d.ReasoningContent,
+	}
+	for _, tc := range d.ToolCalls {
+		delta.ToolCalls = append(delta.ToolCalls, &pb.ToolCallDelta{
+			Index:     tc.Index,
+			Id:        tc.ID,
+			Name:      tc.Function.Name,
+			Arguments: tc.Function.Arguments,
+		})
+	}
+	reply.Message = []byte(d.Content)
+	if delta.Content != "" || delta.ReasoningContent != "" ||
+		len(delta.ToolCalls) > 0 {
+		reply.ChatDeltas = []*pb.ChatDelta{delta}
+	}
+	return reply
+}
+
+// Chat-stream registry: chunk JSON arrives on the engine's delivery thread
+// through one shared C callback; the integer handle in user_data selects the
+// destination channel (never a Go pointer across the ABI).
+var (
+	chatStreamsMu  sync.Mutex
+	chatStreams    = map[uintptr]chan<- *pb.Reply{}
+	chatStreamNext uintptr
+	chatCbOnce     sync.Once
+	chatCbPtr      uintptr
+)
+
+func chatCallback(delta uintptr, finished uintptr, userData uintptr) uintptr {
+	chatStreamsMu.Lock()
+	results := chatStreams[userData]
+	chatStreamsMu.Unlock()
+	if results == nil {
+		return 0
+	}
+	_ = finished // the terminal call carries an empty delta; nothing to emit.
+	payload := goString(delta)
+	if payload == "" {
+		return 1
+	}
+	var chunk chatChunk
+	if err := json.Unmarshal([]byte(payload), &chunk); err != nil {
+		xlog.Error("[vllm-cpp] unparseable chat chunk", "error", err)
+		return 1
+	}
+	results <- chunk.toReply()
+	return 1
+}
+
+func registerChatStream(results chan<- *pb.Reply) uintptr {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	chatStreamNext++
+	chatStreams[chatStreamNext] = results
+	return chatStreamNext
+}
+
+func unregisterChatStream(h uintptr) {
+	chatStreamsMu.Lock()
+	defer chatStreamsMu.Unlock()
+	delete(chatStreams, h)
+}
+
+// PredictRich implements the non-streaming rich path. Without structured
+// messages it falls back to the plain Predict flow (LocalAI-side templating,
+// optional grammar constraint).
+func (v *VllmCpp) PredictRich(opts *pb.PredictOptions) (*pb.Reply, error) {
+	if !useChatPath(opts) {
+		text, err := v.Predict(opts)
+		if err != nil {
+			return nil, err
+		}
+		return &pb.Reply{Message: []byte(text)}, nil
+	}
+	if v.engine == 0 {
+		return nil, fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, false)
+	if err != nil {
+		return nil, err
+	}
+	var out uintptr
+	rc := vllmChat(v.engine, request, unsafe.Pointer(&out)) // #nosec G103 -- char** out-param
+	if rc != vllmOK {
+		return nil, fmt.Errorf("vllm-cpp: chat failed: %s", vllmLastError())
+	}
+	payload := goString(out)
+	vllmStringFree(out)
+	var response chatChunk
+	if err := json.Unmarshal([]byte(payload), &response); err != nil {
+		return nil, fmt.Errorf("vllm-cpp: unparseable chat response: %w", err)
+	}
+	return response.toReply(), nil
+}
+
+// PredictStreamRich implements the streaming rich path. Contract: send into
+// the channel and return when finished; the host closes the channel.
+func (v *VllmCpp) PredictStreamRich(opts *pb.PredictOptions, results chan<- *pb.Reply) error {
+	if !useChatPath(opts) {
+		// Legacy bridge: run the plain stream and wrap deltas.
+		plain := make(chan string)
+		if err := v.PredictStream(opts, plain); err != nil {
+			return err
+		}
+		for delta := range plain {
+			results <- &pb.Reply{Message: []byte(delta)}
+		}
+		return nil
+	}
+	if v.engine == 0 {
+		return fmt.Errorf("vllm-cpp: model not loaded")
+	}
+	request, err := chatRequestJSON(opts, true)
+	if err != nil {
+		return err
+	}
+	chatCbOnce.Do(func() {
+		chatCbPtr = purego.NewCallback(chatCallback)
+	})
+	handle := registerChatStream(results)
+	defer unregisterChatStream(handle)
+	rc := vllmChatStream(v.engine, request, chatCbPtr, handle)
+	if rc != vllmOK {
+		return fmt.Errorf("vllm-cpp: chat stream failed: %s", vllmLastError())
+	}
+	return nil
+}

backend/go/vllm-cpp/chat_test.go

@@ -0,0 +1,158 @@
+package main
+
+import (
+	"encoding/json"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("useChatPath", func() {
+	It("requires tokenizer templating AND structured messages", func() {
+		Expect(useChatPath(&pb.PredictOptions{})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{UseTokenizerTemplate: true})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeFalse())
+		Expect(useChatPath(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages:             []*pb.Message{{Role: "user", Content: "hi"}},
+		})).To(BeTrue())
+	})
+})
+
+var _ = Describe("chatRequestJSON", func() {
+	It("lowers messages, tools, tool_choice and sampling onto one request", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "system", Content: "be brief"},
+				{Role: "user", Content: "weather in Rome?"},
+			},
+			Tools:       `[{"type":"function","function":{"name":"get_weather","parameters":{"type":"object"}}}]`,
+			ToolChoice:  "required",
+			Temperature: 0.2,
+			TopP:        0.9,
+			Tokens:      64,
+			StopPrompts: []string{"<|im_end|>"},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req["messages"]).To(HaveLen(2))
+		Expect(req["tools"]).To(HaveLen(1))
+		Expect(req["tool_choice"]).To(Equal("required"))
+		Expect(req["max_tokens"]).To(BeNumerically("==", 64))
+		Expect(req["top_p"]).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(req["stop"]).To(ConsistOf("<|im_end|>"))
+		Expect(req).NotTo(HaveKey("stream_options"))
+	})
+
+	It("parses a named-function tool_choice object and asks for stream usage", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			Messages:   []*pb.Message{{Role: "user", Content: "hi"}},
+			ToolChoice: `{"type":"function","function":{"name":"get_weather"}}`,
+		}, true)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req map[string]any
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		choice, ok := req["tool_choice"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		Expect(choice["type"]).To(Equal("function"))
+		Expect(req["stream_options"]).To(HaveKeyWithValue("include_usage", true))
+	})
+
+	It("rejects malformed tools JSON", func() {
+		_, err := chatRequestJSON(&pb.PredictOptions{
+			Messages: []*pb.Message{{Role: "user", Content: "hi"}},
+			Tools:    "{not json",
+		}, false)
+		Expect(err).To(HaveOccurred())
+	})
+})
+
+var _ = Describe("chatChunk.toReply", func() {
+	It("maps a streaming tool-call delta onto ChatDelta/ToolCallDelta", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{
+			"tool_calls":[{"index":0,"id":"call_1","function":{"name":"get_weather","arguments":"{\"city\":"}}]
+		},"finish_reason":null}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].ToolCalls).To(HaveLen(1))
+		tc := reply.ChatDeltas[0].ToolCalls[0]
+		Expect(tc.Name).To(Equal("get_weather"))
+		Expect(tc.Id).To(Equal("call_1"))
+		Expect(tc.Arguments).To(Equal(`{"city":`))
+	})
+
+	It("maps a non-stream response message and usage", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion","choices":[{"message":{
+			"role":"assistant","content":"Sunny."},"finish_reason":"stop"}],
+			"usage":{"prompt_tokens":12,"completion_tokens":3}}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+
+		reply := chunk.toReply()
+		Expect(string(reply.Message)).To(Equal("Sunny."))
+		Expect(reply.ChatDeltas).To(HaveLen(1))
+		Expect(reply.ChatDeltas[0].Content).To(Equal("Sunny."))
+		Expect(reply.PromptTokens).To(BeNumerically("==", 12))
+		Expect(reply.Tokens).To(BeNumerically("==", 3))
+	})
+
+	It("emits no ChatDelta for an empty role-only chunk", func() {
+		var chunk chatChunk
+		payload := `{"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":""}}]}`
+		Expect(json.Unmarshal([]byte(payload), &chunk)).To(Succeed())
+		Expect(chunk.toReply().ChatDeltas).To(BeEmpty())
+	})
+})
+
+var _ = Describe("chatRequestJSON multi-turn tool round trip", func() {
+	It("forwards tool_call_id, name, reasoning and assistant tool_calls", func() {
+		out, err := chatRequestJSON(&pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "What is the weather in Rome?"},
+				{
+					Role:             "assistant",
+					ReasoningContent: "need the weather tool",
+					ToolCalls:        `[{"id":"call_1","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Rome\"}"}}]`,
+				},
+				{Role: "tool", ToolCallId: "call_1", Name: "get_weather", Content: `{"temp": 21}`},
+			},
+		}, false)
+		Expect(err).NotTo(HaveOccurred())
+
+		var req struct {
+			Messages []map[string]any `json:"messages"`
+		}
+		Expect(json.Unmarshal([]byte(out), &req)).To(Succeed())
+		Expect(req.Messages).To(HaveLen(3))
+
+		assistant := req.Messages[1]
+		Expect(assistant["reasoning"]).To(Equal("need the weather tool"))
+		calls, ok := assistant["tool_calls"].([]any)
+		Expect(ok).To(BeTrue())
+		Expect(calls).To(HaveLen(1))
+		call := calls[0].(map[string]any)
+		Expect(call["id"]).To(Equal("call_1"))
+
+		tool := req.Messages[2]
+		Expect(tool["role"]).To(Equal("tool"))
+		Expect(tool["tool_call_id"]).To(Equal("call_1"))
+		Expect(tool["name"]).To(Equal("get_weather"))
+		Expect(tool["content"]).To(Equal(`{"temp": 21}`))
+
+		user := req.Messages[0]
+		Expect(user).NotTo(HaveKey("tool_call_id"))
+		Expect(user).NotTo(HaveKey("name"))
+		Expect(user).NotTo(HaveKey("reasoning"))
+	})
+})

backend/go/vllm-cpp/e2e_test.go

@@ -0,0 +1,281 @@
+package main
+
+// E2E over a real model + the built libvllm. Gated on VLLM_CPP_MODEL (a .gguf
+// file or a safetensors model dir): without it the suite skips, so CI runs
+// only the unit specs. test.sh auto-downloads a small GGUF when the gate is
+// unset and the download is allowed.
+
+import (
+	"encoding/json"
+	"os"
+	"runtime"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("vllm-cpp e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	It("refuses a foreign model artefact", func() {
+		other := &VllmCpp{}
+		Expect(other.Load(&pb.ModelOptions{ModelFile: "/nonexistent/foreign.bin"})).NotTo(Succeed())
+	})
+
+	It("completes a prompt (greedy)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "The capital of France is",
+			Tokens:      16,
+			Temperature: 0,
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(BeEmpty())
+	})
+
+	It("is deterministic under greedy decoding", func() {
+		opts := &pb.PredictOptions{Prompt: "1 2 3 4", Tokens: 8, Temperature: 0}
+		a, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		b, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(a).To(Equal(b))
+	})
+
+	It("streams deltas that concatenate to the blocking result", func() {
+		opts := &pb.PredictOptions{Prompt: "Count: one two", Tokens: 12, Temperature: 0}
+		blocking, err := backend.Predict(opts)
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan string)
+		Expect(backend.PredictStream(opts, results)).To(Succeed())
+		var sb strings.Builder
+		for delta := range results {
+			sb.WriteString(delta)
+		}
+		Expect(sb.String()).To(Equal(blocking))
+	})
+
+	It("honors stop words", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "a b c d e f",
+			Tokens:      64,
+			Temperature: 0,
+			StopPrompts: []string{"g"},
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).NotTo(ContainSubstring("g h"))
+	})
+
+	It("constrains generation with a GBNF grammar (tool-call path)", func() {
+		text, err := backend.Predict(&pb.PredictOptions{
+			Prompt:      "Answer strictly yes or no: is water wet?",
+			Tokens:      4,
+			Temperature: 0,
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(text).To(Or(HavePrefix("yes"), HavePrefix("no")))
+	})
+
+	It("serves concurrent streams", func() {
+		const n = 4
+		type result struct {
+			text string
+			err  error
+		}
+		done := make(chan result, n)
+		for i := 0; i < n; i++ {
+			go func() {
+				results := make(chan string)
+				err := backend.PredictStream(&pb.PredictOptions{
+					Prompt: "Hello", Tokens: 8, Temperature: 0,
+				}, results)
+				var sb strings.Builder
+				for delta := range results {
+					sb.WriteString(delta)
+				}
+				done <- result{sb.String(), err}
+			}()
+		}
+		for i := 0; i < n; i++ {
+			r := <-done
+			Expect(r.err).NotTo(HaveOccurred())
+			Expect(r.text).NotTo(BeEmpty())
+		}
+	})
+})
+
+var _ = Describe("vllm-cpp chat e2e", Label("e2e"), Ordered, func() {
+	var backend *VllmCpp
+
+	weatherTools := `[{"type":"function","function":{"name":"get_weather",` +
+		`"description":"Get the current weather for a city.",` +
+		`"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]`
+
+	BeforeAll(func() {
+		modelPath := os.Getenv("VLLM_CPP_MODEL")
+		if modelPath == "" {
+			Skip("VLLM_CPP_MODEL not set; skipping chat e2e")
+		}
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			if runtime.GOOS == "darwin" {
+				lib = "./libvllm.dylib"
+			} else {
+				lib = "./libvllm.so"
+			}
+		}
+		Expect(registerLib(lib)).To(Succeed())
+
+		backend = &VllmCpp{}
+		Expect(backend.Load(&pb.ModelOptions{
+			ModelFile:   modelPath,
+			ContextSize: 2048,
+		})).To(Succeed())
+	})
+
+	AfterAll(func() {
+		if backend != nil {
+			Expect(backend.Free()).To(Succeed())
+		}
+	})
+
+	chatOpts := func() *pb.PredictOptions {
+		return &pb.PredictOptions{
+			UseTokenizerTemplate: true,
+			Messages: []*pb.Message{
+				{Role: "user", Content: "Reply with one short sentence: what is the capital of France?"},
+			},
+			Tokens:      512,
+			Temperature: 0,
+		}
+	}
+
+	It("answers a plain chat turn through the engine-side template", func() {
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(BeEmpty())
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+	})
+
+	It("splits reasoning from content engine-side (auto-detected from the template)", func() {
+		// The Qwen3.5 chat template carries <think>, so the engine auto-selects
+		// the think_auto reasoning parser: a markerless answer stays pure
+		// content; if the model DOES think, the block arrives as
+		// ReasoningContent and never leaks into Message.
+		reply, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(reply.Message)).NotTo(ContainSubstring("<think>"))
+		Expect(string(reply.Message)).To(ContainSubstring("Paris"))
+		for _, d := range reply.ChatDeltas {
+			Expect(d.ReasoningContent).NotTo(ContainSubstring("Paris"),
+				"the user-visible answer must not be swallowed into reasoning")
+		}
+	})
+
+	It("streams chat deltas that concatenate to the blocking answer", func() {
+		blocking, err := backend.PredictRich(chatOpts())
+		Expect(err).NotTo(HaveOccurred())
+
+		results := make(chan *pb.Reply, 64)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(chatOpts(), results)
+			close(results)
+		}()
+		var sb strings.Builder
+		for r := range results {
+			sb.WriteString(string(r.Message))
+		}
+		Expect(<-done).To(Succeed())
+		Expect(sb.String()).To(Equal(string(blocking.Message)))
+	})
+
+	It("emits a parsed tool call when tool_choice requires it", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "What is the weather in Rome right now?"},
+		}
+		opts.Tools = weatherTools
+		opts.ToolChoice = "required"
+		opts.Tokens = 256
+
+		reply, err := backend.PredictRich(opts)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(reply.ChatDeltas).NotTo(BeEmpty())
+		var name, args string
+		for _, d := range reply.ChatDeltas {
+			for _, tc := range d.ToolCalls {
+				if tc.Name != "" {
+					name = tc.Name
+				}
+				args += tc.Arguments
+			}
+		}
+		Expect(name).To(Equal("get_weather"))
+		var parsed map[string]any
+		Expect(json.Unmarshal([]byte(args), &parsed)).To(Succeed(),
+			"tool arguments must be valid JSON: %q", args)
+		Expect(parsed).To(HaveKey("city"))
+	})
+
+	It("lets the engine decide on auto tool choice and streams tool deltas", func() {
+		opts := chatOpts()
+		opts.Messages = []*pb.Message{
+			{Role: "user", Content: "Use the get_weather tool to check the weather in Rome."},
+		}
+		opts.Tools = weatherTools
+		opts.Tokens = 512
+
+		results := make(chan *pb.Reply, 128)
+		done := make(chan error, 1)
+		go func() {
+			done <- backend.PredictStreamRich(opts, results)
+			close(results)
+		}()
+		sawToolDelta := false
+		for r := range results {
+			for _, d := range r.ChatDeltas {
+				if len(d.ToolCalls) > 0 {
+					sawToolDelta = true
+				}
+			}
+		}
+		Expect(<-done).To(Succeed())
+		// tool_choice auto is a LAZY constraint: the model may or may not call.
+		// With an explicit instruction the gate model reliably does; treat a
+		// no-call run as a soft signal rather than a hard failure only if the
+		// engine produced SOME output.
+		Expect(sawToolDelta).To(BeTrue(), "expected the engine to engage the tool")
+	})
+})

backend/go/vllm-cpp/govllmcpp.go

@@ -0,0 +1,182 @@
+package main
+
+// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v2).
+//
+// The structs below are hand-mirrored PODs of the C declarations, with
+// explicit padding so the Go layout matches the C layout on linux/darwin
+// amd64+arm64. Struct-by-value entry points (the *_default helpers) are NOT
+// bound - purego's struct-return support is platform-dependent - so the
+// defaults are replicated here and guarded by the vllm_abi_version check at
+// startup: a library whose ABI differs from what these mirrors were written
+// against is refused before any request runs.
+
+import (
+	"fmt"
+	"unsafe"
+
+	"github.com/ebitengine/purego"
+)
+
+// abiVersion is the VLLM_ABI_VERSION this file mirrors (vllm.h).
+const abiVersion = 5
+
+// vllm_status (vllm.h).
+const (
+	vllmOK = 0
+)
+
+// cModelParams mirrors vllm_model_params.
+type cModelParams struct {
+	ModelPath           uintptr // const char*
+	TokenizerConfigPath uintptr // const char*
+	BlockSize           int32
+	NumBlocks           int32
+	MaxModelLen         int32
+	MaxNumSeqs          int32
+	ToolParser          uintptr // const char*; NULL = auto-detect (ABI v4)
+	ReasoningParser     uintptr // const char*; NULL = auto-detect (ABI v5)
+}
+
+// cSamplingParams mirrors vllm_sampling_params (ABI v2, structured fields
+// included). Padding matches the C compiler's: the uint64 seed is 8-aligned,
+// and each pointer following an int32 is 8-aligned.
+type cSamplingParams struct {
+	Temperature          float32
+	TopP                 float32
+	TopK                 int32
+	MinP                 float32
+	MaxTokens            int32
+	_                    [4]byte
+	Seed                 uint64
+	HasSeed              int32
+	PresencePenalty      float32
+	FrequencyPenalty     float32
+	RepetitionPenalty    float32
+	MinTokens            int32
+	IgnoreEOS            int32
+	Stop                 uintptr // const char* const*
+	NStop                int32
+	_                    [4]byte
+	StructuredJSON       uintptr // const char*
+	StructuredRegex      uintptr // const char*
+	StructuredChoice     uintptr // const char* const*
+	NStructuredChoice    int32
+	_                    [4]byte
+	StructuredGrammar    uintptr // const char*
+	StructuredJSONObject int32
+	_                    [4]byte
+}
+
+// cCompletion mirrors vllm_completion.
+type cCompletion struct {
+	Text             uintptr // char*, caller-owned
+	FinishReason     uintptr // const char*, library-owned
+	PromptTokens     int32
+	CompletionTokens int32
+}
+
+// defaultSamplingParams mirrors vllm_sampling_params_default().
+func defaultSamplingParams() cSamplingParams {
+	return cSamplingParams{
+		Temperature:       1.0,
+		TopP:              1.0,
+		MaxTokens:         16,
+		RepetitionPenalty: 1.0,
+	}
+}
+
+// defaultModelParams mirrors vllm_model_params_default().
+func defaultModelParams() cModelParams {
+	return cModelParams{
+		BlockSize:  32,
+		NumBlocks:  256,
+		MaxNumSeqs: 8,
+	}
+}
+
+var (
+	vllmEngineLoad     func(params, out unsafe.Pointer) int32
+	vllmEngineFree     func(engine uintptr)
+	vllmComplete       func(engine uintptr, prompt string, params, out unsafe.Pointer) int32
+	vllmCompleteStream func(engine uintptr, prompt string, params unsafe.Pointer, cb uintptr, userData uintptr) int32
+	vllmChat           func(engine uintptr, requestJSON string, out unsafe.Pointer) int32
+	vllmChatStream     func(engine uintptr, requestJSON string, cb uintptr, userData uintptr) int32
+	vllmStringFree     func(s uintptr)
+	vllmCompletionFree func(out unsafe.Pointer)
+	vllmLastError      func() string
+	vllmVersion        func() string
+	vllmABIVersion     func() int32
+)
+
+type libFunc struct {
+	ptr  any
+	name string
+}
+
+// registerLib dlopens libvllm and binds the C ABI, refusing an ABI-version
+// mismatch (the struct mirrors above would be undefined behavior against a
+// different layout).
+func registerLib(libName string) error {
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		return fmt.Errorf("vllm-cpp: dlopen %s: %w", libName, err)
+	}
+	for _, lf := range []libFunc{
+		{&vllmEngineLoad, "vllm_engine_load"},
+		{&vllmEngineFree, "vllm_engine_free"},
+		{&vllmComplete, "vllm_complete"},
+		{&vllmCompleteStream, "vllm_complete_stream"},
+		{&vllmChat, "vllm_chat"},
+		{&vllmChatStream, "vllm_chat_stream"},
+		{&vllmStringFree, "vllm_string_free"},
+		{&vllmCompletionFree, "vllm_completion_free"},
+		{&vllmLastError, "vllm_last_error"},
+		{&vllmVersion, "vllm_version"},
+		{&vllmABIVersion, "vllm_abi_version"},
+	} {
+		purego.RegisterLibFunc(lf.ptr, lib, lf.name)
+	}
+	if v := vllmABIVersion(); v != abiVersion {
+		return fmt.Errorf("vllm-cpp: ABI mismatch: library reports v%d, backend built against v%d", v, abiVersion)
+	}
+	return nil
+}
+
+// cString returns a NUL-terminated byte slice for s. The backing array may be
+// passed to C for the duration of a call (the ABI borrows and copies); keep it
+// alive across the call with runtime.KeepAlive.
+func cString(s string) []byte {
+	b := make([]byte, len(s)+1)
+	copy(b, s)
+	return b
+}
+
+// cStringArray builds a NULL-free array of C-string pointers plus the backing
+// buffers that must stay alive for the duration of the C call.
+func cStringArray(ss []string) (ptrs []uintptr, backing [][]byte) {
+	backing = make([][]byte, 0, len(ss))
+	ptrs = make([]uintptr, 0, len(ss))
+	for _, s := range ss {
+		b := cString(s)
+		backing = append(backing, b)
+		ptrs = append(ptrs, uintptr(unsafe.Pointer(&b[0]))) // #nosec G103 -- borrowed by C for the call only
+	}
+	return ptrs, backing
+}
+
+// goString copies a NUL-terminated C string.
+func goString(p uintptr) string {
+	if p == 0 {
+		return ""
+	}
+	//nolint:govet // C-owned pointer handed over by purego, valid for this call
+	base := unsafe.Pointer(p) // #nosec G103 -- C-owned, copied out immediately
+	n := 0
+	for *(*byte)(unsafe.Add(base, n)) != 0 {
+		n++
+	}
+	if n == 0 {
+		return ""
+	}
+	return string(unsafe.Slice((*byte)(base), n))
+}

backend/go/vllm-cpp/main.go

@@ -0,0 +1,35 @@
+package main
+
+// Note: this is started internally by LocalAI and a server is allocated for each model
+import (
+	"flag"
+	"os"
+	"runtime"
+
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+func main() {
+	libName := os.Getenv("VLLM_CPP_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "./libvllm.dylib"
+		} else {
+			libName = "./libvllm.so"
+		}
+	}
+
+	if err := registerLib(libName); err != nil {
+		panic(err)
+	}
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &VllmCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/vllm-cpp/options.go

@@ -0,0 +1,54 @@
+package main
+
+// Engine-sizing knobs carried through the model config's free-form
+// `options:` list ("key:value" entries), mirroring how the other in-house
+// backends pass engine-specific settings that have no proto field.
+
+import (
+	"strconv"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+type loadOptions struct {
+	blockSize  int32 // KV block size (tokens/block); engine default 32.
+	numBlocks  int32 // KV blocks to allocate; engine default 256.
+	maxNumSeqs int32 // max concurrent sequences; engine default 8.
+	// Engine-side parser selection (ABI v4/v5). Empty = the engine
+	// auto-detects from the chat template; "none" disables the reasoning
+	// split; unknown names fail the first chat call.
+	toolParser      string
+	reasoningParser string
+}
+
+func parseOptions(opts *pb.ModelOptions) loadOptions {
+	lo := loadOptions{}
+	for _, o := range opts.GetOptions() {
+		k, v, found := strings.Cut(o, ":")
+		if !found {
+			continue
+		}
+		switch strings.TrimSpace(k) {
+		case "block_size":
+			lo.blockSize = parseInt32(v, lo.blockSize)
+		case "num_blocks":
+			lo.numBlocks = parseInt32(v, lo.numBlocks)
+		case "max_num_seqs":
+			lo.maxNumSeqs = parseInt32(v, lo.maxNumSeqs)
+		case "tool_parser":
+			lo.toolParser = strings.TrimSpace(v)
+		case "reasoning_parser":
+			lo.reasoningParser = strings.TrimSpace(v)
+		}
+	}
+	return lo
+}
+
+func parseInt32(s string, fallback int32) int32 {
+	n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 32)
+	if err != nil || n <= 0 {
+		return fallback
+	}
+	return int32(n)
+}

backend/go/vllm-cpp/package.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avf $CURDIR/vllm-cpp $CURDIR/package/
+cp -fLv $CURDIR/libvllm.so $CURDIR/package/ 2>/dev/null || true
+cp -fLv $CURDIR/libvllm.dylib $CURDIR/package/ 2>/dev/null || true
+cp -fv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ $(uname -s) = "Darwin" ]; then
+    echo "Detected Darwin"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/go/vllm-cpp/run.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath "$0")")
+
+cd /
+
+# vllm.cpp ships ONE portable library per platform (SIMD tiers are per-file
+# with runtime dispatch), so there is no per-CPU variant probing here.
+if [ "$(uname)" = "Darwin" ]; then
+	LIBRARY="$CURDIR/libvllm.dylib"
+	export DYLD_LIBRARY_PATH="$CURDIR"/lib:$DYLD_LIBRARY_PATH
+else
+	LIBRARY="$CURDIR/libvllm.so"
+	export LD_LIBRARY_PATH="$CURDIR"/lib:$LD_LIBRARY_PATH
+fi
+
+export VLLM_CPP_LIBRARY=$LIBRARY
+
+# If there is a lib/ld.so, use it
+if [ -f "$CURDIR"/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using library: $LIBRARY"
+	exec "$CURDIR"/lib/ld.so "$CURDIR"/vllm-cpp "$@"
+fi
+
+echo "Using library: $LIBRARY"
+exec "$CURDIR"/vllm-cpp "$@"

backend/go/vllm-cpp/test.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+cd "$CURDIR"
+
+echo "Running vllm-cpp backend tests..."
+
+# Unit specs always run (struct-mirror layout, option/sampling mapping, load
+# validation). The e2e specs need a real model: set VLLM_CPP_MODEL to a .gguf
+# file or a safetensors model dir to enable them (see e2e_test.go).
+go test -v -timeout 1200s .
+
+echo "All vllm-cpp tests passed."

backend/go/vllm-cpp/vllmcpp_test.go

@@ -0,0 +1,162 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"unsafe"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestVllmCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "vllm-cpp suite")
+}
+
+// The Go POD mirrors must match the C struct layout of vllm.h (ABI v2)
+// byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
+// amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
+var _ = Describe("C ABI struct mirrors", func() {
+	It("cModelParams matches vllm_model_params", func() {
+		var p cModelParams
+		Expect(unsafe.Offsetof(p.ModelPath)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TokenizerConfigPath)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.BlockSize)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.NumBlocks)).To(Equal(uintptr(20)))
+		Expect(unsafe.Offsetof(p.MaxModelLen)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.MaxNumSeqs)).To(Equal(uintptr(28)))
+		Expect(unsafe.Offsetof(p.ToolParser)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.ReasoningParser)).To(Equal(uintptr(40)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(48)))
+	})
+
+	It("cSamplingParams matches vllm_sampling_params (ABI v2)", func() {
+		var p cSamplingParams
+		Expect(unsafe.Offsetof(p.Temperature)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(p.TopP)).To(Equal(uintptr(4)))
+		Expect(unsafe.Offsetof(p.TopK)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(p.MinP)).To(Equal(uintptr(12)))
+		Expect(unsafe.Offsetof(p.MaxTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(p.Seed)).To(Equal(uintptr(24)))
+		Expect(unsafe.Offsetof(p.HasSeed)).To(Equal(uintptr(32)))
+		Expect(unsafe.Offsetof(p.PresencePenalty)).To(Equal(uintptr(36)))
+		Expect(unsafe.Offsetof(p.FrequencyPenalty)).To(Equal(uintptr(40)))
+		Expect(unsafe.Offsetof(p.RepetitionPenalty)).To(Equal(uintptr(44)))
+		Expect(unsafe.Offsetof(p.MinTokens)).To(Equal(uintptr(48)))
+		Expect(unsafe.Offsetof(p.IgnoreEOS)).To(Equal(uintptr(52)))
+		Expect(unsafe.Offsetof(p.Stop)).To(Equal(uintptr(56)))
+		Expect(unsafe.Offsetof(p.NStop)).To(Equal(uintptr(64)))
+		Expect(unsafe.Offsetof(p.StructuredJSON)).To(Equal(uintptr(72)))
+		Expect(unsafe.Offsetof(p.StructuredRegex)).To(Equal(uintptr(80)))
+		Expect(unsafe.Offsetof(p.StructuredChoice)).To(Equal(uintptr(88)))
+		Expect(unsafe.Offsetof(p.NStructuredChoice)).To(Equal(uintptr(96)))
+		Expect(unsafe.Offsetof(p.StructuredGrammar)).To(Equal(uintptr(104)))
+		Expect(unsafe.Offsetof(p.StructuredJSONObject)).To(Equal(uintptr(112)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
+	})
+
+	It("cCompletion matches vllm_completion", func() {
+		var c cCompletion
+		Expect(unsafe.Offsetof(c.Text)).To(Equal(uintptr(0)))
+		Expect(unsafe.Offsetof(c.FinishReason)).To(Equal(uintptr(8)))
+		Expect(unsafe.Offsetof(c.PromptTokens)).To(Equal(uintptr(16)))
+		Expect(unsafe.Offsetof(c.CompletionTokens)).To(Equal(uintptr(20)))
+		Expect(unsafe.Sizeof(c)).To(Equal(uintptr(24)))
+	})
+})
+
+var _ = Describe("parseOptions", func() {
+	It("extracts the engine sizing knobs", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:64", "num_blocks:512", "max_num_seqs:32", "unknown:ignored",
+		}})
+		Expect(lo.blockSize).To(Equal(int32(64)))
+		Expect(lo.numBlocks).To(Equal(int32(512)))
+		Expect(lo.maxNumSeqs).To(Equal(int32(32)))
+	})
+	It("ignores malformed and non-positive values", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			"block_size:abc", "num_blocks:-1", "max_num_seqs", "block_size:0",
+		}})
+		Expect(lo).To(Equal(loadOptions{}))
+	})
+})
+
+var _ = Describe("samplingFromPredict", func() {
+	It("maps the sampling fields onto the C POD", func() {
+		sp, _ := samplingFromPredict(&pb.PredictOptions{
+			Temperature:      0.7,
+			TopP:             0.9,
+			TopK:             40,
+			MinP:             0.05,
+			Tokens:           128,
+			Seed:             42,
+			Penalty:          1.1,
+			PresencePenalty:  0.5,
+			FrequencyPenalty: 0.25,
+			IgnoreEOS:        true,
+		})
+		Expect(sp.Temperature).To(BeNumerically("~", 0.7, 1e-6))
+		Expect(sp.TopP).To(BeNumerically("~", 0.9, 1e-6))
+		Expect(sp.TopK).To(Equal(int32(40)))
+		Expect(sp.MinP).To(BeNumerically("~", 0.05, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(128)))
+		Expect(sp.HasSeed).To(Equal(int32(1)))
+		Expect(sp.Seed).To(Equal(uint64(42)))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.1, 1e-6))
+		Expect(sp.PresencePenalty).To(BeNumerically("~", 0.5, 1e-6))
+		Expect(sp.FrequencyPenalty).To(BeNumerically("~", 0.25, 1e-6))
+		Expect(sp.IgnoreEOS).To(Equal(int32(1)))
+	})
+
+	It("keeps the engine defaults for unset fields and stays unseeded", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{})
+		Expect(sp.TopP).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.RepetitionPenalty).To(BeNumerically("~", 1.0, 1e-6))
+		Expect(sp.MaxTokens).To(Equal(int32(0))) // unbounded, engine-capped.
+		Expect(sp.HasSeed).To(Equal(int32(0)))
+		Expect(sp.Stop).To(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).To(Equal(uintptr(0)))
+		Expect(keep).To(BeEmpty())
+	})
+
+	It("wires stop prompts and the grammar constraint", func() {
+		sp, keep := samplingFromPredict(&pb.PredictOptions{
+			StopPrompts: []string{"</s>", "\n\n"},
+			Grammar:     "root ::= \"yes\" | \"no\"",
+		})
+		Expect(sp.NStop).To(Equal(int32(2)))
+		Expect(sp.Stop).NotTo(Equal(uintptr(0)))
+		Expect(sp.StructuredGrammar).NotTo(Equal(uintptr(0)))
+		Expect(keep).NotTo(BeEmpty())
+	})
+})
+
+var _ = Describe("validModelPath", func() {
+	It("accepts a .gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "model.gguf")
+		Expect(os.WriteFile(p, []byte("GGUF"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).To(Succeed())
+	})
+	It("accepts a directory with config.json", func() {
+		dir := GinkgoT().TempDir()
+		Expect(os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o600)).To(Succeed())
+		Expect(validModelPath(dir)).To(Succeed())
+	})
+	It("refuses a directory without config.json (greedy-probe rule)", func() {
+		Expect(validModelPath(GinkgoT().TempDir())).NotTo(Succeed())
+	})
+	It("refuses a non-gguf file", func() {
+		dir := GinkgoT().TempDir()
+		p := filepath.Join(dir, "weights.bin")
+		Expect(os.WriteFile(p, []byte("x"), 0o600)).To(Succeed())
+		Expect(validModelPath(p)).NotTo(Succeed())
+	})
+	It("refuses a missing path", func() {
+		Expect(validModelPath("/nonexistent/model.gguf")).NotTo(Succeed())
+	})
+})

backend/index.yaml

@@ -156,6 +156,44 @@
     nvidia-cuda-12: "cuda12-whisper"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-whisper"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-whisper"
+- &vllm-cpp
+  name: "vllm-cpp"
+  alias: "vllm-cpp"
+  license: apache-2.0
+  description: |
+    vllm.cpp is a from-scratch C++20 port of vLLM created and maintained by the LocalAI team.
+    It mirrors vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching,
+    scheduler, sampler) on a portable tensor runtime with no Python, PyTorch or ggml at
+    inference time. It loads Hugging Face safetensors and GGUF checkpoints, supports
+    structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine,
+    and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan.
+  urls:
+    - https://github.com/mudler/vllm.cpp
+  tags:
+    - text-to-text
+    - LLM
+    - CPU
+    - GPU
+    - CUDA
+    - metal
+  capabilities:
+    default: "cpu-vllm-cpp"
+    nvidia: "cuda13-vllm-cpp"
+    metal: "metal-vllm-cpp"
+    vulkan: "vulkan-vllm-cpp"
+    nvidia-cuda-13: "cuda13-vllm-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp"
+- !!merge <<: *vllm-cpp
+  name: "vllm-cpp-development"
+  capabilities:
+    default: "cpu-vllm-cpp-development"
+    nvidia: "cuda13-vllm-cpp-development"
+    metal: "metal-vllm-cpp-development"
+    vulkan: "vulkan-vllm-cpp-development"
+    nvidia-cuda-13: "cuda13-vllm-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp-development"
+    nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp-development"
 - &crispasr
   name: "crispasr"
   alias: "crispasr"
@@ -6573,3 +6611,53 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-supertonic"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-supertonic
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cpu-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "cuda13-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "nvidia-l4t-arm64-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "vulkan-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-vllm-cpp
+- !!merge <<: *vllm-cpp
+  name: "metal-vllm-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-vllm-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-vllm-cpp

core/gallery/importers/llama-cpp.go

@@ -37,6 +37,7 @@ func (i *LlamaCPPImporter) AdditionalBackends() []KnownBackendEntry {
 	return []KnownBackendEntry{
 		{Name: "ik-llama-cpp", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with ik-quants"},
 		{Name: "turboquant", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with TurboQuant optimizations"},
+		{Name: "vllm-cpp", Modality: "text", Description: "vLLM-style continuous-batching engine (vllm.cpp) consuming GGUF, by the LocalAI team"},
 	}
 }
 
@@ -135,7 +136,7 @@ func (i *LlamaCPPImporter) Import(details Details) (gallery.ModelConfig, error)
 	backend := "llama-cpp"
 	if b, ok := preferencesMap["backend"].(string); ok {
 		switch b {
-		case "ik-llama-cpp", "turboquant":
+		case "ik-llama-cpp", "turboquant", "vllm-cpp":
 			backend = b
 		}
 	}
@@ -157,6 +158,16 @@ func (i *LlamaCPPImporter) Import(details Details) (gallery.ModelConfig, error)
 		},
 	}
 
+	// vllm-cpp rides the same autoparser-style flow as llama-cpp: the ENGINE
+	// applies the model's chat template (GGUF tokenizer.chat_template) and
+	// decides when a tool call engages (lazy structural-tag constraint +
+	// engine-side parsing), so the tokenizer-template config carries over
+	// verbatim. Only `use_jinja` is dropped - that option is a llama-cpp
+	// backend knob with no meaning for vllm-cpp.
+	if backend == "vllm-cpp" {
+		modelConfig.Options = nil
+	}
+
 	if embeddings != "" && strings.ToLower(embeddings) == "true" || strings.ToLower(embeddings) == "yes" {
 		trueV := true
 		modelConfig.Embeddings = &trueV

core/gallery/importers/llama-cpp_test.go

@@ -181,9 +181,31 @@ var _ = Describe("LlamaCPPImporter", func() {
 			Expect(modelConfig.Files[0].Filename).To(Equal("my-model.gguf"))
 		})
 
+		It("swaps the emitted backend to vllm-cpp when preferred, keeping engine-side templating", func() {
+			preferences := json.RawMessage(`{"backend": "vllm-cpp"}`)
+			details := Details{
+				URI:         "https://example.com/my-model.gguf",
+				Preferences: preferences,
+			}
+
+			modelConfig, err := importer.Import(details)
+
+			Expect(err).ToNot(HaveOccurred())
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: vllm-cpp"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).NotTo(ContainSubstring("backend: llama-cpp\n"), fmt.Sprintf("Model config: %+v", modelConfig))
+			// vllm-cpp rides the autoparser-style flow: the ENGINE templates and
+			// parses tools, so use_tokenizer_template carries over; only the
+			// llama-cpp-specific use_jinja option is dropped.
+			Expect(modelConfig.ConfigFile).NotTo(ContainSubstring("use_jinja"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("use_tokenizer_template: true"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("model: my-model.gguf"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(len(modelConfig.Files)).To(Equal(1))
+			Expect(modelConfig.Files[0].Filename).To(Equal("my-model.gguf"))
+		})
+
 		It("keeps backend: llama-cpp for unknown backend preferences", func() {
 			// Unknown backend values must not leak into the emitted YAML —
-			// we only honour the two curated drop-in replacements.
+			// we only honour the curated drop-in replacements.
 			preferences := json.RawMessage(`{"backend": "something-weird"}`)
 			details := Details{
 				URI:         "https://example.com/my-model.gguf",
@@ -529,7 +551,7 @@ var _ = Describe("LlamaCPPImporter", func() {
 	})
 
 	Context("AdditionalBackends", func() {
-		It("advertises ik-llama-cpp and turboquant as drop-in replacements", func() {
+		It("advertises ik-llama-cpp, turboquant and vllm-cpp as drop-in replacements", func() {
 			entries := importer.AdditionalBackends()
 
 			names := make([]string, 0, len(entries))
@@ -538,15 +560,13 @@ var _ = Describe("LlamaCPPImporter", func() {
 				names = append(names, e.Name)
 				byName[e.Name] = e
 			}
-			Expect(names).To(ConsistOf("ik-llama-cpp", "turboquant"))
+			Expect(names).To(ConsistOf("ik-llama-cpp", "turboquant", "vllm-cpp"))
 
-			ik := byName["ik-llama-cpp"]
-			Expect(ik.Modality).To(Equal("text"))
-			Expect(ik.Description).NotTo(BeEmpty())
-
-			tq := byName["turboquant"]
-			Expect(tq.Modality).To(Equal("text"))
-			Expect(tq.Description).NotTo(BeEmpty())
+			for _, name := range names {
+				e := byName[name]
+				Expect(e.Modality).To(Equal("text"))
+				Expect(e.Description).NotTo(BeEmpty())
+			}
 		})
 	})
 })

core/gallery/importers/vllm.go

@@ -19,6 +19,16 @@ func (i *VLLMImporter) Name() string      { return "vllm" }
 func (i *VLLMImporter) Modality() string  { return "text" }
 func (i *VLLMImporter) AutoDetects() bool { return true }
 
+// AdditionalBackends advertises vllm-cpp (the LocalAI-team C++ port of vLLM)
+// as a preference-only drop-in: it consumes the same safetensors model dirs
+// this importer detects, so selecting it swaps the emitted backend field while
+// reusing the vllm Match/Import pipeline.
+func (i *VLLMImporter) AdditionalBackends() []KnownBackendEntry {
+	return []KnownBackendEntry{
+		{Name: "vllm-cpp", Modality: "text", Description: "C++ port of vLLM by the LocalAI team (safetensors + GGUF, no Python at inference)"},
+	}
+}
+
 func (i *VLLMImporter) Match(details Details) bool {
 	preferences, err := details.Preferences.MarshalJSON()
 	if err != nil {
@@ -31,7 +41,7 @@ func (i *VLLMImporter) Match(details Details) bool {
 	}
 
 	b, ok := preferencesMap["backend"].(string)
-	if ok && b == "vllm" {
+	if ok && (b == "vllm" || b == "vllm-cpp") {
 		return true
 	}
 
@@ -92,15 +102,22 @@ func (i *VLLMImporter) Import(details Details) (gallery.ModelConfig, error) {
 	// Apply per-model-family inference parameter defaults
 	config.ApplyInferenceDefaults(&modelConfig, details.URI)
 
-	// Auto-detect tool_parser and reasoning_parser for known model families.
-	// Surfacing them in the generated YAML lets users see and edit the choices.
-	parsers := config.MatchParserDefaults(details.URI)
-	if parsers != nil {
-		if tp, ok := parsers["tool_parser"]; ok {
-			modelConfig.Options = append(modelConfig.Options, "tool_parser:"+tp)
-		}
-		if rp, ok := parsers["reasoning_parser"]; ok {
-			modelConfig.Options = append(modelConfig.Options, "reasoning_parser:"+rp)
+	if backend == "vllm-cpp" {
+		// vllm-cpp applies the model's chat template ENGINE-side (like the
+		// vllm python backend, so use_tokenizer_template carries over), but
+		// tool/reasoning parsing is the engine's own autoparser pipeline -
+		// the vllm-python tool_parser/reasoning_parser options don't apply.
+	} else {
+		// Auto-detect tool_parser and reasoning_parser for known model families.
+		// Surfacing them in the generated YAML lets users see and edit the choices.
+		parsers := config.MatchParserDefaults(details.URI)
+		if parsers != nil {
+			if tp, ok := parsers["tool_parser"]; ok {
+				modelConfig.Options = append(modelConfig.Options, "tool_parser:"+tp)
+			}
+			if rp, ok := parsers["reasoning_parser"]; ok {
+				modelConfig.Options = append(modelConfig.Options, "reasoning_parser:"+rp)
+			}
 		}
 	}
 

core/gallery/importers/vllm_test.go

@@ -133,6 +133,34 @@ var _ = Describe("VLLMImporter", func() {
 			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: vllm"))
 		})
 
+		It("swaps the emitted backend to vllm-cpp when preferred, keeping engine-side templating", func() {
+			preferences := json.RawMessage(`{"backend": "vllm-cpp"}`)
+			details := Details{
+				URI:         "https://huggingface.co/test/my-model",
+				Preferences: preferences,
+			}
+
+			modelConfig, err := importer.Import(details)
+
+			Expect(err).ToNot(HaveOccurred())
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: vllm-cpp"))
+			// vllm-cpp templates ENGINE-side (use_tokenizer_template carries
+			// over); the vllm-python parser options don't apply.
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("use_tokenizer_template: true"))
+			Expect(modelConfig.ConfigFile).NotTo(ContainSubstring("tool_parser"))
+			Expect(modelConfig.ConfigFile).NotTo(ContainSubstring("reasoning_parser"))
+		})
+
+		It("matches when backend preference is vllm-cpp", func() {
+			preferences := json.RawMessage(`{"backend": "vllm-cpp"}`)
+			details := Details{
+				URI:         "https://huggingface.co/test/my-model",
+				Preferences: preferences,
+			}
+
+			Expect(importer.Match(details)).To(BeTrue())
+		})
+
 		It("should handle invalid JSON preferences", func() {
 			preferences := json.RawMessage(`invalid json`)
 			details := Details{

docs/content/features/backends.md

@@ -124,7 +124,7 @@ For getting started, see the available backends in LocalAI here: https://github.
 
 LocalAI supports various types of backends:
 
-- **LLM Backends**: For running language models (e.g., llama.cpp, vLLM, SGLang, transformers, MLX)
+- **LLM Backends**: For running language models (e.g., llama.cpp, vLLM, vllm.cpp, SGLang, transformers, MLX)
 - **Speech-to-Text Backends**: For transcription and speaker diarization (e.g., whisper.cpp, parakeet.cpp, moss-transcribe.cpp, faster-whisper, NeMo)
 - **Text-to-Speech Backends**: For speech synthesis (e.g., piper, Kokoro, VibeVoice, Qwen3-TTS)
 - **Sound Generation Backends**: For music and audio generation (e.g., ACE-Step)

docs/content/reference/compatibility-table.md

@@ -24,6 +24,7 @@ All backends listed here can be installed on demand from the [Backend Gallery]({
 | [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) | Hard fork of llama.cpp optimized for CPU/hybrid CPU+GPU with IQK quants, custom quant mixes, and MLA for DeepSeek | GPT | yes | yes | CPU (AVX2+) |
 | [turboquant](https://github.com/TheTom/llama-cpp-turboquant) | llama.cpp fork adding the TurboQuant KV-cache quantization scheme | GPT | yes | yes | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Jetson L4T |
 | [ds4](https://github.com/antirez/ds4) | DeepSeek V4 Flash single-model inference engine, optimized for Metal and CUDA | GPT | no | yes | CPU, CUDA 12/13, Metal, Jetson L4T |
+| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM by the LocalAI team: paged KV cache, continuous batching, prefix caching, safetensors + GGUF, engine-enforced structured output, no Python at inference | GPT, Functions | no | yes | CPU, CUDA 12/13 (Blackwell-family), Vulkan, Metal, Jetson L4T (GB10) |
 | [vLLM](https://github.com/vllm-project/vllm) | Fast LLM serving with PagedAttention; GPTQ/AWQ/FP8 quantization | GPT, Functions, Multimodal | no | yes | CUDA 12/13, ROCm, Intel SYCL, Jetson L4T |
 | [vLLM Omni](https://github.com/vllm-project/vllm-omni) | Unified multimodal generation (text, image, video, audio) on top of vLLM | Multimodal GPT, Functions | no | yes | CUDA 12/13, ROCm, Jetson L4T |
 | [SGLang](https://github.com/sgl-project/sglang) | Fast serving framework for LLMs and vision-language models with speculative decoding | GPT, Functions, Multimodal | no | yes | CUDA 12/13, ROCm, Intel SYCL, Jetson L4T |