mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
927206ed53
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks, MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was an outlier and averaging it in overstated us by roughly 1.5 points. Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms against MLX-LM's 532.6, so we remain about 1.5% faster there. Nothing else changes. MLX still wins prefill and loses decode, the shape gate is still the right disposition, and the pin and the flag are still coupled. The gate is worth about 1.7 points over the MLX-off build rather than 2.7. Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
91 lines
4.2 KiB
Markdown
91 lines
4.2 KiB
Markdown
# vllm-cpp backend
|
|
|
|
LocalAI text-generation backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
|
|
the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching,
|
|
safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
|
|
inference time.
|
|
|
|
The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
|
|
ABI v2) through purego:
|
|
|
|
- `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
|
|
directory (`config.json` + safetensors). `context_size` maps to
|
|
`max_model_len`; `options: ["block_size:<n>", "num_blocks:<n>",
|
|
"max_num_seqs:<n>"]` size the KV cache and scheduler admission.
|
|
- `Predict` -> `vllm_complete` (blocking).
|
|
- `PredictStream` -> `vllm_complete_stream`; concurrent gRPC requests batch
|
|
continuously in the engine's shared AsyncLLM scheduler.
|
|
- Chat / tool calling rides the SAME code path as the llama.cpp autoparser:
|
|
with `use_tokenizer_template: true` the backend implements
|
|
`PredictRich`/`PredictStreamRich` over the ABI v3 chat entry points
|
|
(`vllm_chat` / `vllm_chat_stream`). The ENGINE applies the model's chat
|
|
template (GGUF `tokenizer.chat_template` or `tokenizer_config.json`),
|
|
decides when a tool call engages (`tool_choice: auto` lowers to a LAZY
|
|
structural-tag decode constraint; `required`/named force one), parses tool
|
|
calls with its streaming Hermes-style parser, and the backend maps each
|
|
`chat.completion.chunk` onto `ChatDelta`/`ToolCallDelta` protos.
|
|
- Without structured messages the plain path applies:
|
|
`PredictOptions.Grammar` -> the ABI's `structured_grammar` (GBNF) for
|
|
LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
|
|
choice constraints are also exposed by the ABI.
|
|
|
|
Model config example:
|
|
|
|
```yaml
|
|
name: qwen3-vllm
|
|
backend: vllm-cpp
|
|
context_size: 8192
|
|
parameters:
|
|
model: Qwen3-4B # model dir (safetensors) or .gguf file
|
|
options:
|
|
- max_num_seqs:16
|
|
```
|
|
|
|
## Apple Silicon: the MLX GEMM provider (ON by default, gated to prefill)
|
|
|
|
`BUILD_TYPE=metal` builds vllm.cpp's MLX provider for the dense GEMM
|
|
(`VLLM_CPP_MLX=on`, the default here). It is on because upstream now SHAPE-GATES
|
|
it to prefill; it was briefly off in this branch's history, and that was correct
|
|
at the time for an ungated provider.
|
|
|
|
The gate matters more than the flag. MLX's steel GEMM wins prefill but loses
|
|
decode, because the provider pays an `mx::eval` synchronisation plus an output
|
|
memcpy on every call and decode makes ~112 calls *per token*. Measured on an
|
|
Apple M4, Qwen3-1.7B-bf16 warm at p=512 g=128:
|
|
|
|
| configuration | prefill TTFT | warm throughput |
|
|
|---|--:|--:|
|
|
| MLX **gated to prefill** (pin >= 89c46aeb) | **524.5 ms** | **24.37 tok/s, 97.6% of MLX-LM** |
|
|
| MLX ungated (older pins) | 537 ms | 12.7 tok/s |
|
|
| MLX off | 602 ms | 23.9 tok/s, 95.9% |
|
|
|
|
Ratios are against an MLX-LM baseline measured INTERLEAVED with ours over four
|
|
ABBA blocks (its spread 0.34%, ours 0.12%). An earlier revision of this file
|
|
claimed 99.1%; that used a two-run MLX-LM baseline containing an outlier and
|
|
overstated us by about 1.5 points.
|
|
|
|
**`VLLM_CPP_VERSION` and this flag are coupled.** Moving the pin back before
|
|
`89c46aeb` while leaving `VLLM_CPP_MLX=on` would take the middle row — roughly
|
|
half throughput. If you roll the pin back, roll the default back with it.
|
|
|
|
One caveat: MLX's GEMM is not bit-identical to the native kernel, so an MLX build
|
|
produces a different greedy sequence than a non-MLX one. That is a property of the
|
|
provider, not of the gate, and it predates this packaging. Full disposition in
|
|
vllm.cpp `docs/BENCHMARKS.md`.
|
|
|
|
Build knobs:
|
|
|
|
- `VLLM_CPP_MLX=off` builds Metal without the provider: ~124 MB smaller, and
|
|
96.4% of MLX-LM instead of 99.1%.
|
|
- `MLX_VERSION` pins the wheel (default `0.29.3`). MLX is consumed as the
|
|
prebuilt pip wheel because building it from source needs `xcrun metal`, i.e. a
|
|
full Xcode the macOS runners do not have.
|
|
|
|
Packaging vendors `libmlx.dylib`, `mlx.metallib` and MLX's MIT license into
|
|
`package/lib/`, and rewrites `libvllm.dylib`'s rpath to `@loader_path/lib`
|
|
(re-signing it, since `install_name_tool` invalidates the signature). The
|
|
metallib must stay beside `libmlx.dylib`: MLX looks for it there.
|
|
|
|
Testing: `make test` runs the unit specs; export `VLLM_CPP_MODEL=<model>` (and
|
|
optionally `VLLM_CPP_LIBRARY=<libvllm path>`) to enable the e2e specs.
|