mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat(vllm-cpp): bump vllm.cpp and default MLX ON, gated to prefill

Browse Source 
Bumps VLLM_CPP_VERSION from 9e1c9025 to eec09bed and turns VLLM_CPP_MLX back on.
These two must move together, which is why they are one commit.

Upstream now shape-gates the MLX provider to prefill: it declines m < 2, which is
exactly the decode GEMV. MLX's steel GEMM wins prefill, 524.5 ms of TTFT against
602 for the native path, but loses decode badly because the provider pays an
mx::eval synchronisation and an output memcpy on every call while decode makes
about 112 calls per token. Ungated it does both; gated it does only the good half.

Measured on an Apple M4 with Qwen3-1.7B-bf16 warm at p=512 g=128:

  MLX gated to prefill (pin >= 89c46aeb)   TTFT 524.5 ms   24.40 tok/s, 99.1% of MLX-LM
  MLX ungated (older pins)                 TTFT 537 ms     12.7 tok/s
  MLX off                                  TTFT 602 ms     23.9 tok/s

This branch briefly defaulted the provider off, which was the correct call for an
ungated provider at the old pin. The gate is what makes on correct again, so the
pin and the flag are coupled: rolling VLLM_CPP_VERSION back before 89c46aeb while
leaving MLX on would select the middle row and roughly halve throughput. Both the
Makefile comment and the README state that dependency explicitly.

The bump also brings six Metal kernels landed upstream since the old pin — mma
prefill attention, a vectorised decode V accumulation, vectorised attention
staging, a fused qk-norm-RoPE preamble, a simdgroup-per-row softmax and a
simdgroup-per-head preamble — which take the non-MLX Metal path from 89.4% to
96.4% of MLX-LM on their own.

One caveat, recorded in the README: MLX's GEMM is not bit-identical to the native
kernel, so an MLX build produces a different greedy sequence than a non-MLX build.
That is a property of the provider rather than of the gate and predates this
packaging.

Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto
2026-07-28 08:36:30 +00:00
committed by localai-org-maint-bot
parent 1556b08d81
commit d49738b59c
2 changed files with 35 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
backend/go/vllm-cpp/Makefile
View File

@@ -11,18 +11,25 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e
# vllm.cpp version
VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845
VLLM_CPP_VERSION?=eec09bed5a03457837b499781c23d8e44f106813
# MLX GEMM provider (darwin/metal only; see the metal branch below for why).
# Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun
# metal`, i.e. a full Xcode the macOS runners do not have, while the wheel ships
# include/, lib/libmlx.dylib and the compiled mlx.metallib ready to link.
#
# DEFAULT OFF. It was on when this branch opened, on the strength of an A/B that
# had MLX at 1.88x. That measurement is stale: vllm.cpp's own Metal kernels have
# since improved several-fold, and re-measured on the same M4 the provider is now
# 46% SLOWER end to end (11.98 vs 22.06 warm tok/s). See the README.
VLLM_CPP_MLX?=off
# DEFAULT ON, but ONLY because VLLM_CPP_VERSION above is pinned at or past
# vllm.cpp 89c46aeb, which SHAPE-GATES the provider to prefill. The ordering is
# load-bearing, not incidental:
#
# pin >= 89c46aeb, MLX on -> 99.1% of MLX-LM (gated: prefill only)
# pin < 89c46aeb, MLX on -> ~51% (ungated: it also takes decode)
#
# MLX's steel GEMM wins prefill (537 ms TTFT against 602) and loses decode badly,
# because the provider pays an mx::eval sync plus an output memcpy per call and
# decode makes ~112 calls per TOKEN. Ungated it does both; gated it does only the
# good half. So if this pin is ever moved BACKWARDS, this default must go with it.
VLLM_CPP_MLX?=on
MLX_VERSION?=0.29.3
MLX_VENV?=$(abspath ./mlx-venv)
# Resolved lazily (recursive `=`, not `:=`): the glob only matches once the venv

46
backend/go/vllm-cpp/README.md
View File

@@ -41,39 +41,37 @@ options:
- max_num_seqs:16
```
## Apple Silicon: the MLX GEMM provider (OFF by default)
## Apple Silicon: the MLX GEMM provider (ON by default, gated to prefill)
`BUILD_TYPE=metal` can build vllm.cpp's optional MLX provider for the dense GEMM
(`VLLM_CPP_MLX=on`). **It is OFF by default, because it is currently slower.**
`BUILD_TYPE=metal` builds vllm.cpp's MLX provider for the dense GEMM
(`VLLM_CPP_MLX=on`, the default here). It is on because upstream now SHAPE-GATES
it to prefill; it was briefly off in this branch's history, and that was correct
at the time for an ungated provider.
This branch originally shipped it ON, on the strength of an A/B that had MLX at
1.88-2.19x against the native MSL GEMM. That measurement was honest when taken
and is now stale: vllm.cpp's Metal kernels have since improved several-fold
(mma prefill attention, vectorised decode V accumulation, a fused qk-norm-RoPE
preamble and more), so the native path no longer resembles the one MLX was
compared against.
The gate matters more than the flag. MLX's steel GEMM wins prefill but loses
decode, because the provider pays an `mx::eval` synchronisation plus an output
memcpy on every call and decode makes ~112 calls *per token*. Measured on an
Apple M4, Qwen3-1.7B-bf16 warm at p=512 g=128:
Re-measured on the same Apple M4, same binary, arms toggled with
`VT_OP_PROVIDER_DISABLE=mlx`, Qwen3-1.7B-bf16 warm at p=512 g=128:
| | prefill TTFT | warm throughput |
| configuration | prefill TTFT | warm throughput |
|---|--:|--:|
| MLX provider ON | 1370 ms | **11.98 tok/s** |
| MLX provider OFF | 1400 ms | **22.06 tok/s** |
| MLX **gated to prefill** (pin >= 89c46aeb) | **524.5 ms** | **24.40 tok/s — 99.1% of MLX-LM** |
| MLX ungated (older pins) | 537 ms | 12.7 tok/s |
| MLX off | 602 ms | 23.9 tok/s |
MLX's steel GEMM is still ~20% faster than ours in isolation, but the provider
pays a per-op `mx::eval` synchronisation plus an output `memcpy` (it cannot write
into our buffer). On prefill's ~112 GEMMs that overhead leaves +2%; on decode,
where the same sync is paid once per matmul per token, it costs 46%.
**`VLLM_CPP_VERSION` and this flag are coupled.** Moving the pin back before
`89c46aeb` while leaving `VLLM_CPP_MLX=on` would take the middle row — roughly
half throughput. If you roll the pin back, roll the default back with it.
Turning it on is therefore only sensible for prefill-dominated workloads, and
even then the margin is small. Full disposition in vllm.cpp `docs/BENCHMARKS.md`,
"The MLX provider verdict".
One caveat: MLX's GEMM is not bit-identical to the native kernel, so an MLX build
produces a different greedy sequence than a non-MLX one. That is a property of the
provider, not of the gate, and it predates this packaging. Full disposition in
vllm.cpp `docs/BENCHMARKS.md`.
Build knobs:
- `VLLM_CPP_MLX=on` builds the provider in: ~19 MB `libmlx.dylib` plus a ~105 MB
`mlx.metallib`, and currently slower end to end. Off is the default.
- `VLLM_CPP_MLX=off` builds Metal without the provider: ~124 MB smaller, and
96.4% of MLX-LM instead of 99.1%.
- `MLX_VERSION` pins the wheel (default `0.29.3`). MLX is consumed as the
prebuilt pip wheel because building it from source needs `xcrun metal`, i.e. a
full Xcode the macOS runners do not have.