From 1556b08d810abd6f78c16c046490eb8aaddf394a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Mon, 27 Jul 2026 21:45:33 +0000 Subject: [PATCH] fix(vllm-cpp): default the MLX GEMM provider OFF on darwin This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was correct when taken and is now stale: vllm.cpp's own Metal kernels have improved several-fold since, through mma prefill attention, a vectorised decode V accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a simdgroup-per-row softmax. The native path MLX was compared against no longer exists. Re-measured on the same Apple M4, in the same binary, with the arms toggled by VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128: MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s Shipping the previous default would have halved Apple Silicon throughput. MLX's steel GEMM is still about 20% faster than ours in isolation, but the provider pays a per-op mx::eval synchronisation plus an output memcpy, because it cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead leaves a 2% gain; on decode, where the same synchronisation is paid once per matmul per token, it costs 46%. The option is kept for prefill-dominated workloads, where the margin is small but real. The README section is rewritten rather than patched: it previously presented the stale table as the reason for the default, so leaving it in place would have made the new default look arbitrary. Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] Signed-off-by: Ettore Di Giacinto --- backend/go/vllm-cpp/Makefile | 7 +++++- backend/go/vllm-cpp/README.md | 45 +++++++++++++++++++++-------------- 2 files changed, 33 insertions(+), 19 deletions(-) diff --git a/backend/go/vllm-cpp/Makefile b/backend/go/vllm-cpp/Makefile index 62d99396c..8795f621d 100644 --- a/backend/go/vllm-cpp/Makefile +++ b/backend/go/vllm-cpp/Makefile @@ -17,7 +17,12 @@ VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845 # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun # metal`, i.e. a full Xcode the macOS runners do not have, while the wheel ships # include/, lib/libmlx.dylib and the compiled mlx.metallib ready to link. -VLLM_CPP_MLX?=on +# +# DEFAULT OFF. It was on when this branch opened, on the strength of an A/B that +# had MLX at 1.88x. That measurement is stale: vllm.cpp's own Metal kernels have +# since improved several-fold, and re-measured on the same M4 the provider is now +# 46% SLOWER end to end (11.98 vs 22.06 warm tok/s). See the README. +VLLM_CPP_MLX?=off MLX_VERSION?=0.29.3 MLX_VENV?=$(abspath ./mlx-venv) # Resolved lazily (recursive `=`, not `:=`): the glob only matches once the venv diff --git a/backend/go/vllm-cpp/README.md b/backend/go/vllm-cpp/README.md index cd6eb94a0..1589b1ff8 100644 --- a/backend/go/vllm-cpp/README.md +++ b/backend/go/vllm-cpp/README.md @@ -41,30 +41,39 @@ options: - max_num_seqs:16 ``` -## Apple Silicon: the MLX GEMM provider +## Apple Silicon: the MLX GEMM provider (OFF by default) -`BUILD_TYPE=metal` builds the Metal backend with vllm.cpp's optional MLX -provider for the dense GEMM (`VLLM_CPP_MLX=on`, the default here). Upstream keeps -it off because it costs a ~19 MB `libmlx.dylib` plus a ~105 MB `mlx.metallib`; -this backend accepts that because the provider was measured to pay for it on an -Apple M4, against the native MSL GEMM in the SAME binary (arms toggled with -`VT_OP_PROVIDER_DISABLE=mlx`), Qwen3-1.7B-bf16 at p=512 g=128: +`BUILD_TYPE=metal` can build vllm.cpp's optional MLX provider for the dense GEMM +(`VLLM_CPP_MLX=on`). **It is OFF by default, because it is currently slower.** -| Concurrency | MLX agg tok/s | native agg tok/s | speedup | -|--:|--:|--:|--:| -| 1 | 5.79 | 3.08 | 1.88x | -| 8 | 25.70 | 13.69 | 1.88x | -| 16 | 38.65 | 17.69 | 2.19x | +This branch originally shipped it ON, on the strength of an A/B that had MLX at +1.88-2.19x against the native MSL GEMM. That measurement was honest when taken +and is now stale: vllm.cpp's Metal kernels have since improved several-fold +(mma prefill attention, vectorised decode V accumulation, a fused qk-norm-RoPE +preamble and more), so the native path no longer resembles the one MLX was +compared against. -TTFT improves 2x to 3x, peak memory is unchanged, and the GEMM output is -bit-identical to the native kernel on every parity shape. MLX serves the dense -GEMM only: paged attention stays vllm.cpp's own kernel, because MLX has no -paged-KV primitive. Full disposition in vllm.cpp `docs/BENCHMARKS.md`, -"MLX GEMM provider A/B on Apple M4". +Re-measured on the same Apple M4, same binary, arms toggled with +`VT_OP_PROVIDER_DISABLE=mlx`, Qwen3-1.7B-bf16 warm at p=512 g=128: + +| | prefill TTFT | warm throughput | +|---|--:|--:| +| MLX provider ON | 1370 ms | **11.98 tok/s** | +| MLX provider OFF | 1400 ms | **22.06 tok/s** | + +MLX's steel GEMM is still ~20% faster than ours in isolation, but the provider +pays a per-op `mx::eval` synchronisation plus an output `memcpy` (it cannot write +into our buffer). On prefill's ~112 GEMMs that overhead leaves +2%; on decode, +where the same sync is paid once per matmul per token, it costs 46%. + +Turning it on is therefore only sensible for prefill-dominated workloads, and +even then the margin is small. Full disposition in vllm.cpp `docs/BENCHMARKS.md`, +"The MLX provider verdict". Build knobs: -- `VLLM_CPP_MLX=off` builds Metal without the provider: ~124 MB smaller, slower. +- `VLLM_CPP_MLX=on` builds the provider in: ~19 MB `libmlx.dylib` plus a ~105 MB + `mlx.metallib`, and currently slower end to end. Off is the default. - `MLX_VERSION` pins the wheel (default `0.29.3`). MLX is consumed as the prebuilt pip wheel because building it from source needs `xcrun metal`, i.e. a full Xcode the macOS runners do not have.