diff --git a/backend/go/vllm-cpp/Makefile b/backend/go/vllm-cpp/Makefile index 8795f621d..e9b0fc06c 100644 --- a/backend/go/vllm-cpp/Makefile +++ b/backend/go/vllm-cpp/Makefile @@ -11,18 +11,25 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e # vllm.cpp version VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp -VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845 +VLLM_CPP_VERSION?=eec09bed5a03457837b499781c23d8e44f106813 # MLX GEMM provider (darwin/metal only; see the metal branch below for why). # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun # metal`, i.e. a full Xcode the macOS runners do not have, while the wheel ships # include/, lib/libmlx.dylib and the compiled mlx.metallib ready to link. # -# DEFAULT OFF. It was on when this branch opened, on the strength of an A/B that -# had MLX at 1.88x. That measurement is stale: vllm.cpp's own Metal kernels have -# since improved several-fold, and re-measured on the same M4 the provider is now -# 46% SLOWER end to end (11.98 vs 22.06 warm tok/s). See the README. -VLLM_CPP_MLX?=off +# DEFAULT ON, but ONLY because VLLM_CPP_VERSION above is pinned at or past +# vllm.cpp 89c46aeb, which SHAPE-GATES the provider to prefill. The ordering is +# load-bearing, not incidental: +# +# pin >= 89c46aeb, MLX on -> 99.1% of MLX-LM (gated: prefill only) +# pin < 89c46aeb, MLX on -> ~51% (ungated: it also takes decode) +# +# MLX's steel GEMM wins prefill (537 ms TTFT against 602) and loses decode badly, +# because the provider pays an mx::eval sync plus an output memcpy per call and +# decode makes ~112 calls per TOKEN. Ungated it does both; gated it does only the +# good half. So if this pin is ever moved BACKWARDS, this default must go with it. +VLLM_CPP_MLX?=on MLX_VERSION?=0.29.3 MLX_VENV?=$(abspath ./mlx-venv) # Resolved lazily (recursive `=`, not `:=`): the glob only matches once the venv diff --git a/backend/go/vllm-cpp/README.md b/backend/go/vllm-cpp/README.md index 1589b1ff8..3755340d4 100644 --- a/backend/go/vllm-cpp/README.md +++ b/backend/go/vllm-cpp/README.md @@ -41,39 +41,37 @@ options: - max_num_seqs:16 ``` -## Apple Silicon: the MLX GEMM provider (OFF by default) +## Apple Silicon: the MLX GEMM provider (ON by default, gated to prefill) -`BUILD_TYPE=metal` can build vllm.cpp's optional MLX provider for the dense GEMM -(`VLLM_CPP_MLX=on`). **It is OFF by default, because it is currently slower.** +`BUILD_TYPE=metal` builds vllm.cpp's MLX provider for the dense GEMM +(`VLLM_CPP_MLX=on`, the default here). It is on because upstream now SHAPE-GATES +it to prefill; it was briefly off in this branch's history, and that was correct +at the time for an ungated provider. -This branch originally shipped it ON, on the strength of an A/B that had MLX at -1.88-2.19x against the native MSL GEMM. That measurement was honest when taken -and is now stale: vllm.cpp's Metal kernels have since improved several-fold -(mma prefill attention, vectorised decode V accumulation, a fused qk-norm-RoPE -preamble and more), so the native path no longer resembles the one MLX was -compared against. +The gate matters more than the flag. MLX's steel GEMM wins prefill but loses +decode, because the provider pays an `mx::eval` synchronisation plus an output +memcpy on every call and decode makes ~112 calls *per token*. Measured on an +Apple M4, Qwen3-1.7B-bf16 warm at p=512 g=128: -Re-measured on the same Apple M4, same binary, arms toggled with -`VT_OP_PROVIDER_DISABLE=mlx`, Qwen3-1.7B-bf16 warm at p=512 g=128: - -| | prefill TTFT | warm throughput | +| configuration | prefill TTFT | warm throughput | |---|--:|--:| -| MLX provider ON | 1370 ms | **11.98 tok/s** | -| MLX provider OFF | 1400 ms | **22.06 tok/s** | +| MLX **gated to prefill** (pin >= 89c46aeb) | **524.5 ms** | **24.40 tok/s — 99.1% of MLX-LM** | +| MLX ungated (older pins) | 537 ms | 12.7 tok/s | +| MLX off | 602 ms | 23.9 tok/s | -MLX's steel GEMM is still ~20% faster than ours in isolation, but the provider -pays a per-op `mx::eval` synchronisation plus an output `memcpy` (it cannot write -into our buffer). On prefill's ~112 GEMMs that overhead leaves +2%; on decode, -where the same sync is paid once per matmul per token, it costs 46%. +**`VLLM_CPP_VERSION` and this flag are coupled.** Moving the pin back before +`89c46aeb` while leaving `VLLM_CPP_MLX=on` would take the middle row — roughly +half throughput. If you roll the pin back, roll the default back with it. -Turning it on is therefore only sensible for prefill-dominated workloads, and -even then the margin is small. Full disposition in vllm.cpp `docs/BENCHMARKS.md`, -"The MLX provider verdict". +One caveat: MLX's GEMM is not bit-identical to the native kernel, so an MLX build +produces a different greedy sequence than a non-MLX one. That is a property of the +provider, not of the gate, and it predates this packaging. Full disposition in +vllm.cpp `docs/BENCHMARKS.md`. Build knobs: -- `VLLM_CPP_MLX=on` builds the provider in: ~19 MB `libmlx.dylib` plus a ~105 MB - `mlx.metallib`, and currently slower end to end. Off is the default. +- `VLLM_CPP_MLX=off` builds Metal without the provider: ~124 MB smaller, and + 96.4% of MLX-LM instead of 99.1%. - `MLX_VERSION` pins the wheel (default `0.29.3`). MLX is consumed as the prebuilt pip wheel because building it from source needs `xcrun metal`, i.e. a full Xcode the macOS runners do not have.