mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00
Code Issues Packages Projects Releases Wiki Activity

docs(vllm-cpp): correct the MLX-gated figure to 97.6%, from 99.1%

Browse Source 
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That
figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s
averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks,
MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was
an outlier and averaging it in overstated us by roughly 1.5 points.

Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the
MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms
against MLX-LM's 532.6, so we remain about 1.5% faster there.

Nothing else changes. MLX still wins prefill and loses decode, the shape gate is
still the right disposition, and the pin and the flag are still coupled. The gate
is worth about 1.7 points over the MLX-off build rather than 2.7.

Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto
2026-07-28 08:41:18 +00:00
committed by localai-org-maint-bot
parent d49738b59c
commit 927206ed53
1 changed files with 7 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
backend/go/vllm-cpp/README.md
View File

@@ -55,9 +55,14 @@ Apple M4, Qwen3-1.7B-bf16 warm at p=512 g=128:
| configuration | prefill TTFT | warm throughput |
|---|--:|--:|
| MLX **gated to prefill** (pin >= 89c46aeb) | **524.5 ms** | **24.40 tok/s — 99.1% of MLX-LM** |
| MLX **gated to prefill** (pin >= 89c46aeb) | **524.5 ms** | **24.37 tok/s, 97.6% of MLX-LM** |
| MLX ungated (older pins) | 537 ms | 12.7 tok/s |
| MLX off | 602 ms | 23.9 tok/s |
| MLX off | 602 ms | 23.9 tok/s, 95.9% |
Ratios are against an MLX-LM baseline measured INTERLEAVED with ours over four
ABBA blocks (its spread 0.34%, ours 0.12%). An earlier revision of this file
claimed 99.1%; that used a two-run MLX-LM baseline containing an outlier and
overstated us by about 1.5 points.
**`VLLM_CPP_VERSION` and this flag are coupled.** Moving the pin back before
`89c46aeb` while leaving `VLLM_CPP_MLX=on` would take the middle row — roughly