mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-01 11:00:24 -04:00
be09de0692
This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was correct when taken and is now stale: vllm.cpp's own Metal kernels have improved several-fold since, through mma prefill attention, a vectorised decode V accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a simdgroup-per-row softmax. The native path MLX was compared against no longer exists. Re-measured on the same Apple M4, in the same binary, with the arms toggled by VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128: MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s Shipping the previous default would have halved Apple Silicon throughput. MLX's steel GEMM is still about 20% faster than ours in isolation, but the provider pays a per-op mx::eval synchronisation plus an output memcpy, because it cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead leaves a 2% gain; on decode, where the same synchronisation is paid once per matmul per token, it costs 46%. The option is kept for prefill-dominated workloads, where the margin is small but real. The README section is rewritten rather than patched: it previously presented the stale table as the reason for the default, so leaving it in place would have made the new default look arbitrary. Assisted-by: Claude Code:claude-opus-5 [ClaudeCode] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>