bench(dense): FORCE_CUBLAS no-op for dense too (720.8 vs 721.8) - every flag lever exhausted

Confirms parity (dense+MoE, both phases) is strictly the FP4 tensor-core kernel;
no config/flag shortcut remains.

Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto
2026-06-20 03:59:27 +00:00
parent ce60737fc5
commit 19742aee64

View File

@@ -73,6 +73,10 @@ weight-quant comparison; the difference is the compute kernel.
3. **Scope decision (the reason for this benchmark): the Lever-3 kernel track must also deliver a NON-grouped
block-scaled FP4 GEMM for dense**, not only the MoE grouped GEMM. The dense GEMM is the simpler of the two
(a plain CUTLASS dense GEMM), so it's a good first kernel to land — and it benefits every dense model.
- **No cheap lever:** `GGML_CUDA_FORCE_CUBLAS` is a **no-op for dense too** (Q4_K pp512: 720.8 vs 721.8) —
dequant→cuBLAS-BF16 doesn't engage / isn't faster than int8-MMQ on GB10. With ubatch (saturates) and
nwarps (static_assert) already ruled out for MoE, **every config/flag lever is now exhausted** for both
model classes. Parity is strictly the FP4 tensor-core kernel.
4. **Aside:** full NVFP4 (W4A4) is currently unusable for dense on this vLLM/GB10 build — worth revisiting
on a newer vLLM, and a point in llama.cpp's favor (its 4-bit dense path at least *runs*).