From 7c45447c9e9145f9d012d73c10bb895634c80297 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 26 Jun 2026 00:53:09 +0000 Subject: [PATCH] docs(paged): FUTURE_LEVERS - parked decode-parity exploration trail Ranked pick-up points after the 95%-bit-exact plateau: hybrid-precision SSM state (per-head f32/bf16 split - the bf16 error is concentrated in long-memory heads, so a split could capture most of the +25-31% while passing the f32 KL gate), dense CUDA-graph instability, the rms_norm->fp4 fold (flat-risk), datacenter Blackwell sm_100 (no LPDDR5x floor), adaptive prefill budget, MoE-specific recurrence tuning. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto --- .../llama-cpp/patches/paged/FUTURE_LEVERS.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 backend/cpp/llama-cpp/patches/paged/FUTURE_LEVERS.md diff --git a/backend/cpp/llama-cpp/patches/paged/FUTURE_LEVERS.md b/backend/cpp/llama-cpp/patches/paged/FUTURE_LEVERS.md new file mode 100644 index 000000000..86be42f2a --- /dev/null +++ b/backend/cpp/llama-cpp/patches/paged/FUTURE_LEVERS.md @@ -0,0 +1,77 @@ +# Decode-Parity: Parked Levers (future exploration) + +**Context.** The bit-exact decode-parity effort shipped patches **0018-0023**: dense decode +38% -> **95% of vLLM** @npl128 on GB10 / DGX Spark (LPDDR5x ~273 GB/s), every patch +**byte-identical to llama's own f32 output** (md5-gated). The gated-DeltaNet recurrence (the +dominant ~50% kernel) now runs at **84.6% of peak BW = past vLLM's 82.4%**, at the DRAM floor. +bf16 SSM state was fully built and **shelved** (real +25-31% lever but fails the f32 KL gate). + +The remaining non-recurrence kernels (FP4 GEMM, attention, lm_head) are at their bit-exact +floor: any knob changes a reduction order vs the f32 reference. So further *bit-exact* decode +gains are marginal; the levers below are the honest pick-up points, ranked by promise. + +--- + +## 1. Hybrid-precision SSM state (the most promising) + +The bf16 build (`BF16_SSM_STATE_RESULTS.md`) proved the throughput lever is large - +recurrence **-49%/call** (dense 3.38 -> 1.73 ms), dense decode ~**490 t/s = 125% of vLLM** (clean +runs), MoE @128 **+24.9%** - but bf16 fails the f32 KL gate (KLD 0.06-0.17 at >=1024 ctx, +~10% argmax flips). The discrimination showed the error is **intrinsic to bf16 over the +long-memory heads** (exp(g) ~ 1, where the per-step decay does not contract the rounding); +short/fast-decaying heads are fine. + +**Lever:** a per-head (or per-channel) precision split - keep the long-memory heads (g near 1) +in f32, store the fast-decaying heads (g well below 1, where rounding contracts) in bf16. Could +capture most of the speedup while passing the KL gate. Needs a g-magnitude classifier at graph +build + a mixed-dtype recurrent-state cache. **HIGH promise, moderate effort.** The bf16 kernel +plumbing already exists (DGX `~/llama-paged-dev/BF16_SSM_STATE.diff`); this adds the per-head +dtype selection on top. + +*Note:* plain bf16 (no split) is also a legitimate **opt-in for precision-tolerant deployments** - +it is exactly vLLM's own GDN precision (vLLM's recurrent cache is bf16), so "match vLLM speed at +vLLM precision" is a one-flag away if a user wants it. We declined it as the *default* because our +f32 is a strictly higher bar. + +## 2. Dense CUDA-graph instability + +The bf16 dense decode was **bimodal** across runs (287 / 336 / 487 / 498 t/s) - a dense-path +CUDA-graph capture/replay instability (good runs hit ~490). The f32 dense path measured stable +(371-376) but the bimodality is a latent fragility worth root-causing; a robust graph capture on +the dense path could stabilize and possibly lift dense decode. **Moderate promise**, diagnostic. + +## 3. Dense rms_norm -> fp4 producer-fold (~1.5-2.5%, parked as flat-risk) + +The last bit-exact bucket (`RMSNORM_FP4_FOLD.md`). Folding the standalone `quantize_mmq_nvfp4` +into the rms_norm+mul producer at the FFN boundary (f32 output dead -> droppable) could recover +~1.5-2.5% dense. Parked because: the Lever-2 precedent measured **flat**, it has the worst +gain/plumbing ratio (3-op `{RMS_NORM,MUL,MUL_MAT(NVFP4)}` graph fusion + a pre-quantized-src1 +GEMM path + scratch-pool / CUDA-graph-lifetime plumbing), and the gain risks being swallowed by +the ~0.3-0.5% bench noise floor. Revisit only with the inter-node graph-CSE plumbing built and +proven on a same-build flag toggle (decode_agg lift above noise AND md5 == 0023). **LOW promise.** + +## 4. Datacenter Blackwell (sm_100) + +This effort targeted **consumer** Blackwell sm_12x (sm_120 RTX 50-series, sm_121 GB10). Datacenter +Blackwell (B100/B200/GB200, sm_100 / cc 10.0) has HBM3e (much higher BW) and different MMA +characteristics - the LPDDR5x bandwidth floor that dominates GB10 decode does **not** apply, so the +whole calculus changes (likely compute-bound, not BW-bound; the recurrence would not be the binding +kernel). A separate investigation if datacenter Blackwell becomes a target. + +## 5. Prefill / TTFT scheduler + +The chunked-prefill QoS budget (patches 0013/0016, `LLAMA_MAX_BATCH_TOKENS`) bounds TTFT but uses a +single static default. A **dynamic/adaptive** budget (by concurrency + queue depth) could improve the +TTFT-vs-decode tradeoff at high concurrency. **Moderate promise** for the serving experience (not raw +decode tok/s). + +## 6. MoE-specific recurrence tuning + +The occupancy retune (0022) was tuned on the dense path; it lifted MoE +8.3% as a side effect. The +MoE path (`MUL_MAT_ID` grouped GEMM + the shared GDN recurrence, expert routing changes the GEMM +shapes) may have MoE-specific occupancy headroom. Worth a MoE-targeted reprofile. + +--- + +*All shelved per the host handover - experiments parked. Pick up from the linked result docs in this +directory.*