LocalAI/backend/cpp/llama-cpp/patches/paged/FUTURE_LEVERS.md

Decode-Parity: Parked Levers (future exploration)

Context. The bit-exact decode-parity effort shipped patches 0018-0023: dense decode 38% -> 95% of vLLM @npl128 on GB10 / DGX Spark (LPDDR5x ~273 GB/s), every patch byte-identical to llama's own f32 output (md5-gated). The gated-DeltaNet recurrence (the dominant ~50% kernel) now runs at 84.6% of peak BW = past vLLM's 82.4%, at the DRAM floor. bf16 SSM state was fully built and shelved (real +25-31% lever but fails the f32 KL gate).

The remaining non-recurrence kernels (FP4 GEMM, attention, lm_head) are at their bit-exact floor: any knob changes a reduction order vs the f32 reference. So further bit-exact decode gains are marginal; the levers below are the honest pick-up points, ranked by promise.

---

1. Hybrid-precision SSM state (the most promising)

The bf16 build (BF16_SSM_STATE_RESULTS.md) proved the throughput lever is large - recurrence -49%/call (dense 3.38 -> 1.73 ms), dense decode ~490 t/s = 125% of vLLM (clean runs), MoE @128 +24.9% - but bf16 fails the f32 KL gate (KLD 0.06-0.17 at >=1024 ctx, ~10% argmax flips). The discrimination showed the error is intrinsic to bf16 over the long-memory heads (exp(g) ~ 1, where the per-step decay does not contract the rounding); short/fast-decaying heads are fine.

Lever: a per-head (or per-channel) precision split - keep the long-memory heads (g near 1) in f32, store the fast-decaying heads (g well below 1, where rounding contracts) in bf16. Could capture most of the speedup while passing the KL gate. Needs a g-magnitude classifier at graph build + a mixed-dtype recurrent-state cache. HIGH promise, moderate effort. The bf16 kernel plumbing already exists (DGX ~/llama-paged-dev/BF16_SSM_STATE.diff); this adds the per-head dtype selection on top.

Note (precision, corrected): plain bf16 (no split) is a legitimate opt-in for precision-tolerant deployments, but it is below vLLM's recurrent precision, NOT equal to it. vLLM keeps the gated-DeltaNet temporal state in f32 (proven three ways in BITEXACT_VS_VLLM.md; only its tiny conv state is bf16, and llama keeps even that f32). So bf16 here trades below-vLLM precision for above-vLLM throughput. We declined it as the default because both llama's f32 AND vLLM's f32 are a higher bar - and at equal f32 precision llama's recurrence already beats vLLM (84.6% vs 82.4% peak BW), so we do not need bf16 to match vLLM's recurrence.

2. Dense CUDA-graph instability

The bf16 dense decode was bimodal across runs (287 / 336 / 487 / 498 t/s) - a dense-path CUDA-graph capture/replay instability (good runs hit ~490). The f32 dense path measured stable (371-376) but the bimodality is a latent fragility worth root-causing; a robust graph capture on the dense path could stabilize and possibly lift dense decode. Moderate promise, diagnostic.

3. Dense rms_norm -> fp4 producer-fold (~1.5-2.5%, parked as flat-risk)

The last bit-exact bucket (RMSNORM_FP4_FOLD.md). Folding the standalone quantize_mmq_nvfp4 into the rms_norm+mul producer at the FFN boundary (f32 output dead -> droppable) could recover ~1.5-2.5% dense. Parked because: the Lever-2 precedent measured flat, it has the worst gain/plumbing ratio (3-op {RMS_NORM,MUL,MUL_MAT(NVFP4)} graph fusion + a pre-quantized-src1 GEMM path + scratch-pool / CUDA-graph-lifetime plumbing), and the gain risks being swallowed by the ~0.3-0.5% bench noise floor. Revisit only with the inter-node graph-CSE plumbing built and proven on a same-build flag toggle (decode_agg lift above noise AND md5 == 0023). LOW promise.

4. Datacenter Blackwell (sm_100)

This effort targeted consumer Blackwell sm_12x (sm_120 RTX 50-series, sm_121 GB10). Datacenter Blackwell (B100/B200/GB200, sm_100 / cc 10.0) has HBM3e (much higher BW) and different MMA characteristics - the LPDDR5x bandwidth floor that dominates GB10 decode does not apply, so the whole calculus changes (likely compute-bound, not BW-bound; the recurrence would not be the binding kernel). A separate investigation if datacenter Blackwell becomes a target.

5. Prefill / TTFT scheduler + paged-pool burst degradation (HIGH priority - the weakest benchmark number)

The final benchmark (QWEN36_NVFP4_BENCH.md) exposed TTFT as the clear weak spot vs vLLM. Two distinct issues:

• Static decode-first budget tradeoff: the QoS budget (patches 0013/0016, LLAMA_MAX_BATCH_TOKENS=512) maximizes decode tok/s + memory but throttles burst-prefill, so under a synchronized 128-way burst TTFT climbs to 903 s dense / 213 s MoE @npl128 vs vLLM's chunked-prefill 6-18 s. A dynamic/adaptive budget (by concurrency + queue depth), or matching vLLM's chunked-prefill interleave, would rebalance.
• Paged-pool burst-degradation BUG (concrete, found in the benchmark): after a high-npl burst, a server's subsequent lower-npl prefill collapses (fresh npl8 = 507 t/s / 6 s TTFT; npl8 after an npl64 burst = 65 t/s / 64 s). Decode stays robust; only prefill degrades -> root-cause the paged-pool state that persists across the burst.

HIGH promise for the serving experience: decode (dense 90-117%, MoE 77-83% of vLLM) and memory (1.5-3x lower) are already strong; TTFT is the one number holding back a clean public win.

6. MoE-specific recurrence tuning

The occupancy retune (0022) was tuned on the dense path; it lifted MoE +8.3% as a side effect. The MoE path (MUL_MAT_ID grouped GEMM + the shared GDN recurrence, expert routing changes the GEMM shapes) may have MoE-specific occupancy headroom. Worth a MoE-targeted reprofile.

---

All shelved per the host handover - experiments parked. Pick up from the linked result docs in this directory.