diff --git a/backend/cpp/llama-cpp/patches/paged/GDN_DECODE_VERIFY.md b/backend/cpp/llama-cpp/patches/paged/GDN_DECODE_VERIFY.md new file mode 100644 index 000000000..933593cea --- /dev/null +++ b/backend/cpp/llama-cpp/patches/paged/GDN_DECODE_VERIFY.md @@ -0,0 +1,208 @@ +# GDN decode verify: is llama.cpp's Gated-Delta-Net decode O(1) or an O(ctx) re-scan? + +Verdict-first, then the evidence. This closes lever 5 of `VLLM_DECODE_GROUNDING.md` ("Verify +llama's GDN/linear-attention decode path"): on the Qwen3.6 hybrid models, is llama re-scanning the +context (O(ctx)) in the linear-attention layers, or keeping vLLM's O(1)-in-context recurrent state? + +Method: GGUF-metadata + source reading on the `paged` dev tree (`~/llama-paged-dev`, build-cuda +sm_121) on `dgx.casa`, plus nsys CUDA-kernel decode traces on `~/bench/q36-27b-nvfp4.gguf` +(GB10 / DGX Spark, `GGML_CUDA_DISABLE_GRAPHS=1`, paged KV, `-fa on`). Models: +`~/bench/q36-27b-nvfp4.gguf` (dense, arch `qwen35`), `~/bench/q36-35b-a3b-nvfp4.gguf` +(MoE, arch `qwen35moe`). + +## TL;DR verdict + +**llama.cpp's GDN decode is EFFICIENT: it is O(1)-in-context, a single fused CUDA kernel that +reads + updates a fixed-size cached recurrent state, structurally identical to vLLM's +`fused_recurrent_gated_delta_rule`. It is NOT a re-scan, NOT a context-scaling blowup, and NOT a +major contributor to the ~2.4x eager-decode gap.** There is no GDN-specific bottleneck to fix, so +the cheap model-specific lever this probe was hunting for does not exist. The 2.4x is the general +kernel work (the FP4 weight GEMM, which dominates the step, plus the O(ctx) full-attention decode +kernel in the minority of full-attention layers), exactly as `VLLM_DECODE_GROUNDING.md` concluded. + +The decisive datum: at matched batch (npl4), pure decode, 4x more context, the GDN kernel time is +**flat** while the full-attention kernel grows ~3.1x: + +| kernel | ctx 1024 | ctx 4096 | ratio | meaning | +|--------|---------:|---------:|------:|---------| +| `gated_delta_net_cuda` (GDN linear-attn) | 10.3 us/launch | 8.0 us/launch | **~1.0x (flat)** | **O(1) in ctx** | +| `flash_attn_tile` (full-attn layers) | 27.1 us/launch | 85.0 us/launch | **3.1x** | O(ctx), as expected | +| total ms / decode step | 84.9 | 86.0 | 1.01x | GEMM-bound, ctx-independent | + +Identical decode-step counts in both windows (~190 steps, ~9134 GDN launches), so this is a +per-step like-for-like comparison: the GDN layers do **not** get more expensive as context grows. + +## 1. Architecture (confirmed from GGUF metadata + tensor names) + +Both Qwen3.6 models are hybrid: a `full_attention_interval` of 4 means every 4th layer is standard +full attention and the other 3/4 are Gated-Delta-Net (GDN) linear attention with a recurrent state. + +**Dense Qwen3.6-27B (`general.architecture = qwen35`):** +- `block_count = 64`, `full_attention_interval = 4` -> **16 full-attention layers + 48 GDN layers**. +- Full-attn: `head_count = 24`, `head_count_kv = 4` (GQA), `key_length = value_length = 256`, + rope `freq_base = 1e7`, mrope sections `[11,11,10,0]`. +- GDN/SSM: `ssm.state_size = 128`, `ssm.conv_kernel = 4`, `ssm.group_count = 16`, + `ssm.time_step_rank = 48`, `ssm.inner_size = 6144`. So the recurrent state per GDN layer is + `[S_v=128, S_v=128, H_v=48]` per sequence (`H_v = inner_size/state_size = 6144/128 = 48` value + heads), i.e. a 128x128 state matrix per head, ~3.1 MB (F32) per sequence per layer. + +**MoE Qwen3.6-35B-A3B (`general.architecture = qwen35moe`):** +- `block_count = 41`, `full_attention_interval = 4` (~10 full-attn + ~31 GDN layers). +- `head_count = 16`, `head_count_kv = 2`, `key_length = value_length = 256`, + `expert_count = 256`, `expert_used_count = 8`, `expert_feed_forward_length = 512`. +- Same SSM dims: `state_size = 128`, `conv_kernel = 4`, `group_count = 16`, + `inner_size = 4096` -> `H_v = 32` value heads. + +**Tensor names confirm the op split (27B, per-layer dump):** +- GDN layers (e.g. `blk.0.*`): `ssm_alpha`, `ssm_beta`, `ssm_conv1d`, `ssm_a`, `ssm_dt.bias`, + `ssm_norm`, `ssm_out`, plus `attn_qkv` / `attn_gate` (the in/out projections of the linear-attn + block). No `attn_k/v/output`, no per-head q/k norm. +- Full-attn layers (e.g. `blk.3.*`, every 4th): `attn_q`, `attn_k`, `attn_v`, `attn_output`, + `attn_q_norm`, `attn_k_norm`. No `ssm_*`. + +llama loads the GDN layers through the **recurrent memory** (`llama-memory-recurrent`), not the KV +cache: the conv state and the SSM state live in `conv_states_all` / `ssm_states_all` and are read +and written every step. Only the 16/10 full-attention layers use the (paged) KV cache. This is the +SSM-style recurrent path, not standard attention. + +## 2. llama.cpp GDN decode implementation: O(1) recurrent-state update (code-proven) + +Graph build (shared by both models): `src/models/delta-net-base.cpp`, dispatched from +`src/models/qwen35.cpp` and `src/models/qwen35moe.cpp` (the MoE class inherits +`llm_build_delta_net_base` and calls the same `build_recurrent_attn`, qwen35moe.cpp:472). + +**Decode dispatch (`build_delta_net`, delta-net-base.cpp:425-447):** when `n_seq_tokens == 1` +(decode), it takes `build_delta_net_fused` if `cparams.fused_gdn_ar` (the default, see below), else +`build_delta_net_autoregressive`. Both are O(1): + +- `build_delta_net_autoregressive` (delta-net-base.cpp:289-371) is the explicit rank-1 recurrence on + the fixed-size state `s` shaped `[S_v, S_v, H_v, n_seqs]`: `s *= exp(g)` (decay), + `sk = sum_rows(s * k)`, `d = (v - sk^T) * beta`, `s += k (x) d^T` (rank-1 update), + `o = sum_rows(s * q)`. **No loop over past tokens, no KV read** - it touches only the state and + the single new token's q/k/v/g/beta. `GGML_ASSERT(n_tokens == 1)`. +- `build_delta_net_fused` (delta-net-base.cpp:373-423) collapses the same recurrence into one op, + `ggml_gated_delta_net(q, k, v, g, b, s, K=1)`. + +**State is cached across steps, not rebuilt (`build_recurrent_attn`, delta-net-base.cpp:527-606):** +the input state `s` is read from `ssm_states_all` via `build_rs`, and the new state is copied back +with `ggml_cpy(new_state, view(ssm_states_all, ... kv_head ...))` (lines 555-558). The causal-conv +state is handled the same way in `build_conv_state` (449-525): the previous `conv_kernel-1 = 3` +samples are read from `conv_states_all`, the new token is appended, and the last 3 are written back. +So both pieces of GDN state persist in the recurrent cache exactly like a KV cache persists tokens - +this is the recurrent analogue, fixed size, independent of context length. + +**Defaults (`src/llama-context.cpp:200-201`):** `cparams.fused_gdn_ar = true` and +`fused_gdn_ch = true`. They are only auto-disabled if the fused op cannot be scheduled on the same +device as the layer (`device_gdn != device_kv`, lines 540-595); on a single GB10 with `-ngl 99` +that does not happen, so the **fused single-kernel path is what runs**. + +**The CUDA kernel (`ggml/src/ggml-cuda/gated_delta_net.cu`) is the crux, and it is unambiguously +O(1) in context:** +- Launch grid `dim3(H, n_seqs, ceil(S_v/4))` and block `(min(warp,S_v), 4, 1)` (lines 184-185): + the grid spans heads x sequences x state-columns. **There is no context-length dimension and no + context-length argument anywhere in the kernel signature** (q/k/v/g/beta are the new token(s) + `[S_v, H, n_tokens, n_seqs]`; `curr_state` is the fixed `[S_v, S_v, H, n_seqs]`). +- Each warp loads its shard of the fixed-size state into registers **once** (lines 57-61), then + loops `for (t = 0; t < n_tokens; t++)` (line 63). At decode `n_tokens == 1`, so it is a single + iteration: read the one new token, do the rank-1 update + `s_shard[r] = g * s_shard[r] + k[i] * delta_col` and the readout `attn = S^T q` (lines 84-141), + then write the updated state back (lines 161-167). No second loop, no read of any past KV. +- Work per decode step is therefore proportional to `S_v * S_v * H * n_seqs` (the state size x + batch) and **constant in context length**. This is precisely vLLM's + `fused_recurrent_gated_delta_rule_packed_decode_kernel` (one batched launch updating a + fixed-size `[K,V]` state) cited in the grounding doc. + +A chunked GPU kernel for prefill is a TODO (delta-net-base.cpp:181 `//TODO: Add chunked kernel`); +the chunked CPU/graph path (`build_delta_net_chunking`) only runs for multi-token ubatches +(prefill), never at decode. + +## 3. nsys decode profiling: GDN is a small share and does not scale with context + +Qwen3.6-27B NVFP4, sm_121, `GGML_CUDA_DISABLE_GRAPHS=1`, paged KV, `-fa on`, `llama-server` driven +to steady decode by a looping completion client. Kernel time bucketed by name (full classifier and +sqlites under `~/bench/gdn_study/`). + +**(a) Share at the headline batch (npl128, ctx 1024), GPU 92.7% busy:** + +| bucket | % of busy | us/launch | +|--------|----------:|----------:| +| GEMM_weight (`mul_mat_q`/`mul_mat_vec_q`) | 59.2 | - | +| **GDN_recurrent (`gated_delta_net_cuda`)** | **8.9** | 369 | +| GEMM_act_quant (`quantize_mmq_nvfp4`) | 8.2 | - | +| elementwise / act_glu / norm / rope | ~13.5 | - | +| embed_gather (`get_rows`) | 2.9 | - | +| **ATTENTION_full (`flash_attn`, 16 layers)** | **1.8** | 107 | +| copy_cast (`cpy`) | 1.8 | - | +| **GDN_conv (`ssm_conv`)** | **1.5** | - | + +The whole GDN path (recurrent 8.9% + conv 1.5%) is ~10% of the step; full attention is ~2%; the +**weight GEMM dominates at ~67% (59.2% GEMM + 8.2% act-quant requant)**. This is the dense model, +where the grounding predicted the GEMM would be the lever. + +**(b) Share at low batch (npl32, ctx 1024), weight-bandwidth (GEMV) regime, GPU ~100%:** +GEMM_weight 88.7%, GDN_recurrent 0.8%, ATTENTION_full 0.7%, GDN_conv 0.3%. At low batch the +weight-read GEMV swamps everything and GDN is negligible; the GDN share tracks the batch, not the +context. + +**(c) Context-scaling control (the decisive test): matched batch npl4, pure decode, ctx 1024 vs +4096.** Small batch -> fast prefill -> a clean pure-decode capture (verified: GEMM is the M=1 +`mul_mat_vec_q` decode GEMV, and the client completed decode rounds inside the window). Identical +decode-step counts (~190 steps, gated_delta_net launched 9141 vs 9134 times), so per-launch time is +a true per-step comparison: + +| kernel / bucket | ctx 1024 | ctx 4096 | ratio | +|-----------------|---------:|---------:|------:| +| `gated_delta_net_cuda` us/launch | 10.3 | **8.0** | **0.78x (flat)** | +| GDN_recurrent share | 0.6% | 0.4% | flat/down | +| `ssm_conv` (GDN_conv) us/launch | 5.2 | 5.2 | 1.00x | +| `flash_attn_tile` us/launch | 27.1 | **85.0** | **3.14x** | +| ATTENTION_full share | 0.6% | 1.8% | 3.0x up | +| total ms / decode step | 84.9 | 86.0 | 1.01x | + +The GDN kernel time is flat (even a hair faster) across a 4x context increase, while the +full-attention kernel grows ~3x, exactly the O(1)-vs-O(ctx) signature. The total step time barely +moves because at this batch the (context-independent) FP4 weight GEMM is 88% of the step. This is +the empirical confirmation of the code analysis: **llama's GDN decode does not re-scan the context.** + +(An earlier npl32 ctx4096 attempt was discarded: with 32 parallel slots each independently +prefilling ~4100 tokens, the nsys window caught prefill, not steady decode - the `mul_mat_q(M=128)` ++ `flash_attn_ext_f16(ctx4096)` signature gave it away. The npl4 runs above avoid this by keeping +prefill short.) + +## 4. Verdict and fix scope + +**Efficient, not a bottleneck.** llama.cpp runs the Qwen3.6 GDN/linear-attention layers as a fused, +single-CUDA-kernel, O(1)-in-context recurrent-state update, with the conv and SSM state cached in +the recurrent memory across decode steps. It is algorithmically the same as vLLM's O(1) +`fused_recurrent` decode. The probe's worst case (llama re-scanning context => GDN layers ballooning +with context and concurrency) is **falsified**: the GDN kernel is flat across 4x context, and the +op carries no context-length parameter at all. + +**So the GDN path is not the cheap model-specific lever.** It is a small-to-moderate, context-flat +share of the step (~0.4-0.8% at low batch, ~10% including conv at batch 128), and removing it would +not dent the 2.4x. The gap is the general kernel work, confirming `VLLM_DECODE_GROUNDING.md`: +1. the **FP4 weight GEMM** is the dominant bucket (~59% GEMM + ~8% `quantize_mmq_nvfp4` requant that + vLLM fuses away via native FP4-MMA / grouped Marlin); this is the biggest, hardest lever. +2. the **full-attention decode kernel** is the O(ctx) residual (the only thing that grows with + context, ~3x per-launch over 4x ctx), in the minority of full-attention layers. + +If anything on the GDN side is ever worth touching, it is a bounded micro-optimization, not a +complexity fix: the kernel is memory-bound on the F32 recurrent state (state read+write is +`S_v^2 * H * batch` = ~0.79 GB/step over 273 GB/s at batch 128, hence the ~8.9% share), and this +traffic is **intrinsic to the architecture - vLLM pays the identical state I/O**, so it is not a +llama-specific inefficiency. A future win could keep the recurrent state in bf16 or fuse the +`ssm_conv` + gated-norm into the delta-net kernel to shave that ~10%, but the ceiling is small and +it does not close the 2.4x. The throughput effort stays where the grounding put it: the FP4 GEMM +(fused act-quant + native FP4-MMA) and the full-attention decode kernel, with a CUDA-graphed +steady-state step as the bounded host-side add-on. + +## Reproduce + +- Metadata: `python3 gguf-py/gguf/scripts/gguf_dump.py --no-tensors ~/bench/q36-27b-nvfp4.gguf`. +- Code: `src/models/delta-net-base.cpp` (build_delta_net 425, autoregressive 289, fused 373, + build_recurrent_attn 527, build_conv_state 449); `src/llama-context.cpp:200-201,540-595` + (fused_gdn defaults/guard); `ggml/src/ggml-cuda/gated_delta_net.cu` (kernel 4-168, launch grid + 184-185, dispatch 226-312). +- Profiles: `~/bench/gdn_study/drv.sh