From 6e0b9102100195241e2aab264eb7b5b1a460b5c2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 21 Jun 2026 21:32:58 +0000 Subject: [PATCH] analysis: decode gap is GPU/kernel-bound, NOT host overhead (corrects premise) Rigorous re-measurement on pr24423: concurrent decode is GPU-compute-bound (~96% util, sampled), CUDA graphs ARE enabled at npl=128 (94/98 calls replay a captured graph; n_kv padded to 256 keeps topology stable), and graphs ON vs OFF is only +1.5% at npl=128. The earlier '20% GPU util / 170ms host' read was a windowing error (whole-run nsys vs decode-windowed). So no host/graph patch helps. The real 547->667 gap is the quantized DECODE GEMM: mul_mat_q (Q4_K/Q6_K) is ~68% of decode GPU time and runs ~2.1x above the GB10 bandwidth floor (poorly tuned for the thin n=128 shape); vLLM's Marlin int4 runs closer. Lever = a Marlin-style int4 decode kernel for K-quants (or a Marlin-friendly int4 serving format), not host work. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto --- .../cpp/llama-cpp/paged/DECODE_OVERHEAD.md | 196 ++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 backend/cpp/llama-cpp/paged/DECODE_OVERHEAD.md diff --git a/backend/cpp/llama-cpp/paged/DECODE_OVERHEAD.md b/backend/cpp/llama-cpp/paged/DECODE_OVERHEAD.md new file mode 100644 index 000000000..e8d7157cd --- /dev/null +++ b/backend/cpp/llama-cpp/paged/DECODE_OVERHEAD.md @@ -0,0 +1,196 @@ +# llama.cpp multi-user decode overhead on DGX Spark (GB10, sm_121) + +Investigation of the Qwen3-32B concurrent-decode throughput gap (llama.cpp ~547 t/s +vs vLLM ~667 t/s) on the GB10 box, build `~/llama.cpp-pr24423/build` (Release, +sm_121, `LLAMA_MAX_SEQ=256`, flash-attn on), model +`~/bench/q3-32b-gguf/Qwen3-32B-Q4_K_M.gguf`. + +## TL;DR (the result overturns the brief's premise) + +On **this** build the prime suspect is wrong and the host-overhead premise does not +hold: + +1. **CUDA graphs are NOT disabled at high concurrency.** At npl=128, 94 of 98 + decode `graph_compute` calls **replay a captured CUDA graph** (0 resets, stable + key, no property churn post-warmup). The keyed-warmup gate works. +2. **There is no ~170ms/step host hotspot here.** The GPU is **~96% active during + decode with graphs ON and ~96% active with graphs OFF**. Decode at npl=128 is + **GPU-compute-bound**, not host-bound. +3. The brief's "20% GPU util / 66ms GPU / 170ms host per step" was measured on a + different/earlier build (mainline without these graph fixes). It is not + reproducible on `llama.cpp-pr24423`. +4. Because the GPU is the bottleneck, re-enabling graphs cannot lift the number: + the clean A/B shows graphs ON vs OFF = **+1.5% at npl=128** (and +2.9% at + npl=32 - the benefit shrinks as concurrency rises and the GPU saturates). +5. The real gap to vLLM is the **quantized decode GEMM kernel**: `mul_mat_q` + (Q4_K + Q6_K) is ~68% of decode GPU time and runs ~2.1x above the GB10 + memory-bandwidth floor. Closing the gap requires Marlin/Machete-style int4 + GEMM kernels, not host-side work. This is a kernel project (the direction the + prior session's uncommitted `marlin-w4a16.cu` / `fp4-grouped-moe.cu` already + started, though those target w4a16/GPTQ-int4, not the K-quants this GGUF uses). + +## 1. Why CUDA graphs are (not) disabled - exact code + measurement + +### The gate (code) + +PR24423 refactored the CUDA-graph path into a keyed, warmup-based scheme in +`~/llama.cpp-pr24423/ggml/src/ggml-cuda/ggml-cuda.cu`: + +- `ggml_cuda_graph_get_key(cgraph)` (~L3343) keys the cached CUDA graph by + `cgraph->nodes[0]` (first-node pointer). +- `ggml_cuda_graph_check_compability(cgraph)` (~L3301) disables graphs only for: + - **split buffers** (`ggml_backend_buft_is_cuda_split`), and + - **`GGML_OP_MUL_MAT_ID`** when `src0` is non-quantized **or** + `ne[2] > get_mmvq_mmid_max(...)` (MoE expert routing needs a stream sync). + Qwen3-32B is **dense** -> no `MUL_MAT_ID` -> this condition never fires. +- `ggml_backend_cuda_graph_compute` (~L4514) warmup gate: a graph is used only + after **2 consecutive calls with no property change** (`warmup_complete`); any + property change resets warmup. `ggml_cuda_graph_update_required` (~L3347) + detects change by `memcmp` of the full `ggml_tensor` struct + per-src + data-ptr/ne/nb, with a fast path when `cgraph->uid` is unchanged. + +### Why it stays enabled across decode steps + +The graph stays stable because llama.cpp's host-side graph reuse holds during +decode, so node pointers/props (and `cgraph->uid`) do not churn: + +- `llama_kv_cache::get_n_kv` (`src/llama-kv-cache.cpp` L1223-1233) **pads n_kv to + a multiple of 256** ("so that the graph remains constant across batches and can + be reused"). For ntg<=256 within the first KV block, n_kv is constant. +- `can_reuse_kq_mask` (`src/llama-graph.cpp` L43) keeps the KQ-mask dims stable: + `ne=[n_kv, n_tokens/n_stream, 1, n_stream]` = `[256,1,1,128]` every decode step + at npl=128. +- `can_reuse` (`src/llama-context.cpp` L1283) therefore returns true, so the + scheduler is **not** reset/re-split. `graph->uid` is only reassigned inside + `ggml_backend_sched_split_graph` (`ggml/src/ggml-backend.cpp` L1033, L1485), + which is skipped on the reuse path -> stable uid -> CUDA graph replays. + +### Measurement (instrumented build, npl=128, ntg=96) + +Env-gated counters added to `ggml_backend_cuda_graph_compute` / +`ggml_cuda_graph_update_required` (since `GGML_LOG_DEBUG` is compiled out in +Release / NDEBUG). End-of-run summary: + +``` +[GTRACE-SUMMARY] calls=98 notenab=0 warming=3 warmdone=1 RESET=0 USED=94 incompat=0 distinct_keys=1 +``` + +94/98 decode `graph_compute` calls **replayed** a captured CUDA graph; **0** +warmup resets; a **single** distinct graph key for the whole decode; no node +property churn after warmup. Graphs are fully engaged at npl=128. + +(The instrumentation was reverted afterwards; the checkout is back to its +pre-task state and the `.so` rebuilt clean.) + +## 2. The per-step CPU "hotspot" - there isn't one on this build + +GPU utilization during npl=128 decode (ntg=256): + +- **Graphs ON** - `nvidia-smi` sampled every 0.7s through the decode phase: + steady **96% GPU util**, SM clock **2184 MHz** (not throttled), 45-47 W. +- **Graphs OFF** (`GGML_CUDA_DISABLE_GRAPHS=1`) - nsys CUDA trace, 8s window: + total GPU kernel time = `3,983,292,128 ns / 0.516` = **~7.72s of the 8s + window = ~96% GPU-active**. Even with every kernel launched individually from + the host, the GPU is still ~96% busy. There are essentially **no host gaps**. + +Per-step wall = 60.6s / 256 steps = **~237 ms/step**, and the sum of one decode +graph's kernel times (nsys, graphs-on capture) is ~244 ms -> GPU kernel time per +step ~= wall time per step. The host work between steps is in the low single-digit +ms (the ~4% idle), consistent with graphs ON giving only +1.5% at npl=128. + +This directly contradicts the brief's 66ms-GPU / 170ms-host split, which must have +come from a pre-graphs build. + +### Per-step GPU breakdown (nsys, npl=128 decode, graphs off, 8s window) + +| Kernel | % GPU time | ~ms/step | +|--------|-----------:|---------:| +| `mul_mat_q` Q4_K (type 12) | 51.6 | ~118 | +| `flash_attn_ext_f16` | 19.3 | ~44 | +| `mul_mat_q` Q6_K (type 14) | 16.2 | ~37 | +| `unary_gated` silu | 4.1 | ~9 | +| mmq stream-k fixup + quantize_q8_1 | ~5 | ~12 | +| rms_norm / rope / set_rows / add | ~4 | ~10 | + +Quantized matmul = **~68%** of decode GPU time (~155 ms/step). Attention ~19%. + +`perf` could not profile the host (kernel `perf_event_paranoid=4`), but it is moot: +the host is ~4% of the wall, so there is no ~170ms host hotspot to chase. + +## 3. Fix attempt + measured result + +### The requested fix (re-enable graphs / pad the decode batch) is a no-op here + +Graphs are already enabled and the batch is already stable (n_kv padded to 256, +kq_mask dims constant). The clean cold A/B (cooldowns between every run): + +| npl | graphs ON (t/s) | graphs OFF (t/s) | delta | +|----:|----------------:|-----------------:|------:| +| 32 | 242.60 | 235.75 | +2.9% | +| 64 | 398.59 | 389.06 | +2.5% | +| 128 | 543.95 | 535.71 | +1.5% | + +Baseline (separate cold runs, original non-instrumented build): +npl=32 243.9, npl=64 397.1, **npl=128 544.95** (matches the ~546 baseline). + +Graphs help, but the benefit **monotonically shrinks** as concurrency rises and +the GPU saturates. At npl=128 there is only ~1.5% of host launch overhead left to +remove, and GPU util is ~96% in both columns. **You cannot lift npl=128 decode +toward 667 by working on graphs/host overhead - the GPU is the bottleneck.** + +### Where the number actually is, and the real lever + +- vLLM 667 t/s at this concurrency = **192 ms/step**; llama.cpp 547 = **237 + ms/step**. The ~45 ms/step gap maps almost entirely onto the quantized matmul. +- GB10 memory-bandwidth floor for a 32B Q4_K_M (~19.8 GB of weights, read once + per step and shared across the 128 sequences) at ~273 GB/s is **~72 ms/step**. + llama.cpp's `mul_mat_q` spends ~155 ms/step on matmul = **~2.1x the bandwidth + floor**. vLLM's Marlin/Machete int4 GEMMs run much closer to the floor; that + efficiency difference is the ~547 -> 667 gap. +- The Q6_K matmul (`mul_mat_q` type 14) also shows pathological tail latency + (median 0.89 ms, max 5.5 ms) - the MMQ kernel is not well-tuned for the skinny + n=128 decode shape. + +**The lever to beat 547 is a faster quantized decode GEMM**, i.e. a Marlin-style +int4 kernel for the decode shapes. This is exactly the direction of the prior +session's uncommitted `ggml/src/ggml-cuda/marlin-w4a16.cu` and +`fp4-grouped-moe.cu` (already wired via +`if (!split && ggml_cuda_w4a16_mul_mat(...)) return;` in `ggml_cuda_mul_mat`). +Note those target **w4a16 / GPTQ-int4**, while this GGUF is **K-quant (Q4_K/Q6_K)**, +so they are inert for this model - a Marlin path for K-quants (or shipping the +model in a Marlin-friendly int4 format) would be required. That is a multi-day +kernel effort, out of scope for this session, but it is the only lever that can +move the number. + +### Why the "bump LLAMA_MAX_SEQ to 1024 -> 377" data point is consistent + +`llama_batch_allocr` keeps `seq_cpl` as an `LLAMA_MAX_SEQ x LLAMA_MAX_SEQ` table +(`src/llama-batch.cpp`), so per-batch seq bookkeeping scales ~O(MAX_SEQ^2). At +MAX_SEQ=1024 that host cost becomes large enough (~70 ms/step) to dominate and +drop decode to 377. At MAX_SEQ=256 the same term is ~4.4 ms/step (the ~1.5% that +graphs reclaim); lowering to 128 would save ~3 ms/step (~1%). So MAX_SEQ tuning +confirms the host term is real but tiny at 256 - not a path to 667. + +## How this would land in LocalAI + +- **No host/graph patch is warranted** for this build: graphs already engage and + the decode is GPU-bound. A "pad the decode batch / force graph capture" patch + would change nothing measurable at high concurrency. +- The actionable upstream/vendored work is a **Marlin-style int4 decode GEMM** + (extend the prior `marlin-w4a16.cu` to cover K-quants, or quantize the served + model into a Marlin-friendly int4 layout). That is where the ~547 -> 667+ lives. +- If a small host win is still wanted, keep `LLAMA_MAX_SEQ` no larger than the max + concurrency actually used (the per-batch `seq_cpl` table is O(MAX_SEQ^2)). + +## Reproduction + +``` +# baseline / A/B (cold, 30s cooldowns) +llama-batched-bench -m Qwen3-32B-Q4_K_M.gguf -npp 16 -ntg 128 -npl 32,64,128 \ + -ngl 99 -b 2048 -ub 2048 -fa on # graphs on +GGML_CUDA_DISABLE_GRAPHS=1 ...same... # graphs off + +# GPU util (graphs on): sample nvidia-smi during decode -> ~96%, 2184 MHz +# GPU active (graphs off): nsys profile -t cuda --delay=6 --duration=8 ... +# nsys stats --report cuda_gpu_kern_sum -> sum/0.516 ~= 7.72s of 8s = ~96% +```