LocalAI/backend/cpp/llama-cpp/paged/DECODE_OVERHEAD.md

# llama.cpp multi-user decode overhead on DGX Spark (GB10, sm_121)

Investigation of the Qwen3-32B concurrent-decode throughput gap (llama.cpp ~547 t/s
vs vLLM ~667 t/s) on the GB10 box, build `~/llama.cpp-pr24423/build` (Release,
sm_121, `LLAMA_MAX_SEQ=256`, flash-attn on), model
`~/bench/q3-32b-gguf/Qwen3-32B-Q4_K_M.gguf`.

## TL;DR (the result overturns the brief's premise)

On **this** build the prime suspect is wrong and the host-overhead premise does not
hold:

1. **CUDA graphs are NOT disabled at high concurrency.** At npl=128, 94 of 98
   decode `graph_compute` calls **replay a captured CUDA graph** (0 resets, stable
   key, no property churn post-warmup). The keyed-warmup gate works.
2. **There is no ~170ms/step host hotspot here.** The GPU is **~96% active during
   decode with graphs ON and ~96% active with graphs OFF**. Decode at npl=128 is
   **GPU-compute-bound**, not host-bound.
3. The brief's "20% GPU util / 66ms GPU / 170ms host per step" was measured on a
   different/earlier build (mainline without these graph fixes). It is not
   reproducible on `llama.cpp-pr24423`.
4. Because the GPU is the bottleneck, re-enabling graphs cannot lift the number:
   the clean A/B shows graphs ON vs OFF = **+1.5% at npl=128** (and +2.9% at
   npl=32 - the benefit shrinks as concurrency rises and the GPU saturates).
5. The real gap to vLLM is the **quantized decode GEMM kernel**: `mul_mat_q`
   (Q4_K + Q6_K) is ~68% of decode GPU time and runs ~2.1x above the GB10
   memory-bandwidth floor. Closing the gap requires Marlin/Machete-style int4
   GEMM kernels, not host-side work. This is a kernel project (the direction the
   prior session's uncommitted `marlin-w4a16.cu` / `fp4-grouped-moe.cu` already
   started, though those target w4a16/GPTQ-int4, not the K-quants this GGUF uses).

## 1. Why CUDA graphs are (not) disabled - exact code + measurement

### The gate (code)

PR24423 refactored the CUDA-graph path into a keyed, warmup-based scheme in
`~/llama.cpp-pr24423/ggml/src/ggml-cuda/ggml-cuda.cu`:

- `ggml_cuda_graph_get_key(cgraph)` (~L3343) keys the cached CUDA graph by
  `cgraph->nodes[0]` (first-node pointer).
- `ggml_cuda_graph_check_compability(cgraph)` (~L3301) disables graphs only for:
  - **split buffers** (`ggml_backend_buft_is_cuda_split`), and
  - **`GGML_OP_MUL_MAT_ID`** when `src0` is non-quantized **or**
    `ne[2] > get_mmvq_mmid_max(...)` (MoE expert routing needs a stream sync).
  Qwen3-32B is **dense** -> no `MUL_MAT_ID` -> this condition never fires.
- `ggml_backend_cuda_graph_compute` (~L4514) warmup gate: a graph is used only
  after **2 consecutive calls with no property change** (`warmup_complete`); any
  property change resets warmup. `ggml_cuda_graph_update_required` (~L3347)
  detects change by `memcmp` of the full `ggml_tensor` struct + per-src
  data-ptr/ne/nb, with a fast path when `cgraph->uid` is unchanged.

### Why it stays enabled across decode steps

The graph stays stable because llama.cpp's host-side graph reuse holds during
decode, so node pointers/props (and `cgraph->uid`) do not churn:

- `llama_kv_cache::get_n_kv` (`src/llama-kv-cache.cpp` L1223-1233) **pads n_kv to
  a multiple of 256** ("so that the graph remains constant across batches and can
  be reused"). For ntg<=256 within the first KV block, n_kv is constant.
- `can_reuse_kq_mask` (`src/llama-graph.cpp` L43) keeps the KQ-mask dims stable:
  `ne=[n_kv, n_tokens/n_stream, 1, n_stream]` = `[256,1,1,128]` every decode step
  at npl=128.
- `can_reuse` (`src/llama-context.cpp` L1283) therefore returns true, so the
  scheduler is **not** reset/re-split. `graph->uid` is only reassigned inside
  `ggml_backend_sched_split_graph` (`ggml/src/ggml-backend.cpp` L1033, L1485),
  which is skipped on the reuse path -> stable uid -> CUDA graph replays.

### Measurement (instrumented build, npl=128, ntg=96)

Env-gated counters added to `ggml_backend_cuda_graph_compute` /
`ggml_cuda_graph_update_required` (since `GGML_LOG_DEBUG` is compiled out in
Release / NDEBUG). End-of-run summary:

```
[GTRACE-SUMMARY] calls=98 notenab=0 warming=3 warmdone=1 RESET=0 USED=94 incompat=0 distinct_keys=1
```

94/98 decode `graph_compute` calls **replayed** a captured CUDA graph; **0**
warmup resets; a **single** distinct graph key for the whole decode; no node
property churn after warmup. Graphs are fully engaged at npl=128.

(The instrumentation was reverted afterwards; the checkout is back to its
pre-task state and the `.so` rebuilt clean.)

## 2. The per-step CPU "hotspot" - there isn't one on this build

GPU utilization during npl=128 decode (ntg=256):

- **Graphs ON** - `nvidia-smi` sampled every 0.7s through the decode phase:
  steady **96% GPU util**, SM clock **2184 MHz** (not throttled), 45-47 W.
- **Graphs OFF** (`GGML_CUDA_DISABLE_GRAPHS=1`) - nsys CUDA trace, 8s window:
  total GPU kernel time = `3,983,292,128 ns / 0.516` = **~7.72s of the 8s
  window = ~96% GPU-active**. Even with every kernel launched individually from
  the host, the GPU is still ~96% busy. There are essentially **no host gaps**.

Per-step wall = 60.6s / 256 steps = **~237 ms/step**, and the sum of one decode
graph's kernel times (nsys, graphs-on capture) is ~244 ms -> GPU kernel time per
step ~= wall time per step. The host work between steps is in the low single-digit
ms (the ~4% idle), consistent with graphs ON giving only +1.5% at npl=128.

This directly contradicts the brief's 66ms-GPU / 170ms-host split, which must have
come from a pre-graphs build.

### Per-step GPU breakdown (nsys, npl=128 decode, graphs off, 8s window)

| Kernel | % GPU time | ~ms/step |
|--------|-----------:|---------:|
| `mul_mat_q` Q4_K (type 12) | 51.6 | ~118 |
| `flash_attn_ext_f16` | 19.3 | ~44 |
| `mul_mat_q` Q6_K (type 14) | 16.2 | ~37 |
| `unary_gated` silu | 4.1 | ~9 |
| mmq stream-k fixup + quantize_q8_1 | ~5 | ~12 |
| rms_norm / rope / set_rows / add | ~4 | ~10 |

Quantized matmul = **~68%** of decode GPU time (~155 ms/step). Attention ~19%.

`perf` could not profile the host (kernel `perf_event_paranoid=4`), but it is moot:
the host is ~4% of the wall, so there is no ~170ms host hotspot to chase.

## 3. Fix attempt + measured result

### The requested fix (re-enable graphs / pad the decode batch) is a no-op here

Graphs are already enabled and the batch is already stable (n_kv padded to 256,
kq_mask dims constant). The clean cold A/B (cooldowns between every run):

| npl | graphs ON (t/s) | graphs OFF (t/s) | delta |
|----:|----------------:|-----------------:|------:|
| 32  | 242.60 | 235.75 | +2.9% |
| 64  | 398.59 | 389.06 | +2.5% |
| 128 | 543.95 | 535.71 | +1.5% |

Baseline (separate cold runs, original non-instrumented build):
npl=32 243.9, npl=64 397.1, **npl=128 544.95** (matches the ~546 baseline).

Graphs help, but the benefit **monotonically shrinks** as concurrency rises and
the GPU saturates. At npl=128 there is only ~1.5% of host launch overhead left to
remove, and GPU util is ~96% in both columns. **You cannot lift npl=128 decode
toward 667 by working on graphs/host overhead - the GPU is the bottleneck.**

### Where the number actually is, and the real lever

- vLLM 667 t/s at this concurrency = **192 ms/step**; llama.cpp 547 = **237
  ms/step**. The ~45 ms/step gap maps almost entirely onto the quantized matmul.
- GB10 memory-bandwidth floor for a 32B Q4_K_M (~19.8 GB of weights, read once
  per step and shared across the 128 sequences) at ~273 GB/s is **~72 ms/step**.
  llama.cpp's `mul_mat_q` spends ~155 ms/step on matmul = **~2.1x the bandwidth
  floor**. vLLM's Marlin/Machete int4 GEMMs run much closer to the floor; that
  efficiency difference is the ~547 -> 667 gap.
- The Q6_K matmul (`mul_mat_q` type 14) also shows pathological tail latency
  (median 0.89 ms, max 5.5 ms) - the MMQ kernel is not well-tuned for the skinny
  n=128 decode shape.

**The lever to beat 547 is a faster quantized decode GEMM**, i.e. a Marlin-style
int4 kernel for the decode shapes. This is exactly the direction of the prior
session's uncommitted `ggml/src/ggml-cuda/marlin-w4a16.cu` and
`fp4-grouped-moe.cu` (already wired via
`if (!split && ggml_cuda_w4a16_mul_mat(...)) return;` in `ggml_cuda_mul_mat`).
Note those target **w4a16 / GPTQ-int4**, while this GGUF is **K-quant (Q4_K/Q6_K)**,
so they are inert for this model - a Marlin path for K-quants (or shipping the
model in a Marlin-friendly int4 format) would be required. That is a multi-day
kernel effort, out of scope for this session, but it is the only lever that can
move the number.

### Why the "bump LLAMA_MAX_SEQ to 1024 -> 377" data point is consistent

`llama_batch_allocr` keeps `seq_cpl` as an `LLAMA_MAX_SEQ x LLAMA_MAX_SEQ` table
(`src/llama-batch.cpp`), so per-batch seq bookkeeping scales ~O(MAX_SEQ^2). At
MAX_SEQ=1024 that host cost becomes large enough (~70 ms/step) to dominate and
drop decode to 377. At MAX_SEQ=256 the same term is ~4.4 ms/step (the ~1.5% that
graphs reclaim); lowering to 128 would save ~3 ms/step (~1%). So MAX_SEQ tuning
confirms the host term is real but tiny at 256 - not a path to 667.

## How this would land in LocalAI

- **No host/graph patch is warranted** for this build: graphs already engage and
  the decode is GPU-bound. A "pad the decode batch / force graph capture" patch
  would change nothing measurable at high concurrency.
- The actionable upstream/vendored work is a **Marlin-style int4 decode GEMM**
  (extend the prior `marlin-w4a16.cu` to cover K-quants, or quantize the served
  model into a Marlin-friendly int4 layout). That is where the ~547 -> 667+ lives.
- If a small host win is still wanted, keep `LLAMA_MAX_SEQ` no larger than the max
  concurrency actually used (the per-batch `seq_cpl` table is O(MAX_SEQ^2)).

## Reproduction

```
# baseline / A/B (cold, 30s cooldowns)
llama-batched-bench -m Qwen3-32B-Q4_K_M.gguf -npp 16 -ntg 128 -npl 32,64,128 \
  -ngl 99 -b 2048 -ub 2048 -fa on            # graphs on
GGML_CUDA_DISABLE_GRAPHS=1 ...same...        # graphs off

# GPU util (graphs on): sample nvidia-smi during decode -> ~96%, 2184 MHz
# GPU active (graphs off): nsys profile -t cuda --delay=6 --duration=8 ...
#   nsys stats --report cuda_gpu_kern_sum  -> sum/0.516 ~= 7.72s of 8s = ~96%
```

## UPDATE: NVFP4 closes most of the decode gap (no Marlin-for-K-quants needed)

The diagnosis above said the lever is "a more bandwidth-efficient int4 decode GEMM"
and feared a multi-day Marlin-for-K-quants kernel. But the FP4-MMA path is already
that kernel. Measured (npl=128, cold A/B, npp=16 ntg=128):

| quant | decode S_TG (t/s) | vs Q4_K | vs vLLM 667 |
|---|---|---|---|
| Q4_K_M | 547 (548/546) | - | 82% |
| **NVFP4** | **619 (617/622)** | **+13%** | **93%** |

NVFP4's `mul_mat_q<NVFP4>` runs closer to the GB10 bandwidth floor at the thin n=128
decode shape than Q4_K's int8-MMQ (which ran ~2.1x above it). So shipping the model
as NVFP4 closes the decode gap from ~22% to ~7% AND wins prefill (1209 vs Q4 767 /
vLLM 800). Net on GB10: llama.cpp+NVFP4 is ahead on prefill (1.5x) and within ~7% on
decode. The remaining ~7% would be incremental FP4-MMA decode-kernel tuning, NOT a
from-scratch Marlin kernel - a much smaller, optional effort. NVFP4 is the answer to
both the prefill and the decode gap.