LocalAI/backend/cpp/llama-cpp/patches/paged/DECODE_GAP_STUDY.md

llama-server vs vLLM: decode-step gap decomposition (DGX Spark, GB10 / sm_121)

Profiling study (no engine changes). Question: matched apples-to-apples (both batched servers, NVFP4-class weights, prefix caching on, both eager), why is llama-server ~4-6x slower per decode step than vLLM on Qwen3-32B at a 1024-token shared-prefix / batch-32 fan-out, and what is closable vs structural.

Hardware: NVIDIA GB10 (sm_121), unified LPDDR5X. Model: Qwen3-32B, 64 layers. llama side: ~/llama-paged-dev/build-cuda/bin/llama-server, q3-32b-nvfp4-dense.gguf (NVFP4 weights, type-40 FP4-MMA path), -ngl 99 --parallel 32 -c 40960 -fa on, GGML_CUDA_DISABLE_GRAPHS=1 (eager). vLLM 0.23.0 NVFP4A16 (W4A16/Marlin), --enforce-eager. Workload: 1024-token shared prefix + unique 32-token suffix, K=32 concurrent, generate 64. All profiling scripts are dev-tree only (~/bench/decode_study/); minimal in-code timers were not needed (server already reports per-slot eval time, which excludes prompt-eval = pure decode).

TL;DR

1. The real-server decode is GPU-BOUND, not host-bound. During steady decode the GPU is ~94.6% utilized (nvidia-smi, real run) / 85-95% busy (nsys). Per-slot CPU sampling, detokenize, and update_slots are fully hidden: a 5-stage sampler chain gives the identical step time as greedy (1346 vs 1343 ms). The "GPU stalls on the CPU serving loop" hypothesis is refuted for this workload.
2. At 1024 context the decode step is ~84% KV/attention, ~16% weight GEMM - the opposite of the thin-batch-GEMM story. Attention scaling with context length, not the matmul, is the load-bearing cost.
3. The worktree's paged KV engine is a decode REGRESSION: ~1.85x slower than stock at 1024 ctx (paged 1279-1343 ms/step vs stock 650-729 ms/step). It gathers K/V/mask into a contiguous buffer (ggml_get_rows) every layer every step, then runs a dense FA kernel - paying a full extra KV read+copy that vLLM's in-kernel PagedAttention never pays. Paging helps prefix-prefill memory; it hurts decode latency.
4. Even stock llama-server (~650-729 ms/step) is ~4-5x slower than vLLM (~120-185 ms/step). The residual gap is the long-context decode-attention kernel and, secondarily, the thin-batch FP4 weight GEMM - both kernel-maturity gaps vs vLLM's FlashInfer/FA paged-decode + Marlin, not serving-loop gaps.

The measured numbers (batch 32, server-reported pure-decode step time)

server_decode_step_ms = max / mean-of-top-8 of per-slot eval time ms-per-token (the most-contended, full-batch-32 slots; excludes prompt eval).

config	decode step ms (max / top8)	client wall ms/step
paged, ctx 1024, greedy	1343 / 1279	1468
paged, ctx 1024, heavy 5-sampler	1346 / 1280	1470
stock (no paging), ctx 1024, greedy	729 / 650	768
paged, ctx 64 (short), greedy	215 / 215	253
vLLM NVFP4A16, ctx 1024 (K=32)	~120-185 (270 tok/s)	-

The brief's reference ~828 ms/step sits between the stock (650-729) and paged (1279-1343) numbers measured here; the decomposition below is what is robust. Our fan-out shares no prefix across the 32 slots (each slot independently prefills 1056 tokens - confirmed in the log), so the 32 sequences are genuinely concurrent and the "max" slot is maximally contended, which is why our paged max runs a little above 828.

Context sweep - decode step is attention-scaling, not fixed overhead

Pure-decode step vs shared-prefix length (paged, batch 32):

prefix ctx	decode step ms
64	215
128	~290
256	~410
512	~660
1024	~1280

Roughly linear in context length: ~1 ms of added step time per added context token. The 215 ms at ctx 64 is the fixed floor (weight GEMM + activations + norm/rope + loop + sampling, attention negligible). Everything above it scales with KV length = attention + KV plumbing. At 1024 ctx the fixed floor is only ~16% of the step.

Where the ~1280 ms paged decode step goes (nsys, pure-decode window)

nsys profile --delay=70 --duration=25 --trace=cuda windowed onto steady 32-way decode (srv_decode2.nsys-rep; an earlier 25-60s window was discarded because nsys's own slowdown stretched the 32 prefills into it, inflating GEMM to a misleading 58%). GPU busy in-window 85.5% (nsys adds gaps; the real run is ~94.6% by nvidia-smi).

bucket	% GPU time	abs (of ~1280 ms)	what it is
flash_attn_ext_f16 ATTENTION	47.7%	~610 ms	decode attention over the 1056-cell KV
cpy_scalar KV copy/cast	18.3%	~234 ms	KV write + f32->f16 casts
get_rows/set_rows KV gather	17.8%	~228 ms	paged gather of K/V/mask to contiguous
mul_mat_q + quantize_mmq	15.7%	~201 ms	NVFP4 weight GEMM (+ activation requant)
rmsnorm / silu / rope / add	~0.6%	~8 ms	elementwise

Cross-check: the GEMM bucket (~201 ms) matches the ctx-64 floor (215 ms) - i.e. the weight matmul is ~the entire short-context step, and is context-independent, as expected. KV/attention buckets (47.7+18.3+17.8 = 83.8%) match the context-sweep finding that ~84% of the step scales with context.

Power signature: ~33-36 W at 94% "utilization" (GB10 can pull far more). High util%

• low power = the kernels are memory/latency-bound, not compute-saturated - the classic decode signature (stream 19 GB of NVFP4 weights + a growing KV every step).

Stock vs paged decomposition

• Stock (~650 ms): ~215 ms GEMM floor + ~435 ms attention/KV (contiguous KV read directly by the FA kernel, no gather).
• Paged (~1280 ms): same ~215 ms floor + ~610 ms attention + ~455 ms paged gather/copy overhead (the get_rows of K/V/mask plus the extra KV copy that feeds the dense FA kernel). That ~455 ms (~36% of the step) is the paged engine's self-inflicted cost and is the entire ~1.85x stock->paged regression.

vLLM decode architecture mapped onto each llama bucket

vLLM at ~120-185 ms/step is faster on every bucket:

llama bucket (paged)	ms	vLLM equivalent	does vLLM avoid it?
paged KV gather (get_rows)	~228	PagedAttention reads blocks in-kernel via a block table	Yes - entirely. No gather op exists.
KV copy/cast	~234	KV written once into block pool; FA reads it in place	Mostly - no per-step recopy
decode attention	~610	FlashInfer / FA paged-decode GQA kernel, split over KV	Same op, far faster kernel on sm_121
weight GEMM + act quant	~201	fused Marlin/Machete W4A16 dequant+MMA, no separate quant pass	Faster + removes the requant kernel
CPU sampling / loop	~0 (hidden)	on-GPU batched sampling	N/A here - already hidden on llama side too

vLLM's whole-step (~150 ms) is less than llama's GEMM floor alone (~215 ms), so vLLM is ahead on the matmul and the attention and avoids the gather. The gap is a stack of kernel-efficiency wins, not one silver bullet.

Ranked levers - closable vs structural

1. Remove the paged gather regression. [Tractable, ~455 ms / ~36% on the paged path; net-zero risk - it is a regression] The worktree's paged engine makes decode 1.85x slower than stock by gathering K/V/mask to contiguous every layer every step (patch 0003 ggml_get_rows). For latency-bound decode, do not enable paged KV - it only ever helps prefix-prefill memory, never decode latency. Fully recovering this and keeping paging requires reading paged blocks in-kernel like vLLM (a from-scratch paged-attention CUDA kernel) - see lever 2.

2. Long-context decode-attention kernel. [Biggest real lever, ~435 ms of stock / ~610 ms of paged; partly structural] Even stock is attention-bound at 1024 ctx. llama.cpp's flash_attn_ext_f16 decode path is ~4-5x slower than vLLM's FlashInfer/FA paged-decode GQA kernel on this Blackwell-class part. This is the cost that grows with context - exactly the regime the brief targets. Tractable in principle (a proper flash-decoding / split-K-over-KV kernel, and a true in-kernel paged read that also kills lever 1's gather), but it is deep CUDA work on a new arch and partly gated by kernel maturity on sm_121. Highest-impact, hardest.

3. Thin-batch FP4 weight GEMM floor. [Tractable, ~201-215 ms / 15-30%; bounded] The NVFP4 mul_mat_q + separate quantize_mmq activation pass is memory-bound and less efficient than vLLM's fused Marlin/Machete W4A16. Fusing dequant into the MMA and folding the activation quant into the GEMM is tractable kernel work. Bounded impact: the floor cannot drop below weight-read-bound (~19 GB / HBM BW per step).

4. Host serving loop / per-slot sampling. [NOT a lever] Measured zero: greedy == heavy-sampler step time; GPU 94.6% busy. On-GPU/batched sampling buys nothing until the kernels (levers 1-3) get fast enough to expose host overhead. Refutes the "host-bound serving loop" hypothesis for this decode-bound workload.

5. Continuous-batch scheduler. [NOT the gap / structural elsewhere] llama-server already fuses all 32 slots into one decode step (one set of kernels per step over batch 32 - confirmed in the trace). vLLM's continuous/chunked-prefill batching wins on mixed prefill+decode overlap, but the steady decode-step gap measured here is kernel-bound, not scheduler-bound.

Honest bottom line

The ~4-6x per-step gap is GPU-kernel-bound, and it decomposes as:

• ~36% of the paged step is a self-inflicted gather regression - remove it (don't run paged for decode-latency workloads).
• The remaining ~4-5x vs vLLM (true even for stock) is kernel efficiency: llama.cpp's long-context decode-attention and thin-batch FP4 GEMM are slower than vLLM's PagedAttention + Marlin on GB10. That is a kernel project (in-kernel paged attention + flash-decoding + fused W4A16 GEMM), not a serving-loop project.
• Sampling, detokenize, update_slots, and the continuous-batch scheduler are not the gap; the GPU is ~95% busy on memory-bound kernels the whole step.

What is closable: lever 1 (immediately, by not paging), lever 3 (bounded, with kernel work). What is structural / hard: lever 2 (the decode-attention kernel + a real in-kernel paged read), which is where the context-scaling gap actually lives and where any serious effort to approach vLLM on GB10 must go.

Reproduction (dev-tree only, ~/bench/decode_study/)

• launch_srv.sh / runcfg.sh - launch llama-server (paged on/off) and a config.
• client.py - K=32 token-id fan-out (1024 prefix + 32 suffix), SAMP=greedy|heavy.
• d2drv.sh - nsys pure-decode window (delay 70s past prefill) -> srv_decode2.nsys-rep.
• cat2.py - kernel-time categorization from the sqlite export.
• vLLM side: ~/bench/run_vllm.sh + vllm_prefix.py (K=32, ~270 tok/s).