# GB10 same-day head-to-head server sweep: llama-server (paged) vs vLLM

Date: 2026-06-23. Hardware: GB10 / DGX Spark (sm_121, 128 GB LPDDR5x unified, ~273 GB/s
weight-read floor). GPU otherwise idle (sibling vLLM had exited; LocalAI docker workers
stopped for the run).

This sweep **replaces** the stale carried "~75-80% of vLLM" figure (commit 07985ba4,
pre-co-batching, single-point). It measures *real serving* steady-state aggregate decode
throughput across the full concurrency curve, for three model classes, with one identical
client driving both engines.

## Method

- **llama**: `llama-server` from the paged dev tree (`~/llama-paged-dev/build-cuda`, HEAD =
  patch 0013 / commit 17d97cb), `LLAMA_KV_PAGED=1`, `-fa on -ngl 999 --parallel 128 -c 65536`.
- **vLLM**: 0.23.0, `vllm serve --enforce-eager --enable-prefix-caching --max-num-seqs >=128
  --max-model-len 4096` (APC on, eager per the GB10 no-CUDA-graphs edge).
- **Client** (`sweep_client2.py`): N concurrent **non-streaming** `/v1/completions`, short
  shared prompt, `max_tokens=min_tokens=256`, `ignore_eos=true`. Aggregate decode tok/s =
  total generated tokens / wall. Non-streaming keeps the Python client off the critical path
  (one JSON parse per request, not per token), so the **server** is the bottleneck. Validated:
  vLLM pushed 4227 tok/s through the exact same client where llama topped out at 2087, so the
  client is not the cap. Both engines use the identical client + prompt -> apples-to-apples.
- npl (concurrency) sweep: 8 / 32 / 64 / 128.

Quant parity:
- Dense: llama **NVFP4-dense GGUF** (weight-only FP4, 16-bit compute) vs vLLM **NVFP4A16**
  (weight FP4, 16-bit activation) -> matched precision class.
- Small: llama **Q8_0** vs vLLM **bf16** (closest loadable form).
- MoE: llama **mxfp4** GGUF. **vLLM could not serve this MoE on GB10 at all** (see below), so
  there is no vLLM MoE column.

## Results: aggregate decode tok/s (higher is better)

### Dense 32B  (llama NVFP4-dense  vs  vLLM NVFP4A16)

| npl | llama (NVFP4) | vLLM (NVFP4A16) | llama % of vLLM |
|----:|--------------:|----------------:|----------------:|
|   8 |          83.2 |            85.9 |          **96.9%** |
|  32 |         228.9 |           301.3 |          76.0%  |
|  64 |         367.1 |           507.8 |          72.3%  |
| 128 |         520.6 |           604.0 |          86.2%  |

Plateau: neither has plateaued at 128 (both still climbing, weight-read bound). llama is at
**parity at batch-8** (97%), dips to ~72% mid-curve (npl 32-64), recovers to 86% at 128.

### Small  Qwen3-0.6B  (llama Q8_0  vs  vLLM bf16)

| npl | llama (Q8_0) | vLLM (bf16) | llama % of vLLM |
|----:|-------------:|------------:|----------------:|
|   8 |        911.3 |       923.0 |        **98.7%** |
|  32 |       1701.6 |      2531.4 |        67.2%  |
|  64 |       1911.7 |      3497.1 |        54.7%  |
| 128 |       2087.6 |      4227.6 |        49.4%  |

Plateau: **llama plateaus hard** at ~2.0-2.1k by npl 64-128 (+9% from 64->128). vLLM keeps
scaling (3497 -> 4227). For a tiny runtime-bound model, vLLM's scheduler/batching amortizes
better; llama-server's per-token host cost (sampling, detok, slot mgmt) caps it. This is the
worst llama-vs-vLLM ratio in the sweep (down to 49%).

### MoE  Qwen3-Coder-30B-A3B  (llama mxfp4; vLLM = NOT SERVABLE on GB10)

| npl | llama (mxfp4) | vLLM |
|----:|--------------:|-----:|
|   8 |         290.0 |  n/a |
|  32 |         582.5 |  n/a |
|  64 |         931.8 |  n/a |
| 128 |        1041.3 |  n/a |

llama-server scales cleanly to **1041 tok/s** at npl 128 with **no npl-128 expert-activation
cliff** (unlike the prior `llama-batched-bench` MoE numbers 253/505/830/620 that peaked at 64;
short-prompt continuous batching in the server avoids it).

**vLLM could not serve this MoE on GB10 (two independent failures):**
1. **bf16** (`Qwen/Qwen3-Coder-30B-A3B-Instruct`, the only HF form on the box): loads the
   56.9 GB of weights, then **hangs at the MoE warmup** (`Using MoEPrepareAndFinalize
   NoDPEPModular` -> `Model loading took ...`), GPU 0% util, and **takes the whole box down
   (hard reboot)**. Reproduced twice. With tight `--gpu-memory-utilization` it still hangs at
   the same step before the API server ever comes up.
2. **mxfp4 GGUF** (same weights llama uses): vLLM 0.23.0's GGUF loader **cannot map the fused
   qwen3moe expert tensors** (`RuntimeError: Failed to map GGUF parameters (48):
   ['model.layers.N.mlp.experts.gate_up_proj', ...]`). Engine init fails outright.

So on GB10, llama.cpp is the *only* engine of the two that serves this 30B-A3B MoE at all -
an availability win, independent of throughput.

## Batch-8 anomaly triage (dense NVFP4) -- RESOLVED

The prior mixed-load run reported llama batch-8 steady decode at **471 ms/step (~19% of vLLM
aggregate, ~17 tok/s)**. This sweep does **not** reproduce it. Clean isolated batch-8 decode:

- `llama-server` batch-8 dense paged = **83.2 tok/s** aggregate = ~96 ms/step = **96.9% of
  vLLM's 85.9** (parity, both at the LPDDR5x weight-read floor).

`llama-batched-bench` cross-check, dense NVFP4, `-npp 16 -ntg 128 -npl 1,8`, the three
hypotheses isolated (S_TG = decode tok/s aggregate at batch 8):

| config                | batch-8 S_TG t/s | ms/decode-step |
|-----------------------|-----------------:|---------------:|
| paged,  ctx 65536     |            90.32 |          88.6  |
| stock,  ctx 65536     |            88.39 |          90.5  |
| paged,  ctx 163840    |            89.33 |          89.6  |
| stock,  ctx 163840    |            87.72 |          91.2  |

Conclusion: clean batch-8 dense decode is **~88-90 tok/s (~89 ms/step) regardless of all three
suspects**:
- **Paged overhead?** No -- paged is within 2% of stock, and at ctx 65k paged is *faster*
  (90.3 vs 88.4). The decode path is not paying a paged penalty at batch-8.
- **The 163840-token ctx allocation?** No -- ctx 163840 == ctx 65536 within 1% (89.3 vs 90.3).
  The large allocation does not slow steady-state decode.
- **NVFP4 decode cost?** This *is* the cost -- ~89 ms/step is the GB10 weight-read floor for a
  32B at batch-8 (it matches vLLM's 86 tok/s server and exceeds it at the kernel level: 90 vs
  86). It is the hardware ceiling, not a bug.

The 471 ms/step is ~5.3x slower than this clean floor and is explained by none of the three.
It was a **mixed-load artifact**: the 8 decoders were time-sharing the GPU with a concurrent
prefill (a large prompt / chunked prefill landing on the same steps). That decode-vs-prefill
contention is exactly the stall **patch 0013 (`LLAMA_PREFILL_BUDGET`)** bounds. In steady-state
isolated decode, batch-8 dense is at **parity with vLLM (97%)**, not 19%.

## Aggregate map (replaces the carried 75-80%)

llama-server (paged) as a fraction of vLLM, by regime:

- **Low concurrency (batch-8): parity, 97-99%** on both measurable classes. Both engines sit on
  the LPDDR5x weight-read floor; there is nothing to win.
- **Dense 32B, mid-to-high concurrency: 72-86%.** Dips to ~72% at npl 32-64, recovers to 86% at
  128. Both still climbing (weight-bound), neither plateaus by 128.
- **Small 0.6B, mid-to-high concurrency: 49-67%.** llama plateaus ~2.0k; vLLM scales to 4.2k.
  Runtime/scheduler-bound regime -- vLLM's batching wins; this is llama's weakest ratio.
- **MoE 30B-A3B: llama-only.** vLLM cannot serve it on GB10 (bf16 reboots the box at MoE
  warmup; GGUF expert tensors unmappable). llama serves it at 290 -> 1041 tok/s, scaling
  cleanly with no npl-128 cliff.

Net: the single "75-80%" number is replaced by a curve. It is *roughly* right only for the
dense mid-band; it is too optimistic for the small model at high concurrency (49%) and moot for
MoE (where llama is the only option). The headline is parity at low concurrency and a hardware
(not engine) ceiling on dense decode.