LocalAI

mirror of https://github.com/mudler/LocalAI.git synced 2026-06-23 16:19:07 -04:00

Files

Ettore Di Giacinto 4dcbcfcf92 docs(paged): decode-step gap study vs vLLM on GB10

Profiling decomposition of the llama-server batch-32 / 1024-ctx decode step
vs vLLM on a DGX Spark (GB10, sm_121). Findings: decode is GPU-bound (~95%
busy, sampling/loop fully hidden); at 1024 ctx the step is ~84% KV/attention
and ~16% weight GEMM; the paged KV engine is a ~1.85x decode regression vs
stock (per-layer gather-to-contiguous); even stock is ~4-5x slower than vLLM,
gated by the long-context decode-attention and thin-batch FP4 GEMM kernels,
not by the serving loop. Ranked closable-vs-structural levers included.

Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

2026-06-22 15:44:24 +00:00

paged

feat(paged): target-readiness for 2xH200 - correctness PASS, load-gen harness, projection

2026-06-21 23:16:28 +00:00

patches

docs(paged): decode-step gap study vs vLLM on GB10

2026-06-22 15:44:24 +00:00

CMakeLists.txt

fix(turboquant): resolve common.h by detecting llama-common vs common target (#9413 )

2026-04-18 20:30:28 +02:00

grpc-server.cpp

feat(llama-cpp): expose paged KV cache as a per-server option (patch 0005)