LocalAI

mirror of https://github.com/mudler/LocalAI.git synced 2026-06-24 16:49:06 -04:00

Files

Ettore Di Giacinto e4c63179e0 docs(paged): verify llama.cpp GDN decode is O(1)-in-context, not a 2.4x lever

Closes lever 5 of VLLM_DECODE_GROUNDING.md. GGUF metadata + source reading on
the paged dev tree plus nsys decode traces on Qwen3.6-27B NVFP4 (GB10 sm_121)
confirm the Gated-Delta-Net linear-attention layers decode as a fused single
CUDA kernel (gated_delta_net.cu) updating a fixed-size cached recurrent state:
no context-length parameter, no KV re-scan. Matched-batch context-scaling
control (npl4, pure decode) shows the GDN kernel flat (10.3 -> 8.0 us/launch)
across 4x context while full-attention grows 3.1x (27 -> 85 us). GDN is a small,
context-flat share (~0.4-10%% by batch); the FP4 weight GEMM dominates (~67%).
Verdict: GDN decode is efficient, not the cheap model-specific fix; the 2.4x is
the general GEMM + full-attention kernel work, as the grounding concluded.

Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

2026-06-24 11:21:44 +00:00

paged

feat(paged): target-readiness for 2xH200 - correctness PASS, load-gen harness, projection

2026-06-21 23:16:28 +00:00

patches

docs(paged): verify llama.cpp GDN decode is O(1)-in-context, not a 2.4x lever

2026-06-24 11:21:44 +00:00

CMakeLists.txt

fix(turboquant): resolve common.h by detecting llama-common vs common target (#9413 )

2026-04-18 20:30:28 +02:00

grpc-server.cpp

feat(llama-cpp/paged): dynamic decode-first prefill budget (patch 0016, continuous-batch P1)