LocalAI

mirror of https://github.com/mudler/LocalAI.git synced 2026-06-23 16:19:07 -04:00

Files

Ettore Di Giacinto 2a500c371f bench(paged): fresh GB10 head-to-head vs vLLM - two distinct gaps

Prefill 6-48x behind and does NOT scale with B (kernel-bound, paging can't fix).
Decode: we win at B=1; 2.5-3.7x behind at B>=8 - THAT concurrency gap is the
engine's domain (0004 pool + 0005 continuous batching target it). Baseline for
the series to improve on.

Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

2026-06-19 23:20:22 +00:00

paged

docs(paged): upstream survey - no FP4 MoE GEMM to patch in; phase 3 is from-scratch

2026-06-19 22:44:11 +00:00

patches

bench(paged): fresh GB10 head-to-head vs vLLM - two distinct gaps

2026-06-19 23:20:22 +00:00

CMakeLists.txt

fix(turboquant): resolve common.h by detecting llama-common vs common target (#9413 )

2026-04-18 20:30:28 +02:00

grpc-server.cpp

feat: generic chat_template_kwargs (model config + per-request metadata) (#10359 )