LocalAI

mirror of https://github.com/mudler/LocalAI.git synced 2026-06-25 09:09:07 -04:00

Files

Ettore Di Giacinto 5ce2f1df51 feat(paged): qwen35 gated-DeltaNet in-place SSM state write-back (patch 0018)

Mirror of the llama-paged-dev patch 0018 engine change plus the measured
results. Per SSM layer per step decode no longer D2D-copies the full ~225 MB
recurrent state into the cache: the fused gated_delta_net op writes the final
state in place at the active sequences cache slot (new
ggml_gated_delta_net_inplace, src[6] = state_dst), mirroring vLLM
fused_recurrent_gated_delta_rule. SSM math unchanged (bit-identical greedy).

Measured (decode_agg S_TG, npp128 ntg128, -fa on, paged on):
  q36-27b-nvfp4 dense: npl32 113.74 -> 136.39 (+19.9 percent),
    npl128 146.23 -> 180.53 (+23.5 percent, = predicted copy-removal ceiling).
  q36-35b-a3b-nvfp4 MoE: npl128 313.36 -> 372.62 (+18.9 percent).
nsys D2D memcpy bucket 18.9 -> 0.23 percent (356 -> 2.93 GB). vLLM share
(391 @128) 37.4 -> 46.2 percent. See SSM_DECODE_FIX_RESULTS.md.

Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

2026-06-24 22:45:49 +00:00

paged

feat(paged): target-readiness for 2xH200 - correctness PASS, load-gen harness, projection

2026-06-21 23:16:28 +00:00

patches

feat(paged): qwen35 gated-DeltaNet in-place SSM state write-back (patch 0018)

2026-06-24 22:45:49 +00:00

CMakeLists.txt

fix(turboquant): resolve common.h by detecting llama-common vs common target (#9413 )

2026-04-18 20:30:28 +02:00

grpc-server.cpp

feat(llama-cpp/paged): dynamic decode-first prefill budget (patch 0016, continuous-batch P1)