mirror of
https://github.com/mudler/LocalAI.git
synced 2026-06-27 09:57:14 -04:00
78fac9a28f
Move ALL paged-attention content out of the stock backend/cpp/llama-cpp backend and into backend/cpp/llama-cpp-localai-paged, so the stock backend is pure upstream llama.cpp and the paged backend owns and applies its own vendored patch series. - Delete the dead early-exploration scaffold backend/cpp/llama-cpp/paged/ (kernel/w4a16 Marlin scaffold, standalone paged_kv_manager, bench/loadgen, its own 0001-0002 patches, dense-era design docs, tests). Zero references repo-wide. - Move backend/cpp/llama-cpp/patches/ (the 28-patch paged series + paged/README + 3 operational docs, plus the kernel/ scaffold patch and the top-level paged README/BENCHMARKS) to backend/cpp/llama-cpp-localai-paged/patches/. The stock backend keeps no patches/ dir; it had no non-paged base patches. - Purify the stock backend: remove the LLAMA_PAGED make variable, the patches/paged apply loop, and the LLAMA_PAGED passthrough to prepare.sh; remove the paged-series handling from prepare.sh. The stock llama.cpp target now only clones the pin and applies its own (currently empty) base patches/ series. The runtime paged option hooks in the shared grpc-server.cpp are untouched (inert without the patches). - The paged backend's Makefile now applies its OWN patches/paged/0*.patch onto each freshly cloned tree via strict git apply (apply-paged-patches), after the copied stock infra clones the pin and applies base patches. - Repoint every reference to the old patches/paged path: the upstream canary workflow + apply script, bump_deps.yaml, gallery/index.yaml, the docs, backend/index.yaml, backend-matrix.yml, the top-level Makefile comments, and the moved PIN_SYNC / README docs. Drop the now-removed LLAMA_PAGED=on build-toggle from comments. Verified: the full 28-patch series applies strict-clean (git apply, exit 0) to a clean ggml-org/llama.cpp checkout at the pinned c299a92c, and the repointed canary apply script resolves and applies the series end to end. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
983 B
983 B
| 1 | model | engine | npl | decode_agg_tps | decode_perseq_tps | prefill_tps | ttft_mean_ms | peak_gb |
|---|---|---|---|---|---|---|---|---|
| 2 | q36-27b-nvfp4 | llama | 8 | 82.5 | 9.57 | 507.3 | 6038.1 | 53.51 |
| 3 | q36-27b-nvfp4 | llama | 32 | 192.6 | 4.79 | 115.0 | 133551.7 | 69.63 |
| 4 | q36-27b-nvfp4 | llama | 64 | 277.8 | 3.09 | 95.9 | 321618.8 | 83.96 |
| 5 | q36-27b-nvfp4 | llama | 128 | 384.6 | 1.86 | 69.7 | 902762.7 | 93.82 |
| 6 | q36-27b-nvfp4 | vllm | 8 | 70.4 | 8.76 | 2096.2 | 1861.1 | 110.92 |
| 7 | q36-27b-nvfp4 | vllm | 32 | 211.8 | 6.28 | 2182.6 | 5353.2 | 110.87 |
| 8 | q36-27b-nvfp4 | vllm | 64 | 309.1 | 4.38 | 2088.9 | 9512.4 | 110.88 |
| 9 | q36-27b-nvfp4 | vllm | 128 | 418.8 | 2.79 | 1929.1 | 18449.5 | 110.95 |
| 10 | q36-35b-a3b-nvfp4 | llama | 8 | 211.8 | 24.45 | 1236.4 | 2477.1 | 39.66 |
| 11 | q36-35b-a3b-nvfp4 | llama | 32 | 393.0 | 10.02 | 1213.9 | 8225.2 | 47.11 |
| 12 | q36-35b-a3b-nvfp4 | llama | 64 | 527.0 | 6.15 | 1152.3 | 15849.5 | 57.13 |
| 13 | q36-35b-a3b-nvfp4 | llama | 128 | 726.4 | 3.73 | 276.8 | 213017.2 | 61.51 |
| 14 | q36-35b-a3b-nvfp4 | vllm | 8 | 256.5 | 31.84 | 5186.5 | 768.8 | 109.62 |
| 15 | q36-35b-a3b-nvfp4 | vllm | 32 | 500.8 | 14.90 | 6223.4 | 1830.4 | 109.63 |
| 16 | q36-35b-a3b-nvfp4 | vllm | 64 | 686.1 | 9.83 | 5926.5 | 3224.4 | 109.63 |
| 17 | q36-35b-a3b-nvfp4 | vllm | 128 | 882.2 | 6.05 | 5300.5 | 6487.7 | 109.64 |