mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-03 04:46:54 -04:00
Code Issues Packages Projects Releases Wiki Activity

docs(paged): reject admission budget sweep

Browse Source 
Assisted-by: Codex:gpt-5
This commit is contained in:
Ettore Di Giacinto
2026-07-01 09:27:20 +00:00
parent 2aa76702df
commit 347a5c05bd
4 changed files with 175 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
backend/cpp/llama-cpp-localai-paged/docs/GB10_PARITY_PHASE0_RESULTS.md
View File

@@ -2938,3 +2938,41 @@ Decision:
per-step histogram trace to test whether prefill chunking/admission can reduce
TTFT without regressing aggregate throughput. Do not move to another GDN/GEMM
rewrite until this scheduler hypothesis is tested.
## Phase 53 Admission Budget Sweep
Phase 53 tests the existing default-off admission knobs exposed by patch 0016:
`LLAMA_MAX_BATCH_TOKENS` and `LLAMA_PREFILL_CAP`. The question was whether a
simple smaller token budget improves dense `n=128` TTFT or aggregate throughput.
Artifact:
- `/home/mudler/bench/phase53_dense_admission_budget_sweep/20260701_111915`
Pre/post gates:
| phase | MoE md5 | dense md5 | `MUL_MAT` | `MUL_MAT_ID` |
|-------|---------|-----------|-----------|--------------|
| pre | `8cb0ce23777bf55f92f63d0292c756b0` | `5951a5b4d624ce891e22ab5fca9bc439` | `1146/1146` | `806/806` |
| post | `8cb0ce23777bf55f92f63d0292c756b0` | `5951a5b4d624ce891e22ab5fca9bc439` | `1146/1146` | `806/806` |
Results:
| variant | agg t/s | decode agg t/s | decode per-seq t/s | prefill t/s | TTFT mean ms | wall s | steps | max waiting prompt slots |
|---------|---------|-----------------|---------------------|-------------|--------------|--------|-------|--------------------------|
| default Phase52 | `139.0` | `360.5` | `1.93` | `629.5` | `23171.5` | `58.921` | `76` | `35` |
| `LLAMA_MAX_BATCH_TOKENS=1536 LLAMA_PREFILL_CAP=512` | `134.4` | `376.7` | `1.82` | `607.0` | `22263.7` | `60.968` | `81` | `26` |
| `LLAMA_MAX_BATCH_TOKENS=1024 LLAMA_PREFILL_CAP=512` | `130.0` | `392.4` | `1.82` | `565.2` | `23234.3` | `63.003` | `89` | `16` |
Decision:
- Smaller admission budgets reduce the maximum number of waiting prompt slots
and raise the h2h `decode_agg_tps` metric, but they reduce aggregate
throughput and prefill throughput.
- `T=1536` gave only a small TTFT improvement (`23171.5 -> 22263.7 ms`) while
worsening wall time and aggregate throughput.
- `T=1024` worsened TTFT and aggregate throughput despite the highest
`decode_agg_tps`.
- Do not promote simple budget shrinkage as a parity lever. The next useful
scheduler work is a richer per-step histogram trace or a targeted first-token
admission policy, not a static lower `LLAMA_MAX_BATCH_TOKENS`.

14
backend/cpp/llama-cpp-localai-paged/docs/PARITY_HANDOFF.md
View File

@@ -652,6 +652,19 @@ matches h2h exactly, so this is the target request. The next GB10 lever should
be a default-off scheduler/admission A/B or a per-step histogram trace, not an
immediate GDN/GEMM rewrite.
Phase 53 tested the existing runtime admission-budget knobs instead of adding
new code. Artifact:
`/home/mudler/bench/phase53_dense_admission_budget_sweep/20260701_111915`.
Pre/post gates stayed green. Dense `n=128` results: default Phase52 `agg=139.0`,
`decode_agg=360.5`, `prefill=629.5`, `TTFT=23171.5ms`, wall `58.921s`;
`T=1536 cap=512` `agg=134.4`, `decode_agg=376.7`, `prefill=607.0`,
`TTFT=22263.7ms`, wall `60.968s`; `T=1024 cap=512` `agg=130.0`,
`decode_agg=392.4`, `prefill=565.2`, `TTFT=23234.3ms`, wall `63.003s`.
Decision: simple budget shrinkage is rejected. It raises h2h decode-agg while
lowering aggregate/prefill throughput, and it does not materially solve TTFT.
Next scheduler work should be per-step histograms or a targeted first-token
admission policy.
---
## 5. METHODOLOGY LESSONS (so you do not repeat the mistakes)
@@ -747,6 +760,7 @@ Only pursue if (a)+(b) are not options and someone explicitly wants the residual
- `~/bench/phase50_dense_true_decode/20260701_103120` - dense graph-node difference-method profile at `npl=128`, `npp=128`; `build-cuda` pre/post md5 and op gates green; true decode paged `383.66 t/s`, vLLM `435.00 t/s`, ratio `0.8820`, pointing next at serving admission/scheduler tracing.
- `~/bench/phase51_serving_admission_trace/20260701_110130` - default-off `LLAMA_SERVING_TRACE=1` fork commit `c6cb8460e`; DGX patched `build-cuda` CTest and md5/op gates green; push and LocalAI patch-series mirror pending approval.
- `~/bench/phase52_dense_admission_trace/20260701_111017` - clean dense `n=128` admission trace; pre/post gates green; `decode_only_steps=0`, `prompt_tokens=22785`, `max_waiting_prompt_slots=35`; next lever is scheduler/admission A/B or per-step histogram trace.
- `~/bench/phase53_dense_admission_budget_sweep/20260701_111915` - runtime sweep of `LLAMA_MAX_BATCH_TOKENS=1536/1024` with `LLAMA_PREFILL_CAP=512`; pre/post gates green; simple budget shrinkage rejected because aggregate/prefill throughput regressed and TTFT did not materially improve.
- Per-engine logs `~/bench/COMBINED_{paged,vllm}_{MOE,DENSE}_server.log`; `~/bench/BENCHMARK_PROGRESS.md`.
- Graph-node-traced high-N profiles: `~/highN_prof2/*.nsys-rep` (paged npl=256), `~/highN_vllm/*.nsys-rep` (vLLM), 2026-06-30.
- A/B dirs: `~/bench/marlin_gate/`, `~/bench/gdn_p1_ab/`.

24
backend/cpp/llama-cpp-localai-paged/docs/VLLM_PARITY_LEVER_MAP.md
View File

@@ -1267,6 +1267,30 @@ supports the Phase50 conclusion that the remaining high-N serving gap is
scheduler/admission and TTFT shaped. Next lever should be a default-off
admission-policy A/B or per-step histogram trace, not immediate kernel work.
### Phase 53 admission budget sweep
Phase53 tested the already-existing default-off budget knobs:
`LLAMA_MAX_BATCH_TOKENS=1536/1024` with `LLAMA_PREFILL_CAP=512`, using the same
dense `n=128`, `ptok=128`, `gen=64` traced serving shape. Artifact:
`/home/mudler/bench/phase53_dense_admission_budget_sweep/20260701_111915`.
Pre/post md5 and op gates stayed green: MoE
`8cb0ce23777bf55f92f63d0292c756b0`, dense
`5951a5b4d624ce891e22ab5fca9bc439`, `MUL_MAT` `1146/1146`, and
`MUL_MAT_ID` `806/806`.
| variant | agg t/s | decode agg t/s | prefill t/s | TTFT mean ms | wall s | max waiting prompt slots |
|---------|---------|-----------------|-------------|--------------|--------|--------------------------|
| default Phase52 | `139.0` | `360.5` | `629.5` | `23171.5` | `58.921` | `35` |
| `T=1536 cap=512` | `134.4` | `376.7` | `607.0` | `22263.7` | `60.968` | `26` |
| `T=1024 cap=512` | `130.0` | `392.4` | `565.2` | `23234.3` | `63.003` | `16` |
Decision: simple budget shrinkage is rejected as a parity lever. It improves
the h2h decode-agg metric by starving/slimming prompt admission, but aggregate
throughput and prefill throughput fall, and TTFT does not materially improve.
Next scheduler work should collect per-step histograms or test a targeted
first-token admission policy.
Relevant files (all absolute): `/home/mudler/_git/LocalAI/.claude/worktrees/feat+paged-attention/backend/cpp/llama-cpp-localai-paged/docs/{DECODE_SERVING_SCOPE.md,PREFILL_GEMM_SCOPE.md,PREFILL_GEMM_RESULTS.md,TENSORCORE_GDN_SCOPE.md,final_benchmark.csv}`, `.../README.md`, `.../patches/paged/0034-feat-paged-native-NVFP4-W4A4-FP4-MMA-large-M-prefill.patch` (P1/P2), `.../patches/paged/0042-feat-paged-fused-residual-add-RMS-norm-weight-multip.patch` (P7), `.../patches/paged/0031` (P4), `0025` (D1), `0018/0022` (D4/D5), `0009/0010` (D3/D6/D7); graph source `/home/mudler/_git/LocalAI/backend/cpp/llama-cpp-paged-dev/src/{models/qwen35moe.cpp,models/delta-net-base.cpp,llama-graph.cpp}`.
### Phase 10 GDN C32 slab update