diff --git a/backend/cpp/llama-cpp-localai-paged/docs/GB10_PARITY_PHASE0_RESULTS.md b/backend/cpp/llama-cpp-localai-paged/docs/GB10_PARITY_PHASE0_RESULTS.md index d51ddea36..3d74498d0 100644 --- a/backend/cpp/llama-cpp-localai-paged/docs/GB10_PARITY_PHASE0_RESULTS.md +++ b/backend/cpp/llama-cpp-localai-paged/docs/GB10_PARITY_PHASE0_RESULTS.md @@ -2371,11 +2371,11 @@ Decision: Any future C1 rerun must push beyond this tested point and keep the same md5 plus `MUL_MAT`/`MUL_MAT_ID` gates. -## Phase 41 Low-Concurrency D1 Check +## Phase 41 Low-Concurrency Serving Check Phase 41 measured the opposite serving regime after Phase40 rejected the tested max-concurrency shortcut: low concurrency and latency-sensitive decode. This is -the regime where the D1/full-step graph-capture direction should matter most. +the regime where any remaining host/scheduler gap should be most visible. Artifacts: @@ -2433,11 +2433,39 @@ Ratios: Decision: -- D1/full-step graph capture remains relevant for low-concurrency and latency - work, but this current-stack snapshot does not show an easy parity bridge: - paged is about `0.75x` vLLM decode at `n=1/8` and `0.665x` at `n=32`. +- The low-concurrency gap is real, but Phase41 does not reopen D1/full-step graph + capture. Patch `0043` already ships that behavior default-on, and Phase34 + route tracing found `host_sync=0/4096` for the current n128 serving path. + Paged is about `0.75x` vLLM decode at `n=1/8` and `0.665x` at `n=32`. - TTFT is the bigger user-visible low-concurrency gap, especially by `n=8/32`; prefill GDN and MoE GEMM work therefore still matters even in a decode-focused serving discussion. -- The next implementation phase should require a separately built A/B and the - same md5 plus `MUL_MAT`/`MUL_MAT_ID` gates before claiming any D1 improvement. +- Do not fund another D1 graph-capture patch on GB10 unless a fresh route trace + first proves a host-sync fallback or graph-disable condition has returned. The + next implementation target should be a measured non-D1 bucket, gated by the + same md5 plus `MUL_MAT`/`MUL_MAT_ID` checks. + +## Phase 42 D1/GDN/GEMM Target Reconciliation + +Phase 42 challenged the Phase41 wording against the patch stack and read-only +subagent analysis. It resolves the next-target decision before any source work. + +Evidence: + +| track | evidence | decision | +|-------|----------|----------| +| D1/full-step graph capture | Patch `0043` is default-on for grouped MMQ decode and opt-out via `LLAMA_MOE_NO_FORCE_GRAPHS=1`; Phase34 route trace found `host_sync=0/4096`; `VLLM_PARITY_FINAL.md` marks D1 shipped and the host-sync premise refuted | closed on current GB10 path | +| S3 decode-shape-stable scheduling | Patch `0041` is shipped default-off after end-to-end A/B showed worse TTFT and lower throughput despite better per-step decode metrics | keep opt-in only | +| GDN prefill | Patches `0046`/`0047` are the shipped GB10 GDN wins; C32 slab, QS-early, and Global-Ai32 were md5-clean but slower | do not add another low-conflict GB10 GDN reorder | +| W4A16 / prefill GEMM | Patches `0033`/`0034`/`0035` are default-off; `0048`-`0050` improved forced W4A16 only marginally and did not beat default MMQ | do not add another small W4A16 body/metadata tweak | + +Next target: + +- The only small incremental candidate left from the current evidence is the + persistent/load-time F32 combined gate projection scoped in Phase38/39: + combine `ffn_gate_inp.weight` and `ffn_gate_inp_shexp.weight` once, run one + F32 gate matmul, and split/view the output. Do not use graph-time + `ggml_concat()`. +- It must be default-off, fork-first, and validated with MoE/dense md5, + `MUL_MAT`, `MUL_MAT_ID`, and KL if either md5 changes before any serving + benchmark. diff --git a/backend/cpp/llama-cpp-localai-paged/docs/PARITY_HANDOFF.md b/backend/cpp/llama-cpp-localai-paged/docs/PARITY_HANDOFF.md index 15e87c511..f6d0cf92c 100644 --- a/backend/cpp/llama-cpp-localai-paged/docs/PARITY_HANDOFF.md +++ b/backend/cpp/llama-cpp-localai-paged/docs/PARITY_HANDOFF.md @@ -525,16 +525,28 @@ and `OPS=MUL_MAT,MUL_MAT_ID`. Pre/post gates stayed green: MoE concurrency at this prompt/gen length and `n<=256`; a future C1 retry must push beyond this tested point and keep the same md5/op gates. -Phase 41 records the low-concurrency counterpart for D1. Artifact: +Phase 41 records the low-concurrency counterpart to the Phase40 high-concurrency +check. Artifact: `/home/mudler/bench/phase41_low_concurrency/20260701_091437`. The snapshot ran with `PARALLEL=32`, `CTX=32768`, `PTOK=128`, `GEN=64`, `NPL="1 8 32"`, and `OPS=MUL_MAT,MUL_MAT_ID`. Pre/post gates stayed green: MoE `8cb0ce23777bf55f92f63d0292c756b0`, dense `5951a5b4d624ce891e22ab5fca9bc439`, `MUL_MAT` `1146/1146`, `MUL_MAT_ID` `806/806`. Paged is about `0.75x` vLLM decode at `n=1/8` and `0.665x` at -`n=32`; TTFT is `1.38x`, `3.14x`, and `3.40x` vLLM respectively. Keep D1 in -scope for low-concurrency/latency, but require a separately built A/B and the -same md5/op gates before claiming improvement. +`n=32`; TTFT is `1.38x`, `3.14x`, and `3.40x` vLLM respectively. Do not reopen +D1 from this result: `0043` already ships grouped-MMQ full-step graph capture +default-on, Phase34 found `host_sync=0/4096`, and S3 is default-off because it +regressed TTFT/end-to-end throughput. + +Phase 42 reconciles the target list after parallel read-only review. D1 is +closed on the current GB10 path; GDN low-conflict work is exhausted after +`0046`/`0047` plus the rejected C32/QS-early/Global-Ai32 follow-ups; W4A16/GEMM +micro-tweaks are exhausted after `0033`-`0035` and `0048`-`0050`. The next small +GB10 source candidate is the Phase38/39 persistent/load-time F32 combined gate +projection: combine `ffn_gate_inp.weight` and `ffn_gate_inp_shexp.weight` once, +run one F32 gate matmul, split/view outputs, default-off, no graph-time +`ggml_concat()`, and gate with MoE/dense md5 plus `MUL_MAT`/`MUL_MAT_ID` before +benchmarking. If md5 changes, run KL first. --- @@ -619,7 +631,7 @@ Only pursue if (a)+(b) are not options and someone explicitly wants the residual - `~/bench/phase39_gate_sgemm_profile/20260701_085211` - short completion profile, diagnostic only because `-n 32` is not a canonical md5 gate; useful for confirming graph-time concat is a real kernel path. - `~/bench/phase39_gate_sgemm_profile/phase27_reanalysis` - Phase27 serving profile re-analysis used to reject graph-time fused gate weight concat; `concat_layout=459.84ms` (`2.29%`) in the serving kernel window. - `~/bench/phase40_max_concurrency/20260701_090012` - max-concurrency C1 check at `NPL=128/192/256`, `PTOK=128`, `GEN=64`, `PARALLEL=256`, `CTX=262144`; pre/post MoE/dense md5 and `MUL_MAT`/`MUL_MAT_ID` gates green, but vLLM also fit at `n=256` and stayed ahead (`paged_decode_over_vllm=0.6354`, `paged_agg_over_vllm=0.4721`). -- `~/bench/phase41_low_concurrency/20260701_091437` - low-concurrency D1 check at `NPL=1/8/32`, `PTOK=128`, `GEN=64`, `PARALLEL=32`, `CTX=32768`; pre/post MoE/dense md5 and `MUL_MAT`/`MUL_MAT_ID` gates green; paged is `0.7493`, `0.7518`, and `0.6649` of vLLM decode at `n=1/8/32`, with TTFT still much worse by `n=8/32`. +- `~/bench/phase41_low_concurrency/20260701_091437` - low-concurrency serving check at `NPL=1/8/32`, `PTOK=128`, `GEN=64`, `PARALLEL=32`, `CTX=32768`; pre/post MoE/dense md5 and `MUL_MAT`/`MUL_MAT_ID` gates green; paged is `0.7493`, `0.7518`, and `0.6649` of vLLM decode at `n=1/8/32`, with TTFT still much worse by `n=8/32`; does not reopen D1. - Per-engine logs `~/bench/COMBINED_{paged,vllm}_{MOE,DENSE}_server.log`; `~/bench/BENCHMARK_PROGRESS.md`. - Graph-node-traced high-N profiles: `~/highN_prof2/*.nsys-rep` (paged npl=256), `~/highN_vllm/*.nsys-rep` (vLLM), 2026-06-30. - A/B dirs: `~/bench/marlin_gate/`, `~/bench/gdn_p1_ab/`. diff --git a/backend/cpp/llama-cpp-localai-paged/docs/VLLM_PARITY_LEVER_MAP.md b/backend/cpp/llama-cpp-localai-paged/docs/VLLM_PARITY_LEVER_MAP.md index 7764614c5..0f7a4113e 100644 --- a/backend/cpp/llama-cpp-localai-paged/docs/VLLM_PARITY_LEVER_MAP.md +++ b/backend/cpp/llama-cpp-localai-paged/docs/VLLM_PARITY_LEVER_MAP.md @@ -1038,10 +1038,10 @@ the memory-footprint advantage as a parity claim at this tested point; any future C1 retry must push beyond it and keep md5 plus `MUL_MAT`/`MUL_MAT_ID` gates. -### Phase 41 low-concurrency D1 check +### Phase 41 low-concurrency serving check -Phase 41 measured the low-concurrency serving regime where D1/full-step graph -capture should be most useful. Artifact: +Phase 41 measured the low-concurrency serving regime where any remaining +host/scheduler gap should be most visible. Artifact: `/home/mudler/bench/phase41_low_concurrency/20260701_091437`. The run used `PARALLEL=32`, `CTX=32768`, `PTOK=128`, `GEN=64`, `NPL="1 8 32"`, and `OPS=MUL_MAT,MUL_MAT_ID`. @@ -1058,10 +1058,35 @@ Result: | 8 | `0.7518` | `0.7398` | `0.6334` | `3.1425` | | 32 | `0.6649` | `0.6397` | `0.5282` | `3.4014` | -Decision: D1 remains a real low-concurrency/latency lever, but Phase41 does not -make it a shortcut to parity. The implementation gate remains a separately built -A/B with md5 plus `MUL_MAT`/`MUL_MAT_ID` checks, and TTFT evidence keeps prefill -GDN/MoE work in scope for serving quality. +Decision: low-concurrency remains a gap, but Phase41 does not reopen +D1/full-step graph capture. Patch `0043` already ships grouped-MMQ full-step +decode graph capture default-on, Phase34 found `host_sync=0/4096`, and S3 is +intentionally default-off because it hurts TTFT/end-to-end throughput. Treat +D1 as closed on the current GB10 path unless a fresh route trace proves a +host-sync fallback or graph-disable condition has returned. TTFT evidence keeps +prefill GDN/MoE work in scope for serving quality. + +### Phase 42 target reconciliation + +Phase 42 challenged the current target list with three read-only subagent +reviews: + +- D1/full-step graph capture: closed on current GB10 path. `0040` S1 is + default-on graph reuse, `0041` S3 is opt-in only, and `0043` D1 is default-on + grouped-MMQ full-step CUDA graph capture. +- GDN prefill: the shipped GB10 wins are `0046`/`0047`; later C32 slab, + QS-early, and Global-Ai32 variants were correctness-clean but slower. Do not + add another low-conflict GDN reorder on GB10. +- W4A16 / prefill GEMM: `0033`/`0034`/`0035` remain default-off; `0048`-`0050` + improved forced W4A16 only marginally and still did not beat default MMQ. Do + not add another small W4A16 body/metadata tweak. + +The next small source candidate, if we stay on GB10, is the persistent/load-time +F32 combined gate projection from Phase38/39: combine `ffn_gate_inp.weight` and +`ffn_gate_inp_shexp.weight` once, issue one F32 gate matmul, and split/view the +outputs. It must be default-off, avoid graph-time `ggml_concat()`, and pass +MoE/dense md5 plus `MUL_MAT`/`MUL_MAT_ID`; if md5 changes, run KL before any +serving benchmark. Relevant files (all absolute): `/home/mudler/_git/LocalAI/.claude/worktrees/feat+paged-attention/backend/cpp/llama-cpp-localai-paged/docs/{DECODE_SERVING_SCOPE.md,PREFILL_GEMM_SCOPE.md,PREFILL_GEMM_RESULTS.md,TENSORCORE_GDN_SCOPE.md,final_benchmark.csv}`, `.../README.md`, `.../patches/paged/0034-feat-paged-native-NVFP4-W4A4-FP4-MMA-large-M-prefill.patch` (P1/P2), `.../patches/paged/0042-feat-paged-fused-residual-add-RMS-norm-weight-multip.patch` (P7), `.../patches/paged/0031` (P4), `0025` (D1), `0018/0022` (D4/D5), `0009/0010` (D3/D6/D7); graph source `/home/mudler/_git/LocalAI/backend/cpp/llama-cpp-paged-dev/src/{models/qwen35moe.cpp,models/delta-net-base.cpp,llama-graph.cpp}`. diff --git a/docs/superpowers/plans/2026-07-01-low-concurrency-phase41.md b/docs/superpowers/plans/2026-07-01-low-concurrency-phase41.md index 15bd380b4..6ca840aec 100644 --- a/docs/superpowers/plans/2026-07-01-low-concurrency-phase41.md +++ b/docs/superpowers/plans/2026-07-01-low-concurrency-phase41.md @@ -93,7 +93,7 @@ vllm 32 454.6 592.4 17.43 5376.5 818.6 - [x] **Step 4: Record decision** -Decision: Phase41 confirms D1 remains relevant for low-concurrency/latency work, but the measured current-stack gap is around `0.75x` vLLM at `n=1/8` and `0.665x` at `n=32`, not an immediate parity bridge. TTFT remains the larger user-visible gap. +Decision: Phase41 confirms a low-concurrency/latency gap, but it does not reopen D1/full-step graph capture. Patch `0043` already ships grouped-MMQ full-step graph capture default-on, and Phase34 found `host_sync=0/4096`. The measured current-stack gap is around `0.75x` vLLM at `n=1/8` and `0.665x` at `n=32`; TTFT remains the larger user-visible gap. ### Task 3: Update Handoff Docs diff --git a/docs/superpowers/plans/2026-07-01-target-reconciliation-phase42.md b/docs/superpowers/plans/2026-07-01-target-reconciliation-phase42.md new file mode 100644 index 000000000..2b401839f --- /dev/null +++ b/docs/superpowers/plans/2026-07-01-target-reconciliation-phase42.md @@ -0,0 +1,108 @@ +# Target Reconciliation Phase42 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Reconcile the post-Phase41 target list so the next parity phase does not chase a closed D1/GDN/W4A16 premise. + +**Architecture:** Use read-only parallel subagent analysis over D1 graph capture, GDN prefill, and W4A16/MoE prefill GEMM. Record the resulting target decision in the parity docs. + +**Tech Stack:** LocalAI docs, llama.cpp patch mirrors, `/home/mudler/_git/llama.cpp` fork, Git. + +--- + +### Task 1: Run Parallel Target Reviews + +**Files:** +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0040-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0041-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0043-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0031-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0046-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0047-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0033-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0034-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0035-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0048-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0049-*.patch` +- Read: `backend/cpp/llama-cpp-localai-paged/patches/paged/0050-*.patch` + +- [x] **Step 1: Review D1** + +Ask a read-only explorer to reconcile whether D1/full-step graph capture is shipped or still open. + +Observed: + +```text +D1/full-step MoE decode CUDA graph capture is shipped and default-on. +The host-sync premise is closed/refuted for current GB10 NVFP4 grouped-MMQ decode. +``` + +- [x] **Step 2: Review GDN** + +Ask a read-only explorer to inspect GDN tensor-core/chunking state. + +Observed: + +```text +0046/0047 are shipped GB10 wins. +0031 scalar chunking stayed opt-in/slower. +C32 slab, QS-early, and Global-Ai32 were correctness-clean but slower. +Do not add another GDN GB10 patch. +``` + +- [x] **Step 3: Review W4A16/GEMM** + +Ask a read-only explorer to inspect the prefill GEMM / W4A16 state. + +Observed: + +```text +0033/0034/0035 are default-off. +0048/0049/0050 improve forced W4A16 only marginally. +Production defaults still use FP4-MMQ. +Do not add another small W4A16 body/metadata patch. +``` + +### Task 2: Record Phase42 Decision + +**Files:** +- Modify: `backend/cpp/llama-cpp-localai-paged/docs/GB10_PARITY_PHASE0_RESULTS.md` +- Modify: `backend/cpp/llama-cpp-localai-paged/docs/VLLM_PARITY_LEVER_MAP.md` +- Modify: `backend/cpp/llama-cpp-localai-paged/docs/PARITY_HANDOFF.md` +- Modify: `docs/superpowers/plans/2026-07-01-low-concurrency-phase41.md` +- Create: `docs/superpowers/plans/2026-07-01-target-reconciliation-phase42.md` + +- [x] **Step 1: Correct Phase41 D1 wording** + +Change Phase41 from "D1 remains relevant" to "low-concurrency remains a gap, but D1 graph capture is already shipped/default-on and not reopened." + +- [x] **Step 2: Add Phase42 decision** + +Record: + +```text +D1: closed on current GB10 path. +GDN: low-conflict GB10 work exhausted. +W4A16/GEMM: micro-patch track exhausted. +Next small GB10 source candidate: persistent/load-time F32 combined gate projection. +``` + +- [x] **Step 3: Verify and commit** + +Run: + +```bash +git diff --check +git status --short +``` + +Commit with: + +```bash +git add backend/cpp/llama-cpp-localai-paged/docs/GB10_PARITY_PHASE0_RESULTS.md \ + backend/cpp/llama-cpp-localai-paged/docs/VLLM_PARITY_LEVER_MAP.md \ + backend/cpp/llama-cpp-localai-paged/docs/PARITY_HANDOFF.md \ + docs/superpowers/plans/2026-07-01-low-concurrency-phase41.md +git add -f docs/superpowers/plans/2026-07-01-target-reconciliation-phase42.md +git commit -m "docs(paged): reconcile next parity target" -m "Assisted-by: Codex:gpt-5" +```