mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 22:33:54 -04:00
refactor(paged): stock llama-cpp is patch-free; paged backend owns its patch series
Move ALL paged-attention content out of the stock backend/cpp/llama-cpp backend and into backend/cpp/llama-cpp-localai-paged, so the stock backend is pure upstream llama.cpp and the paged backend owns and applies its own vendored patch series. - Delete the dead early-exploration scaffold backend/cpp/llama-cpp/paged/ (kernel/w4a16 Marlin scaffold, standalone paged_kv_manager, bench/loadgen, its own 0001-0002 patches, dense-era design docs, tests). Zero references repo-wide. - Move backend/cpp/llama-cpp/patches/ (the 28-patch paged series + paged/README + 3 operational docs, plus the kernel/ scaffold patch and the top-level paged README/BENCHMARKS) to backend/cpp/llama-cpp-localai-paged/patches/. The stock backend keeps no patches/ dir; it had no non-paged base patches. - Purify the stock backend: remove the LLAMA_PAGED make variable, the patches/paged apply loop, and the LLAMA_PAGED passthrough to prepare.sh; remove the paged-series handling from prepare.sh. The stock llama.cpp target now only clones the pin and applies its own (currently empty) base patches/ series. The runtime paged option hooks in the shared grpc-server.cpp are untouched (inert without the patches). - The paged backend's Makefile now applies its OWN patches/paged/0*.patch onto each freshly cloned tree via strict git apply (apply-paged-patches), after the copied stock infra clones the pin and applies base patches. - Repoint every reference to the old patches/paged path: the upstream canary workflow + apply script, bump_deps.yaml, gallery/index.yaml, the docs, backend/index.yaml, backend-matrix.yml, the top-level Makefile comments, and the moved PIN_SYNC / README docs. Drop the now-removed LLAMA_PAGED=on build-toggle from comments. Verified: the full 28-patch series applies strict-clean (git apply, exit 0) to a clean ggml-org/llama.cpp checkout at the pinned c299a92c, and the repointed canary apply script resolves and applies the series end to end. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
1 parent
fb2dc33d52
commit
78fac9a28f
87 files changed
+109
-3997
No files matched your search
@@ -5073,7 +5073,7 @@ includeDarwin:
|
||||
lang: "go"
|
||||
# llama-cpp-localai-paged on Darwin: same bespoke CPU_ALL_VARIANTS + Metal build
|
||||
# as stock llama-cpp (driven by make backends/llama-cpp-localai-paged-darwin),
|
||||
# reusing backend/cpp/llama-cpp sources with LLAMA_PAGED=on. lang=go selects the
|
||||
# reusing backend/cpp/llama-cpp sources, with the paged patch series applied by the wrapper. lang=go selects the
|
||||
# runner/toolchain only; the source path is C++. Metal delivers paged-KV (the
|
||||
# NVFP4 FP4-MMA fast path is CUDA/Blackwell-only) and the GDN/conv fused ops have
|
||||
# no Metal kernel, so a gated-DeltaNet (qwen35) model falls back to the CPU
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# paged-canary-apply.sh - apply the vendored paged-attention patch series
|
||||
# (backend/cpp/llama-cpp/patches/paged/0001-0030) to a llama.cpp checkout, the
|
||||
# (backend/cpp/llama-cpp-localai-paged/patches/paged/0001-0030) to a llama.cpp checkout, the
|
||||
# same way the build does, but tolerating the ONE known-benign pre-existing
|
||||
# quirk in the series. Used by the early-warning canary
|
||||
# (.github/workflows/llama-cpp-paged-canary.yml) so it only goes red on a REAL
|
||||
# upstream break, never on that quirk.
|
||||
#
|
||||
# Usage: paged-canary-apply.sh <llama.cpp-checkout-dir> <patches-dir>
|
||||
# <patches-dir> is normally backend/cpp/llama-cpp/patches (it holds the
|
||||
# <patches-dir> is normally backend/cpp/llama-cpp-localai-paged/patches (it holds the
|
||||
# top-level base series 0*.patch, currently empty, and the paged/ subseries).
|
||||
#
|
||||
# Exit 0 = the whole series applied -> patches still fit upstream.
|
||||
@@ -27,7 +27,7 @@
|
||||
# missing-file hunk rejects the whole patch - and because 0021/0022/0026/0028
|
||||
# build on 0019's code, the rejection cascades to them too. This is a
|
||||
# PRE-EXISTING shipped-series defect, present identically on every pin, NOT an
|
||||
# upstream break (see backend/cpp/llama-cpp/patches/paged/PIN_SYNC_c299a92c.md
|
||||
# upstream break (see backend/cpp/llama-cpp-localai-paged/patches/paged/PIN_SYNC_c299a92c.md
|
||||
# and README.md). We exclude ONLY that dev-doc path and still
|
||||
# apply 0019's real code hunks atomically, so a genuine code-hunk break in 0019
|
||||
# still fails the canary. prepare.sh tolerates the same hunk via
|
||||
@@ -53,7 +53,7 @@ apply_one() {
|
||||
echo "paged-canary: applying $(basename "$p")"
|
||||
if ! git apply --verbose "$@" "$p"; then
|
||||
echo "::error::paged patch no longer applies to the upstream llama.cpp tip: $(basename "$p")"
|
||||
echo "::error::upstream drifted past the vendored paged series - run a PIN_SYNC (backend/cpp/llama-cpp/patches/paged/PIN_SYNC_c299a92c.md), do NOT bump the pin blindly"
|
||||
echo "::error::upstream drifted past the vendored paged series - run a PIN_SYNC (backend/cpp/llama-cpp-localai-paged/patches/paged/PIN_SYNC_c299a92c.md), do NOT bump the pin blindly"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
@@ -11,7 +11,7 @@ jobs:
|
||||
matrix:
|
||||
# NOTE: there is intentionally NO entry for the llama-cpp-localai-paged
|
||||
# backend. It carries a vendored paged-attention patch series
|
||||
# (backend/cpp/llama-cpp/patches/paged/) hand-verified bit-exact against
|
||||
# (backend/cpp/llama-cpp-localai-paged/patches/paged/) hand-verified bit-exact against
|
||||
# ONE specific llama.cpp tip; a naive nightly bump would move the tip out
|
||||
# from under the patches and break `git apply` at build time. Its pin is
|
||||
# therefore decoupled (its own LLAMA_VERSION in
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
name: 'llama.cpp paged patches: upstream canary'
|
||||
|
||||
# EARLY-WARNING CANARY for the vendored paged-attention patch series
|
||||
# (backend/cpp/llama-cpp/patches/paged/0001-0030).
|
||||
# (backend/cpp/llama-cpp-localai-paged/patches/paged/0001-0030).
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# The paged backend (backend/cpp/llama-cpp-localai-paged) pins its OWN verified
|
||||
@@ -17,7 +17,7 @@ name: 'llama.cpp paged patches: upstream canary'
|
||||
# RED HERE means: time to run a PIN_SYNC (rebase the patches onto the new tip,
|
||||
# pass the bit-exact gate on the GPU, re-export the .patch files, THEN advance
|
||||
# the pin in backend/cpp/llama-cpp-localai-paged/Makefile). See
|
||||
# backend/cpp/llama-cpp/patches/paged/PIN_SYNC_c299a92c.md.
|
||||
# backend/cpp/llama-cpp-localai-paged/patches/paged/PIN_SYNC_c299a92c.md.
|
||||
#
|
||||
# SIGNAL-ONLY: this workflow moves no pinned version, ships nothing, and is fully
|
||||
# decoupled from bump_deps - so the main dep-bump PR stays green regardless. A
|
||||
@@ -91,7 +91,7 @@ jobs:
|
||||
run: |
|
||||
bash .github/scripts/paged-canary-apply.sh \
|
||||
/tmp/llama.cpp \
|
||||
"$PWD/backend/cpp/llama-cpp/patches"
|
||||
"$PWD/backend/cpp/llama-cpp-localai-paged/patches"
|
||||
echo "- apply: full paged series applies to the upstream tip :white_check_mark:" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
compile:
|
||||
@@ -141,12 +141,16 @@ jobs:
|
||||
cp -a /opt/grpc/. /usr/local/
|
||||
|
||||
# Pre-populate the llama.cpp checkout at the latest tip with the
|
||||
# paged series applied via the tolerant canary apply (so the benign
|
||||
# 0019 dev-doc hunk does not abort the build). Because
|
||||
# backend/cpp/llama-cpp/llama.cpp now exists, the Makefile
|
||||
# llama.cpp target (strict clone + git apply) is skipped and
|
||||
# prepare.sh sees the paged sentinel and skips re-applying - so we
|
||||
# drive the REAL grpc-server build path on top of our apply.
|
||||
# paged series applied via the tolerant canary apply. Because
|
||||
# backend/cpp/llama-cpp/llama.cpp now exists, the stock Makefile's
|
||||
# llama.cpp target (clone + base-patch apply) is skipped and the
|
||||
# now patch-free prepare.sh only copies the grpc-server sources -
|
||||
# so we drive the REAL grpc-server build path on top of our paged
|
||||
# apply. The stock llama-cpp backend no longer carries the paged
|
||||
# series (it lives in backend/cpp/llama-cpp-localai-paged/patches/
|
||||
# paged); we build it here in the stock dir only because that is
|
||||
# where the shared build infra (Makefile / grpc-server.cpp /
|
||||
# CMakeLists.txt / prepare.sh) lives.
|
||||
cd backend/cpp/llama-cpp/
|
||||
mkdir -p llama.cpp
|
||||
cd llama.cpp
|
||||
@@ -157,15 +161,16 @@ jobs:
|
||||
cd /LocalAI
|
||||
bash .github/scripts/paged-canary-apply.sh \
|
||||
backend/cpp/llama-cpp/llama.cpp \
|
||||
"$PWD/backend/cpp/llama-cpp/patches"
|
||||
"$PWD/backend/cpp/llama-cpp-localai-paged/patches"
|
||||
|
||||
# Cheapest real CUDA build that proves the patches compile: one
|
||||
# CUDA arch, cublas, paged on. CMAKE_ARGS is passed via the
|
||||
# environment (not as a make arg) so the Makefile += flags are
|
||||
# still appended, exactly like .docker/llama-cpp-localai-paged-compile.sh.
|
||||
# CUDA arch, cublas. CMAKE_ARGS is passed via the environment (not
|
||||
# as a make arg) so the Makefile += flags are still appended,
|
||||
# exactly like .docker/llama-cpp-localai-paged-compile.sh. The paged
|
||||
# series is already applied to the checkout above, so the stock
|
||||
# build just compiles the patched tree.
|
||||
cd backend/cpp/llama-cpp/
|
||||
BUILD_TYPE=cublas \
|
||||
LLAMA_PAGED=on \
|
||||
CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=80" \
|
||||
make grpc-server
|
||||
test -x grpc-server
|
||||
|
||||
@@ -1142,8 +1142,8 @@ backends/llama-cpp-darwin: build
|
||||
./local-ai backends install "ocifile://$(abspath ./backend-images/llama-cpp.tar)"
|
||||
|
||||
# llama-cpp-localai-paged on Darwin: same bespoke CPU_ALL_VARIANTS + Metal build as
|
||||
# stock llama-cpp (otool dylib bundling), driven through the paged wrapper Makefile
|
||||
# with LLAMA_PAGED=on. Mirrors backends/llama-cpp-darwin.
|
||||
# stock llama-cpp (otool dylib bundling), driven through the paged wrapper Makefile,
|
||||
# which applies its own vendored paged patch series. Mirrors backends/llama-cpp-darwin.
|
||||
backends/llama-cpp-localai-paged-darwin: build
|
||||
bash ./scripts/build/llama-cpp-localai-paged-darwin.sh
|
||||
./local-ai backends install "ocifile://$(abspath ./backend-images/llama-cpp-localai-paged.tar)"
|
||||
@@ -1198,7 +1198,7 @@ BACKEND_IK_LLAMA_CPP = ik-llama-cpp|ik-llama-cpp|.|false|false
|
||||
# Reuses backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
|
||||
BACKEND_TURBOQUANT = turboquant|turboquant|.|false|false
|
||||
# llama-cpp-localai-paged = stock llama.cpp grpc-server + the LocalAI paged-attention
|
||||
# patch series (LLAMA_PAGED=on). Reuses backend/cpp/llama-cpp sources via a thin
|
||||
# patch series (vendored in this wrapper backend). Reuses backend/cpp/llama-cpp sources via a thin
|
||||
# wrapper Makefile (same upstream pin as stock llama-cpp; no fork, no patch-grpc-server).
|
||||
BACKEND_LLAMA_CPP_LOCALAI_PAGED = llama-cpp-localai-paged|llama-cpp-localai-paged|.|false|false
|
||||
# ds4 is antirez/ds4, a DeepSeek V4 Flash-specific inference engine.
|
||||
|
||||
@@ -1,33 +1,36 @@
|
||||
|
||||
# llama-cpp-localai-paged is LocalAI's paged-attention llama.cpp variant. It
|
||||
# builds upstream llama.cpp with the LocalAI paged-attention patch series
|
||||
# (backend/cpp/llama-cpp/patches/paged/) applied on top (LLAMA_PAGED=on). It
|
||||
# reuses backend/cpp/llama-cpp's grpc-server.cpp / CMakeLists.txt / prepare.sh
|
||||
# sources verbatim via a thin wrapper.
|
||||
# (patches/paged/, vendored in THIS backend) applied on top. It reuses
|
||||
# backend/cpp/llama-cpp's grpc-server.cpp / CMakeLists.txt / prepare.sh / Makefile
|
||||
# sources verbatim via a thin wrapper - the stock llama-cpp backend is pure
|
||||
# upstream and carries NONE of the paged patches; this backend OWNS them.
|
||||
#
|
||||
# Pin handling (mirrors the turboquant wrapper, the precedent this is modelled
|
||||
# on): the paged patch series is hand-verified bit-exact against ONE specific
|
||||
# llama.cpp tip and re-exported by the manual PIN_SYNC process
|
||||
# (backend/cpp/llama-cpp/patches/paged/PIN_SYNC_*.md). A naive pin bump would
|
||||
# move the tip out from under the patches and break `git apply` at build time,
|
||||
# so this backend OWNS its pin (LLAMA_VERSION below) instead of inheriting the
|
||||
# auto-bumped stock pin from backend/cpp/llama-cpp/Makefile. The override is
|
||||
# forced into every copied build via `LLAMA_VERSION=$(LLAMA_VERSION)`. There is
|
||||
# deliberately NO bump_deps.yaml entry for it: it is advanced ONLY by PIN_SYNC,
|
||||
# never nightly. (turboquant CAN auto-bump because its fork branch carries the
|
||||
# patches; the paged series is vendored as .patch files here, so it cannot.)
|
||||
# (patches/paged/PIN_SYNC_*.md). A naive pin bump would move the tip out from
|
||||
# under the patches and break `git apply` at build time, so this backend OWNS
|
||||
# its pin (LLAMA_VERSION below) instead of inheriting the auto-bumped stock pin
|
||||
# from backend/cpp/llama-cpp/Makefile. The override is forced into every copied
|
||||
# build via `LLAMA_VERSION=$(LLAMA_VERSION)`. There is deliberately NO
|
||||
# bump_deps.yaml entry for it: it is advanced ONLY by PIN_SYNC, never nightly.
|
||||
# (turboquant CAN auto-bump because its fork branch carries the patches; the
|
||||
# paged series is vendored as .patch files here, so it cannot.)
|
||||
#
|
||||
# - NO patch-grpc-server.sh and NO apply-patches.sh: the shared
|
||||
# grpc-server.cpp already carries the (runtime-gated) paged option hooks,
|
||||
# and the paged patch series is applied by the copied llama-cpp Makefile's
|
||||
# own `llama.cpp` target whenever LLAMA_PAGED=on (which we force below).
|
||||
# - NO patch-grpc-server.sh and NO apply-patches.sh: the shared grpc-server.cpp
|
||||
# already carries the (runtime-gated) paged option hooks, and the paged patch
|
||||
# series (patches/paged/) is applied by THIS Makefile's own apply step onto
|
||||
# the freshly cloned tree, using the same strict `git apply` method the stock
|
||||
# build uses for base patches. The stock llama-cpp Makefile applies only its
|
||||
# own (currently empty) base patches/ series, never the paged one.
|
||||
|
||||
# Manually pin-synced llama.cpp tip the paged patch series is verified against.
|
||||
# Decoupled from the auto-bumped stock pin in backend/cpp/llama-cpp/Makefile so
|
||||
# the nightly llama.cpp bump cannot silently break the vendored paged patches.
|
||||
# Advance ONLY via the PIN_SYNC process (rebase patches + bit-exact gate +
|
||||
# re-export), then update this value. See:
|
||||
# backend/cpp/llama-cpp/patches/paged/PIN_SYNC_*.md
|
||||
# backend/cpp/llama-cpp-localai-paged/patches/paged/PIN_SYNC_*.md
|
||||
#
|
||||
# This pin = the manual, verified sync. The signal telling you WHEN to do the
|
||||
# next sync is the early-warning canary
|
||||
@@ -47,28 +50,49 @@ ARCH?=$(shell uname -m)
|
||||
|
||||
CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
LLAMA_CPP_DIR := $(CURRENT_MAKEFILE_DIR)/../llama-cpp
|
||||
# OUR vendored paged-attention patch series. Owned by this backend; the stock
|
||||
# llama-cpp backend no longer carries it. Applied onto each freshly cloned
|
||||
# llama.cpp tree by apply-paged-patches below (strict git apply).
|
||||
PAGED_PATCHES_DIR := $(CURRENT_MAKEFILE_DIR)/patches/paged
|
||||
|
||||
GREEN := \033[0;32m
|
||||
RESET := \033[0m
|
||||
|
||||
# Apply OUR vendored paged-attention patch series (patches/paged/0*.patch) onto a
|
||||
# freshly cloned llama.cpp tree ($(1)) using the SAME strict git-apply method the
|
||||
# stock build uses for its base patches (backend/cpp/llama-cpp/Makefile `llama.cpp`
|
||||
# target). Strict: any patch that no longer applies aborts the build (exit 1) -
|
||||
# that is the signal to run a PIN_SYNC, never to bump the pin blindly. The series
|
||||
# is owned by THIS backend, not by the now-pure stock llama-cpp backend.
|
||||
define apply-paged-patches
|
||||
cd $(1) && \
|
||||
for p in $(PAGED_PATCHES_DIR)/0*.patch; do \
|
||||
[ -e "$$p" ] || continue; \
|
||||
echo "applying llama.cpp PAGED patch: $$p"; \
|
||||
git apply --verbose "$$p" || { echo "paged patch failed: $$p"; exit 1; }; \
|
||||
done
|
||||
endef
|
||||
|
||||
# Each flavor target:
|
||||
# 1. copies backend/cpp/llama-cpp/ (grpc-server.cpp + prepare.sh +
|
||||
# CMakeLists.txt + Makefile) into a sibling
|
||||
# llama-cpp-localai-paged-<flavor>-build directory;
|
||||
# 2. clones the SAME upstream llama.cpp pin into that copy and applies the
|
||||
# base AND paged patch series via the copy's own `llama.cpp` target with
|
||||
# LLAMA_PAGED=on;
|
||||
# 3. runs the copy's `grpc-server` target (LLAMA_PAGED=on) and copies the
|
||||
# produced binary up as llama-cpp-localai-paged-<flavor>.
|
||||
# We patch only the *copy*, never the original under backend/cpp/llama-cpp/, so
|
||||
# the stock llama-cpp build stays untouched.
|
||||
# 2. clones OUR pinned upstream llama.cpp into that copy via the copy's own
|
||||
# `llama.cpp` target (which applies the stock base patches/ series, normally
|
||||
# empty), then applies THIS backend's paged patch series (patches/paged/)
|
||||
# onto the cloned tree with strict `git apply` (apply-paged-patches);
|
||||
# 3. runs the copy's `grpc-server` target and copies the produced binary up as
|
||||
# llama-cpp-localai-paged-<flavor>.
|
||||
# We clone+patch only the *copy*, never the original under backend/cpp/llama-cpp/,
|
||||
# so the stock llama-cpp build stays untouched and patch-free.
|
||||
define paged-build
|
||||
rm -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build
|
||||
cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build
|
||||
$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build purge
|
||||
$(info $(GREEN)I llama-cpp-localai-paged build info:$(1)$(RESET))
|
||||
LLAMA_VERSION=$(LLAMA_VERSION) LLAMA_PAGED=on $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build llama.cpp
|
||||
CMAKE_ARGS="$(CMAKE_ARGS) $(2)" TARGET="$(3)" LLAMA_VERSION=$(LLAMA_VERSION) LLAMA_PAGED=on \
|
||||
LLAMA_VERSION=$(LLAMA_VERSION) $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build llama.cpp
|
||||
$(call apply-paged-patches,$(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build/llama.cpp)
|
||||
CMAKE_ARGS="$(CMAKE_ARGS) $(2)" TARGET="$(3)" LLAMA_VERSION=$(LLAMA_VERSION) \
|
||||
$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build grpc-server
|
||||
cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-$(1)-build/grpc-server llama-cpp-localai-paged-$(1)
|
||||
endef
|
||||
@@ -97,8 +121,9 @@ llama-cpp-localai-paged-cpu-all:
|
||||
cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-cpu-all-build
|
||||
$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-cpu-all-build purge
|
||||
$(info $(GREEN)I llama-cpp-localai-paged build info:cpu-all-variants$(RESET))
|
||||
LLAMA_VERSION=$(LLAMA_VERSION) LLAMA_PAGED=on $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-cpu-all-build llama.cpp
|
||||
SHARED_LIBS=ON EXTRA_CMAKE_ARGS="-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON" TARGET="--target grpc-server --target ggml" LLAMA_VERSION=$(LLAMA_VERSION) LLAMA_PAGED=on \
|
||||
LLAMA_VERSION=$(LLAMA_VERSION) $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-cpu-all-build llama.cpp
|
||||
$(call apply-paged-patches,$(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-cpu-all-build/llama.cpp)
|
||||
SHARED_LIBS=ON EXTRA_CMAKE_ARGS="-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON" TARGET="--target grpc-server --target ggml" LLAMA_VERSION=$(LLAMA_VERSION) \
|
||||
$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-cpu-all-build grpc-server
|
||||
cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-localai-paged-cpu-all-build/grpc-server llama-cpp-localai-paged-cpu-all
|
||||
rm -rf ggml-shared-libs && mkdir -p ggml-shared-libs
|
||||
|
||||
File renamed without changes.
+11
-7
@@ -16,7 +16,7 @@ patch needs fixing, and the failure points at exactly which step the upstream ch
|
||||
|
||||
| # | Patch | What | Verifies |
|
||||
|---|-------|------|----------|
|
||||
| 0001 | `0001-vendor-paged-kv-manager.patch` | Add `src/paged-kv-manager.{h,cpp}` (vLLM-parity block manager, CPU foundation) + CMake; no behavior change | builds; unit-tested separately under `../paged/` |
|
||||
| 0001 | `0001-vendor-paged-kv-manager.patch` | Add `src/paged-kv-manager.{h,cpp}` (vLLM-parity block manager, CPU foundation) + CMake; no behavior change | builds; unit-tested separately |
|
||||
| 0002 | `0002-paged-kv-storage.patch` | Shared block-pool KV tensor + `set_rows`-by-slot writes, behind `LLAMA_KV_PAGED` | builds; write/gather round-trip |
|
||||
| 0003 | `0003-paged-gather-read.patch` | `build_attn_paged` gather-read in `llama-graph.cpp` | **Gate 0**: token-identical greedy gen, single + multi-seq |
|
||||
| 0004 | `0004-paged-ondemand-alloc.patch` | On-demand block allocation via PagedKVManager | max concurrent seqs before OOM |
|
||||
@@ -35,21 +35,25 @@ git checkout <LLAMA_VERSION from ../Makefile>
|
||||
git checkout -b paged
|
||||
|
||||
# 2. apply the current series (each becomes a commit), or develop the next patch
|
||||
git am /path/to/backend/cpp/llama-cpp/patches/00*.patch # or `git apply` + commit per patch
|
||||
git am /path/to/backend/cpp/llama-cpp-localai-paged/patches/paged/00*.patch # or `git apply` + commit per patch
|
||||
|
||||
# 3. iterate a phase as ONE commit, then export the whole series 1:1
|
||||
git format-patch <LLAMA_VERSION>..paged -o /path/to/backend/cpp/llama-cpp/patches/ --zero-commit -N
|
||||
git format-patch <LLAMA_VERSION>..paged -o /path/to/backend/cpp/llama-cpp-localai-paged/patches/paged/ --zero-commit -N
|
||||
|
||||
# 4. on a pin bump: rebase `paged` onto the new pin; only conflicting patches need edits; re-export.
|
||||
```
|
||||
|
||||
## Build integration
|
||||
|
||||
`../Makefile`'s `llama.cpp:` target runs, after `git checkout -b build $(LLAMA_VERSION)`:
|
||||
The series is owned by this backend (`backend/cpp/llama-cpp-localai-paged`), not by the stock
|
||||
`llama-cpp` backend, which is pure upstream. `../Makefile` (the paged wrapper) clones the pinned
|
||||
`llama.cpp` via the copied stock build infra, then applies this series onto the cloned tree with the
|
||||
same strict `git apply` the stock build uses for base patches:
|
||||
```
|
||||
for p in $(CURRENT_MAKEFILE_DIR)/patches/0*.patch; do git apply --verbose "$p"; done
|
||||
for p in $(PAGED_PATCHES_DIR)/0*.patch; do git apply --verbose "$p" || exit 1; done
|
||||
```
|
||||
All variants (avx/avx2/avx512/cuda/…) copy the patched `llama.cpp/` tree, so the series ships everywhere.
|
||||
All variants (avx/avx2/avx512/cuda/…) clone + apply into their own build copy, so the series ships
|
||||
everywhere without ever touching the stock `llama-cpp` source tree.
|
||||
|
||||
## Status
|
||||
|
||||
@@ -78,5 +82,5 @@ by itself reach vLLM throughput parity, because the measured prefill bottleneck
|
||||
(Lever 3: `mul_mat_q<MXFP4>` ~22 TFLOP/s, ~27× behind vLLM) — a *per-token compute* gap that paging does not
|
||||
touch. Paged attention closes the **concurrency/memory** gap (more sequences, prefix reuse); the prefill/throughput
|
||||
gap additionally needs the tcgen05/CUTLASS grouped-GEMM (deferred, upstream-grade, no shortcut — see
|
||||
`../paged/UPSTREAM_GGML_ISSUE.md` and `DGX_BLACKWELL_PLAN.md`). So full vLLM parity = this series **AND** the
|
||||
`paged/README.md`). So full vLLM parity = this series **AND** the
|
||||
kernel; neither alone suffices.
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
+3
-2
@@ -21,7 +21,8 @@ Unlike the `9d5d882d` sync (which needed 4 patch re-exports), this bump required
|
||||
**zero patch changes**. The already-shipped source-only series (the result of the
|
||||
`7e1832b8` strip that removed all stray dev-doc hunks) applies to a fresh clean
|
||||
`ggml-org/llama.cpp` checkout at `c299a92c` with the build's own **strict
|
||||
`git apply`** (the `llama.cpp` target in `backend/cpp/llama-cpp/Makefile`:
|
||||
`git apply`** (the `apply-paged-patches` step in
|
||||
`backend/cpp/llama-cpp-localai-paged/Makefile`:
|
||||
`git apply --verbose "$p" || exit 1`) and reaches **exit 0** - every one of the
|
||||
28 patches reported "Applied patch ... cleanly", the sentinel
|
||||
`src/paged-kv-manager.cpp` was created, and there are **zero** stray
|
||||
@@ -94,7 +95,7 @@ here to keep the pin-bump diff minimal.
|
||||
|
||||
## Source of truth
|
||||
|
||||
The shipped `.patch` files under `backend/cpp/llama-cpp/patches/paged/` are the
|
||||
The shipped `.patch` files under `backend/cpp/llama-cpp-localai-paged/patches/paged/` are the
|
||||
source of truth and are unchanged by this bump. The DGX dev tree
|
||||
(`~/llama-paged-dev`, branch `paged`) was advanced to `c299a92c` for consistency;
|
||||
the pre-bump state is retained at `paged-prebump-9d5d882d-backup`.
|
||||
File renamed without changes.
File renamed without changes.
|
500 Internal Server Error
Gitea Version: 1.28.0+dev-477-g8b6ad49a5f |