mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 01:48:06 -04:00
fix(ci): rebuild backends when shared build inputs change (#10975)
The backend matrix path filter only matched files under a backend's own directory, so a change to shared build infrastructure rebuilt nothing at all: an empty matrix, every job green, and the change reaching no image. PR #10946 fixed scripts/build/package-gpu-libs.sh shipping a partial 4-of-8 cuDNN library set, which mixed versions with the venv's pip cuDNN and produced CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at inference time. It merged 1h48m after the weekly full-matrix cron had already run, so no backend image ever received the fix and nothing signalled that it had been un-shipped. Add a SHARED_BUILD_INPUTS table mapping each shared path to the narrowest set of matrix entries it can honestly invalidate, plus a generic rule for backend/Dockerfile.<x> (which each entry already names). A full matrix is 417 Linux + 56 Darwin builds, so package-gpu-libs.sh now rebuilds the 176 Python entries rather than everything. Unclassified files under scripts/build/ fall back to a full rebuild deliberately: over-building is recoverable, silently shipping nothing is not. Extract the filtering logic to scripts/lib/backend-filter.mjs so it can be unit-tested without bun, js-yaml or a GitHub API round-trip, and run those tests from the existing lint workflow via `make test-ci-scripts`. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]
committed by
GitHub
GitHub
parent
465d488c90
commit
6e52d0c2ef
@@ -34,7 +34,7 @@ The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `
|
||||
|
||||
**Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
|
||||
|
||||
**`scripts/changed-backends.js` registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/changed-backends.js:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
|
||||
**Path-filter registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/lib/backend-filter.mjs:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
|
||||
|
||||
```js
|
||||
if (item.dockerfile.endsWith("<your-dockerfile-suffix>")) {
|
||||
@@ -54,7 +54,9 @@ for (const e of m.include.filter(e => e.backend === '<your-backend>')) {
|
||||
}"
|
||||
```
|
||||
|
||||
A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/changed-backends.js`.
|
||||
A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/lib/backend-filter.mjs`.
|
||||
|
||||
If your backend consumes a *shared* build input that lives outside its own directory (a new script under `scripts/build/`, a new file copied into every image), add a rule to `SHARED_BUILD_INPUTS` in the same file — the per-backend prefix match cannot see those, and a miss ships your change to no image at all. See `scripts/lib/backend-filter_test.mjs` for the pattern; `make test-ci-scripts` runs it.
|
||||
|
||||
**`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=<sha>` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`):
|
||||
|
||||
@@ -115,7 +117,7 @@ Wiring a backend into `includeDarwin:` is more than the matrix entry:
|
||||
|
||||
1. **`includeDarwin:` entry** — `tag-suffix: "-metal-darwin-arm64-<backend>"`, `build-type: "metal"`, `lang: "go"` for go+ggml backends; omit `build-type` for the bespoke C++ ones (llama-cpp / ds4 / privacy-filter). Match an existing entry of the same shape.
|
||||
2. **`backend/index.yaml`** — add `metal:` to the backend's `capabilities` map (main and `-development`) and concrete `metal-<backend>` / `metal-<backend>-development` image entries pointing at the `-metal-darwin-arm64-<backend>` images.
|
||||
3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/changed-backends.js` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
|
||||
3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/lib/backend-filter.mjs` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
|
||||
4. **C++ proto gotcha** — if the backend compiles the generated gRPC/protobuf in a separate CMake target (e.g. `hw_grpc_proto`), that target must link `protobuf::libprotobuf` + `gRPC::grpc++` so the Homebrew include dirs propagate; otherwise macOS fails with `google/protobuf/runtime_version.h not found` (Linux hides this because apt headers sit in `/usr/include`).
|
||||
|
||||
The CI path filter only builds a backend on a PR when a file under its directory changes, so a darwin-only YAML edit builds nothing — touch a file under `backend/<lang>/<backend>/` (a one-line comment is enough) in the same PR.
|
||||
@@ -243,7 +245,7 @@ After adding a new backend, verify:
|
||||
|
||||
- [ ] Backend directory structure is complete with all necessary files
|
||||
- [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
|
||||
- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
|
||||
- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case (in `scripts/lib/backend-filter.mjs`) for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
|
||||
- [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
|
||||
- [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
|
||||
- [ ] Tag suffixes match between workflow file and index.yaml
|
||||
|
||||
@@ -114,6 +114,24 @@ Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynami
|
||||
- **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
|
||||
- **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
|
||||
|
||||
### Shared build inputs
|
||||
|
||||
The per-backend prefix match only sees files under a backend's own directory, so a change to shared build infrastructure would rebuild *nothing* — an empty matrix, every job green, and the change reaching no image. That silently un-shipped PR #10946 (a partial-cuDNN packaging fix in `scripts/build/package-gpu-libs.sh`), which merged 1h48m after the weekly cron and so sat unbuilt for a week.
|
||||
|
||||
`SHARED_BUILD_INPUTS` in `scripts/lib/backend-filter.mjs` closes that hole. Each rule maps a shared path to the narrowest set of matrix entries it can honestly invalidate, since a full matrix is 417 Linux + 56 Darwin builds:
|
||||
|
||||
| Changed path | Rebuilds |
|
||||
|---|---|
|
||||
| `backend/backend.proto` | everything (all languages compile or copy it) |
|
||||
| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
|
||||
| `backend/python/common/` | Python, Linux + Darwin |
|
||||
| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
|
||||
| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
|
||||
| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
|
||||
| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
|
||||
|
||||
Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
|
||||
|
||||
The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
|
||||
|
||||
## The `DEPS_REFRESH` cache-buster (Python backends)
|
||||
|
||||
2
.github/backend-matrix.yml
vendored
2
.github/backend-matrix.yml
vendored
@@ -23,7 +23,7 @@
|
||||
# checklist in .agents/adding-backends.md (includeDarwin entry, the index.yaml
|
||||
# `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD
|
||||
# branch for C/C++ backends, and the inferBackendPathDarwin case in
|
||||
# scripts/changed-backends.js so the path filter actually builds it).
|
||||
# scripts/lib/backend-filter.mjs so the path filter actually builds it).
|
||||
|
||||
# Linux matrix (consumed by backend-jobs).
|
||||
include:
|
||||
|
||||
9
.github/workflows/lint.yml
vendored
9
.github/workflows/lint.yml
vendored
@@ -57,3 +57,12 @@ jobs:
|
||||
- uses: actions/checkout@v7
|
||||
- name: run packaging script tests
|
||||
run: make test-build-scripts
|
||||
|
||||
# The backend matrix path filter fails silently: a miss emits an empty
|
||||
# matrix, every job goes green, and the change reaches no image (#10946).
|
||||
# Its tests need only node, so they ride along with this job.
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
- name: run CI script tests
|
||||
run: make test-ci-scripts
|
||||
|
||||
7
Makefile
7
Makefile
@@ -215,6 +215,13 @@ test-backend-cpp:
|
||||
test-build-scripts:
|
||||
@set -e; for t in scripts/build/*_test.sh; do echo "== $$t"; bash "$$t"; done
|
||||
|
||||
## Runs the unit tests for the CI helper scripts under scripts/lib/. Currently
|
||||
## the backend matrix path filter, whose failure mode is invisible in CI: it
|
||||
## emits an empty matrix, every job goes green, and the change ships to no
|
||||
## image at all (see PR #10946). Plain `node --test`, no dependencies.
|
||||
test-ci-scripts:
|
||||
@set -e; for t in scripts/lib/*_test.mjs; do echo "== $$t"; node --test "$$t"; done
|
||||
|
||||
## Runs the core suite ($(TEST_PATHS)) with statement-coverage instrumentation
|
||||
## and writes a merged profile to $(COVERAGE_PROFILE). Deliberately omits
|
||||
## --fail-fast so a single failure doesn't truncate the coverage number, and
|
||||
|
||||
@@ -2,6 +2,11 @@ import fs from "fs";
|
||||
import * as yaml from "js-yaml";
|
||||
import { Octokit } from "@octokit/core";
|
||||
|
||||
import {
|
||||
getAllBackendPaths,
|
||||
filterMatrix,
|
||||
} from "./lib/backend-filter.mjs";
|
||||
|
||||
// Matrix data lives in a small data-only YAML so both backend.yml (master push)
|
||||
// and backend_pr.yml (pull_request) can use a dynamic `matrix: ${{ fromJson(...) }}`
|
||||
// for the live job, while this script remains the single source of truth for
|
||||
@@ -13,125 +18,7 @@ const includesDarwin = matrixYml.includeDarwin;
|
||||
const eventPath = process.env.GITHUB_EVENT_PATH;
|
||||
const event = JSON.parse(fs.readFileSync(eventPath, "utf8"));
|
||||
|
||||
// Infer backend path
|
||||
function inferBackendPath(item) {
|
||||
if (item.dockerfile.endsWith("python")) {
|
||||
return `backend/python/${item.backend}/`;
|
||||
}
|
||||
// parakeet-cpp is a Go backend (Dockerfile.golang) wrapping the parakeet.cpp
|
||||
// ggml port via purego. It lives in backend/go/parakeet-cpp/; this explicit
|
||||
// branch (placed before the generic golang one, which would also resolve it
|
||||
// correctly) documents the mapping and guards against a future
|
||||
// dockerfile-suffix change.
|
||||
if (item.backend === "parakeet-cpp") {
|
||||
return `backend/go/parakeet-cpp/`;
|
||||
}
|
||||
// ced is a Go backend (Dockerfile.golang) wrapping the ced.cpp ggml port via
|
||||
// purego, living in backend/go/ced/. Same explicit-branch rationale as
|
||||
// parakeet-cpp above: the generic golang fallthrough would also resolve it,
|
||||
// but this documents the mapping and guards a future dockerfile-suffix change.
|
||||
if (item.backend === "ced") {
|
||||
return `backend/go/ced/`;
|
||||
}
|
||||
// moss-transcribe-cpp is a Go backend (Dockerfile.golang) wrapping the
|
||||
// moss-transcribe.cpp ggml port via purego, living in
|
||||
// backend/go/moss-transcribe-cpp/. Same explicit-branch rationale as
|
||||
// parakeet-cpp / ced: the generic golang fallthrough would also resolve it,
|
||||
// but this documents the mapping and guards a future dockerfile-suffix change.
|
||||
if (item.backend === "moss-transcribe-cpp") {
|
||||
return `backend/go/moss-transcribe-cpp/`;
|
||||
}
|
||||
// moss-tts-cpp is a Go backend (Dockerfile.golang) wrapping the moss-tts.cpp
|
||||
// ggml port via purego, living in backend/go/moss-tts-cpp/. Same
|
||||
// explicit-branch rationale as parakeet-cpp / ced / moss-transcribe-cpp: the
|
||||
// generic golang fallthrough would also resolve it, but this documents the
|
||||
// mapping and guards a future dockerfile-suffix change.
|
||||
if (item.backend === "moss-tts-cpp") {
|
||||
return `backend/go/moss-tts-cpp/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("golang")) {
|
||||
return `backend/go/${item.backend}/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("rust")) {
|
||||
return `backend/rust/${item.backend}/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("ik-llama-cpp")) {
|
||||
return `backend/cpp/ik-llama-cpp/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("turboquant")) {
|
||||
// turboquant is a llama.cpp fork that reuses backend/cpp/llama-cpp sources
|
||||
// via a thin wrapper Makefile. Changes to either dir should retrigger it.
|
||||
return `backend/cpp/turboquant/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("bonsai")) {
|
||||
// bonsai is a llama.cpp fork that reuses backend/cpp/llama-cpp sources
|
||||
// via a thin wrapper Makefile. Changes to either dir should retrigger it.
|
||||
return `backend/cpp/bonsai/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("privacy-filter")) {
|
||||
return `backend/cpp/privacy-filter/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("ds4")) {
|
||||
return `backend/cpp/ds4/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("llama-cpp")) {
|
||||
return `backend/cpp/llama-cpp/`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function inferBackendPathDarwin(item) {
|
||||
// llama-cpp on Darwin builds from the C++ sources, not a backend/go/llama-cpp
|
||||
// tree (which doesn't exist). The Darwin job is matrix-driven with lang=go
|
||||
// for runner/toolchain selection, but the source path is C++.
|
||||
if (item.backend === "llama-cpp") {
|
||||
return `backend/cpp/llama-cpp/`;
|
||||
}
|
||||
// ds4 is C++ too (built via `make backends/ds4-darwin`); the matrix entry
|
||||
// carries lang=go for runner/toolchain selection, but the source is C++.
|
||||
if (item.backend === "ds4") {
|
||||
return `backend/cpp/ds4/`;
|
||||
}
|
||||
// privacy-filter is C++ too (built via `make backends/privacy-filter-darwin`);
|
||||
// same lang=go-for-runner convention, source under backend/cpp.
|
||||
if (item.backend === "privacy-filter") {
|
||||
return `backend/cpp/privacy-filter/`;
|
||||
}
|
||||
if (!item.lang) {
|
||||
return `backend/python/${item.backend}/`;
|
||||
}
|
||||
|
||||
return `backend/${item.lang}/${item.backend}/`;
|
||||
}
|
||||
|
||||
// Build a deduplicated map of backend name -> path prefix from all matrix entries
|
||||
function getAllBackendPaths() {
|
||||
const paths = new Map();
|
||||
for (const item of includes) {
|
||||
const p = inferBackendPath(item);
|
||||
if (p && !paths.has(item.backend)) {
|
||||
paths.set(item.backend, p);
|
||||
}
|
||||
}
|
||||
for (const item of includesDarwin) {
|
||||
const p = inferBackendPathDarwin(item);
|
||||
if (p && !paths.has(item.backend)) {
|
||||
paths.set(item.backend, p);
|
||||
}
|
||||
}
|
||||
return paths;
|
||||
}
|
||||
|
||||
const allBackendPaths = getAllBackendPaths();
|
||||
|
||||
function backendChanged(backend, pathPrefix, changedFiles) {
|
||||
if (changedFiles.some(file => file.startsWith(pathPrefix))) return true;
|
||||
|
||||
// Fork backends reuse backend/cpp/llama-cpp sources via thin wrappers;
|
||||
// changes to either directory must retrigger their pipelines.
|
||||
return (backend === "turboquant" || backend === "bonsai") &&
|
||||
changedFiles.some(file => file.startsWith("backend/cpp/llama-cpp/"));
|
||||
}
|
||||
const allBackendPaths = getAllBackendPaths(includes, includesDarwin);
|
||||
|
||||
const token = process.env.GITHUB_TOKEN;
|
||||
const octokit = new Octokit({ auth: token });
|
||||
@@ -316,15 +203,10 @@ function emitFullMatrix() {
|
||||
function emitFilteredMatrix(changedFiles) {
|
||||
console.log("Changed files:", changedFiles);
|
||||
|
||||
const filtered = includes.filter(item => {
|
||||
const backendPath = inferBackendPath(item);
|
||||
if (!backendPath) return false;
|
||||
return backendChanged(item.backend, backendPath, changedFiles);
|
||||
});
|
||||
|
||||
const filteredDarwin = includesDarwin.filter(item => {
|
||||
const backendPath = inferBackendPathDarwin(item);
|
||||
return changedFiles.some(file => file.startsWith(backendPath));
|
||||
const { filtered, filteredDarwin, changedBackends } = filterMatrix({
|
||||
includes,
|
||||
includesDarwin,
|
||||
changedFiles,
|
||||
});
|
||||
|
||||
console.log("Filtered files:", filtered);
|
||||
@@ -350,8 +232,8 @@ function emitFilteredMatrix(changedFiles) {
|
||||
emitSinglearchShards(singlearch);
|
||||
|
||||
// Per-backend boolean outputs
|
||||
for (const [backend, pathPrefix] of allBackendPaths) {
|
||||
const changed = backendChanged(backend, pathPrefix, changedFiles);
|
||||
for (const backend of allBackendPaths.keys()) {
|
||||
const changed = changedBackends.has(backend);
|
||||
fs.appendFileSync(process.env.GITHUB_OUTPUT, `${backend}=${changed ? 'true' : 'false'}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
287
scripts/lib/backend-filter.mjs
Normal file
287
scripts/lib/backend-filter.mjs
Normal file
@@ -0,0 +1,287 @@
|
||||
// Pure matrix-filtering logic for scripts/changed-backends.js.
|
||||
//
|
||||
// Split out of that script so it can be unit-tested: changed-backends.js reads
|
||||
// GITHUB_EVENT_PATH and talks to the GitHub API at import time, and pulls in
|
||||
// js-yaml + @octokit/core, none of which a test of "which backends does this
|
||||
// diff invalidate?" should need. Everything here is dependency-free and
|
||||
// side-effect-free.
|
||||
|
||||
// Infer backend path
|
||||
export function inferBackendPath(item) {
|
||||
if (item.dockerfile.endsWith("python")) {
|
||||
return `backend/python/${item.backend}/`;
|
||||
}
|
||||
// parakeet-cpp is a Go backend (Dockerfile.golang) wrapping the parakeet.cpp
|
||||
// ggml port via purego. It lives in backend/go/parakeet-cpp/; this explicit
|
||||
// branch (placed before the generic golang one, which would also resolve it
|
||||
// correctly) documents the mapping and guards against a future
|
||||
// dockerfile-suffix change.
|
||||
if (item.backend === "parakeet-cpp") {
|
||||
return `backend/go/parakeet-cpp/`;
|
||||
}
|
||||
// ced is a Go backend (Dockerfile.golang) wrapping the ced.cpp ggml port via
|
||||
// purego, living in backend/go/ced/. Same explicit-branch rationale as
|
||||
// parakeet-cpp above: the generic golang fallthrough would also resolve it,
|
||||
// but this documents the mapping and guards a future dockerfile-suffix change.
|
||||
if (item.backend === "ced") {
|
||||
return `backend/go/ced/`;
|
||||
}
|
||||
// moss-transcribe-cpp is a Go backend (Dockerfile.golang) wrapping the
|
||||
// moss-transcribe.cpp ggml port via purego, living in
|
||||
// backend/go/moss-transcribe-cpp/. Same explicit-branch rationale as
|
||||
// parakeet-cpp / ced: the generic golang fallthrough would also resolve it,
|
||||
// but this documents the mapping and guards a future dockerfile-suffix change.
|
||||
if (item.backend === "moss-transcribe-cpp") {
|
||||
return `backend/go/moss-transcribe-cpp/`;
|
||||
}
|
||||
// moss-tts-cpp is a Go backend (Dockerfile.golang) wrapping the moss-tts.cpp
|
||||
// ggml port via purego, living in backend/go/moss-tts-cpp/. Same
|
||||
// explicit-branch rationale as parakeet-cpp / ced / moss-transcribe-cpp: the
|
||||
// generic golang fallthrough would also resolve it, but this documents the
|
||||
// mapping and guards a future dockerfile-suffix change.
|
||||
if (item.backend === "moss-tts-cpp") {
|
||||
return `backend/go/moss-tts-cpp/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("golang")) {
|
||||
return `backend/go/${item.backend}/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("rust")) {
|
||||
return `backend/rust/${item.backend}/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("ik-llama-cpp")) {
|
||||
return `backend/cpp/ik-llama-cpp/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("turboquant")) {
|
||||
// turboquant is a llama.cpp fork that reuses backend/cpp/llama-cpp sources
|
||||
// via a thin wrapper Makefile. Changes to either dir should retrigger it.
|
||||
return `backend/cpp/turboquant/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("bonsai")) {
|
||||
// bonsai is a llama.cpp fork that reuses backend/cpp/llama-cpp sources
|
||||
// via a thin wrapper Makefile. Changes to either dir should retrigger it.
|
||||
return `backend/cpp/bonsai/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("privacy-filter")) {
|
||||
return `backend/cpp/privacy-filter/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("ds4")) {
|
||||
return `backend/cpp/ds4/`;
|
||||
}
|
||||
if (item.dockerfile.endsWith("llama-cpp")) {
|
||||
return `backend/cpp/llama-cpp/`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function inferBackendPathDarwin(item) {
|
||||
// llama-cpp on Darwin builds from the C++ sources, not a backend/go/llama-cpp
|
||||
// tree (which doesn't exist). The Darwin job is matrix-driven with lang=go
|
||||
// for runner/toolchain selection, but the source path is C++.
|
||||
if (item.backend === "llama-cpp") {
|
||||
return `backend/cpp/llama-cpp/`;
|
||||
}
|
||||
// ds4 is C++ too (built via `make backends/ds4-darwin`); the matrix entry
|
||||
// carries lang=go for runner/toolchain selection, but the source is C++.
|
||||
if (item.backend === "ds4") {
|
||||
return `backend/cpp/ds4/`;
|
||||
}
|
||||
// privacy-filter is C++ too (built via `make backends/privacy-filter-darwin`);
|
||||
// same lang=go-for-runner convention, source under backend/cpp.
|
||||
if (item.backend === "privacy-filter") {
|
||||
return `backend/cpp/privacy-filter/`;
|
||||
}
|
||||
if (!item.lang) {
|
||||
return `backend/python/${item.backend}/`;
|
||||
}
|
||||
|
||||
return `backend/${item.lang}/${item.backend}/`;
|
||||
}
|
||||
|
||||
// Build a deduplicated map of backend name -> path prefix from all matrix entries
|
||||
export function getAllBackendPaths(includes, includesDarwin) {
|
||||
const paths = new Map();
|
||||
for (const item of includes) {
|
||||
const p = inferBackendPath(item);
|
||||
if (p && !paths.has(item.backend)) {
|
||||
paths.set(item.backend, p);
|
||||
}
|
||||
}
|
||||
for (const item of includesDarwin) {
|
||||
const p = inferBackendPathDarwin(item);
|
||||
if (p && !paths.has(item.backend)) {
|
||||
paths.set(item.backend, p);
|
||||
}
|
||||
}
|
||||
return paths;
|
||||
}
|
||||
|
||||
export function backendChanged(backend, pathPrefix, changedFiles) {
|
||||
if (changedFiles.some(file => file.startsWith(pathPrefix))) return true;
|
||||
|
||||
// Fork backends reuse backend/cpp/llama-cpp sources via thin wrappers;
|
||||
// changes to either directory must retrigger their pipelines.
|
||||
return (backend === "turboquant" || backend === "bonsai") &&
|
||||
changedFiles.some(file => file.startsWith("backend/cpp/llama-cpp/"));
|
||||
}
|
||||
|
||||
// Darwin entries carry `lang` only for runner/toolchain selection; an entry
|
||||
// without it is a Python backend (see .github/backend-matrix.yml).
|
||||
const isDarwinPython = item => !item.lang;
|
||||
|
||||
// backend_build_darwin.yml routes llama-cpp, ds4 and privacy-filter to their
|
||||
// own bespoke make targets; every other lang=go entry goes through
|
||||
// `make build-darwin-go-backend` -> scripts/build/golang-darwin.sh.
|
||||
const DARWIN_BESPOKE_BUILDERS = new Set(["llama-cpp", "ds4", "privacy-filter"]);
|
||||
const isDarwinGenericGo = item =>
|
||||
!!item.lang && !DARWIN_BESPOKE_BUILDERS.has(item.backend);
|
||||
|
||||
const isLinuxPython = item => item.dockerfile.endsWith("python");
|
||||
|
||||
const never = () => false;
|
||||
const always = () => true;
|
||||
|
||||
// Shared build inputs: files that end up in, or decide the contents of, images
|
||||
// belonging to backends whose own directory they do not live under. The
|
||||
// per-backend prefix match in filterMatrix() structurally cannot see these, so
|
||||
// before this table a change to any of them rebuilt *nothing at all*.
|
||||
//
|
||||
// That is not hypothetical. PR #10946 fixed scripts/build/package-gpu-libs.sh
|
||||
// shipping a partial 4-of-8 cuDNN library set (which mixed versions with the
|
||||
// venv's pip cuDNN and produced CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at
|
||||
// inference time). It merged 1h48m after the weekly full-matrix cron had
|
||||
// already run, so no backend image ever received the fix and nothing signalled
|
||||
// that it had been un-shipped.
|
||||
//
|
||||
// Each rule names the narrowest set of matrix entries it can honestly
|
||||
// invalidate, because a full backend matrix is a large CI spend. Rules are
|
||||
// first-match-wins per changed file, so specific entries must precede the
|
||||
// scripts/build/ catch-all at the bottom.
|
||||
export const SHARED_BUILD_INPUTS = [
|
||||
{
|
||||
// Every language consumes backend.proto: Dockerfile.python COPYs it, Go
|
||||
// backends regenerate their stubs from it via `make protogen-go`, the C++
|
||||
// CMakeLists compile backend.pb.cc from it, and the Rust crate Makefile
|
||||
// copies it in before build.rs runs.
|
||||
matches: file => file === "backend/backend.proto",
|
||||
linux: always,
|
||||
darwin: always,
|
||||
},
|
||||
{
|
||||
// COPY'd into every Python image (Dockerfile.python) and into every Darwin
|
||||
// Python build (scripts/build/python-darwin.sh).
|
||||
matches: file => file.startsWith("backend/python/common/"),
|
||||
linux: isLinuxPython,
|
||||
darwin: isDarwinPython,
|
||||
},
|
||||
{
|
||||
// The reusable build workflows own build-args, packaging and push for
|
||||
// their whole OS family, so a change there can alter what lands in every
|
||||
// image that family builds.
|
||||
matches: file => file === ".github/workflows/backend_build.yml",
|
||||
linux: always,
|
||||
darwin: never,
|
||||
},
|
||||
{
|
||||
matches: file => file === ".github/workflows/backend_build_darwin.yml",
|
||||
linux: never,
|
||||
darwin: always,
|
||||
},
|
||||
{
|
||||
// Stages the CUDA/ROCm runtime libraries into every Python image's lib/.
|
||||
// COPY'd and run by Dockerfile.python only. This is the #10946 case.
|
||||
matches: file => file === "scripts/build/package-gpu-libs.sh",
|
||||
linux: isLinuxPython,
|
||||
darwin: never,
|
||||
},
|
||||
{
|
||||
matches: file => file === "scripts/build/python-darwin.sh",
|
||||
linux: never,
|
||||
darwin: isDarwinPython,
|
||||
},
|
||||
{
|
||||
matches: file => file === "scripts/build/golang-darwin.sh",
|
||||
linux: never,
|
||||
darwin: isDarwinGenericGo,
|
||||
},
|
||||
{
|
||||
matches: file => file === "scripts/build/llama-cpp-darwin.sh",
|
||||
linux: never,
|
||||
darwin: item => item.backend === "llama-cpp",
|
||||
},
|
||||
{
|
||||
matches: file => file === "scripts/build/ds4-darwin.sh",
|
||||
linux: never,
|
||||
darwin: item => item.backend === "ds4",
|
||||
},
|
||||
{
|
||||
matches: file => file === "scripts/build/privacy-filter-darwin.sh",
|
||||
linux: never,
|
||||
darwin: item => item.backend === "privacy-filter",
|
||||
},
|
||||
{
|
||||
// Catch-all, deliberately last and deliberately broad: anything else under
|
||||
// scripts/build/ is an unclassified image-packaging input. Rebuild
|
||||
// everything until someone narrows it with a rule above. Erring toward an
|
||||
// expensive full matrix beats the failure mode that hid #10946 — a new
|
||||
// shared script silently shipping to nothing. `*_test.sh` are the tests
|
||||
// for those scripts and never reach an image.
|
||||
matches: file =>
|
||||
file.startsWith("scripts/build/") && !file.endsWith("_test.sh"),
|
||||
linux: always,
|
||||
darwin: always,
|
||||
},
|
||||
];
|
||||
|
||||
// The matrix stores dockerfiles as "./backend/Dockerfile.python"; changed-file
|
||||
// lists are repo-relative.
|
||||
function dockerfilePath(item) {
|
||||
return (item.dockerfile || "").replace(/^\.\//, "");
|
||||
}
|
||||
|
||||
// A backend/Dockerfile.<x> edit invalidates exactly the Linux entries built
|
||||
// from it. These files sit outside every backend directory, so the prefix match
|
||||
// never sees them — same blind spot as SHARED_BUILD_INPUTS, but expressible
|
||||
// precisely because each entry already names its dockerfile.
|
||||
function dockerfileChanged(item, changedFiles) {
|
||||
const df = dockerfilePath(item);
|
||||
return df !== "" && changedFiles.includes(df);
|
||||
}
|
||||
|
||||
function matchedSharedRules(changedFiles) {
|
||||
const rules = [];
|
||||
for (const file of changedFiles) {
|
||||
const rule = SHARED_BUILD_INPUTS.find(r => r.matches(file));
|
||||
if (rule && !rules.includes(rule)) rules.push(rule);
|
||||
}
|
||||
return rules;
|
||||
}
|
||||
|
||||
// Filter both matrices against a changed-file list. Returns the surviving
|
||||
// entries plus the set of backend names considered changed, which drives the
|
||||
// per-backend boolean outputs consumed by test-extra.yml.
|
||||
export function filterMatrix({ includes, includesDarwin, changedFiles }) {
|
||||
const sharedRules = matchedSharedRules(changedFiles);
|
||||
|
||||
const filtered = includes.filter(item => {
|
||||
const backendPath = inferBackendPath(item);
|
||||
if (!backendPath) return false;
|
||||
if (backendChanged(item.backend, backendPath, changedFiles)) return true;
|
||||
if (dockerfileChanged(item, changedFiles)) return true;
|
||||
return sharedRules.some(rule => rule.linux(item));
|
||||
});
|
||||
|
||||
const filteredDarwin = includesDarwin.filter(item => {
|
||||
const backendPath = inferBackendPathDarwin(item);
|
||||
if (changedFiles.some(file => file.startsWith(backendPath))) return true;
|
||||
return sharedRules.some(rule => rule.darwin(item));
|
||||
});
|
||||
|
||||
const changedBackends = new Set();
|
||||
for (const item of filtered) changedBackends.add(item.backend);
|
||||
for (const item of filteredDarwin) changedBackends.add(item.backend);
|
||||
for (const [backend, pathPrefix] of getAllBackendPaths(includes, includesDarwin)) {
|
||||
if (backendChanged(backend, pathPrefix, changedFiles)) changedBackends.add(backend);
|
||||
}
|
||||
|
||||
return { filtered, filteredDarwin, changedBackends };
|
||||
}
|
||||
189
scripts/lib/backend-filter_test.mjs
Normal file
189
scripts/lib/backend-filter_test.mjs
Normal file
@@ -0,0 +1,189 @@
|
||||
// Unit tests for the backend matrix path filter (scripts/lib/backend-filter.mjs).
|
||||
//
|
||||
// Run with `make test-ci-scripts` (plain `node --test`, no dependencies), which
|
||||
// is what .github/workflows/lint.yml runs. The production entrypoint
|
||||
// scripts/changed-backends.js needs bun + js-yaml + @octokit/core; the logic
|
||||
// under test here needs none of that.
|
||||
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { filterMatrix } from "./backend-filter.mjs";
|
||||
|
||||
// A representative slice of .github/backend-matrix.yml: one entry per build
|
||||
// path that the filter treats differently. Kept inline rather than parsed from
|
||||
// the real YAML so these tests stay dependency-free and stable when backends
|
||||
// are added or removed.
|
||||
const includes = [
|
||||
{
|
||||
backend: "diffusers",
|
||||
dockerfile: "./backend/Dockerfile.python",
|
||||
"tag-suffix": "-nvidia-cuda-13-diffusers",
|
||||
},
|
||||
{
|
||||
backend: "vllm",
|
||||
dockerfile: "./backend/Dockerfile.python",
|
||||
"tag-suffix": "-nvidia-cuda-13-vllm",
|
||||
},
|
||||
{
|
||||
backend: "ced",
|
||||
dockerfile: "./backend/Dockerfile.golang",
|
||||
"tag-suffix": "-ced",
|
||||
},
|
||||
{
|
||||
backend: "kokoros",
|
||||
dockerfile: "./backend/Dockerfile.rust",
|
||||
"tag-suffix": "-kokoros",
|
||||
},
|
||||
{
|
||||
backend: "llama-cpp",
|
||||
dockerfile: "./backend/Dockerfile.llama-cpp",
|
||||
"tag-suffix": "-llama-cpp",
|
||||
},
|
||||
{
|
||||
backend: "turboquant",
|
||||
dockerfile: "./backend/Dockerfile.turboquant",
|
||||
"tag-suffix": "-turboquant",
|
||||
},
|
||||
];
|
||||
|
||||
const includesDarwin = [
|
||||
{ backend: "diffusers", "tag-suffix": "-metal-darwin-arm64-diffusers" },
|
||||
{ backend: "mlx", "tag-suffix": "-metal-darwin-arm64-mlx" },
|
||||
{ backend: "whisper", lang: "go", "tag-suffix": "-metal-darwin-arm64-whisper" },
|
||||
{ backend: "llama-cpp", lang: "go", "tag-suffix": "-metal-darwin-arm64-llama-cpp" },
|
||||
{ backend: "ds4", lang: "go", "tag-suffix": "-metal-darwin-arm64-ds4" },
|
||||
];
|
||||
|
||||
const run = changedFiles =>
|
||||
filterMatrix({ includes, includesDarwin, changedFiles });
|
||||
|
||||
const names = entries => entries.map(e => e.backend).sort();
|
||||
|
||||
test("a change to only package-gpu-libs.sh rebuilds every Python image", () => {
|
||||
// The PR #10946 regression: this script is COPY'd and run by
|
||||
// Dockerfile.python for every Python backend, but lives under scripts/, so
|
||||
// the per-backend prefix match produced an empty matrix and the cuDNN
|
||||
// packaging fix shipped to nothing.
|
||||
const { filtered, filteredDarwin, changedBackends } = run([
|
||||
"scripts/build/package-gpu-libs.sh",
|
||||
]);
|
||||
|
||||
assert.notEqual(filtered.length, 0, "expected a non-empty Linux matrix");
|
||||
assert.deepEqual(names(filtered), ["diffusers", "vllm"]);
|
||||
assert.ok(changedBackends.has("vllm"));
|
||||
|
||||
// Darwin Python builds never invoke it (see scripts/build/python-darwin.sh).
|
||||
assert.deepEqual(filteredDarwin, []);
|
||||
});
|
||||
|
||||
test("docs- and README-only changes never produce a matrix", () => {
|
||||
const { filtered, filteredDarwin, changedBackends } = run([
|
||||
"README.md",
|
||||
"docs/content/docs/overview.md",
|
||||
]);
|
||||
|
||||
assert.deepEqual(filtered, []);
|
||||
assert.deepEqual(filteredDarwin, []);
|
||||
assert.equal(changedBackends.size, 0);
|
||||
});
|
||||
|
||||
test("a backend's own directory still triggers only that backend", () => {
|
||||
const { filtered, changedBackends } = run(["backend/python/vllm/backend.py"]);
|
||||
|
||||
assert.deepEqual(names(filtered), ["vllm"]);
|
||||
assert.deepEqual([...changedBackends], ["vllm"]);
|
||||
});
|
||||
|
||||
test("Dockerfile.python rebuilds Python images and nothing else", () => {
|
||||
const { filtered, filteredDarwin } = run(["backend/Dockerfile.python"]);
|
||||
|
||||
assert.deepEqual(names(filtered), ["diffusers", "vllm"]);
|
||||
// Darwin doesn't build from any Dockerfile.
|
||||
assert.deepEqual(filteredDarwin, []);
|
||||
});
|
||||
|
||||
test("Dockerfile.golang rebuilds Go images and nothing else", () => {
|
||||
const { filtered } = run(["backend/Dockerfile.golang"]);
|
||||
|
||||
assert.deepEqual(names(filtered), ["ced"]);
|
||||
});
|
||||
|
||||
test("a per-family Dockerfile rebuilds only its own family", () => {
|
||||
const { filtered } = run(["backend/Dockerfile.llama-cpp"]);
|
||||
|
||||
assert.deepEqual(names(filtered), ["llama-cpp"]);
|
||||
});
|
||||
|
||||
test("backend.proto rebuilds every backend on every OS", () => {
|
||||
const { filtered, filteredDarwin } = run(["backend/backend.proto"]);
|
||||
|
||||
assert.equal(filtered.length, includes.length);
|
||||
assert.equal(filteredDarwin.length, includesDarwin.length);
|
||||
});
|
||||
|
||||
test("backend/python/common rebuilds Python images on Linux and Darwin", () => {
|
||||
const { filtered, filteredDarwin } = run([
|
||||
"backend/python/common/libbackend.sh",
|
||||
]);
|
||||
|
||||
assert.deepEqual(names(filtered), ["diffusers", "vllm"]);
|
||||
assert.deepEqual(names(filteredDarwin), ["diffusers", "mlx"]);
|
||||
});
|
||||
|
||||
test("the reusable Linux build workflow rebuilds Linux only", () => {
|
||||
const { filtered, filteredDarwin } = run([
|
||||
".github/workflows/backend_build.yml",
|
||||
]);
|
||||
|
||||
assert.equal(filtered.length, includes.length);
|
||||
assert.deepEqual(filteredDarwin, []);
|
||||
});
|
||||
|
||||
test("the reusable Darwin build workflow rebuilds Darwin only", () => {
|
||||
const { filtered, filteredDarwin } = run([
|
||||
".github/workflows/backend_build_darwin.yml",
|
||||
]);
|
||||
|
||||
assert.deepEqual(filtered, []);
|
||||
assert.equal(filteredDarwin.length, includesDarwin.length);
|
||||
});
|
||||
|
||||
test("golang-darwin.sh skips backends with bespoke Darwin build scripts", () => {
|
||||
const { filtered, filteredDarwin } = run([
|
||||
"scripts/build/golang-darwin.sh",
|
||||
]);
|
||||
|
||||
assert.deepEqual(filtered, []);
|
||||
assert.deepEqual(names(filteredDarwin), ["whisper"]);
|
||||
});
|
||||
|
||||
test("a bespoke Darwin build script rebuilds only its own backend", () => {
|
||||
const { filteredDarwin } = run(["scripts/build/ds4-darwin.sh"]);
|
||||
|
||||
assert.deepEqual(names(filteredDarwin), ["ds4"]);
|
||||
});
|
||||
|
||||
test("an unclassified scripts/build/ file conservatively rebuilds everything", () => {
|
||||
const { filtered, filteredDarwin } = run([
|
||||
"scripts/build/package-something-new.sh",
|
||||
]);
|
||||
|
||||
assert.equal(filtered.length, includes.length);
|
||||
assert.equal(filteredDarwin.length, includesDarwin.length);
|
||||
});
|
||||
|
||||
test("tests for the packaging scripts do not rebuild anything", () => {
|
||||
const { filtered, filteredDarwin } = run([
|
||||
"scripts/build/package-gpu-libs_test.sh",
|
||||
]);
|
||||
|
||||
assert.deepEqual(filtered, []);
|
||||
assert.deepEqual(filteredDarwin, []);
|
||||
});
|
||||
|
||||
test("turboquant still retriggers on llama-cpp source changes", () => {
|
||||
const { filtered } = run(["backend/cpp/llama-cpp/grpc-server.cpp"]);
|
||||
|
||||
assert.deepEqual(names(filtered), ["llama-cpp", "turboquant"]);
|
||||
});
|
||||
Reference in New Issue
Block a user