mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-04 12:22:22 -04:00
Compare commits
1 Commits
bot/issue-
...
models/qih
| Author | SHA1 | Date | |
|---|---|---|---|
|
455aee4eaf |
@@ -1,382 +0,0 @@
|
||||
# Adding a New Backend
|
||||
|
||||
When adding a new backend to LocalAI, you need to update several files to ensure the backend is properly built, tested, and registered. Here's a step-by-step guide based on the pattern used for adding backends like `moonshine`:
|
||||
|
||||
## 1. Create Backend Directory Structure
|
||||
|
||||
Create the backend directory under the appropriate location:
|
||||
- **Python backends**: `backend/python/<backend-name>/`
|
||||
- **Go backends**: `backend/go/<backend-name>/`
|
||||
- **C++ backends**: `backend/cpp/<backend-name>/`
|
||||
- **Rust backends**: `backend/rust/<backend-name>/`
|
||||
|
||||
For Python backends, you'll typically need:
|
||||
- `backend.py` - Main gRPC server implementation
|
||||
- `Makefile` - Build configuration
|
||||
- `install.sh` - Installation script for dependencies
|
||||
- `protogen.sh` - Protocol buffer generation script
|
||||
- `requirements.txt` - Python dependencies
|
||||
- `run.sh` - Runtime script
|
||||
- `test.py` / `test.sh` - Test files
|
||||
|
||||
For Rust backends, you'll typically need (see `backend/rust/kokoros/` as a reference):
|
||||
- `Cargo.toml` - Crate manifest; depend on the upstream project as a submodule under `sources/`
|
||||
- `build.rs` - Invokes `tonic_build` to generate gRPC stubs from `backend/backend.proto` (use the `BACKEND_PROTO_PATH` env var so the Makefile can inject the canonical copy)
|
||||
- `src/` - The gRPC server implementation (implement `Backend` via `tonic`)
|
||||
- `Makefile` - Copies `backend.proto` into the crate, runs `cargo build --release`, then `package.sh`
|
||||
- `package.sh` - Uses `ldd` to bundle the binary's dynamic deps and `ld.so` into `package/lib/`
|
||||
- `run.sh` - Sets `LD_LIBRARY_PATH`/`SSL_CERT_DIR` and execs the binary via the bundled `lib/ld.so`
|
||||
- `sources/<UpstreamProject>/` - Git submodule with the upstream Rust crate
|
||||
|
||||
## 2. Add Build Configurations to `.github/backend-matrix.yml`
|
||||
|
||||
The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `backend.yml` itself). `backend.yml` (master push) and `backend_pr.yml` (PR) load it via `scripts/changed-backends.js`, which also handles per-file path filtering so only touched backends rebuild on PRs and master pushes alike. Add build matrix entries to `.github/backend-matrix.yml` for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
|
||||
|
||||
**Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
|
||||
|
||||
**Path-filter registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/lib/backend-filter.mjs:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
|
||||
|
||||
```js
|
||||
if (item.dockerfile.endsWith("<your-dockerfile-suffix>")) {
|
||||
return `backend/cpp/<your-backend>/`; // or backend/python|go|rust/...
|
||||
}
|
||||
```
|
||||
|
||||
The `endsWith()` test is against the matrix entry's `dockerfile:` value (e.g. `./backend/Dockerfile.ds4` → `endsWith("ds4")`). Specificity order matters here just like it does for importers: more-specific suffixes go BEFORE more-generic ones (e.g. `ds4` before `llama-cpp` even though both end with letters, because some upstream might one day call itself `super-ds4-llama-cpp`). Verify locally before pushing:
|
||||
|
||||
```bash
|
||||
# Confirm your dockerfile suffix is unique enough
|
||||
node -e "
|
||||
const yaml = require('js-yaml'); const fs = require('fs');
|
||||
const m = yaml.load(fs.readFileSync('.github/backend-matrix.yml','utf8'));
|
||||
for (const e of m.include.filter(e => e.backend === '<your-backend>')) {
|
||||
console.log(e.dockerfile, '->', e.dockerfile.endsWith('<suffix>'));
|
||||
}"
|
||||
```
|
||||
|
||||
A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/lib/backend-filter.mjs`.
|
||||
|
||||
If your backend consumes a *shared* build input that lives outside its own directory (a new script under `scripts/build/`, a new file copied into every image), add a rule to `SHARED_BUILD_INPUTS` in the same file — the per-backend prefix match cannot see those, and a miss ships your change to no image at all. See `scripts/lib/backend-filter_test.mjs` for the pattern; `make test-ci-scripts` runs it.
|
||||
|
||||
**`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=<sha>` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`):
|
||||
|
||||
```yaml
|
||||
# .github/workflows/bump_deps.yaml
|
||||
matrix:
|
||||
include:
|
||||
- repository: "antirez/ds4"
|
||||
variable: "DS4_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/cpp/ds4/Makefile"
|
||||
```
|
||||
|
||||
And the corresponding Makefile shape (mirror `backend/cpp/llama-cpp/Makefile`):
|
||||
|
||||
```makefile
|
||||
DS4_VERSION?=ae302c2fa18cc6d9aefc021d0f27ae03c9ad2fc0
|
||||
DS4_REPO?=https://github.com/antirez/ds4
|
||||
...
|
||||
ds4:
|
||||
mkdir -p ds4
|
||||
cd ds4 && git init -q && \
|
||||
git remote add origin $(DS4_REPO) && \
|
||||
git fetch --depth 1 origin $(DS4_VERSION) && \
|
||||
git checkout FETCH_HEAD
|
||||
```
|
||||
|
||||
If you have a `prepare.sh` doing the clone, delete it — the recipe belongs in the Makefile target so `make purge && make` works as a clean-and-rebuild and so the bump bot finds the pin.
|
||||
|
||||
**Placement in file:**
|
||||
- CPU builds: Add after other CPU builds (e.g., after `cpu-chatterbox`)
|
||||
- CUDA 12 builds: Add after other CUDA 12 builds (e.g., after `gpu-nvidia-cuda-12-chatterbox`)
|
||||
- CUDA 13 builds: Add after other CUDA 13 builds (e.g., after `gpu-nvidia-cuda-13-chatterbox`)
|
||||
|
||||
**Additional build types you may need:**
|
||||
- ROCm/HIP: Use `build-type: 'hipblas'` with `base-image: "rocm/dev-ubuntu-24.04:7.2.1"`
|
||||
- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"`
|
||||
- L4T (ARM): Use `build-type: 'l4t'` with `platforms: 'linux/arm64'` and `runs-on: 'ubuntu-24.04-arm'`
|
||||
|
||||
**Per-arch native builds (`linux/amd64` + `linux/arm64`):**
|
||||
|
||||
Multi-arch backends are NOT a single matrix entry with `platforms: 'linux/amd64,linux/arm64'`. Instead, add **two** entries — one with `platforms: 'linux/amd64'` + `platform-tag: 'amd64'` + `runs-on: 'ubuntu-latest'`, one with `platforms: 'linux/arm64'` + `platform-tag: 'arm64'` + `runs-on: 'ubuntu-24.04-arm'` — both sharing the same `tag-suffix`. The script detects the shared `tag-suffix` and emits a `merge-matrix` entry, so `backend-merge-jobs` (in `backend.yml`/`backend_pr.yml`) automatically assembles the manifest list from per-arch digest artifacts. See `-cpu-faster-whisper` in `.github/backend-matrix.yml` for a reference shape.
|
||||
|
||||
**llama-cpp / ik-llama-cpp / turboquant variants only — `builder-base-image`:**
|
||||
|
||||
Entries whose `dockerfile` is `./backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` must also set a `builder-base-image` field pointing at a prebuilt base from `quay.io/go-skynet/ci-cache:base-grpc-*` (CI builds these via `.github/workflows/base-images.yml`). The mapping is by `(build-type, platforms)` — see existing entries for the pattern. CI uses these prebuilt bases to skip the gRPC compile (~25–35 min cold). Local `make backends/<name>` ignores `builder-base-image` and uses the from-source path inside the Dockerfile, so you don't need quay access for local builds.
|
||||
|
||||
### Cover every OS the project supports (Linux **and** Darwin)
|
||||
|
||||
`.github/backend-matrix.yml` has two matrices, and they are the source of truth for which OS a backend ships on:
|
||||
|
||||
- `include:` — the **Linux** matrix (x86_64 + arm64; CPU and CUDA / ROCm / SYCL / Vulkan).
|
||||
- `includeDarwin:` — the **macOS / Apple Silicon** matrix (arm64; Metal where the engine supports it, otherwise a native arm64 CPU build).
|
||||
|
||||
**A new backend must target every OS it can build for — do not ship Linux-only by default.** A backend that appears only under `include:` is silently unavailable on macOS even when its code would run there. Most C/C++/GGML engines build on Darwin out of the box (ggml defaults `GGML_METAL=ON` on Apple, so a plain build is Metal-enabled), and many Python backends do too (CPU / MPS wheels). If a backend genuinely cannot support an OS (e.g. CUDA-only, no CPU variant), state that in the PR description instead of omitting it silently.
|
||||
|
||||
Wiring a backend into `includeDarwin:` is more than the matrix entry:
|
||||
|
||||
1. **`includeDarwin:` entry** — `tag-suffix: "-metal-darwin-arm64-<backend>"`, `build-type: "metal"`, `lang: "go"` for go+ggml backends; omit `build-type` for the bespoke C++ ones (llama-cpp / ds4 / privacy-filter). Match an existing entry of the same shape.
|
||||
2. **`backend/index.yaml`** — add `metal:` to the backend's `capabilities` map (main and `-development`) and concrete `metal-<backend>` / `metal-<backend>-development` image entries pointing at the `-metal-darwin-arm64-<backend>` images.
|
||||
3. **C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/lib/backend-filter.mjs` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
|
||||
4. **C++ proto gotcha** — if the backend compiles the generated gRPC/protobuf in a separate CMake target (e.g. `hw_grpc_proto`), that target must link `protobuf::libprotobuf` + `gRPC::grpc++` so the Homebrew include dirs propagate; otherwise macOS fails with `google/protobuf/runtime_version.h not found` (Linux hides this because apt headers sit in `/usr/include`).
|
||||
|
||||
The CI path filter only builds a backend on a PR when a file under its directory changes, so a darwin-only YAML edit builds nothing — touch a file under `backend/<lang>/<backend>/` (a one-line comment is enough) in the same PR.
|
||||
|
||||
## 3. Add Backend Metadata to `backend/index.yaml`
|
||||
|
||||
**Step 3a: Add Meta Definition**
|
||||
|
||||
Add a YAML anchor definition in the `## metas` section (around line 2-300). Look for similar backends to use as a template such as `diffusers` or `chatterbox`
|
||||
|
||||
**Step 3b: Add Image Entries**
|
||||
|
||||
Add image entries at the end of the file, following the pattern of similar backends such as `diffusers` or `chatterbox`. Include both `latest` (production) and `master` (development) tags.
|
||||
|
||||
**Note on integrity:** OCI backends installed from a gallery whose `verification:` block is set are verified against a keyless-cosign policy before extraction; tarball/HTTP backends use the optional `sha256:` field. New backends do not need any extra YAML — the gallery-level `verification:` block covers every entry. See [.agents/backend-signing.md](backend-signing.md) for the producer-side CI step.
|
||||
|
||||
## 4. Update the Makefile
|
||||
|
||||
The Makefile needs to be updated in several places to support building and testing the new backend:
|
||||
|
||||
**Step 4a: Add to `.NOTPARALLEL`**
|
||||
|
||||
Add `backends/<backend-name>` to the `.NOTPARALLEL` line (around line 2) to prevent parallel execution conflicts:
|
||||
|
||||
```makefile
|
||||
.NOTPARALLEL: ... backends/<backend-name>
|
||||
```
|
||||
|
||||
**Step 4b: Add to `prepare-test-extra`**
|
||||
|
||||
Add the backend to the `prepare-test-extra` target to prepare it for testing. Use the path matching your language bucket (`backend/python/`, `backend/go/`, `backend/rust/`, …):
|
||||
|
||||
```makefile
|
||||
prepare-test-extra: protogen-python
|
||||
...
|
||||
$(MAKE) -C backend/<lang>/<backend-name>
|
||||
```
|
||||
|
||||
For Rust backends the target is usually the crate build target itself (e.g. `$(MAKE) -C backend/rust/<backend-name> <backend-name>-grpc`) so the binary is in place before `test` runs.
|
||||
|
||||
**Step 4c: Add to `test-extra`**
|
||||
|
||||
Add the backend to the `test-extra` target to run its tests — applies to Go and Rust backends too, not only Python:
|
||||
|
||||
```makefile
|
||||
test-extra: prepare-test-extra
|
||||
...
|
||||
$(MAKE) -C backend/<lang>/<backend-name> test
|
||||
```
|
||||
|
||||
Each backend's own `Makefile` should define a `test` target so this line works regardless of language. Integration tests that need large model downloads should be gated behind an env var (see `backend/rust/kokoros/`'s `KOKOROS_MODEL_PATH` pattern) so CI only runs unit tests.
|
||||
|
||||
**Step 4d: Add Backend Definition**
|
||||
|
||||
Add a backend definition variable in the backend definitions section (around line 428-457). The format depends on the backend type:
|
||||
|
||||
**For Python backends with root context** (like `faster-whisper`, `coqui`):
|
||||
```makefile
|
||||
BACKEND_<BACKEND_NAME> = <backend-name>|python|.|false|true
|
||||
```
|
||||
|
||||
**For Python backends with `./backend` context** (like `chatterbox`, `moonshine`):
|
||||
```makefile
|
||||
BACKEND_<BACKEND_NAME> = <backend-name>|python|./backend|false|true
|
||||
```
|
||||
|
||||
**For Go backends**:
|
||||
```makefile
|
||||
BACKEND_<BACKEND_NAME> = <backend-name>|golang|.|false|true
|
||||
```
|
||||
|
||||
**For Rust backends**:
|
||||
```makefile
|
||||
BACKEND_<BACKEND_NAME> = <backend-name>|rust|.|false|true
|
||||
```
|
||||
|
||||
The language field (`python`/`golang`/`rust`/…) must match a `backend/Dockerfile.<lang>` file.
|
||||
|
||||
**Step 4e: Generate Docker Build Target**
|
||||
|
||||
Add an eval call to generate the docker-build target (around line 480-501):
|
||||
|
||||
```makefile
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_<BACKEND_NAME>)))
|
||||
```
|
||||
|
||||
**Step 4f: Add to `docker-build-backends`**
|
||||
|
||||
Add `docker-build-<backend-name>` to the `docker-build-backends` target (around line 507):
|
||||
|
||||
```makefile
|
||||
docker-build-backends: ... docker-build-<backend-name>
|
||||
```
|
||||
|
||||
**Determining the Context:**
|
||||
|
||||
- If the backend is in `backend/python/<backend-name>/` and uses `./backend` as context in the workflow file, use `./backend` context
|
||||
- If the backend is in `backend/python/<backend-name>/` but uses `.` as context in the workflow file, use `.` context
|
||||
- Check similar backends to determine the correct context
|
||||
|
||||
## Engine preference for gallery model variants
|
||||
|
||||
A gallery entry can declare `variants`, alternative builds of the same weights,
|
||||
and LocalAI picks one per host: it drops builds whose backend cannot run here or
|
||||
that do not fit memory, then ranks the survivors by **engine preference
|
||||
first, serving feature second, size third** (`SelectVariant` in
|
||||
`core/gallery/resolve_variant.go`).
|
||||
|
||||
Ask whether your backend should outrank another one on some hardware. If it
|
||||
should, add it to `engineNamePreferenceRules` in `pkg/system/capabilities.go`,
|
||||
best engine first for that capability:
|
||||
|
||||
```go
|
||||
{Nvidia, []string{engineVLLM, engineSGLang, engineLlamaCpp}},
|
||||
+ {Nvidia, []string{engineVLLM, engineSGLang, engineMyEngine, engineLlamaCpp}},
|
||||
```
|
||||
|
||||
That is the ENGINE NAME table, matched as a substring of a gallery entry's
|
||||
`backend:` value. Two sibling tables in the same file speak different
|
||||
vocabularies and are matched against different things:
|
||||
|
||||
| Table | Vocabulary | Matched against | Consumer |
|
||||
|-------|-----------|-----------------|----------|
|
||||
| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` |
|
||||
| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking |
|
||||
| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine |
|
||||
|
||||
**Putting a token in the wrong table matches nothing and does not error**: every
|
||||
candidate scores equal and the next sort key decides, so the preference silently
|
||||
stops existing. The block comment above all three tables spells the contract out.
|
||||
|
||||
The serving feature table is the odd one: it is not keyed by capability, because
|
||||
no hardware prefers a plain build over an equivalent faster build of the same
|
||||
weights. It reads a declared tag and nothing else. The entry name was the
|
||||
original signal and is gone: a naming convention is not a contract, and names
|
||||
are author-supplied free text where a short marker like `mtp` turns up inside
|
||||
unrelated words or on weights whose entry enables nothing.
|
||||
`overrides.options` was rejected for the mirror-image reason: `spec_type:` is
|
||||
llama.cpp's config vocabulary, whereas a cross-backend ranking decision must
|
||||
work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`.
|
||||
|
||||
**If your backend can serve the same weights faster** (speculative decoding,
|
||||
multi-token prediction), say so in the docs for its gallery entries so curators
|
||||
tag them: the tagging rule and the per-backend evidence table live in
|
||||
[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to
|
||||
appear in the token table itself; it ranks builds, not engines.
|
||||
|
||||
Leaving your backend out is a valid choice when no ordering can be justified for
|
||||
it. It then ranks below every known engine and selection falls back to size,
|
||||
which is the behaviour that predates preference.
|
||||
|
||||
**Leaving a whole capability out is not.** A missing row gives that host an
|
||||
empty preference list, so size alone decides among everything that survives the
|
||||
filters, and the filter will not save you: `IsBackendCompatible` derives hardware
|
||||
support from the engine NAME, so `vllm` and `sglang` carry no darwin, cuda, rocm
|
||||
or sycl token and are never dropped on a host with no GPU. That is why `default`
|
||||
(no usable accelerator, including a GPU under the 4 GiB VRAM floor) and
|
||||
`darwin-x86` both have rows putting `llama-cpp` first. Every capability
|
||||
`getSystemCapabilities()` can return needs a row unless every engine really is
|
||||
equally at home there. When you add one, enumerate the engines you are demoting
|
||||
rather than relying on them falling through unmatched: unmatched engines all tie
|
||||
with each other, so size decides among them.
|
||||
|
||||
## Documenting the backend (README + docs)
|
||||
|
||||
A backend is not "added" until it is discoverable. Update the user-facing docs:
|
||||
|
||||
- **`docs/content/features/backends.md`** - add the backend to the right
|
||||
category in the "LocalAI supports various types of backends" list (and add a
|
||||
new category if it introduces a new modality, e.g. sound classification).
|
||||
- If the backend introduces a **new API surface** (a new endpoint or a realtime
|
||||
capability), document it under `docs/content/` where its area lives (audio,
|
||||
vision, etc.) and follow the api-endpoints checklist in
|
||||
[api-endpoints-and-auth.md](api-endpoints-and-auth.md).
|
||||
|
||||
**If the backend is a native C/C++/GGML engine created and maintained by the
|
||||
LocalAI team** (a from-scratch port like `parakeet.cpp`, `ced.cpp`,
|
||||
`vibevoice.cpp`, `rf-detr.cpp`, not a wrapper around a third-party runtime), it
|
||||
ALSO belongs in the top-level **`README.md`** table under "native C/C++/GGML
|
||||
engines ... developed and maintained by the LocalAI project itself". Add a row
|
||||
linking the upstream engine repo with a one-line description. This is the
|
||||
project's showcase of its own engines; a new in-house backend that is missing
|
||||
from it is a documentation bug.
|
||||
|
||||
## 5. Verification Checklist
|
||||
|
||||
After adding a new backend, verify:
|
||||
|
||||
- [ ] Backend directory structure is complete with all necessary files
|
||||
- [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
|
||||
- [ ] **OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml` `metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case (in `scripts/lib/backend-filter.mjs`) for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
|
||||
- [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
|
||||
- [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
|
||||
- [ ] Tag suffixes match between workflow file and index.yaml
|
||||
- [ ] Makefile updated with all 6 required changes (`.NOTPARALLEL`, `prepare-test-extra`, `test-extra`, backend definition, docker-build target eval, `docker-build-backends`)
|
||||
- [ ] No YAML syntax errors (check with linter)
|
||||
- [ ] No Makefile syntax errors (check with linter)
|
||||
- [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
|
||||
- [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`.
|
||||
- [ ] **Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide.
|
||||
- [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`)
|
||||
- [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md`
|
||||
|
||||
## Bundling runtime shared libraries (`package.sh`)
|
||||
|
||||
The final `Dockerfile.python` stage is `FROM scratch` — there is no system `libc`, no `apt`, no fallback library path. Only files explicitly copied from the builder stage end up in the backend image. That means any runtime `dlopen` your backend (or its Python deps) needs **must** be packaged into `${BACKEND}/lib/`.
|
||||
|
||||
Pattern:
|
||||
|
||||
1. Make sure the library is installed in the builder stage of `backend/Dockerfile.python` (add it to the top-level `apt-get install`).
|
||||
2. Drop a `package.sh` in your backend directory that copies the library — and its soname symlinks — into `$(dirname $0)/lib`. See `backend/python/vllm/package.sh` for a reference implementation that walks `/usr/lib/x86_64-linux-gnu`, `/usr/lib/aarch64-linux-gnu`, etc.
|
||||
3. `Dockerfile.python` already runs `package.sh` automatically if it exists, after `package-gpu-libs.sh`.
|
||||
4. `libbackend.sh` automatically prepends `${EDIR}/lib` to `LD_LIBRARY_PATH` at run time, so anything packaged this way is found by `dlopen`.
|
||||
|
||||
How to find missing libs: when a Python module silently fails to register torch ops or you see `AttributeError: '_OpNamespace' '...' object has no attribute '...'`, run the backend image's Python with `LD_DEBUG=libs` to see which `dlopen` failed. The filename in the error message (e.g. `libnuma.so.1`) is what you need to package.
|
||||
|
||||
To verify packaging works without trusting the host:
|
||||
|
||||
```bash
|
||||
make docker-build-<backend>
|
||||
CID=$(docker create --entrypoint=/run.sh local-ai-backend:<backend>)
|
||||
docker cp $CID:/lib /tmp/check && docker rm $CID
|
||||
ls /tmp/check # expect the bundled .so files + symlinks
|
||||
```
|
||||
|
||||
Then boot it inside a fresh `ubuntu:24.04` (which intentionally does *not* have the lib installed) to confirm it actually loads from the backend dir.
|
||||
|
||||
## Importer integration
|
||||
|
||||
When you add a new backend, you MUST also make it importable via the model import form (`/import-model`). The import form dropdown is sourced dynamically from `GET /backends/known` — it reads the importer registry at `core/gallery/importers/importers.go`, so the steps below are the ONLY way to make your backend show up.
|
||||
|
||||
Required steps:
|
||||
|
||||
1. **If your backend has unambiguous detection signals** (unique file extension, HF `pipeline_tag`, unique repo name pattern, unique artefact like `modules.json`):
|
||||
- Create an importer file at `core/gallery/importers/<backend>.go` following the Match/Import pattern in `llama-cpp.go`.
|
||||
- Register it in `importers.go:defaultImporters` in **specificity order** — more specific detectors must appear BEFORE more generic ones (e.g. `sentencetransformers` before `transformers`, `stablediffusion-ggml` before `llama-cpp`, `vllm-omni` before `vllm`). First match wins.
|
||||
2. **If your backend is a drop-in replacement** (same artefacts as another backend, e.g. `ik-llama-cpp` and `turboquant` both consume GGUF the same way `llama-cpp` does):
|
||||
- Do NOT create a new importer. Extend the existing importer's `Import()` to swap the emitted `backend:` field when `preferences.backend` matches. See `llama-cpp.go` for the pattern.
|
||||
3. **If your backend has no reliable auto-detect signal** (preference-only — e.g. `sglang`, `tinygrad`, `whisperx`):
|
||||
- Do NOT create an importer. Instead add the backend name to the curated pref-only slice in `core/http/endpoints/localai/backend.go` that feeds `/backends/known`. A single line addition.
|
||||
4. **Always** add a table-driven test in `core/gallery/importers/importers_test.go` (Ginkgo/Gomega):
|
||||
- Use a real public HuggingFace repo URI as the test fixture (existing tests already hit the live HF API — follow that pattern).
|
||||
- Cover detection (auto-match without preferences), preference-override (explicit `backend:` in preferences wins), and — if the backend's modality has a common `pipeline_tag` but ambiguous artefacts — an ambiguity test asserting `errors.Is(err, importers.ErrAmbiguousImport)`.
|
||||
|
||||
Rules of thumb:
|
||||
|
||||
- When in doubt, lean pref-only. A wrong auto-detect is worse than a forced preference.
|
||||
- Never silently emit a modality mismatch (e.g. emit `llama-cpp` for a TTS repo because `.gguf` is present). Return `ErrAmbiguousImport` instead.
|
||||
- Registration order is the single most common source of bugs. Check by running `go test ./core/gallery/importers/...` — the existing suite will fail if you've shadowed a pre-existing detector.
|
||||
|
||||
## 6. Example: Adding a Python Backend
|
||||
|
||||
For reference, when `moonshine` was added:
|
||||
- **Files created**: `backend/python/moonshine/{backend.py, Makefile, install.sh, protogen.sh, requirements.txt, run.sh, test.py, test.sh}`
|
||||
- **Workflow entries**: 3 build configurations (CPU, CUDA 12, CUDA 13)
|
||||
- **Index entries**: 1 meta definition + 6 image entries (cpu, cuda12, cuda13 x latest/development)
|
||||
- **Makefile updates**:
|
||||
- Added to `.NOTPARALLEL` line
|
||||
- Added to `prepare-test-extra` and `test-extra` targets
|
||||
- Added `BACKEND_MOONSHINE = moonshine|python|./backend|false|true`
|
||||
- Added eval for docker-build target generation
|
||||
- Added `docker-build-moonshine` to `docker-build-backends`
|
||||
@@ -1,213 +0,0 @@
|
||||
# Adding GGUF Models from HuggingFace to the Gallery
|
||||
|
||||
When adding a GGUF model from HuggingFace to the LocalAI model gallery, follow this guide.
|
||||
|
||||
## Gallery file
|
||||
|
||||
All models are defined in `gallery/index.yaml`. Find the appropriate section (embedding models near other embeddings, chat models near similar chat models) and add a new entry.
|
||||
|
||||
## Getting the SHA256
|
||||
|
||||
GGUF files on HuggingFace expose their SHA256 via the `x-linked-etag` HTTP header. Fetch it with:
|
||||
|
||||
```bash
|
||||
curl -sI "https://huggingface.co/<org>/<repo>/resolve/main/<filename>.gguf" | grep -i x-linked-etag
|
||||
```
|
||||
|
||||
The value (without quotes) is the SHA256 hash. Example:
|
||||
|
||||
```bash
|
||||
curl -sI "https://huggingface.co/ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/resolve/main/embeddinggemma-300m-qat-Q8_0.gguf" | grep -i x-linked-etag
|
||||
# x-linked-etag: "6fa0c02a9c302be6f977521d399b4de3a46310a4f2621ee0063747881b673f67"
|
||||
```
|
||||
|
||||
**Important**: Pay attention to exact filename casing — HuggingFace filenames are case-sensitive (e.g., `Q8_0` vs `q8_0`). Check the repo's file listing to get the exact name.
|
||||
|
||||
## Entry format — Embedding models
|
||||
|
||||
Embedding models use `gallery/virtual.yaml` as the base config and set `embeddings: true`:
|
||||
|
||||
```yaml
|
||||
- name: "model-name"
|
||||
url: github:mudler/LocalAI/gallery/virtual.yaml@master
|
||||
urls:
|
||||
- https://huggingface.co/<original-model-org>/<original-model-name>
|
||||
- https://huggingface.co/<gguf-org>/<gguf-repo-name>
|
||||
description: |
|
||||
Short description of the model, its size, and capabilities.
|
||||
tags:
|
||||
- embeddings
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
embeddings: true
|
||||
parameters:
|
||||
model: <filename>.gguf
|
||||
files:
|
||||
- filename: <filename>.gguf
|
||||
uri: huggingface://<gguf-org>/<gguf-repo-name>/<filename>.gguf
|
||||
sha256: <sha256-hash>
|
||||
```
|
||||
|
||||
## Entry format — Chat/LLM models
|
||||
|
||||
Chat models typically reference a template config (e.g., `gallery/gemma.yaml`, `gallery/chatml.yaml`) that defines the prompt format. Use YAML anchors (`&name` / `*name`) if adding multiple quantization variants of the same model:
|
||||
|
||||
```yaml
|
||||
- &model-anchor
|
||||
url: "github:mudler/LocalAI/gallery/<template>.yaml@master"
|
||||
name: "model-name"
|
||||
icon: https://example.com/icon.png
|
||||
license: <license>
|
||||
urls:
|
||||
- https://huggingface.co/<org>/<model>
|
||||
- https://huggingface.co/<gguf-org>/<gguf-repo>
|
||||
description: |
|
||||
Model description.
|
||||
tags:
|
||||
- llm
|
||||
- gguf
|
||||
- gpu
|
||||
- cpu
|
||||
overrides:
|
||||
parameters:
|
||||
model: <filename>-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: <filename>-Q4_K_M.gguf
|
||||
sha256: <sha256>
|
||||
uri: huggingface://<gguf-org>/<gguf-repo>/<filename>-Q4_K_M.gguf
|
||||
```
|
||||
|
||||
To add a variant (e.g., different quantization), use YAML merge:
|
||||
|
||||
```yaml
|
||||
- !!merge <<: *model-anchor
|
||||
name: "model-name-q8"
|
||||
overrides:
|
||||
parameters:
|
||||
model: <filename>-Q8_0.gguf
|
||||
files:
|
||||
- filename: <filename>-Q8_0.gguf
|
||||
sha256: <sha256>
|
||||
uri: huggingface://<gguf-org>/<gguf-repo>/<filename>-Q8_0.gguf
|
||||
```
|
||||
|
||||
## Offering several builds of one model (`variants`)
|
||||
|
||||
When the same model is published in more than one quantization, or is also
|
||||
servable by another engine, add each build as its own ordinary gallery entry and
|
||||
then point one of them at the others with `variants`:
|
||||
|
||||
```yaml
|
||||
- !!merge <<: *chatml
|
||||
name: "nanbeige4.1-3b-q4"
|
||||
# ... the usual urls / overrides / files for the Q4 build ...
|
||||
variants:
|
||||
- model: nanbeige4.1-3b-q8
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- The declaring entry is a **complete, normal entry**. It keeps its own
|
||||
`files`/`overrides` and stays installable on every host and by every older
|
||||
LocalAI release, which simply ignore `variants`.
|
||||
- A variant references another gallery entry **by name**. That entry must exist
|
||||
and must not declare `variants` of its own.
|
||||
- **A referenced entry keeps its own gallery row by default.** It is hidden only
|
||||
in the collapsed listing (`collapse_variants=true`, which the web UI requests
|
||||
by default), where the declaring entry stands in for it. Searching there still
|
||||
matches the referenced entry and answers with the entry declaring it, so
|
||||
referencing an entry never makes it unfindable; turning the collapse off
|
||||
returns it under its own name.
|
||||
- **Order carries no meaning.** Do not try to encode a preference; write the
|
||||
list in whatever order reads best.
|
||||
- **A variant may be smaller than the declaring entry.** Offering a downgrade
|
||||
for small hosts is a normal shape: the declaring entry's own build competes
|
||||
like every other candidate, so a large host keeps the large build.
|
||||
- **Do not describe hardware.** At install time LocalAI drops variants whose
|
||||
backend cannot run on the host, then drops those that do not fit available
|
||||
memory. The declaring entry's own build is exempt from both filters, so
|
||||
selection always terminates on something installable. Sizes are measured live
|
||||
from the weights and cached, so nothing has to be written down.
|
||||
- **Engine preference outranks size.** Among the builds that survive the
|
||||
filters, the host's preferred engine wins first and only then does the larger
|
||||
footprint win. On NVIDIA a vLLM build beats a larger llama.cpp one; on Apple
|
||||
silicon an MLX build beats a larger GGUF one; on a host with no preference for
|
||||
either engine the larger build wins, since a bigger footprint is a higher
|
||||
quality quantization of the same weights. Predict what a user gets by asking
|
||||
which engine the host prefers before asking which build is biggest. The
|
||||
per-capability order lives in `engineNamePreferenceRules`
|
||||
(`pkg/system/capabilities.go`); see
|
||||
[adding-backends.md](adding-backends.md) for how a backend gets into it.
|
||||
- **Serving feature preference sits between engine and size.** Among builds on
|
||||
an equally preferred engine, one that speculates or predicts several tokens
|
||||
per step beats the plain build of the same weights, because it answers faster
|
||||
for the same output: a `dflash` build beats an `mtp` one, and either beats a
|
||||
plain build. The order lives in `servingFeaturePreferenceTokens`
|
||||
(`pkg/system/capabilities.go`) and is matched against the entry's `tags:` and
|
||||
**nothing else**: not the entry name, not `overrides.options`. See
|
||||
[the tagging rule](#the-dflash--mtp-tagging-rule) below. Engine deliberately
|
||||
outranks it: a serving feature makes the right engine faster, it does not make
|
||||
a wrong engine right. Fit still outranks both, so a drafter pairing (strictly
|
||||
larger than the plain build, since it ships a drafter alongside it) is dropped
|
||||
on a host too small for it before this order is ever consulted.
|
||||
- A variant is nothing but a name; there is no per-variant memory field. When
|
||||
the measured size for a build is wrong, correct it on the referenced entry by
|
||||
setting that entry's own `size:` (e.g. `size: "20GiB"`). The estimator prefers
|
||||
a declared size over its own guesswork, so the fix applies everywhere the size
|
||||
is shown or compared rather than only to variant selection.
|
||||
|
||||
Users can override the automatic choice with `variant` on `POST /models/apply`,
|
||||
`local-ai models install --variant`, or the `install_model` MCP tool. See
|
||||
`docs/content/features/model-gallery.md`.
|
||||
|
||||
The gallery lint specs live in `core/gallery`, so run that suite after adding a
|
||||
`variants` list.
|
||||
|
||||
### The `dflash` / `mtp` tagging rule
|
||||
|
||||
**Tag an entry `dflash` or `mtp` when the entry actually configures that
|
||||
feature. Variant ranking reads the tag and nothing else.**
|
||||
|
||||
Decide by looking at what the entry configures, in whatever vocabulary its
|
||||
backend uses:
|
||||
|
||||
| Backend | Configures the feature when it declares |
|
||||
|---------|------------------------------------------|
|
||||
| `llama-cpp` | `overrides.options` contains `spec_type:draft-dflash` or `spec_type:draft-mtp` |
|
||||
| `ds4` | `overrides.options` contains `mtp_path:` / `mtp_draft:` |
|
||||
| `sglang` | the referenced `gallery/*.yaml` sets `speculative_algorithm:` |
|
||||
|
||||
That check is curation-time only. `spec_type` is llama.cpp's config vocabulary,
|
||||
and a cross-backend ranking decision must not depend on one backend's option
|
||||
syntax, which is precisely why the ranker reads the tag instead of the options.
|
||||
|
||||
Two mistakes the rule exists to prevent:
|
||||
|
||||
- **Weights that carry the heads are not an entry that enables them.** The
|
||||
NVFP4 GGUF entries ship MTP-bearing weights but set only `use_jinja:true`, so
|
||||
they enable no speculative decoding and must NOT be tagged. Tagging them wins
|
||||
them the feature axis without being any faster.
|
||||
- **A name is not a declaration.** An entry whose name spells `-mtp` while
|
||||
configuring nothing gets no tag, and an entry that configures the feature is
|
||||
tagged even when its name says nothing (`hy3`, `glm-5.2`). Ranking never reads
|
||||
the name, so an untagged build that does enable the feature is simply ranked
|
||||
as plain rather than promoted on a marker nobody meant.
|
||||
|
||||
## Available template configs
|
||||
|
||||
Look at existing `.yaml` files in `gallery/` to find the right prompt template for your model architecture:
|
||||
|
||||
- `gemma.yaml` — Gemma-family models (gemma, embeddinggemma, etc.)
|
||||
- `chatml.yaml` — ChatML format (many Mistral/OpenHermes models)
|
||||
- `deepseek.yaml` — DeepSeek models
|
||||
- `virtual.yaml` — Minimal base (good for embedding models that don't need chat templates)
|
||||
|
||||
## Checklist
|
||||
|
||||
1. **Find the GGUF file** on HuggingFace — note exact filename (case-sensitive)
|
||||
2. **Get the SHA256** using the `curl -sI` + `x-linked-etag` method above
|
||||
3. **Choose the right template** config from `gallery/` based on model architecture
|
||||
4. **Add the entry** to `gallery/index.yaml` near similar models
|
||||
5. **Set `embeddings: true`** if it's an embedding model
|
||||
6. **Include both URLs** — the original model page and the GGUF repo
|
||||
7. **Write a description** — mention model size, capabilities, and quantization type
|
||||
@@ -1,101 +0,0 @@
|
||||
# AI Coding Assistants
|
||||
|
||||
This document provides guidance for AI tools and developers using AI
|
||||
assistance when contributing to LocalAI.
|
||||
|
||||
**LocalAI follows the same guidelines as the Linux kernel project for
|
||||
AI-assisted contributions.** See the upstream policy here:
|
||||
<https://docs.kernel.org/process/coding-assistants.html>
|
||||
|
||||
The rules below mirror that policy, adapted to LocalAI's license and
|
||||
project layout. If anything is unclear, the kernel document is the
|
||||
authoritative reference for intent.
|
||||
|
||||
AI tools helping with LocalAI development should follow the standard
|
||||
project development process:
|
||||
|
||||
- [CONTRIBUTING.md](../CONTRIBUTING.md) — development workflow, commit
|
||||
conventions, and PR guidelines
|
||||
- [.agents/coding-style.md](coding-style.md) — code style, editorconfig,
|
||||
logging, and documentation conventions
|
||||
- [.agents/building-and-testing.md](building-and-testing.md) — build and
|
||||
test procedures
|
||||
|
||||
## Licensing and Legal Requirements
|
||||
|
||||
All contributions must comply with LocalAI's licensing requirements:
|
||||
|
||||
- LocalAI is licensed under the **MIT License** — see the [LICENSE](../LICENSE)
|
||||
file
|
||||
- New source files should use the SPDX license identifier `MIT` where
|
||||
applicable to the file type
|
||||
- Contributions must be compatible with the MIT License and must not
|
||||
introduce code under incompatible licenses (e.g., GPL) without an
|
||||
explicit discussion with maintainers
|
||||
|
||||
## Signed-off-by and Developer Certificate of Origin
|
||||
|
||||
**AI agents MUST NOT add `Signed-off-by` tags.** Only humans can legally
|
||||
certify the Developer Certificate of Origin (DCO). The human submitter
|
||||
is responsible for:
|
||||
|
||||
- Reviewing all AI-generated code
|
||||
- Ensuring compliance with licensing requirements
|
||||
- Adding their own `Signed-off-by` tag (when the project requires DCO)
|
||||
to certify the contribution
|
||||
- Taking full responsibility for the contribution
|
||||
|
||||
AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
|
||||
A human reviewer owns the contribution; the AI's involvement is recorded
|
||||
via `Assisted-by` (see below).
|
||||
|
||||
## Attribution
|
||||
|
||||
When AI tools contribute to LocalAI development, proper attribution helps
|
||||
track the evolving role of AI in the development process. Contributions
|
||||
should include an `Assisted-by` tag in the commit message trailer in the
|
||||
following format:
|
||||
|
||||
```
|
||||
Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
|
||||
```
|
||||
|
||||
Where:
|
||||
|
||||
- `AGENT_NAME` — name of the AI tool or framework (e.g., `Claude`,
|
||||
`Copilot`, `Cursor`)
|
||||
- `MODEL_VERSION` — specific model version used (e.g.,
|
||||
`claude-opus-4-7`, `gpt-5`)
|
||||
- `[TOOL1] [TOOL2]` — optional specialized analysis tools invoked by the
|
||||
agent (e.g., `golangci-lint`, `staticcheck`, `go vet`)
|
||||
|
||||
Basic development tools (git, go, make, editors) should **not** be listed.
|
||||
|
||||
### Example
|
||||
|
||||
```
|
||||
fix(llama-cpp): handle empty tool call arguments
|
||||
|
||||
Previously the parser panicked when the model returned a tool call with
|
||||
an empty arguments object. Fall back to an empty JSON object in that
|
||||
case so downstream consumers receive a valid payload.
|
||||
|
||||
Assisted-by: Claude:claude-opus-4-7 golangci-lint
|
||||
Signed-off-by: Jane Developer <jane@example.com>
|
||||
```
|
||||
|
||||
## Scope and Responsibility
|
||||
|
||||
Using an AI assistant does not reduce the contributor's responsibility.
|
||||
The human submitter must:
|
||||
|
||||
- Understand every line that lands in the PR
|
||||
- Verify that generated code compiles, passes tests, and follows the
|
||||
project style
|
||||
- Confirm that any referenced APIs, flags, or file paths actually exist
|
||||
in the current tree (AI models may hallucinate identifiers)
|
||||
- Not submit AI output verbatim without review
|
||||
|
||||
Reviewers may ask for clarification on any change regardless of how it
|
||||
was produced. "An AI wrote it" is not an acceptable answer to a design
|
||||
question.
|
||||
@@ -1,355 +0,0 @@
|
||||
# API Endpoints and Authentication
|
||||
|
||||
This guide covers how to add new API endpoints and properly integrate them with the auth/permissions system.
|
||||
|
||||
> **Before you ship a new endpoint or capability surface**, re-read the [checklist at the bottom of this file](#checklist). LocalAI advertises its feature surface in several independent places — miss any one of them and clients/admins/UI won't know the endpoint exists.
|
||||
|
||||
## Architecture overview
|
||||
|
||||
Authentication and authorization flow through three layers:
|
||||
|
||||
1. **Global auth middleware** (`core/http/auth/middleware.go` → `auth.Middleware`) — applied to every request in `core/http/app.go`. Handles session cookies, Bearer tokens, API keys, and legacy API keys. Populates `auth_user` and `auth_role` in the Echo context.
|
||||
2. **Feature middleware** (`auth.RequireFeature`) — per-feature access control applied to route groups or individual routes. Checks if the authenticated user has the specific feature enabled.
|
||||
3. **Admin middleware** (`auth.RequireAdmin`) — restricts endpoints to admin users only.
|
||||
|
||||
When auth is disabled (no auth DB, no legacy API keys), all middleware becomes pass-through (`auth.NoopMiddleware`).
|
||||
|
||||
## Adding a new API endpoint
|
||||
|
||||
### Step 1: Create the handler
|
||||
|
||||
Write the endpoint handler in the appropriate package under `core/http/endpoints/`. Follow existing patterns:
|
||||
|
||||
```go
|
||||
// core/http/endpoints/localai/my_feature.go
|
||||
func MyFeatureEndpoint(app *application.Application) echo.HandlerFunc {
|
||||
return func(c echo.Context) error {
|
||||
// Use auth.GetUser(c) to get the authenticated user (may be nil if auth is disabled)
|
||||
user := auth.GetUser(c)
|
||||
|
||||
// Your logic here
|
||||
return c.JSON(http.StatusOK, result)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Register routes
|
||||
|
||||
Add routes in the appropriate file under `core/http/routes/`. The file you use depends on the endpoint category:
|
||||
|
||||
| File | Category |
|
||||
|------|----------|
|
||||
| `routes/openai.go` | OpenAI-compatible API endpoints (`/v1/...`) |
|
||||
| `routes/localai.go` | LocalAI-specific endpoints (`/api/...`, `/models/...`, `/backends/...`) |
|
||||
| `routes/agents.go` | Agent pool endpoints (`/api/agents/...`) |
|
||||
| `routes/auth.go` | Auth endpoints (`/api/auth/...`) |
|
||||
| `routes/ui_api.go` | UI backend API endpoints |
|
||||
|
||||
### Step 3: Apply the right middleware
|
||||
|
||||
Choose the appropriate protection level:
|
||||
|
||||
#### No auth required (public)
|
||||
Exempt paths bypass auth entirely. Add to `isExemptPath()` in `middleware.go` or use the `/api/auth/` prefix (always exempt). Use sparingly — most endpoints should require auth.
|
||||
|
||||
#### Standard auth (any authenticated user)
|
||||
The global middleware already handles this. API paths (`/api/`, `/v1/`, etc.) automatically require authentication when auth is enabled. You don't need to add any extra middleware.
|
||||
|
||||
```go
|
||||
router.GET("/v1/my-endpoint", myHandler) // auth enforced by global middleware
|
||||
```
|
||||
|
||||
#### Admin only
|
||||
Pass `adminMiddleware` to the route. This is set up in `app.go` and passed to `Register*Routes` functions:
|
||||
|
||||
```go
|
||||
// In the Register function signature, accept the middleware:
|
||||
func RegisterMyRoutes(router *echo.Echo, app *application.Application, adminMiddleware echo.MiddlewareFunc) {
|
||||
router.POST("/models/apply", myHandler, adminMiddleware)
|
||||
}
|
||||
```
|
||||
|
||||
#### Feature-gated
|
||||
For endpoints that should be toggleable per-user, use feature middleware. There are two approaches:
|
||||
|
||||
**Approach A: Route-level middleware** (preferred for groups of related endpoints)
|
||||
|
||||
```go
|
||||
// In app.go, create the feature middleware:
|
||||
myFeatureMw := auth.RequireFeature(application.AuthDB(), auth.FeatureMyFeature)
|
||||
|
||||
// Pass it to the route registration function:
|
||||
routes.RegisterMyRoutes(e, app, myFeatureMw)
|
||||
|
||||
// In the routes file, apply to a group:
|
||||
g := e.Group("/api/my-feature", myFeatureMw)
|
||||
g.GET("", listHandler)
|
||||
g.POST("", createHandler)
|
||||
```
|
||||
|
||||
**Approach B: RouteFeatureRegistry** (preferred for individual OpenAI-compatible endpoints)
|
||||
|
||||
Add an entry to `RouteFeatureRegistry` in `core/http/auth/features.go`. The `RequireRouteFeature` global middleware will automatically enforce it:
|
||||
|
||||
```go
|
||||
var RouteFeatureRegistry = []RouteFeature{
|
||||
// ... existing entries ...
|
||||
{"POST", "/v1/my-endpoint", FeatureMyFeature},
|
||||
}
|
||||
```
|
||||
|
||||
## Adding a new feature
|
||||
|
||||
When you need a new toggleable feature (not just a new endpoint under an existing feature):
|
||||
|
||||
### 1. Define the feature constant
|
||||
|
||||
Add to `core/http/auth/permissions.go`:
|
||||
|
||||
```go
|
||||
const (
|
||||
// Add to the appropriate group:
|
||||
// Agent features (default OFF for new users)
|
||||
FeatureMyFeature = "my_feature"
|
||||
|
||||
// OR API features (default ON for new users)
|
||||
FeatureMyFeature = "my_feature"
|
||||
)
|
||||
```
|
||||
|
||||
Then add it to the appropriate slice:
|
||||
|
||||
```go
|
||||
// Default OFF — user must be explicitly granted access:
|
||||
var AgentFeatures = []string{..., FeatureMyFeature}
|
||||
|
||||
// Default ON — user has access unless explicitly revoked:
|
||||
var APIFeatures = []string{..., FeatureMyFeature}
|
||||
```
|
||||
|
||||
### 2. Add feature metadata
|
||||
|
||||
In `core/http/auth/features.go`, add to the appropriate `FeatureMetas` function so the admin UI can display it:
|
||||
|
||||
```go
|
||||
func AgentFeatureMetas() []FeatureMeta {
|
||||
return []FeatureMeta{
|
||||
// ... existing ...
|
||||
{FeatureMyFeature, "My Feature", false}, // false = default OFF
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Wire up the middleware
|
||||
|
||||
In `core/http/app.go`:
|
||||
|
||||
```go
|
||||
myFeatureMw := auth.RequireFeature(application.AuthDB(), auth.FeatureMyFeature)
|
||||
```
|
||||
|
||||
Then pass it to the route registration function.
|
||||
|
||||
### 4. Register route-feature mappings (if applicable)
|
||||
|
||||
If your feature gates standard API endpoints (like `/v1/...`), add entries to `RouteFeatureRegistry` in `features.go` instead of using per-route middleware.
|
||||
|
||||
## Accessing the authenticated user in handlers
|
||||
|
||||
```go
|
||||
import "github.com/mudler/LocalAI/core/http/auth"
|
||||
|
||||
func MyHandler(c echo.Context) error {
|
||||
// Get the user (nil when auth is disabled or unauthenticated)
|
||||
user := auth.GetUser(c)
|
||||
if user == nil {
|
||||
// Handle unauthenticated — or let middleware handle it
|
||||
}
|
||||
|
||||
// Check role
|
||||
if user.Role == auth.RoleAdmin {
|
||||
// admin-specific logic
|
||||
}
|
||||
|
||||
// Check feature access programmatically (when you need conditional behavior, not full blocking)
|
||||
if auth.HasFeatureAccess(db, user, auth.FeatureMyFeature) {
|
||||
// feature-specific logic
|
||||
}
|
||||
|
||||
// Check model access
|
||||
if !auth.IsModelAllowed(db, user, modelName) {
|
||||
return c.JSON(http.StatusForbidden, ...)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Middleware composition patterns
|
||||
|
||||
Middleware can be composed at different levels. Here are the patterns used in the codebase:
|
||||
|
||||
### Group-level middleware (agents pattern)
|
||||
```go
|
||||
// All routes in the group share the middleware
|
||||
g := e.Group("/api/agents", poolReadyMw, agentsMw)
|
||||
g.GET("", listHandler)
|
||||
g.POST("", createHandler)
|
||||
```
|
||||
|
||||
### Per-route middleware (localai pattern)
|
||||
```go
|
||||
// Individual routes get middleware as extra arguments
|
||||
router.POST("/models/apply", applyHandler, adminMiddleware)
|
||||
router.GET("/metrics", metricsHandler, adminMiddleware)
|
||||
```
|
||||
|
||||
### Middleware slice (openai pattern)
|
||||
```go
|
||||
// Build a middleware chain for a handler
|
||||
chatMiddleware := []echo.MiddlewareFunc{
|
||||
usageMiddleware,
|
||||
traceMiddleware,
|
||||
modelFilterMiddleware,
|
||||
}
|
||||
app.POST("/v1/chat/completions", chatHandler, chatMiddleware...)
|
||||
```
|
||||
|
||||
## Error response format
|
||||
|
||||
Always use `schema.ErrorResponse` for auth/permission errors to stay consistent with the OpenAI-compatible API:
|
||||
|
||||
```go
|
||||
return c.JSON(http.StatusForbidden, schema.ErrorResponse{
|
||||
Error: &schema.APIError{
|
||||
Message: "feature not enabled for your account",
|
||||
Code: http.StatusForbidden,
|
||||
Type: "authorization_error",
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
Use these HTTP status codes:
|
||||
- `401 Unauthorized` — no valid credentials provided
|
||||
- `403 Forbidden` — authenticated but lacking permission
|
||||
- `429 Too Many Requests` — rate limited (auth endpoints)
|
||||
|
||||
## Usage tracking
|
||||
|
||||
If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
|
||||
|
||||
## Advertising surfaces — where to register a new capability
|
||||
|
||||
Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.
|
||||
|
||||
### 1. Swagger `@Tags` annotation (mandatory)
|
||||
|
||||
Every handler needs a swagger block so the endpoint appears in `/swagger/index.html` and in the `/api/instructions` output. The `@Tags` value is what groups the endpoint into a capability area:
|
||||
|
||||
```go
|
||||
// MyEndpoint does X.
|
||||
// @Summary Do X.
|
||||
// @Tags my-capability
|
||||
// @Param request body schema.MyRequest true "payload"
|
||||
// @Success 200 {object} schema.MyResponse "Response"
|
||||
// @Router /v1/my-endpoint [post]
|
||||
func MyEndpoint(...) echo.HandlerFunc { ... }
|
||||
```
|
||||
|
||||
Use an existing tag when the endpoint extends an existing area (e.g. `audio`, `images`, `face-recognition`). Create a new tag only when the endpoint introduces a genuinely new capability surface — and in that case, also register it in step 2.
|
||||
|
||||
After adding endpoints, regenerate the embedded spec so the runtime serves it:
|
||||
|
||||
```bash
|
||||
make protogen-go # ensures gRPC codegen is fresh first
|
||||
make swagger # regenerates swagger/swagger.json
|
||||
```
|
||||
|
||||
### 2. `/api/instructions` registry (for new capability areas)
|
||||
|
||||
`core/http/endpoints/localai/api_instructions.go` defines `instructionDefs` — a lightweight, machine-readable index of capability areas that groups swagger endpoints by tag. It's the primary discovery surface for agents and SDKs ("what can this server do?").
|
||||
|
||||
**When to update:** only when adding a new capability area (a new swagger tag). Existing-tag additions automatically surface without any change here.
|
||||
|
||||
Add an entry to `instructionDefs`:
|
||||
|
||||
```go
|
||||
{
|
||||
Name: "my-capability", // URL segment at /api/instructions/my-capability
|
||||
Description: "Short sentence describing the capability",
|
||||
Tags: []string{"my-capability"}, // must match swagger @Tags
|
||||
Intro: "Optional gotcha/context that isn't in the swagger descriptions (caveats, defaults, cross-references to other endpoints).",
|
||||
},
|
||||
```
|
||||
|
||||
Also bump the expected-length count in `api_instructions_test.go` and add the name to the `ContainElements` assertion.
|
||||
|
||||
### 3. `capabilities.js` symbol (for new model-config FLAG_* flags)
|
||||
|
||||
If your feature needs a new `FLAG_*` usecase flag in `core/config/model_config.go` (so users can filter gallery models by it, and so `/v1/models` surfaces it), you need to update **all** of:
|
||||
|
||||
- `Usecase<Name>` string constant in `core/config/backend_capabilities.go`
|
||||
- `UsecaseInfoMap` entry mapping the string to its flag + gRPC method
|
||||
- `FLAG_<NAME>` bitmask in `core/config/model_config.go`
|
||||
- `GetAllModelConfigUsecases()` map entry (otherwise the YAML loader silently ignores the string)
|
||||
- `ModalityGroups` membership if the flag should affect `IsMultimodal()` (e.g. realtime_audio is in both speech-input and audio-output groups so a lone flag still reads as multimodal)
|
||||
- `GuessUsecases()` branch listing the backends that own this capability
|
||||
- `usecaseFilters` in `core/http/routes/ui_api.go` (drives the gallery filter dropdown)
|
||||
- `Models.jsx` `FILTERS` array + matching `filters.<camelCase>` i18n key in `core/http/react-ui/public/locales/en/models.json`
|
||||
- `core/http/react-ui/src/utils/capabilities.js`:
|
||||
|
||||
```js
|
||||
export const CAP_MY_CAPABILITY = 'FLAG_MY_CAPABILITY'
|
||||
```
|
||||
|
||||
React pages that want to filter the ModelSelector by capability import this symbol. Declare it even if you're not building the UI page yet — the declaration keeps the Go/JS vocabularies in sync.
|
||||
|
||||
### 4. `docs/content/` (user-facing documentation)
|
||||
|
||||
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features and an entry in `docs/content/whats-new.md`. See the pattern used by `face-recognition.md` / `object-detection.md`.
|
||||
|
||||
## Path protection rules
|
||||
|
||||
The global auth middleware classifies paths as API paths or non-API paths:
|
||||
|
||||
- **API paths** (always require auth when auth is enabled): `/api/`, `/v1/`, `/models/`, `/backends/`, `/backend/`, `/tts`, `/vad`, `/video`, `/stores/`, `/system`, `/ws/`, `/metrics`
|
||||
- **Exempt paths** (never require auth): `/api/auth/` prefix, anything in `appConfig.PathWithoutAuth`
|
||||
- **Non-API paths** (UI, static assets): pass through without auth — the React UI handles login redirects client-side
|
||||
|
||||
If you add endpoints under a new top-level path prefix, add it to `isAPIPath()` in `middleware.go` to ensure it requires authentication.
|
||||
|
||||
## Checklist
|
||||
|
||||
When adding a new endpoint:
|
||||
|
||||
**Routing & auth**
|
||||
- [ ] Handler in `core/http/endpoints/`
|
||||
- [ ] Route registered in appropriate `core/http/routes/` file
|
||||
- [ ] Auth level chosen: public / standard / admin / feature-gated
|
||||
- [ ] Entry added to `RouteFeatureRegistry` in `core/http/auth/features.go` (one row per route/method — all /v1/* routes gate through this, not per-route middleware)
|
||||
- [ ] If new feature: constant in `permissions.go`, added to the right slice (`APIFeatures` default-ON / `AgentFeatures` default-OFF), metadata in `features.go` `*FeatureMetas()`
|
||||
- [ ] If feature uses group middleware: wired in `core/http/app.go` and passed to the route registration function
|
||||
- [ ] If new path prefix: added to `isAPIPath()` in `middleware.go`
|
||||
- [ ] If token-counting: `usageMiddleware` added to middleware chain
|
||||
|
||||
**Advertising surfaces (easy to miss — see the [Advertising surfaces](#advertising-surfaces--where-to-register-a-new-capability) section)**
|
||||
- [ ] Swagger block on the handler: `@Summary`, `@Tags`, `@Param`, `@Success`, `@Router`
|
||||
- [ ] If new capability area (new swagger tag): entry in `instructionDefs` in `core/http/endpoints/localai/api_instructions.go` + test count bumped in `api_instructions_test.go`
|
||||
- [ ] If new `FLAG_*` usecase flag: matching `CAP_*` symbol exported from `core/http/react-ui/src/utils/capabilities.js`
|
||||
- [ ] `docs/content/features/<feature>.md` created; cross-links from related feature pages; entry in `docs/content/whats-new.md`
|
||||
|
||||
**Quality**
|
||||
- [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
|
||||
- [ ] Tests cover both authenticated and unauthenticated access
|
||||
- [ ] Swagger regenerated (`make swagger`) if you changed any `@Router`/`@Tags`/`@Param` annotation
|
||||
|
||||
## Companion: MCP admin tool surface
|
||||
|
||||
**Required for admin endpoints.** Every new admin endpoint MUST be considered for the MCP admin tool surface — the REST API and the MCP tool catalog can drift silently otherwise, and both the LocalAI Assistant chat modality and the standalone `local-ai mcp-server` rely on `pkg/mcp/localaitools/` to mirror REST.
|
||||
|
||||
Two outcomes are acceptable; one is not:
|
||||
|
||||
- **Tool added.** The new endpoint is something an admin would manage conversationally (install, list, edit, toggle, upgrade). Follow the full checklist in [.agents/localai-assistant-mcp.md](localai-assistant-mcp.md): add a `LocalAIClient` interface method, implement it in both `inproc` and `httpapi`, register the tool with a `Tool*` constant, update the skill prompts, **and add the route to `toolToHTTPRoute` in `pkg/mcp/localaitools/coverage_test.go`**.
|
||||
- **Tool deliberately skipped.** The endpoint is internal/diagnostic and adding a chat path would be misleading. Document the decision in the PR description; no code action.
|
||||
- **Forgot.** This breaks the contract. The `TestToolHTTPRouteMappingComplete` test in `pkg/mcp/localaitools` is a partial guard (it checks every `Tool*` has a route mapping), but it does NOT detect new REST endpoints without a tool — that's still a process check on the PR author.
|
||||
|
||||
**Add to the bottom of the checklist below**:
|
||||
- [ ] If admin: decided whether MCP coverage is needed; if yes, tool registered + map updated; if no, skip-reason in PR description.
|
||||
@@ -1,126 +0,0 @@
|
||||
# Backend image signing & verification
|
||||
|
||||
LocalAI verifies backend OCI images against a per-gallery keyless-cosign
|
||||
policy. This page documents the trust model, the producer side
|
||||
(`.github/workflows/backend_merge.yml` in this repo), and the consumer
|
||||
side (`pkg/oci/cosignverify` plus the gallery YAML).
|
||||
|
||||
## Trust model
|
||||
|
||||
- **Producer:** `.github/workflows/backend_merge.yml` signs each pushed
|
||||
manifest list with `cosign sign --recursive` in keyless mode after
|
||||
`docker buildx imagetools create`. The signing cert is issued by
|
||||
Fulcio bound to the workflow's OIDC identity. There is no long-lived
|
||||
signing key. `--recursive` signs both the manifest list and every
|
||||
per-arch entry — needed because our consumer resolves a tag to a
|
||||
per-arch manifest before checking signatures.
|
||||
- **Storage:** Signatures are written as OCI 1.1 referrers
|
||||
(`--registry-referrers-mode=oci-1-1`) in the new Sigstore bundle format
|
||||
(current cosign releases do this by default; no `--new-bundle-format`
|
||||
flag). No `:sha256-<hex>.sig` tag clutter.
|
||||
- **Consumer:** `pkg/oci/cosignverify` discovers the bundle via the
|
||||
referrers API, hands it to `sigstore-go`, and verifies it against the
|
||||
policy declared in the gallery YAML (`Gallery.Verification`).
|
||||
- **Revocation:** Keyless cosign certs are ephemeral (10-minute Fulcio
|
||||
validity), so revocation is policy-side, not CA-side. The gallery's
|
||||
`verification.not_before` (RFC3339) is the kill-switch — advance it to
|
||||
invalidate every signature produced before a known compromise window.
|
||||
|
||||
## Producer setup
|
||||
|
||||
`backend_merge.yml` is the workflow that joins per-arch digests into the
|
||||
multi-arch manifest list users actually pull, so it's also the right place
|
||||
to sign. The job needs:
|
||||
|
||||
- `permissions: { id-token: write, contents: read }` at the job level so
|
||||
the runner can exchange its GitHub OIDC token for a Fulcio cert.
|
||||
- `sigstore/cosign-installer@v3` step (current cosign releases already
|
||||
default to the new bundle format).
|
||||
- After each `docker buildx imagetools create`, resolve the resulting
|
||||
list digest with `docker buildx imagetools inspect <tag> --format
|
||||
'{{.Manifest.Digest}}'` and sign:
|
||||
|
||||
```sh
|
||||
cosign sign --yes --recursive \
|
||||
--registry-referrers-mode=oci-1-1 \
|
||||
"${REGISTRY_REPO}@${DIGEST}"
|
||||
```
|
||||
|
||||
Sign by digest, never by tag — signing by tag binds the signature to
|
||||
whatever the tag points at *now*, and a subsequent tag push orphans it.
|
||||
|
||||
`--registry-referrers-mode=oci-1-1` is still gated behind
|
||||
`COSIGN_EXPERIMENTAL=1` in cosign v2.4.x (set at the job env level in
|
||||
`backend_merge.yml`). Re-evaluate when bumping the pinned cosign release
|
||||
— newer versions are expected to graduate this flag and the env var can
|
||||
then be dropped.
|
||||
|
||||
`backend_build_darwin.yml` builds and pushes single-arch darwin images
|
||||
that bypass the manifest-list merge. If/when those entries get a gallery
|
||||
`verification:` policy, the equivalent cosign step has to land there
|
||||
too.
|
||||
|
||||
## Consumer setup (in `mudler/LocalAI` gallery YAML)
|
||||
|
||||
Once CI is signing, add a `verification:` block to the backend gallery
|
||||
entry (`backend/index.yaml`):
|
||||
|
||||
```yaml
|
||||
- name: localai
|
||||
url: github:mudler/LocalAI/backend/index.yaml@master
|
||||
verification:
|
||||
issuer: "https://token.actions.githubusercontent.com"
|
||||
identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/heads/master$"
|
||||
# Optional revocation cutoff; advance during incident response.
|
||||
# not_before: "2026-06-01T00:00:00Z"
|
||||
```
|
||||
|
||||
Identity matching pins the OIDC subject Fulcio issued the signing cert
|
||||
to. Without this, any image signed by *anyone* with a Fulcio cert would
|
||||
pass — the regex is what makes a signature mean "produced by our CI".
|
||||
|
||||
## Strict mode
|
||||
|
||||
Default behaviour: OCI backends without a `verification:` block install
|
||||
with a warning (logs include `installing OCI backend without signature
|
||||
verification`). Tarball/HTTP backends without a `sha256` field log a
|
||||
similar warning.
|
||||
|
||||
For production, set `LOCALAI_REQUIRE_BACKEND_INTEGRITY=1` (or pass
|
||||
`--require-backend-integrity` to `local-ai run` / `local-ai backends
|
||||
install` / `local-ai models install`). The warning becomes a hard error
|
||||
and unverifiable backends refuse to install.
|
||||
|
||||
## Revocation playbook
|
||||
|
||||
If `backend_merge.yml` (or any workflow with `id-token: write`) is
|
||||
compromised and we've shipped malicious signed images:
|
||||
|
||||
1. **Identify the compromise window.** Find the earliest IntegratedTime
|
||||
from the bad signatures (Rekor search by `subject` filter).
|
||||
2. **Set `verification.not_before`** in `backend/index.yaml` to a
|
||||
timestamp just *after* that window's start.
|
||||
3. **Push the YAML.** Deployed LocalAI instances pick it up on next
|
||||
gallery refresh (1-hour cache in `core/gallery/gallery.go`).
|
||||
4. **Fix the underlying compromise** in the workflow and re-sign images
|
||||
with the new build, which will have IntegratedTime > `not_before`.
|
||||
5. **Optional:** for absolute decisiveness, also rotate to a new
|
||||
workflow path (`backend_merge_v2.yml`) and update `identity_regex`.
|
||||
|
||||
## Where the code lives
|
||||
|
||||
- `pkg/oci/cosignverify/` — verifier, policy, OCI referrer fetch, NotBefore enforcement.
|
||||
- `pkg/downloader/uri.go` — `WithImageVerifier` option threaded through `DownloadFileWithContext`.
|
||||
- `core/gallery/backends.go` — `backendDownloadOptions` builds the verifier from the gallery's policy.
|
||||
- `core/config/gallery.go` — `Gallery.Verification` YAML schema.
|
||||
- `core/cli/run.go`, `core/cli/backends.go`, `core/cli/models.go` — `--require-backend-integrity` flag propagation.
|
||||
- `.github/workflows/backend_merge.yml` — producer-side `cosign sign --recursive` after each multi-arch manifest list push.
|
||||
|
||||
## Out of scope (follow-ups)
|
||||
|
||||
- **Signing the gallery YAML itself.** The index is fetched over HTTPS
|
||||
from GitHub; we trust the host. A cosign blob signature on the YAML
|
||||
would close that gap but adds key-management overhead. Revisit this
|
||||
page if/when added.
|
||||
- **Tarball/HTTP backend signing.** Cosign can sign arbitrary blobs, but
|
||||
for now non-OCI backends keep using the `sha256:` field in YAML.
|
||||
@@ -1,47 +0,0 @@
|
||||
# Build and Testing
|
||||
|
||||
Building and testing the project depends on the components involved and the platform where development is taking place. Due to the amount of context required it's usually best not to try building or testing the project unless the user requests it. If you must build the project then inspect the Makefile in the project root and the Makefiles of any backends that are effected by changes you are making. In addition the workflows in .github/workflows can be used as a reference when it is unclear how to build or test a component. The primary Makefile contains targets for building inside or outside Docker, if the user has not previously specified a preference then ask which they would like to use.
|
||||
|
||||
## Building a specified backend
|
||||
|
||||
Let's say the user wants to build a particular backend for a given platform. For example let's say they want to build coqui for ROCM/hipblas
|
||||
|
||||
- The Makefile has targets like `docker-build-coqui` created with `generate-docker-build-target` at the time of writing. Recently added backends may require a new target.
|
||||
- At a minimum we need to set the BUILD_TYPE, BASE_IMAGE build-args
|
||||
- Use `.github/backend-matrix.yml` as a reference — it's the data-only YAML that lists every backend variant's `build-type`, `base-image`, `platforms`, etc. (`backend.yml` and `backend_pr.yml` consume it via `scripts/changed-backends.js`).
|
||||
- l4t and cublas also require the CUDA major and minor version.
|
||||
- For llama-cpp / ik-llama-cpp / turboquant the matrix also sets `builder-base-image` pointing at a prebuilt `quay.io/go-skynet/ci-cache:base-grpc-*` tag. Local `make backends/<name>` defaults to `BUILDER_TARGET=builder-fromsource` and doesn't need it — the Dockerfile's from-source stage installs everything itself.
|
||||
- You can pretty print a command like `DOCKER_MAKEFLAGS=-j$(nproc --ignore=1) BUILD_TYPE=hipblas BASE_IMAGE=rocm/dev-ubuntu-24.04:7.2.1 make docker-build-coqui`
|
||||
- Unless the user specifies that they want you to run the command, then just print it because not all agent frontends handle long running jobs well and the output may overflow your context
|
||||
- The user may say they want to build AMD or ROCM instead of hipblas, or Intel instead of SYCL or NVIDIA insted of l4t or cublas. Ask for confirmation if there is ambiguity.
|
||||
- Sometimes the user may need extra parameters to be added to `docker build` (e.g. `--platform` for cross-platform builds or `--progress` to view the full logs), in which case you can generate the `docker build` command directly.
|
||||
|
||||
## Test coverage gate
|
||||
|
||||
The core Go suites (`./pkg`, `./core`, plus the in-process integration suite `./tests/e2e`) are covered by a **strict, monotonic coverage ratchet**:
|
||||
|
||||
- `make test-coverage` — runs the suites with `covermode=atomic` instrumentation and writes a merged profile to `coverage/coverage.out`. Uses the same prerequisites as `make test`.
|
||||
- **`--coverpkg` (`COVERAGE_COVERPKG = core/...,pkg/...`):** coverage is attributed to the core+pkg packages, not just the package under test. This is what lets the in-process `tests/e2e` suite (which drives the real HTTP server over loopback via `application.New`) credit the `core/http/endpoints/...` handlers it exercises — folding it in roughly doubled endpoint coverage (e.g. `endpoints/openai` 13.6% → 52%). The denominator is therefore *all* of `core`+`pkg` (minus generated proto, dropped via `COVERAGE_EXCLUDE_RE`), so the number isn't comparable to a plain per-package figure.
|
||||
- **Integration suites (`COVERAGE_E2E_ROOTS = ./tests/e2e`)** run non-recursively (excludes `tests/e2e/distributed`, which needs containers) with `--label-filter=!real-models` (those need a downloaded model) against the mock backend built by `prepare-test`. `tests/integration` is deliberately excluded — it needs `make backends/local-store`, which the coverage CI job doesn't build.
|
||||
- **Flake note:** folding integration tests into a *strict* gate means a hard e2e failure (or a spec that silently stops running) can fail the coverage gate, not just the test. `--flake-attempts` absorbs transient retryable failures; covermode=atomic keeps line coverage deterministic otherwise.
|
||||
- **Why one ginkgo run per root (`scripts/run-coverage.sh`):** passing several recursive roots to a *single* ginkgo invocation (e.g. `ginkgo -r ./pkg ./core`) only merges **one** root's coverprofile into `--output-dir`/`--coverprofile` — the others are silently dropped. Verified with ginkgo 2.29.0: `-r ./pkg ./core` yields only `./pkg` coverage, while `-r ./core` alone yields all 34 core packages. So the script runs each root separately and concatenates the (disjoint) profiles. Don't "simplify" it back to a single multi-root invocation — that's how `core/` (including all of `core/http`, ~7.4k statements) silently vanished from the number before.
|
||||
- **Build tags (`COVERAGE_TAGS`, passed via `GINKGO_TAGS`):** defaults to `debug auth`. The `auth` tag is required to compile the real (sqlite-backed) auth implementation and its ~150 `//go:build auth` tests — without it those files aren't built, the tests don't run, and the gate scores auth against a stub (~3.7% instead of ~38%). If you add new tag-gated tests, extend `COVERAGE_TAGS` or they won't count (and likely won't run in CI at all).
|
||||
- `make test-coverage-check` — runs `test-coverage`, then `scripts/coverage-check.sh` fails the build if total coverage is **below** the committed baseline in `coverage-baseline.txt`. The Linux job in `.github/workflows/test.yml` runs this instead of `make test`.
|
||||
- `make test-coverage-baseline` — regenerates and overwrites `coverage-baseline.txt` from the current run.
|
||||
|
||||
### React UI coverage
|
||||
|
||||
The React UI (`core/http/react-ui/`) has **no component/unit tests** — its only tests are the Playwright e2e specs in `e2e/`, which run against the real app served by `tests/e2e-ui/ui-test-server` (the dist is `//go:embed`ed, so the server is rebuilt per coverage run). Those specs do genuinely exercise the UI (clicks, `fill`, `setInputFiles`, `getByRole`/`getByText`, visibility/value assertions).
|
||||
|
||||
- `make test-ui-coverage` — builds an istanbul-instrumented bundle (`COVERAGE=true`, via `vite-plugin-istanbul` with `forceBuildInstrument: true` — the plugin skips production builds otherwise), re-embeds it into `ui-test-server` (the dist is `//go:embed`ed), runs the Playwright specs, and writes an `nyc` report to `core/http/react-ui/coverage/`. The specs import `{ test, expect }` from `e2e/coverage-fixtures.js` (re-exports Playwright's, plus harvests `window.__coverage__` into `.nyc_output/` after each test). Instrumentation is off unless `COVERAGE=true`, so dev/prod builds and plain `make test-ui-e2e` are unaffected (the fixture no-ops when `window.__coverage__` is absent).
|
||||
- **Browser:** the flake dev shell ships `chromium` and exports `PLAYWRIGHT_CHROMIUM_PATH`; `playwright.config.js` uses it via `launchOptions.executablePath`, and the Makefile skips `playwright install` when it's set. This avoids Playwright's downloaded browser, which can't resolve system libs (`libglib-2.0`, …) on NixOS. In CI (no `PLAYWRIGHT_CHROMIUM_PATH`) the Makefile falls back to `playwright install --with-deps chromium`.
|
||||
- The app is a React SPA, so coverage accumulates across in-app navigation within a test; a full `page.goto`/reload resets it.
|
||||
- `.nycrc.json` uses `all: true`, so **every `src/**` file is in the report**, including 0%-coverage ones — that's how you spot features with no test at all (sort the HTML report or `coverage-summary.json` by line% ascending).
|
||||
- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`).
|
||||
- **Why it has a tolerance (unlike the strict Go gate):** UI e2e coverage is *non-deterministic*. Specs that assert on state and end while async/lazy render work is still in flight collect those lines only when the render beats the coverage teardown — so the total drifts with machine speed/load (a fast local box reads higher than a slow CI runner), diffusely across many specs. The tolerance absorbs that drift, so set the baseline *below* the slow-CI floor, never to a fast-local `make test-ui-coverage-baseline` number, or CI flaps.
|
||||
- **Raising coverage is cheap:** a *render-smoke* spec (navigate to a route, assert its header renders) mounts a lazy page and runs its full render + initial effects, capturing most of its lines in a few lines of test — see `e2e/page-render-smoke.spec.js`. Auth is disabled in the test server (`isAdmin=true`), so `RequireAdmin`/`RequireFeature` routes render without a mock. The most *deterministic* win is removing a race: make a spec `await` a rendered element before ending (see `e2e/agents.spec.js` → AgentCreate) so its lines count every run.
|
||||
|
||||
Rules (both gates):
|
||||
- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
|
||||
- If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
|
||||
- The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.
|
||||
@@ -1,268 +0,0 @@
|
||||
# CI Build Caching
|
||||
|
||||
Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache plus a layered set of prebuilt base images. This file explains how the cache is laid out, what invalidates it, and how to bypass it.
|
||||
|
||||
## Workflow surfaces
|
||||
|
||||
| Workflow | Purpose | Triggers |
|
||||
|---|---|---|
|
||||
| `.github/workflows/backend.yml` | Backend container images on master | `push` to master + tags, weekly Sunday cron, `workflow_dispatch` |
|
||||
| `.github/workflows/backend_pr.yml` | Backend container images on PRs | `pull_request` |
|
||||
| `.github/workflows/backend_build.yml` | Reusable: builds one backend (one arch) by digest | `workflow_call` from above |
|
||||
| `.github/workflows/backend_merge.yml` | Reusable: assembles per-arch digests into a multi-arch manifest list | `workflow_call` |
|
||||
| `.github/workflows/backend_build_darwin.yml` | Reusable: macOS-native backend builds | `workflow_call` |
|
||||
| `.github/workflows/image.yml` / `image-pr.yml` | Root LocalAI image (push / PR) | push / PR |
|
||||
| `.github/workflows/image_build.yml` / `image_merge.yml` | Reusable: per-arch root-image build + merge | `workflow_call` |
|
||||
| `.github/workflows/base-images.yml` | Builds the prebuilt `base-grpc-*` builder bases | Saturdays 05:00 UTC cron, `workflow_dispatch`, master push touching `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/apt-mirror.sh`, or this workflow |
|
||||
|
||||
The matrix that drives `backend.yml` / `backend_pr.yml` lives in **`.github/backend-matrix.yml`** (data-only YAML, not embedded in the workflow). `scripts/changed-backends.js` parses it, applies path-filter logic against the PR diff (PR events) or the GitHub Compare API (push events), and emits the filtered matrix plus a `merge-matrix` for backends with multiple per-arch entries.
|
||||
|
||||
## Cache layout
|
||||
|
||||
- **Cache registry**: `quay.io/go-skynet/ci-cache`
|
||||
- **One tag per matrix entry per arch**, derived from `tag-suffix` and `platform-tag`:
|
||||
- Backend builds (`backend_build.yml`): `cache<tag-suffix>-<platform-tag>`
|
||||
- e.g. `cache-cpu-faster-whisper-amd64`, `cache-cpu-faster-whisper-arm64`, `cache-gpu-nvidia-cuda-13-llama-cpp-amd64`
|
||||
- Root image builds (`image_build.yml`): `cache-localai<tag-suffix>-<platform-tag>` (with a `-core` placeholder when `tag-suffix` is empty, so `cache-localai-core-amd64` for the core image)
|
||||
- Pre-built base images (`base-images.yml`): `cache-base-grpc-<variant>` (one per `(BUILD_TYPE, arch)` permutation)
|
||||
- Each tag stores a multi-arch BuildKit cache manifest (`mode=max`), so every intermediate stage is re-usable, not just the final image.
|
||||
|
||||
The per-arch suffix exists because amd64 and arm64 builds produce different intermediate content; sharing one cache key would thrash on every cross-arch rebuild.
|
||||
|
||||
## Read/write semantics
|
||||
|
||||
| Trigger | `cache-from` | `cache-to` |
|
||||
|---|---|---|
|
||||
| `push` to `master` / tag / cron / dispatch | yes | yes (`mode=max,ignore-error=true`) |
|
||||
| `pull_request` | yes | **no** |
|
||||
|
||||
PR builds read master's warm cache but never write — this prevents PRs from polluting the shared cache with their experimental state. After merge, the master build for that matrix entry refreshes the cache.
|
||||
|
||||
`ignore-error=true` on the write side means a transient quay push failure does not fail the build; the next master push retries.
|
||||
|
||||
## Pre-built base images (`base-grpc-*`)
|
||||
|
||||
The C++ backend Dockerfiles (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) compile gRPC from source. On a cold build that's ~25–35 min before any LocalAI source compiles. To skip that on CI, `.github/workflows/base-images.yml` builds and pushes a set of pre-prepped builder bases:
|
||||
|
||||
| Tag | Contents |
|
||||
|---|---|
|
||||
| `base-grpc-amd64` / `base-grpc-arm64` | Ubuntu 24.04 + apt build deps + protoc + cmake + gRPC at `/opt/grpc` |
|
||||
| `base-grpc-cuda-12-amd64` | the above + CUDA 12.8 toolkit |
|
||||
| `base-grpc-cuda-13-amd64` | the above + CUDA 13.0 toolkit (Ubuntu 22.04 base) |
|
||||
| `base-grpc-cuda-13-arm64` | the above + CUDA 13.0 sbsa toolkit (Ubuntu 24.04 base) |
|
||||
| `base-grpc-l4t-cuda-12-arm64` | JetPack r36.4.0 base (CUDA preinstalled, `SKIP_DRIVERS=true`) + gRPC |
|
||||
| `base-grpc-rocm-amd64` | rocm/dev-ubuntu-24.04:7.2.1 base + hipblas/hipblaslt/rocblas + gRPC |
|
||||
| `base-grpc-vulkan-amd64` / `base-grpc-vulkan-arm64` | Ubuntu 24.04 + Vulkan SDK 1.4.335 + gRPC |
|
||||
| `base-grpc-intel-amd64` | intel/oneapi-basekit:2025.3.2 base + gRPC |
|
||||
|
||||
**Single source of truth**: the install logic for all 10 variants lives in `.docker/install-base-deps.sh`. Both `Dockerfile.base-grpc-builder` AND each variant Dockerfile's `builder-fromsource` stage bind-mount and execute the same script — so the prebuilt CI base and the local from-source path are bit-equivalent by construction.
|
||||
|
||||
### How variant Dockerfiles consume the base
|
||||
|
||||
`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` are multi-target. Three stages plus a final aliasing stage:
|
||||
|
||||
- `builder-fromsource` — `FROM ${BASE_IMAGE}` then runs `install-base-deps.sh` and the per-backend compile script. Used when `BUILDER_TARGET=builder-fromsource` (the default; local `make backends/<name>`).
|
||||
- `builder-prebuilt` — `FROM ${BUILDER_BASE_IMAGE}` (one of the prebuilt `base-grpc-*` tags) and runs only the per-backend compile script. Used when `BUILDER_TARGET=builder-prebuilt` (CI when the matrix entry sets `builder-base-image`).
|
||||
- `FROM ${BUILDER_TARGET} AS builder` — alias resolves the ARG-selected stage to a fixed name (BuildKit doesn't allow ARG expansion in `COPY --from=`).
|
||||
- `FROM scratch` + `COPY --from=builder ...package/. ./` — emits the final scratch image with just the package contents.
|
||||
|
||||
BuildKit prunes the unreferenced builder stage, so each build only runs the path it needs. `backend_build.yml` derives `BUILDER_TARGET=builder-prebuilt` automatically when the matrix entry has a non-empty `builder-base-image`; otherwise it defaults to `builder-fromsource`.
|
||||
|
||||
The matrix `(build-type, platforms)` → `builder-base-image` mapping for llama-cpp / ik-llama-cpp / turboquant entries:
|
||||
|
||||
| `build-type` | `platforms` | tag |
|
||||
|---|---|---|
|
||||
| `''` | `linux/amd64` | `base-grpc-amd64` |
|
||||
| `''` | `linux/arm64` | `base-grpc-arm64` |
|
||||
| `cublas` cuda 12 | `linux/amd64` | `base-grpc-cuda-12-amd64` |
|
||||
| `cublas` cuda 13 | `linux/amd64` | `base-grpc-cuda-13-amd64` |
|
||||
| `cublas` cuda 13 | `linux/arm64` | `base-grpc-cuda-13-arm64` |
|
||||
| `cublas` cuda 12 + JetPack base | `linux/arm64` | `base-grpc-l4t-cuda-12-arm64` |
|
||||
| `hipblas` | `linux/amd64` | `base-grpc-rocm-amd64` |
|
||||
| `vulkan` | `linux/amd64` | `base-grpc-vulkan-amd64` |
|
||||
| `vulkan` | `linux/arm64` | `base-grpc-vulkan-arm64` |
|
||||
| `sycl_*` | `linux/amd64` | `base-grpc-intel-amd64` |
|
||||
|
||||
### Bootstrap order when adding a new variant
|
||||
|
||||
If you add a new entry to `base-images.yml`'s matrix, the new tag does not exist on quay until the workflow runs. To consume it from a variant entry safely, dispatch the base-images workflow on the branch first:
|
||||
|
||||
```bash
|
||||
gh workflow run base-images.yml --ref <feature-branch>
|
||||
```
|
||||
|
||||
Wait for the new variant to push, then merge the consumer change. Otherwise the consumer's CI fails with "image not found."
|
||||
|
||||
## Per-arch native builds + manifest merge
|
||||
|
||||
Multi-arch backends (and the core LocalAI image) build natively per arch instead of running both arches under QEMU emulation on a single x86 runner. The pattern:
|
||||
|
||||
- The matrix has TWO entries per multi-arch backend, sharing the same `tag-suffix` but distinct `platforms` + `platform-tag` + `runs-on`. Example: `-cpu-faster-whisper` has one amd64 entry on `ubuntu-latest` and one arm64 entry on `ubuntu-24.04-arm`.
|
||||
- Each per-arch build pushes by **canonical digest only** (no tags) via `outputs: type=image,push-by-digest=true,name-canonical=true,push=true`. The digest is uploaded as an artifact named `digests<tag-suffix>-<platform-tag>` (or `digests-localai<...>` for root-image builds).
|
||||
- `scripts/changed-backends.js` detects shared `tag-suffix` and emits a `merge-matrix` output. `backend.yml` / `backend_pr.yml` have a `backend-merge-jobs` job that consumes it and calls `backend_merge.yml`.
|
||||
- `backend_merge.yml` downloads all matching digest artifacts and runs `docker buildx imagetools create` to publish the final tagged manifest list pointing at both per-arch digests. Same `docker/metadata-action` config as the original monolithic build, so consumers see no tag-shape change.
|
||||
- `image_merge.yml` is the equivalent for the root LocalAI image (`-core` placeholder when `tag-suffix` is empty so the artifact-name glob doesn't over-match across `core` and `gpu-vulkan`).
|
||||
|
||||
**`provenance: false` is required on multi-registry digest pushes**: with the default `mode=max` provenance attestation, BuildKit bundles a per-registry attestation manifest into each registry's manifest list, making the resulting list digest diverge across registries. `steps.build.outputs.digest` only matches one of them and the merge step's `imagetools create <reg>@sha256:<digest>` lookup fails on the other. Setting `provenance: false` keeps the digest content-only and identical across registries.
|
||||
|
||||
## Path filter on master push
|
||||
|
||||
Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynamically through `scripts/changed-backends.js`:
|
||||
|
||||
- **PR events**: paginated `pulls/{n}/files` API → filter the matrix to entries whose `dockerfile` path prefix matches the PR diff.
|
||||
- **Push events**: GitHub Compare API (`/repos/{owner}/{repo}/compare/{before}...{after}`) → same path-filter logic. Falls back to "run everything" on first-branch push (`event.before` zero), API truncation (≥300 changed files), missing API token, or any thrown error.
|
||||
- **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
|
||||
- **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
|
||||
|
||||
### Shared build inputs
|
||||
|
||||
The per-backend prefix match only sees files under a backend's own directory, so a change to shared build infrastructure would rebuild *nothing* — an empty matrix, every job green, and the change reaching no image. That silently un-shipped PR #10946 (a partial-cuDNN packaging fix in `scripts/build/package-gpu-libs.sh`), which merged 1h48m after the weekly cron and so sat unbuilt for a week.
|
||||
|
||||
`SHARED_BUILD_INPUTS` in `scripts/lib/backend-filter.mjs` closes that hole. Each rule maps a shared path to the narrowest set of matrix entries it can honestly invalidate, since a full matrix is 417 Linux + 56 Darwin builds:
|
||||
|
||||
| Changed path | Rebuilds |
|
||||
|---|---|
|
||||
| `backend/backend.proto` | everything (all languages compile or copy it) |
|
||||
| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
|
||||
| `backend/python/common/` | Python, Linux + Darwin |
|
||||
| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
|
||||
| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
|
||||
| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
|
||||
| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
|
||||
|
||||
Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
|
||||
|
||||
The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
|
||||
|
||||
## The `DEPS_REFRESH` cache-buster (Python backends)
|
||||
|
||||
Every Python backend goes through the shared `backend/Dockerfile.python`, which ends with:
|
||||
|
||||
```dockerfile
|
||||
ARG DEPS_REFRESH=initial
|
||||
RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
|
||||
```
|
||||
|
||||
Most Python backends ship `requirements*.txt` files that **do not pin every transitive dep** (`torch`, `transformers`, `vllm`, `diffusers`, etc. are listed without a `==` pin, or with `>=` lower bounds only). With a warm BuildKit cache, the `make` layer hashes only on Dockerfile instructions + COPYed source — not on what `pip install` resolves at runtime. So a warm cache would ship the *first* version of `vllm` ever cached and never pick up upstream releases.
|
||||
|
||||
`DEPS_REFRESH` defends against that:
|
||||
|
||||
- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W19`) before each build and passes it as a build-arg.
|
||||
- The `RUN ... make` layer's BuildKit hash now includes that string, so the layer invalidates **at most once per week**, automatically picking up newer wheels.
|
||||
- Within a week, builds stay warm.
|
||||
|
||||
This applies only to `Dockerfile.python` because:
|
||||
- Go (`Dockerfile.golang`) pins versions in `go.mod` / `go.sum`.
|
||||
- Rust (`Dockerfile.rust`) pins via `Cargo.lock`.
|
||||
- C++ backends pin gRPC (`v1.65.0`) and llama.cpp at a specific commit; their inputs don't drift between rebuilds.
|
||||
|
||||
### Adjusting the cadence
|
||||
|
||||
Bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`) for faster refreshes. For one-shot rebuilds without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
|
||||
|
||||
## ccache for C++ backend builds
|
||||
|
||||
`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` declare a BuildKit cache mount on `/root/.ccache`:
|
||||
|
||||
```dockerfile
|
||||
RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
```
|
||||
|
||||
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
|
||||
|
||||
On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
|
||||
|
||||
## Composite actions
|
||||
|
||||
Two composite actions handle runner-side prep:
|
||||
|
||||
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
|
||||
- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
|
||||
|
||||
Both actions run before any docker buildx step.
|
||||
|
||||
## Concurrency
|
||||
|
||||
All `backend.yml` / `image.yml` / `test.yml` / etc. workflows use:
|
||||
|
||||
```yaml
|
||||
concurrency:
|
||||
group: ci-<workflow>-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
```
|
||||
|
||||
- **PR events** group by PR number → newer pushes to the same PR cancel old runs (intended).
|
||||
- **Push events** group by `github.sha` → each master commit gets its own run; rapid-fire merges don't cancel each other (this was a real issue prior — two master pushes 11 seconds apart would cancel the first's CI).
|
||||
|
||||
## Self-warming, no separate populator
|
||||
|
||||
There is no cron job that pre-warms the BuildKit cache for individual backends. The production builds *are* the populators. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in the variant `builder-fromsource` stage or skipped entirely when consuming `base-grpc-*`, Python wheel installs, etc.). The base-images workflow's weekly cron is the closest thing to a populator and only refreshes the prebuilt builder bases.
|
||||
|
||||
## Manually evicting cache
|
||||
|
||||
To force a fully cold build for one backend or the whole image:
|
||||
|
||||
```bash
|
||||
# Delete a single tag (requires quay credentials with admin on the repo)
|
||||
curl -X DELETE \
|
||||
-H "Authorization: Bearer ${QUAY_TOKEN}" \
|
||||
https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm-amd64
|
||||
|
||||
# List all tags
|
||||
curl -s -H "Authorization: Bearer ${QUAY_TOKEN}" \
|
||||
"https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/?limit=100" | jq '.tags[].name'
|
||||
```
|
||||
|
||||
Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry per arch so a stale tag never bleeds into a different build.
|
||||
|
||||
## What the cache does **not** cover
|
||||
|
||||
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
|
||||
- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
|
||||
- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
|
||||
|
||||
## Darwin native caches
|
||||
|
||||
`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
|
||||
|
||||
| Cache | Path(s) | Key | Scope |
|
||||
|---|---|---|---|
|
||||
| Go modules + build | `~/go/pkg/mod`, `~/Library/Caches/go-build` | `go.sum` (managed by `actions/setup-go@v5` `cache: true`) | All darwin jobs |
|
||||
| Homebrew | `~/Library/Caches/Homebrew/downloads`, selected `/opt/homebrew/Cellar/*` | hash of `backend_build_darwin.yml` | All darwin jobs |
|
||||
| ccache (llama.cpp CMake) | `~/Library/Caches/ccache` | pinned `LLAMA_VERSION` from `backend/cpp/llama-cpp/Makefile` | `inputs.backend == 'llama-cpp'` only |
|
||||
| Python wheels (uv + pip) | `~/Library/Caches/pip`, `~/Library/Caches/uv` | `inputs.backend` + ISO week (`+%Y-W%V`) + hash of that backend's `requirements*.txt` | `inputs.lang == 'python'` only |
|
||||
|
||||
Read/write semantics match the BuildKit cache: `actions/cache/restore` runs every time, `actions/cache/save` is gated on `github.event_name != 'pull_request'`. PRs read master's warm cache but never write back.
|
||||
|
||||
The Python wheel cache uses the same ISO-week cache-buster as the Linux `DEPS_REFRESH` build-arg — same problem (unpinned `torch`/`mlx`/`diffusers`/`transformers` resolve to fresh wheels weekly), same ~one-cold-rebuild-per-week solution.
|
||||
|
||||
The brew Cellar cache requires `HOMEBREW_NO_AUTO_UPDATE=1` and `HOMEBREW_NO_INSTALL_CLEANUP=1` (set as job-level env). Without those, `brew install` would mutate the very directories that were just restored, defeating the cache.
|
||||
|
||||
**Force-link after cache restore**: `actions/cache` restores `/opt/homebrew/Cellar/*` but NOT the `/opt/homebrew/bin/*` symlinks. After a cache hit, `brew install` sees the Cellar entries and decides "already installed" without re-running its link step, leaving the formulas off PATH. The Dependencies step explicitly runs `brew link --overwrite` for every cached formula afterwards to ensure the symlinks exist.
|
||||
|
||||
For ccache, the workflow exports `CMAKE_ARGS=… -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache` via `$GITHUB_ENV` before running `make build-darwin-go-backend`. The Makefile in `backend/cpp/llama-cpp/` already forwards `CMAKE_ARGS` through to each variant build (`fallback`, `grpc`, `rpc-server`), so no script changes are needed. The three variants share most TUs, so ccache dedupes object files across them.
|
||||
|
||||
`backend_build_darwin.yml` also has a llama-cpp-specific build-step branch that runs `make backends/llama-cpp-darwin` (the bespoke script that compiles three CMake variants and bundles dylibs via `otool`), distinct from the generic `make build-darwin-${lang}-backend` path. This was consolidated from a previously-bespoke top-level `llama-cpp-darwin` job in `backend.yml` so llama-cpp on Darwin honors the same path filter as the other 34 Darwin backends.
|
||||
|
||||
### Cache budget on Darwin
|
||||
|
||||
GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~800 MB Go cache + ~2 GB brew Cellar + up to 2 GB ccache + ~1.5 GB × 5 python backends. If the cap is hit, prefer collapsing the per-backend Python keys into a shared `pyenv-darwin-shared-<week>` key (accepts more cross-backend churn for a smaller footprint) before reducing other caches.
|
||||
|
||||
## Self-hosted runners
|
||||
|
||||
`.github/backend-matrix.yml` has zero references to `arc-runner-set` or `bigger-runner` — all backends run on GHA free-tier hosted runners (`ubuntu-latest` for amd64, `ubuntu-24.04-arm` for arm64 native, `macos-14` for Darwin). The migration off self-hosted relied on the per-arch native split (no QEMU emulation) plus `setup-build-disk`'s `/mnt` relocation (~100 GB working space, enough for ROCm dev image + vLLM/torch installs).
|
||||
|
||||
One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
|
||||
|
||||
## Touching the cache pipeline
|
||||
|
||||
When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
|
||||
|
||||
1. **Don't drop `DEPS_REFRESH=...` from the build-args** without a replacement strategy (lockfiles, pinned requirements). Otherwise master will silently freeze on whichever versions were cached at the time.
|
||||
2. **Keep `(tag-suffix, platform-tag)` unique per matrix entry** — together they're the cache namespace. Two matrix entries sharing a key would clobber each other's cache.
|
||||
3. **Keep `cache-to` gated on `github.event_name != 'pull_request'`** — PRs must not write.
|
||||
4. **Keep `ignore-error=true` on `cache-to`** — quay registry hiccups must not fail builds.
|
||||
5. **Keep `provenance: false` on push-by-digest steps** — multi-registry digest divergence is the Bug We Already Fixed; reintroducing provenance attestation re-breaks the merge.
|
||||
6. **`install-base-deps.sh` is the single source of truth for base contents.** Both `Dockerfile.base-grpc-builder` (CI) and the variant Dockerfiles' `builder-fromsource` (local) bind-mount and execute it. If you add a package to one path, add it to the script — don't fork the logic into a Dockerfile RUN.
|
||||
7. **After adding a `base-images.yml` matrix variant, run the workflow on your branch before merging consumer changes** that depend on the new tag — otherwise the consumer's CI fails "image not found."
|
||||
@@ -1,72 +0,0 @@
|
||||
# Coding Style
|
||||
|
||||
The project has the following .editorconfig:
|
||||
|
||||
```
|
||||
root = true
|
||||
|
||||
[*]
|
||||
indent_style = space
|
||||
indent_size = 2
|
||||
end_of_line = lf
|
||||
charset = utf-8
|
||||
trim_trailing_whitespace = true
|
||||
insert_final_newline = true
|
||||
|
||||
[*.go]
|
||||
indent_style = tab
|
||||
|
||||
[Makefile]
|
||||
indent_style = tab
|
||||
|
||||
[*.proto]
|
||||
indent_size = 2
|
||||
|
||||
[*.py]
|
||||
indent_size = 4
|
||||
|
||||
[*.js]
|
||||
indent_size = 2
|
||||
|
||||
[*.yaml]
|
||||
indent_size = 2
|
||||
|
||||
[*.md]
|
||||
trim_trailing_whitespace = false
|
||||
```
|
||||
|
||||
- Use comments sparingly to explain why code does something, not what it does. Comments are there to add context that would be difficult to deduce from reading the code.
|
||||
- Prefer modern Go e.g. use `any` not `interface{}`
|
||||
|
||||
## Logging
|
||||
|
||||
Use `github.com/mudler/xlog` for logging which has the same API as slog.
|
||||
|
||||
## Go tests
|
||||
|
||||
All Go tests — including backend tests — must use [Ginkgo](https://onsi.github.io/ginkgo/) (v2) with Gomega matchers, not the stdlib `testing` package with `t.Run` / `t.Errorf`. A test file should register a suite with `RegisterFailHandler(Fail)` in a `TestXxx(t *testing.T)` bootstrap and use `Describe`/`Context`/`It` blocks for the actual cases. Look at any existing `*_test.go` under `core/` or `pkg/` for a template.
|
||||
|
||||
Do not mix styles within a package. If you are extending tests in a package that already uses Ginkgo, keep using Ginkgo. If you find stdlib-style Go tests in the tree, treat them as tech debt to be migrated rather than as a pattern to follow.
|
||||
|
||||
This is enforced by `golangci-lint` via the `forbidigo` linter (see `.golangci.yml`); calls like `t.Errorf` / `t.Fatalf` / `t.Run` / `t.Skip` / `t.Logf` are flagged. Run `make lint` locally before submitting; the same check runs in CI (`.github/workflows/lint.yml`).
|
||||
|
||||
## Outbound HTTP
|
||||
|
||||
All outbound HTTP must go through `github.com/mudler/LocalAI/pkg/httpclient` rather than the standard library's default client. Use `httpclient.New(...)` (no body deadline — safe for streaming/SSE) or `httpclient.NewWithTimeout(d, ...)` (simple request/response). Both **refuse redirects by default** and set a TLS 1.2 floor.
|
||||
|
||||
The reason is GHSA-3mj3-57v2-4636: the std default client follows redirects, and on a *cross-host* redirect Go forwards custom credential headers (e.g. Anthropic's `x-api-key`) to the redirect target, leaking the secret. `httpclient` fails closed instead.
|
||||
|
||||
- Need to follow redirects (download CDNs, registry blobs, GitHub asset URLs)? Pass `httpclient.WithFollowRedirects()` — it still strips credential headers on any cross-host hop.
|
||||
- Have a custom transport (IP-pinned dialer, HTTP/2 tuning, a credential-injecting `RoundTripper`)? Pass `httpclient.WithTransport(rt)`, basing the transport on `httpclient.HardenedTransport()` to keep the TLS floor. Handed a `*http.Client` by a library? `httpclient.Harden(c)` applies the policy in place.
|
||||
|
||||
This is enforced by `forbidigo` (see `.golangci.yml`): `http.DefaultClient` and `http.Get`/`Post`/`PostForm`/`Head` are flagged. The `&http.Client{}` composite literal can't be matched precisely by forbidigo without also flagging legitimate `*http.Client` type references, so that form is caught by review — don't construct raw clients.
|
||||
|
||||
## Documentation
|
||||
|
||||
The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.
|
||||
|
||||
- **Docs-with-code rule**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. The PR template carries a checklist item for this.
|
||||
- **Feature Documentation**: If you add a new feature (like a new backend or API endpoint), create a new markdown file in `docs/content/features/` explaining what it is, how to configure it, and how to use it.
|
||||
- **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
|
||||
- **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.
|
||||
- **Shortcodes**: Use `{{% notice note %}}`, `{{% notice tip %}}`, or `{{% notice warning %}}` for callout boxes. Do **not** use `{{% alert %}}` — that shortcode does not exist in this project's Hugo theme and will break the docs build.
|
||||
@@ -1,141 +0,0 @@
|
||||
# Debugging and Rebuilding Backends
|
||||
|
||||
When a backend fails at runtime (e.g. a gRPC method error, a Python import error, or a dependency conflict), use this guide to diagnose, fix, and rebuild.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
- **Source directory**: `backend/python/<name>/` (or `backend/go/<name>/`, `backend/cpp/<name>/`)
|
||||
- **Installed directory**: `backends/<name>/` — this is what LocalAI actually runs. It is populated by `make backends/<name>` which builds a Docker image, exports it, and installs it via `local-ai backends install`.
|
||||
- **Virtual environment**: `backends/<name>/venv/` — the installed Python venv (for Python backends). The Python binary is at `backends/<name>/venv/bin/python`.
|
||||
|
||||
Editing files in `backend/python/<name>/` does **not** affect the running backend until you rebuild with `make backends/<name>`.
|
||||
|
||||
## Diagnosing Failures
|
||||
|
||||
### 1. Check the logs
|
||||
|
||||
Backend gRPC processes log to LocalAI's stdout/stderr. Look for lines tagged with the backend's model ID:
|
||||
|
||||
```
|
||||
GRPC stderr id="trl-finetune-127.0.0.1:37335" line="..."
|
||||
```
|
||||
|
||||
Common error patterns:
|
||||
- **"Method not implemented"** — the backend is missing a gRPC method that the Go side calls. The model loader (`pkg/model/initializers.go`) always calls `LoadModel` after `Health`; fine-tuning backends must implement it even as a no-op stub.
|
||||
- **Python import errors / `AttributeError`** — usually a dependency version mismatch (e.g. `pyarrow` removing `PyExtensionType`).
|
||||
- **"failed to load backend"** — the gRPC process crashed or never started. Check stderr lines for the traceback.
|
||||
|
||||
### 2. Test the Python environment directly
|
||||
|
||||
You can run the installed venv's Python to check imports without starting the full server:
|
||||
|
||||
```bash
|
||||
backends/<name>/venv/bin/python -c "import datasets; print(datasets.__version__)"
|
||||
```
|
||||
|
||||
If `pip` is missing from the venv, bootstrap it:
|
||||
|
||||
```bash
|
||||
backends/<name>/venv/bin/python -m ensurepip
|
||||
```
|
||||
|
||||
Then use `backends/<name>/venv/bin/python -m pip install ...` to test fixes in the installed venv before committing them to the source requirements.
|
||||
|
||||
### 3. Check upstream dependency constraints
|
||||
|
||||
When you hit a dependency conflict, check what the main library expects. For example, TRL's upstream `requirements.txt`:
|
||||
|
||||
```
|
||||
https://github.com/huggingface/trl/blob/main/requirements.txt
|
||||
```
|
||||
|
||||
Pin minimum versions in the backend's requirements files to match upstream.
|
||||
|
||||
## Common Fixes
|
||||
|
||||
### Missing gRPC methods
|
||||
|
||||
If the Go side calls a method the backend doesn't implement (e.g. `LoadModel`), add a no-op stub in `backend.py`:
|
||||
|
||||
```python
|
||||
def LoadModel(self, request, context):
|
||||
"""No-op — actual loading happens elsewhere."""
|
||||
return backend_pb2.Result(success=True, message="OK")
|
||||
```
|
||||
|
||||
The gRPC contract requires `LoadModel` to succeed for the model loader to return a usable client, even if the backend doesn't need upfront model loading.
|
||||
|
||||
### Dependency version conflicts
|
||||
|
||||
Python backends often break when a transitive dependency releases a breaking change (e.g. `pyarrow` removing `PyExtensionType`). Steps:
|
||||
|
||||
1. Identify the broken import in the logs
|
||||
2. Test in the installed venv: `backends/<name>/venv/bin/python -c "import <module>"`
|
||||
3. Check upstream requirements for version constraints
|
||||
4. Update **all** requirements files in `backend/python/<name>/`:
|
||||
- `requirements.txt` — base deps (grpcio, protobuf)
|
||||
- `requirements-cpu.txt` — CPU-specific (includes PyTorch CPU index)
|
||||
- `requirements-cublas12.txt` — CUDA 12
|
||||
- `requirements-cublas13.txt` — CUDA 13
|
||||
5. Rebuild: `make backends/<name>`
|
||||
|
||||
### PyTorch index conflicts (uv resolver)
|
||||
|
||||
The Docker build uses `uv` for pip installs. When `--extra-index-url` points to the PyTorch wheel index, `uv` may refuse to fetch packages like `requests` from PyPI if it finds a different version on the PyTorch index first. Fix this by adding `--index-strategy=unsafe-first-match` to `install.sh`:
|
||||
|
||||
```bash
|
||||
EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
|
||||
installRequirements
|
||||
```
|
||||
|
||||
Most Python backends already do this — check `backend/python/transformers/install.sh` or similar for reference.
|
||||
|
||||
## Rebuilding
|
||||
|
||||
### Rebuild a single backend
|
||||
|
||||
```bash
|
||||
make backends/<name>
|
||||
```
|
||||
|
||||
This runs the Docker build (`Dockerfile.python`), exports the image to `backend-images/<name>.tar`, and installs it into `backends/<name>/`. It also rebuilds the `local-ai` Go binary (without extra tags).
|
||||
|
||||
**Important**: If you were previously running with `GO_TAGS=auth`, the `make backends/<name>` step will overwrite your binary without that tag. Rebuild the Go binary afterward:
|
||||
|
||||
```bash
|
||||
GO_TAGS=auth make build
|
||||
```
|
||||
|
||||
### Rebuild and restart
|
||||
|
||||
After rebuilding a backend, you must restart LocalAI for it to pick up the new backend files. The backend gRPC process is spawned on demand when the model is first loaded.
|
||||
|
||||
```bash
|
||||
# Kill existing process
|
||||
kill <pid>
|
||||
|
||||
# Restart
|
||||
./local-ai run --debug [your flags]
|
||||
```
|
||||
|
||||
### Quick iteration (skip Docker rebuild)
|
||||
|
||||
For fast iteration on a Python backend's `backend.py` without a full Docker rebuild, you can edit the installed copy directly:
|
||||
|
||||
```bash
|
||||
# Edit the installed copy
|
||||
vim backends/<name>/backend.py
|
||||
|
||||
# Restart LocalAI to respawn the gRPC process
|
||||
```
|
||||
|
||||
This is useful for testing but **does not persist** — the next `make backends/<name>` will overwrite it. Always commit fixes to the source in `backend/python/<name>/`.
|
||||
|
||||
## Verification
|
||||
|
||||
After fixing and rebuilding:
|
||||
|
||||
1. Start LocalAI and confirm the backend registers: look for `Registering backend name="<name>"` in the logs
|
||||
2. Trigger the operation that failed (e.g. start a fine-tuning job)
|
||||
3. Watch the GRPC stderr/stdout lines for the backend's model ID
|
||||
4. Confirm no errors in the traceback
|
||||
@@ -1,145 +0,0 @@
|
||||
# Working on the ds4 Backend
|
||||
|
||||
`antirez/ds4` is a single-model inference engine for DeepSeek V4 Flash.
|
||||
LocalAI wraps the engine's C API (`ds4/ds4.h`) with a fresh C++ gRPC server at
|
||||
`backend/cpp/ds4/` - NOT a fork of llama-cpp's grpc-server.cpp.
|
||||
|
||||
## Pin
|
||||
|
||||
`backend/cpp/ds4/Makefile` pins `DS4_VERSION?=<sha>` at the top. The `ds4`
|
||||
target in the Makefile clones `antirez/ds4` at that commit (mirroring the
|
||||
llama-cpp / ik-llama-cpp / turboquant pattern). The bump-deps bot
|
||||
(`.github/workflows/bump_deps.yaml`) finds this pin via grep and opens a
|
||||
daily PR to update it. To bump manually: edit the `DS4_VERSION?=` line,
|
||||
then `make purge && make` (or rely on CI's clean build).
|
||||
|
||||
## Wire shape
|
||||
|
||||
| RPC | Implementation |
|
||||
|---|---|
|
||||
| Health, Free, Status | Trivial; no engine dependency for Health |
|
||||
| LoadModel | `ds4_engine_open` + `ds4_session_create`; backend is compile-time (DS4_NO_GPU → CPU, __APPLE__ → Metal, otherwise CUDA) |
|
||||
| TokenizeString | `ds4_tokenize_text` |
|
||||
| Predict | `ds4_engine_generate_argmax` + `DsmlParser` → one ChatDelta with content / reasoning_content / tool_calls[] |
|
||||
| PredictStream | Same, per-token ChatDelta writes |
|
||||
|
||||
## DSML
|
||||
|
||||
ds4 emits tool calls as literal text markers (`<｜DSML｜tool_calls>` etc.) -
|
||||
NOT special tokens. `dsml_parser.{h,cpp}` is our streaming state machine that
|
||||
classifies token bytes into CONTENT / REASONING / TOOL_START / TOOL_ARGS / TOOL_END
|
||||
events. `dsml_renderer.{h,cpp}` does the prompt direction: turns
|
||||
OpenAI tool_calls + role=tool messages back into DSML for the next turn.
|
||||
|
||||
## Thinking modes
|
||||
|
||||
`PredictOptions.Metadata["enable_thinking"]` gates thinking on/off (default ON).
|
||||
`["reasoning_effort"] == "max" | "xhigh"` selects `DS4_THINK_MAX`; anything else
|
||||
maps to `DS4_THINK_HIGH`. We pass the chosen mode to `ds4_chat_append_assistant_prefix`.
|
||||
|
||||
## Disk KV cache
|
||||
|
||||
`kv_cache.{h,cpp}` implements an SHA1-keyed file cache using ds4's public
|
||||
`ds4_session_save_payload` / `ds4_session_load_payload` API. Enable per request
|
||||
via `ModelOptions.Options[] = "kv_cache_dir:/some/path"`. Format is **our own** -
|
||||
NOT bit-compatible with ds4-server's KVC files (interop is a follow-up plan).
|
||||
|
||||
## Engine options (LoadModel)
|
||||
|
||||
`LoadModel` maps `ModelOptions.Options[]` (`"key:value"`, from model-YAML
|
||||
`options:`) onto `ds4_engine_options` through a **declarative table**
|
||||
(`kEngineOptSpecs` + `apply_engine_option` in `grpc-server.cpp`). The struct is
|
||||
plain C with no reflection, so the field set is enumerated once in the table;
|
||||
adding a future engine knob is a one-line table row, not a new branch. Unknown
|
||||
keys are ignored (back-compat). A bare flag (`ssd_streaming` with no value)
|
||||
means `true`. Path-type values (`mtp_path`, `expert_profile_path`,
|
||||
`directional_steering_file`) resolve **relative to the model directory**, so a
|
||||
gallery entry can reference a companion file it downloaded by bare filename;
|
||||
absolute values pass through. `ds4_role` / `ds4_layers` / `ds4_listen` /
|
||||
`ds4_route_timeout` / `kv_cache_dir` keep their dedicated handling (validation
|
||||
+ coordinator wiring) and are not in the table.
|
||||
|
||||
Wired keys: `mtp_path`, `mtp_draft`, `mtp_margin`, `prefill_chunk`,
|
||||
`power_percent`, `warm_weights`, `quality`, `ssd_streaming`,
|
||||
`ssd_streaming_cold`, `ssd_streaming_preload_experts`,
|
||||
`ssd_streaming_cache_experts` (count or `NGB`, sets both experts+bytes via
|
||||
`ds4_parse_streaming_cache_experts_arg`), `simulate_used_memory` (`NGB` via
|
||||
`ds4_parse_gib_arg`), `expert_profile_path`, `directional_steering_file`,
|
||||
`directional_steering_attn`, `directional_steering_ffn`.
|
||||
|
||||
## SSD streaming (running models larger than RAM)
|
||||
|
||||
ds4's **SSD streaming** keeps non-routed weights resident and streams routed MoE
|
||||
experts from the GGUF on cache misses, turning "does it fit in RAM" into a speed
|
||||
spectrum. **Metal (Darwin) only** - it is a no-op on CUDA/CPU. Enable with
|
||||
`options: ["ssd_streaming"]`; size the routed-expert cache with
|
||||
`ssd_streaming_cache_experts:NGB` (omit for ds4's automatic 80%-of-working-set
|
||||
budget). Gallery entries built on this: `deepseek-v4-flash-q4-ssd` (153 GB Flash
|
||||
on a 128 GB Mac) and `deepseek-v4-pro-q2-ssd` (433 GB Pro, experimental).
|
||||
|
||||
## Build matrix
|
||||
|
||||
| Build | Where | Notes |
|
||||
|---|---|---|
|
||||
| `cpu-ds4` (amd64 + arm64) | Linux GHA | ds4 considers CPU debug-only; useful only for wiring tests |
|
||||
| `cuda13-ds4` (amd64 + arm64) | Linux GHA + DGX Spark validation | Primary production path on Linux |
|
||||
| `ds4-darwin` (arm64) | macOS GHA runners | Metal; uses `scripts/build/ds4-darwin.sh` like llama-cpp-darwin |
|
||||
|
||||
cuda12 is intentionally omitted. ROCm / Vulkan / SYCL are not applicable.
|
||||
|
||||
## Hardware-gated validation
|
||||
|
||||
`tests/e2e-backends/backend_test.go` in `BACKEND_BINARY` mode:
|
||||
|
||||
```
|
||||
BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \
|
||||
BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \
|
||||
BACKEND_TEST_CAPS=health,load,predict,stream,tools \
|
||||
BACKEND_TEST_TOOL_PROMPT="What's the weather in Paris?" \
|
||||
go test -count=1 -timeout=30m -v ./tests/e2e-backends/...
|
||||
```
|
||||
|
||||
CI does not load the model; the suite is opt-in via env vars.
|
||||
|
||||
## Distributed mode
|
||||
|
||||
ds4 supports **layer-split** distributed inference (a model too big for one host,
|
||||
split by transformer layer; the GGUF must be present on every machine, each loads
|
||||
only its slice). Topology is **inverted** vs llama.cpp: the coordinator listens,
|
||||
workers dial in.
|
||||
|
||||
- **`ds4-worker` binary**: built and packaged next to `grpc-server` (`package.sh`
|
||||
copies it into `package/`). Links the same engine objects plus `ds4_distributed.o`;
|
||||
**no gRPC/protobuf dependency** (speaks ds4's own TCP transport), so it builds
|
||||
even where `grpc-server` can't. Runs the worker serving loop (`ds4_dist_run`).
|
||||
- **Coordinator wiring**: the ds4 `grpc-server` acts as coordinator when `LoadModel`
|
||||
`ModelOptions.Options` (from model-YAML `options:`) carry:
|
||||
- `ds4_role:coordinator` (enables distributed mode; absent → single-node, back-compat)
|
||||
- `ds4_layers:0:19` (coordinator's own slice, inclusive; `N:output` includes the head)
|
||||
- `ds4_listen:0.0.0.0:1234` (address workers dial into)
|
||||
- `ds4_route_timeout:60` (optional; seconds Predict/PredictStream wait for the route
|
||||
to form before returning gRPC `UNAVAILABLE`; default 60)
|
||||
- **Worker CLI**: `local-ai worker ds4-distributed -- <ds4-worker args>` resolves the
|
||||
ds4 backend and execs the packaged `ds4-worker` (raw passthrough), e.g.
|
||||
`--role worker --model /models/ds4flash.gguf --layers 20:output --coordinator <host> 1234`.
|
||||
|
||||
Opt-in e2e in `tests/e2e-backends/backend_test.go`, gated by
|
||||
`BACKEND_TEST_DS4_DISTRIBUTED=1` (plus `BACKEND_TEST_DS4_WORKER_BINARY`,
|
||||
`BACKEND_TEST_DS4_WORKER_LAYERS`, `BACKEND_TEST_DS4_COORDINATOR_LAYERS`,
|
||||
`BACKEND_TEST_DS4_LISTEN`). Design spec:
|
||||
`docs/superpowers/specs/2026-05-30-ds4-distributed-inference-design.md`.
|
||||
|
||||
## Importer
|
||||
|
||||
`core/gallery/importers/ds4.go` (`DS4Importer`) auto-detects ds4 weights by
|
||||
matching the `antirez/deepseek-v4-gguf` repo URI or the
|
||||
`DeepSeek-V4-Flash-*.gguf` filename pattern. **Registered BEFORE
|
||||
`LlamaCPPImporter`** in `defaultImporters` - both match `.gguf` but ds4 is more
|
||||
specific, and first-match-wins. The importer emits `backend: ds4`, uses
|
||||
`ds4flash.gguf` as the local filename (matches ds4's own CLI default), and
|
||||
disables the Go-side automatic tool-parsing fallback (the C++ backend emits
|
||||
ChatDelta.tool_calls natively via `DsmlParser`).
|
||||
|
||||
ds4 is also listed in `core/http/endpoints/localai/backend.go`'s pref-only
|
||||
slice so the `/import-model` UI surfaces it as a manual choice for users who
|
||||
want to force the backend on a non-canonical URI.
|
||||
@@ -1,83 +0,0 @@
|
||||
# llama.cpp Backend
|
||||
|
||||
The llama.cpp backend (`backend/cpp/llama-cpp/grpc-server.cpp`) is a gRPC adaptation of the upstream HTTP server (`llama.cpp/tools/server/server.cpp`). It uses the same underlying server infrastructure from `llama.cpp/tools/server/server-context.cpp`.
|
||||
|
||||
## Building and Testing
|
||||
|
||||
- Test llama.cpp backend compilation: `make backends/llama-cpp`
|
||||
- The backend is built as part of the main build process
|
||||
- Check `backend/cpp/llama-cpp/Makefile` for build configuration
|
||||
|
||||
## Architecture
|
||||
|
||||
- **grpc-server.cpp**: gRPC server implementation, adapts HTTP server patterns to gRPC
|
||||
- Uses shared server infrastructure: `server-context.cpp`, `server-task.cpp`, `server-queue.cpp`, `server-common.cpp`
|
||||
- The gRPC server mirrors the HTTP server's functionality but uses gRPC instead of HTTP
|
||||
|
||||
## Common Issues When Updating llama.cpp
|
||||
|
||||
When fixing compilation errors after upstream changes:
|
||||
1. Check how `server.cpp` (HTTP server) handles the same change
|
||||
2. Look for new public APIs or getter methods
|
||||
3. Store copies of needed data instead of accessing private members
|
||||
4. Update function calls to match new signatures
|
||||
5. Test with `make backends/llama-cpp`
|
||||
|
||||
## Key Differences from HTTP Server
|
||||
|
||||
- gRPC uses `BackendServiceImpl` class with gRPC service methods
|
||||
- HTTP server uses `server_routes` with HTTP handlers
|
||||
- Both use the same `server_context` and task queue infrastructure
|
||||
- gRPC methods: `LoadModel`, `Predict`, `PredictStream`, `Embedding`, `Rerank`, `TokenizeString`, `GetMetrics`, `Health`
|
||||
|
||||
## Tool Call Parsing Maintenance
|
||||
|
||||
When working on JSON/XML tool call parsing functionality, always check llama.cpp for reference implementation and updates:
|
||||
|
||||
### Checking for XML Parsing Changes
|
||||
|
||||
1. **Review XML Format Definitions**: Check `llama.cpp/common/chat-parser-xml-toolcall.h` for `xml_tool_call_format` struct changes
|
||||
2. **Review Parsing Logic**: Check `llama.cpp/common/chat-parser-xml-toolcall.cpp` for parsing algorithm updates
|
||||
3. **Review Format Presets**: Check `llama.cpp/common/chat-parser.cpp` for new XML format presets (search for `xml_tool_call_format form`)
|
||||
4. **Review Model Lists**: Check `llama.cpp/common/chat.h` for `COMMON_CHAT_FORMAT_*` enum values that use XML parsing:
|
||||
- `COMMON_CHAT_FORMAT_GLM_4_5`
|
||||
- `COMMON_CHAT_FORMAT_MINIMAX_M2`
|
||||
- `COMMON_CHAT_FORMAT_KIMI_K2`
|
||||
- `COMMON_CHAT_FORMAT_QWEN3_CODER_XML`
|
||||
- `COMMON_CHAT_FORMAT_APRIEL_1_5`
|
||||
- `COMMON_CHAT_FORMAT_XIAOMI_MIMO`
|
||||
- Any new formats added
|
||||
|
||||
### Model Configuration Options
|
||||
|
||||
Always check `llama.cpp` for new model configuration options that should be supported in LocalAI:
|
||||
|
||||
1. **Check Server Context**: Review `llama.cpp/tools/server/server-context.cpp` for new parameters
|
||||
2. **Check Chat Params**: Review `llama.cpp/common/chat.h` for `common_chat_params` struct changes
|
||||
3. **Check Server Options**: Review `llama.cpp/tools/server/server.cpp` for command-line argument changes
|
||||
4. **Examples of options to check**:
|
||||
- `ctx_shift` - Context shifting support
|
||||
- `parallel_tool_calls` - Parallel tool calling
|
||||
- `reasoning_format` - Reasoning format options
|
||||
- Any new flags or parameters
|
||||
|
||||
### Speculative Decoding Types
|
||||
|
||||
The `spec_type` option in `grpc-server.cpp` delegates to upstream's `common_speculative_types_from_names()`, so new speculative types added to the `common_speculative_type_from_name` map in `common/speculative.cpp` are picked up automatically with no code changes - only docs need an entry in `docs/content/advanced/model-configuration.md`. Current values: `none`, `draft-simple`, `draft-eagle3`, `draft-mtp`, `ngram-simple`, `ngram-map-k`, `ngram-map-k4v`, `ngram-mod`, `ngram-cache`.
|
||||
|
||||
`draft-mtp` (Multi-Token Prediction, [ggml-org/llama.cpp#22673](https://github.com/ggml-org/llama.cpp/pull/22673)) does not need a separate draft GGUF: when `spec_type` includes `draft-mtp` and `draftmodel` is empty, the upstream server creates an MTP context off the target model itself. LocalAI's gRPC layer needs no changes for this — it works through the existing `params.speculative.types` plumbing and the derived `cparams.n_rs_seq = params.speculative.need_n_rs_seq()` in `common_context_params_to_llama`.
|
||||
|
||||
### Implementation Guidelines
|
||||
|
||||
1. **Feature Parity**: Always aim for feature parity with llama.cpp's implementation
|
||||
2. **Test Coverage**: Add tests for new features matching llama.cpp's behavior
|
||||
3. **Documentation**: Update relevant documentation when adding new formats or options
|
||||
4. **Backward Compatibility**: Ensure changes don't break existing functionality
|
||||
|
||||
### Files to Monitor
|
||||
|
||||
- `llama.cpp/common/chat-parser-xml-toolcall.h` - Format definitions
|
||||
- `llama.cpp/common/chat-parser-xml-toolcall.cpp` - Parsing logic
|
||||
- `llama.cpp/common/chat-parser.cpp` - Format presets and model-specific handlers
|
||||
- `llama.cpp/common/chat.h` - Format enums and parameter structures
|
||||
- `llama.cpp/tools/server/server-context.cpp` - Server configuration options
|
||||
@@ -1,97 +0,0 @@
|
||||
# LocalAI Assistant — admin MCP server
|
||||
|
||||
This document is the contract for **anyone** (human or AI agent) touching LocalAI's admin REST surface, the in-process MCP server that wraps it, or the embedded skill prompts that teach the assistant how to use it. Read this before adding/removing/renaming admin endpoints, MCP tools, or skill recipes.
|
||||
|
||||
## What this feature is
|
||||
|
||||
`pkg/mcp/localaitools/` is a public Go package that exposes LocalAI's admin/management surface as an MCP server. It is used in two ways:
|
||||
|
||||
1. **In-process**: when an admin opens a chat with `metadata.localai_assistant=true`, the chat handler injects the in-memory MCP server (paired `net.Pipe()` transport, no HTTP loopback) so the LLM can install models, manage backends and edit configs by chatting.
|
||||
2. **Standalone**: the `local-ai mcp-server --target=…` subcommand serves the same MCP server over stdio, talking HTTP to a remote LocalAI instance.
|
||||
|
||||
The two modes share **all** tool definitions and skill prompts. They differ only in their `LocalAIClient` implementation (`inproc/` calls services directly; `httpapi/` calls REST).
|
||||
|
||||
## The three things you must keep in sync
|
||||
|
||||
When you change LocalAI's admin surface, three layers must stay aligned:
|
||||
|
||||
1. **REST endpoint** in `core/http/endpoints/localai/*.go`.
|
||||
2. **MCP tool registration** in `pkg/mcp/localaitools/tools_*.go`, plus a method on `LocalAIClient` (in `client.go`) and implementations in both `inproc/client.go` **and** `httpapi/client.go`.
|
||||
3. **Skill prompt** under `pkg/mcp/localaitools/prompts/skills/*.md` — the markdown that teaches the LLM how to use the new tool. If the new tool fits an existing recipe, update that recipe; otherwise add a new file.
|
||||
|
||||
If you ship a REST endpoint without (2) and (3), conversational admins won't see the feature.
|
||||
|
||||
## Checklist for adding a new admin endpoint
|
||||
|
||||
- [ ] REST endpoint exists in `core/http/endpoints/localai/*.go` and is gated by `auth.RequireAdmin()` in `core/http/routes/localai.go`.
|
||||
- [ ] `LocalAIClient` interface in `pkg/mcp/localaitools/client.go` has a method covering the new operation.
|
||||
- [ ] DTOs added/updated in `pkg/mcp/localaitools/dto.go` (JSON-tagged; never expose raw service types).
|
||||
- [ ] `inproc/client.go` implements the new method by calling the service directly (not via HTTP loopback).
|
||||
- [ ] `httpapi/client.go` implements the new method by calling the REST endpoint.
|
||||
- [ ] Tool registration added in the appropriate `pkg/mcp/localaitools/tools_*.go`. Mutating tools must reference safety rule 1 in the description.
|
||||
- [ ] If the tool is mutating, ensure `Options{DisableMutating: true}` skips it (mirror the pattern in `tools_models.go`).
|
||||
- [ ] Skill prompt added or updated under `pkg/mcp/localaitools/prompts/skills/`. The prompt must instruct the LLM when to call the tool, what to ask the user first, and what to do on error.
|
||||
- [ ] Tests:
|
||||
- `pkg/mcp/localaitools/server_test.go` adds the tool name to `expectedFullCatalog` and `expectedReadOnlyCatalog` (if read-only).
|
||||
- Tool dispatch is added to `TestEachToolDispatchesToClient`.
|
||||
- `pkg/mcp/localaitools/httpapi/client_test.go` covers the new HTTP path.
|
||||
|
||||
## Adding a new skill recipe (no new tool)
|
||||
|
||||
Sometimes you want to teach the LLM a new pattern that uses existing tools. Drop a markdown file under `pkg/mcp/localaitools/prompts/skills/<verb>_<noun>.md`. The file is automatically embedded by `//go:embed` and assembled into the system prompt in lexicographic order. No Go changes needed.
|
||||
|
||||
Conventions:
|
||||
- Filename: `<verb>_<noun>.md` (e.g. `install_chat_model.md`, `upgrade_backend.md`).
|
||||
- First line: `# Skill: <Title Case description>`.
|
||||
- Number the steps. Reference exact tool names in backticks.
|
||||
- If the skill mutates state, remind the LLM to confirm with the user.
|
||||
|
||||
## Code conventions
|
||||
|
||||
These rules guard against the magic-literal drift that surfaced in the first audit. Do not re-introduce bare strings.
|
||||
|
||||
- **Tool names** always come from the `Tool*` constants in `pkg/mcp/localaitools/tools.go`. Tool registrations, the test catalog (`server_test.go`'s `expectedFullCatalog` / `expectedReadOnlyCatalog`), and dispatch tables reference the constants. The embedded skill prompts under `prompts/` keep bare strings — that's the one allowed exception, and `TestPromptsContainSafetyAnchors` enforces alignment.
|
||||
- **Toggle/pin actions** use the `modeladmin.Action` type (`pkg/mcp/localaitools` and `core/services/modeladmin`). Use `ActionEnable`/`ActionDisable`/`ActionPin`/`ActionUnpin`; never bare `"enable"`/`"pin"` strings.
|
||||
- **Capability tags** for `list_installed_models` use the `localaitools.Capability` type (`capability.go`). The `LocalAIClient.ListInstalledModels` interface takes a typed `Capability`, and the `inproc` switch only accepts canonical values (`"embed"`/`"embedding"` are not aliases — only `CapabilityEmbeddings`).
|
||||
- **HTTP error checks** in `httpapi.Client` use `errors.Is(err, ErrHTTPNotFound)`, not substring matches on `err.Error()`. The typed `*HTTPError` carries `StatusCode` and `Body`; add new sentinel errors as needed rather than re-introducing string matching.
|
||||
- **Channel sends** to `GalleryService.ModelGalleryChannel` / `BackendGalleryChannel` from inproc clients MUST select on `ctx.Done()` so a cancelled chat completion releases the goroutine. See `inproc.sendModelOp` / `sendBackendOp`.
|
||||
- **Disk writes** of model config YAML go through `modeladmin.writeFileAtomic` (temp file + `os.Rename`). `os.WriteFile` truncates on crash and corrupts the model.
|
||||
- **MCP server lifecycle**: every initialised holder MUST register `Close()` with `signals.RegisterGracefulTerminationHandler`. The standalone `mcp-server` CLI uses `signal.NotifyContext` to honour SIGINT/SIGTERM.
|
||||
|
||||
## File map (where to look)
|
||||
|
||||
```
|
||||
pkg/mcp/localaitools/
|
||||
client.go # LocalAIClient interface + DTO registry
|
||||
dto.go # JSON-tagged DTOs shared by both client impls
|
||||
server.go # NewServer(client, opts) — registers tools
|
||||
tools.go # Tool* name constants (single source of truth)
|
||||
capability.go # Capability type + constants
|
||||
tools_models.go # gallery_search, install_model, import_model_uri, ...
|
||||
tools_backends.go
|
||||
tools_config.go
|
||||
tools_system.go
|
||||
tools_state.go
|
||||
prompts.go # //go:embed loader + SystemPrompt(opts)
|
||||
prompts/00_role.md
|
||||
prompts/10_safety.md # SAFETY RULES — change with care
|
||||
prompts/20_tools.md # curated tool catalog with one-liners
|
||||
prompts/skills/*.md
|
||||
inproc/client.go # in-process LocalAIClient (services-direct)
|
||||
httpapi/client.go # REST LocalAIClient (for standalone CLI / remote)
|
||||
core/http/endpoints/mcp/
|
||||
localai_assistant.go # process-wide holder + LocalToolExecutor
|
||||
core/cli/mcp_server.go # local-ai mcp-server subcommand
|
||||
```
|
||||
|
||||
## Why two clients
|
||||
|
||||
The in-process MCP server runs inside the same LocalAI binary that serves chat. Going over HTTP loopback would (a) require minting a synthetic admin API key for the server to authenticate against itself, (b) double-marshal every tool dispatch, and (c) lose access to in-process channels (e.g. `GalleryService.ModelGalleryChannel` for streaming install progress). So in-process uses `inproc.Client`. The standalone stdio CLI talks to a *remote* LocalAI; HTTP is the only option, so it uses `httpapi.Client`. Both implement the same `LocalAIClient` interface, and the parity test in `pkg/mcp/localaitools/parity_test.go` (when present) keeps their output equivalent.
|
||||
|
||||
## Why prompt-enforced confirmation, not code gates
|
||||
|
||||
The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.md` rule 1) that requires the LLM to summarise the action and wait for explicit user confirmation before calling it. There is no `plan_*`/`apply_*` two-step in code. If you add a mutating tool, do **not** add per-tool confirmation logic in Go — instead, list the new tool name in `prompts/10_safety.md` so the LLM knows it falls under the confirmation rule.
|
||||
|
||||
## Distributed mode
|
||||
|
||||
The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period.
|
||||
@@ -1,62 +0,0 @@
|
||||
# Working on the SGLang Backend
|
||||
|
||||
The SGLang backend lives at `backend/python/sglang/backend.py` (async gRPC). It wraps SGLang's `Engine` (`sglang.srt.entrypoints.engine.Engine`) and translates LocalAI's gRPC `PredictOptions` into SGLang sampling params + outputs into `Reply.chat_deltas`. Structurally it mirrors `backend/python/vllm/backend.py` — keep them shaped the same so changes in one have an obvious analog in the other.
|
||||
|
||||
## `engine_args` is the universal escape hatch
|
||||
|
||||
A small fixed set of fields on `ModelOptions` is mapped to typed SGLang kwargs in `LoadModel` (model, quantization, load_format, gpu_memory_utilization → mem_fraction_static, trust_remote_code, enforce_eager → disable_cuda_graph, tensor_parallel_size → tp_size, max_model_len → context_length, dtype). **Everything else** flows through the `engine_args:` YAML map.
|
||||
|
||||
Validation happens in `_apply_engine_args`. Keys are checked against `dataclasses.fields(ServerArgs)` (`sglang.srt.server_args.ServerArgs` is a flat `@dataclass` with ~380 fields). Unknown keys raise `ValueError` at LoadModel time with a `difflib.get_close_matches` suggestion — same shape as the vLLM backend.
|
||||
|
||||
**Precedence:** typed `ModelOptions` fields populate `engine_kwargs` first, then `engine_args` overrides them. So a YAML that sets both `gpu_memory_utilization: 0.9` and `engine_args.mem_fraction_static: 0.5` ends up at `0.5`. Document this when answering "why didn't my YAML field stick?".
|
||||
|
||||
**ServerArgs is flat.** Unlike vLLM, where speculative decoding is nested under `engine_args.speculative_config: {...}`, SGLang exposes flat top-level fields: `speculative_algorithm`, `speculative_draft_model_path`, `speculative_num_steps`, `speculative_eagle_topk`, `speculative_num_draft_tokens`, `speculative_dflash_block_size`, etc. There is no `speculative_config:` dict. Same goes for compilation, kv-transfer, attention — all flat.
|
||||
|
||||
The canonical reference is `python/sglang/srt/server_args.py:ServerArgs` (line ~304). When SGLang adds new flags, no LocalAI code change is needed — they're automatically available via `engine_args:`. The validator picks them up because it introspects the live dataclass.
|
||||
|
||||
## Speculative decoding cheatsheet
|
||||
|
||||
`--speculative-algorithm` accepts `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`, `DFLASH`. `NEXTN` is silently rewritten to `EAGLE` in `ServerArgs.__post_init__` (`server_args.py:3286-3287`). MTP (Multi-Token Prediction) is the same EAGLE path with `num_steps=1, eagle_topk=1, num_draft_tokens=2` against a target whose architecture has multi-token heads (e.g. MiMo-7B-RL, DeepSeek-V3-MTP).
|
||||
|
||||
| Algorithm | Drafter requirement | Gallery demo target | Gallery demo drafter |
|
||||
|-----------|--------------------|---------------------|----------------------|
|
||||
| `NEXTN` / `EAGLE` (MTP) | Assistant drafter or built-in heads | google/gemma-4-E2B-it, google/gemma-4-E4B-it | google/gemma-4-E2B-it-assistant, google/gemma-4-E4B-it-assistant |
|
||||
| `EAGLE3` | EAGLE3 draft head | (no gallery entry yet) | e.g. jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B |
|
||||
| `DFLASH` | Block-diffusion drafter | (no gallery entry yet) | e.g. z-lab/Qwen3-4B-DFlash-b16 |
|
||||
| `STANDALONE` | Smaller LLM as drafter | (no gallery entry yet) | any smaller chat-tuned LLM in the same family |
|
||||
| `NGRAM` | None — uses prefix history | (no gallery entry yet) | n/a |
|
||||
|
||||
The Gemma 4 demos use `mem_fraction_static: 0.85` (cookbook default) and the cookbook's `num_steps=5, num_draft_tokens=6, eagle_topk=1` parameters. Other algorithms are reachable from any user YAML via `engine_args:` but don't have shipped demos yet — that's a deliberate gallery scope choice, not a backend limitation.
|
||||
|
||||
Gemma 4 support requires sglang built from a commit that includes [PR #21952](https://github.com/sgl-project/sglang/pull/21952). LocalAI's pinned release for cublas12 / cublas13 includes it. The `l4t13` (JetPack 7 / sbsa cu130) build floors at `sglang>=0.5.0` because the `pypi.jetson-ai-lab.io` mirror still ships only `0.5.1.post2` as of 2026-05-06 — Gemma 4 / MTP recipes are therefore not available on l4t13 until that mirror catches up. `backend.py` keeps backward compat with the 0.5.x → 0.5.11 `SamplingParams.seed` → `sampling_seed` rename via runtime detection.
|
||||
|
||||
Compatibility caveats per the SGLang docs: DFLASH and NGRAM are incompatible with `enable_dp_attention`; DFLASH requires `pp_size == 1`; STANDALONE is incompatible with `enable_dp_attention`; NGRAM is CUDA-only and disables the overlap scheduler.
|
||||
|
||||
### `mem_fraction_static` + quantization + MTP on consumer GPUs
|
||||
|
||||
When combining online weight quantization (`engine_args.quantization: fp8` / `awq` / etc.) with built-in-head MTP (`speculative_algorithm: EAGLE`/`NEXTN`) on a tight VRAM budget, sglang's default `mem_fraction_static: 0.85` will OOM during draft-worker init. The reason: sglang quantizes the **target** model's transformer blocks but loads the **MTP draft worker's vocab embedding** at the source dtype (typically bf16). For a 7 B-class model with a 150k-token vocab × 4096 hidden, that's another ~1.2 GiB allocated *after* the static pool is reserved. At 0.85 fraction on a 16 GB card there's no room left.
|
||||
|
||||
Workaround: drop `mem_fraction_static` to ~0.7 so the post-static heap can absorb the MTP embedding alloc + CUDA graph private pools. Verified end-to-end on MiMo-7B-RL + fp8 + MTP on a 16 GB RTX 5070 Ti (`gallery/sglang-mimo-7b-mtp.yaml`) at ~88 tok/s. Models with larger vocabs or more MTP layers (e.g. DeepSeek-V3-MTP) need an even smaller fraction.
|
||||
|
||||
This isn't documented anywhere upstream as of 2026-05-06 — the SGLang Gemma 4 cookbook uses 0.85 because their MTP path doesn't go through `eagle_worker_v2.py` for an embedding-bearing draft module. Don't blanket-apply 0.7 across all sglang YAMLs; only when MTP-with-built-in-heads + quantization combine.
|
||||
|
||||
## Tool-call and reasoning parsers stay on `Options[]`
|
||||
|
||||
ServerArgs has `tool_call_parser` and `reasoning_parser` fields, and the backend does pass them through to `Engine` so SGLang's own HTTP/OAI surface keeps working. But for the **LocalAI** request path the backend constructs fresh per-request parser instances in `_make_parsers` (`backend.py:286`) because the parsers are stateful — the streaming and non-streaming paths each need their own.
|
||||
|
||||
So the user-facing knob stays on `Options[]`:
|
||||
|
||||
```yaml
|
||||
options:
|
||||
- tool_parser:hermes
|
||||
- reasoning_parser:deepseek_r1
|
||||
```
|
||||
|
||||
Putting these in `engine_args:` will set them on `ServerArgs` but the LocalAI-level streaming `ChatDelta` will not pick them up. Don't recommend that path.
|
||||
|
||||
## What's missing today (out of scope, but worth tracking)
|
||||
|
||||
- `core/config/hooks_sglang.go` — there is no SGLang equivalent of `hooks_vllm.go`. The vLLM hook auto-selects parsers for known model families from `parser_defaults.json` and seeds production engine_args defaults. A symmetric hook for SGLang could reuse the same `parser_defaults.json` (the SGLang parser names are different but the family detection is shared) and seed defaults like `enable_metrics: true` or attention-backend choices.
|
||||
- `core/gallery/importers/sglang.go` — vLLM has an importer that resolves model architecture → parser defaults at gallery-import time. A matching importer for SGLang would let `local-ai install` populate sensible parsers automatically.
|
||||
|
||||
These should be a follow-up PR, not a blocker for the engine_args feature.
|
||||
@@ -1,120 +0,0 @@
|
||||
# Testing MCP Apps (Interactive Tool UIs)
|
||||
|
||||
MCP Apps is an extension to MCP where tools declare interactive HTML UIs via `_meta.ui.resourceUri`. When the LLM calls such a tool, the UI renders the app in a sandboxed iframe inline in the chat. The app communicates bidirectionally with the host via `postMessage` (JSON-RPC) and can call server tools, send messages, and update model context.
|
||||
|
||||
Spec: https://modelcontextprotocol.io/extensions/apps/overview
|
||||
|
||||
## Quick Start: Run a Test MCP App Server
|
||||
|
||||
The `@modelcontextprotocol/server-basic-react` npm package is a ready-to-use test server that exposes a `get-time` tool with an interactive React clock UI. It requires Node >= 20, so run it in Docker:
|
||||
|
||||
```bash
|
||||
docker run -d --name mcp-app-test -p 3001:3001 node:22-slim \
|
||||
sh -c 'npx -y @modelcontextprotocol/server-basic-react'
|
||||
```
|
||||
|
||||
Wait ~10 seconds for it to start, then verify:
|
||||
|
||||
```bash
|
||||
# Check it's running
|
||||
docker logs mcp-app-test
|
||||
# Expected: "MCP server listening on http://localhost:3001/mcp"
|
||||
|
||||
# Verify MCP protocol works
|
||||
curl -s -X POST http://localhost:3001/mcp \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Accept: application/json, text/event-stream' \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"test","version":"1.0.0"}}}'
|
||||
|
||||
# List tools — should show get-time with _meta.ui.resourceUri
|
||||
curl -s -X POST http://localhost:3001/mcp \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'Accept: application/json, text/event-stream' \
|
||||
-d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'
|
||||
```
|
||||
|
||||
The `tools/list` response should contain:
|
||||
```json
|
||||
{
|
||||
"name": "get-time",
|
||||
"_meta": {
|
||||
"ui": { "resourceUri": "ui://get-time/mcp-app.html" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Testing in LocalAI's UI
|
||||
|
||||
1. Make sure LocalAI is running (e.g. `http://localhost:8080`)
|
||||
2. Build the React UI: `cd core/http/react-ui && npm install && npm run build`
|
||||
3. Open the Chat page in your browser
|
||||
4. Click **"Client MCP"** in the chat header
|
||||
5. Add a new client MCP server:
|
||||
- **URL**: `http://localhost:3001/mcp`
|
||||
- **Use CORS proxy**: enabled (default) — required because the browser can't hit `localhost:3001` directly due to CORS; LocalAI's proxy at `/api/cors-proxy` handles it
|
||||
6. The server should connect and discover the `get-time` tool
|
||||
7. Select a model and send: **"What time is it?"**
|
||||
8. The LLM should call the `get-time` tool
|
||||
9. The tool result should render the interactive React clock app in an iframe as a standalone chat message (not inside the collapsed activity group)
|
||||
|
||||
## What to Verify
|
||||
|
||||
- [ ] Tool appears in the connected tools list (not filtered — `get-time` is callable by the LLM)
|
||||
- [ ] The iframe renders as a standalone chat message with a puzzle-piece icon
|
||||
- [ ] The app loads and is interactive (clock UI, buttons work)
|
||||
- [ ] No "Reconnect to MCP server" overlay (connection is live)
|
||||
- [ ] Console logs show bidirectional communication:
|
||||
- `tools/call` messages from app to host (app calling server tools)
|
||||
- `ui/message` notifications (app sending messages)
|
||||
- [ ] After the app renders, the LLM continues and produces a text response with the time
|
||||
- [ ] Non-UI tools continue to work normally (text-only results)
|
||||
- [ ] Page reload shows the HTML statically with a reconnect overlay until you reconnect
|
||||
|
||||
## Console Log Patterns
|
||||
|
||||
Healthy bidirectional communication looks like:
|
||||
|
||||
```
|
||||
Parsed message { jsonrpc: "2.0", id: N, result: {...} } // Bridge init
|
||||
get-time result: { content: [...] } // Tool result received
|
||||
Calling get-time tool... // App calls tool
|
||||
Sending message { method: "tools/call", ... } // App -> host -> server
|
||||
Parsed message { jsonrpc: "2.0", id: N, result: {...} } // Server response
|
||||
Sending message text to Host: ... // App sends message
|
||||
Sending message { method: "ui/message", ... } // Message notification
|
||||
Message accepted // Host acknowledged
|
||||
```
|
||||
|
||||
Benign warnings to ignore:
|
||||
- `Source map error: ... about:srcdoc` — browser devtools can't find source maps for srcdoc iframes
|
||||
- `Ignoring message from unknown source` — duplicate postMessage from iframe navigation
|
||||
- `notifications/cancelled` — app cleaning up previous requests
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **No server-side changes needed** — the MCP App protocol runs entirely in the browser
|
||||
- `PostMessageTransport` wraps `window.postMessage` between host and `srcdoc` iframe
|
||||
- `AppBridge` (from `@modelcontextprotocol/ext-apps`) auto-forwards `tools/call`, `resources/read`, `resources/list` from the app to the MCP server via the host's `Client`
|
||||
- The iframe uses `sandbox="allow-scripts allow-forms"` (no `allow-same-origin`) — opaque origin, no access to host cookies/DOM/localStorage
|
||||
- App-only tools (`_meta.ui.visibility: "app-only"`) are filtered from the LLM's tool list but remain callable by the app iframe
|
||||
|
||||
## Key Files
|
||||
|
||||
- `core/http/react-ui/src/components/MCPAppFrame.jsx` — iframe + AppBridge component
|
||||
- `core/http/react-ui/src/hooks/useMCPClient.js` — MCP client hook with app UI helpers (`hasAppUI`, `getAppResource`, `getClientForTool`, `getToolDefinition`)
|
||||
- `core/http/react-ui/src/hooks/useChat.js` — agentic loop, attaches `appUI` to tool_result messages
|
||||
- `core/http/react-ui/src/pages/Chat.jsx` — renders MCPAppFrame as standalone chat messages
|
||||
|
||||
## Other Test Servers
|
||||
|
||||
The `@modelcontextprotocol/ext-apps` repo has many example servers:
|
||||
- `@modelcontextprotocol/server-basic-react` — simple clock (React)
|
||||
- More examples at https://github.com/modelcontextprotocol/ext-apps/tree/main/examples
|
||||
|
||||
All examples support both stdio and HTTP transport. Run without `--stdio` for HTTP mode on port 3001.
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
docker rm -f mcp-app-test
|
||||
```
|
||||
@@ -1,115 +0,0 @@
|
||||
# Working on the vLLM Backend
|
||||
|
||||
The vLLM backend lives at `backend/python/vllm/backend.py` (async gRPC) and the multimodal variant at `backend/python/vllm-omni/backend.py` (sync gRPC). Both wrap vLLM's `AsyncLLMEngine` / `Omni` and translate the LocalAI gRPC `PredictOptions` into vLLM `SamplingParams` + outputs into `Reply.chat_deltas`.
|
||||
|
||||
This file captures the non-obvious bits — most of the bring-up was a single PR (`feat/vllm-parity`) and the things below are easy to get wrong.
|
||||
|
||||
## Tool calling and reasoning use vLLM's *native* parsers
|
||||
|
||||
Do not write regex-based tool-call extractors for vLLM. vLLM ships:
|
||||
|
||||
- `vllm.tool_parsers.ToolParserManager` — 50+ registered parsers (`hermes`, `llama3_json`, `llama4_pythonic`, `mistral`, `qwen3_xml`, `deepseek_v3`, `granite4`, `openai`, `kimi_k2`, `glm45`, …)
|
||||
- `vllm.reasoning.ReasoningParserManager` — 25+ registered parsers (`deepseek_r1`, `qwen3`, `mistral`, `gemma4`, …)
|
||||
|
||||
Both can be used standalone: instantiate with a tokenizer, call `extract_tool_calls(text, request=None)` / `extract_reasoning(text, request=None)`. The backend stores the parser *classes* on `self.tool_parser_cls` / `self.reasoning_parser_cls` at LoadModel time and instantiates them per request.
|
||||
|
||||
**Selection:** vLLM does *not* auto-detect parsers from model name — neither does the LocalAI backend. The user (or `core/config/hooks_vllm.go`) must pick one and pass it via `Options[]`:
|
||||
|
||||
```yaml
|
||||
options:
|
||||
- tool_parser:hermes
|
||||
- reasoning_parser:qwen3
|
||||
```
|
||||
|
||||
Auto-defaults for known model families live in `core/config/parser_defaults.json` and are applied:
|
||||
- at gallery import time by `core/gallery/importers/vllm.go`
|
||||
- at model load time by the `vllm` / `vllm-omni` backend hook in `core/config/hooks_vllm.go`
|
||||
|
||||
User-supplied `tool_parser:`/`reasoning_parser:` in the config wins over defaults — the hook checks for existing entries before appending.
|
||||
|
||||
**When to update `parser_defaults.json`:** any time vLLM ships a new tool or reasoning parser, or you onboard a new model family that LocalAI users will pull from HuggingFace. The file is keyed by *family pattern* matched against `normalizeModelID(cfg.Model)` (lowercase, org-prefix stripped, `_`→`-`). Patterns are checked **longest-first** — keep `qwen3.5` before `qwen3`, `llama-3.3` before `llama-3`, etc., or the wrong family wins. Add a covering test in `core/config/hooks_test.go`.
|
||||
|
||||
**Sister file — `core/config/inference_defaults.json`:** same pattern but for sampling parameters (temperature, top_p, top_k, min_p, repeat_penalty, presence_penalty). Loaded by `core/config/inference_defaults.go` and applied by `ApplyInferenceDefaults()`. The schema is `map[string]float64` only — *strings don't fit*, which is why parser defaults needed their own JSON file. The inference file is **auto-generated from unsloth** via `go generate ./core/config/` (see `core/config/gen_inference_defaults/`) — don't hand-edit it; instead update the upstream source or regenerate. Both files share `normalizeModelID()` and the longest-first pattern ordering.
|
||||
|
||||
**Constructor compatibility gotcha:** the abstract `ToolParser.__init__` accepts `tools=`, but several concrete parsers (Hermes2ProToolParser, etc.) override `__init__` and *only* accept `tokenizer`. Always:
|
||||
|
||||
```python
|
||||
try:
|
||||
tp = self.tool_parser_cls(self.tokenizer, tools=tools)
|
||||
except TypeError:
|
||||
tp = self.tool_parser_cls(self.tokenizer)
|
||||
```
|
||||
|
||||
## ChatDelta is the streaming contract
|
||||
|
||||
The Go side (`core/backend/llm.go`, `pkg/functions/chat_deltas.go`) consumes `Reply.chat_deltas` to assemble the OpenAI response. For tool calls to surface in `chat/completions`, the Python backend **must** populate `Reply.chat_deltas[].tool_calls` with `ToolCallDelta{index, id, name, arguments}`. Returning the raw `<tool_call>...</tool_call>` text in `Reply.message` is *not* enough — the Go regex fallback exists for llama.cpp, not for vllm.
|
||||
|
||||
Same story for `reasoning_content` — emit it on `ChatDelta.reasoning_content`, not as part of `content`.
|
||||
|
||||
## Message conversion to chat templates
|
||||
|
||||
`tokenizer.apply_chat_template()` expects a list of dicts, not proto Messages. The shared helper in `backend/python/common/vllm_utils.py` (`messages_to_dicts`) handles the mapping including:
|
||||
|
||||
- `tool_call_id` and `name` for `role="tool"` messages
|
||||
- `tool_calls` JSON-string field → parsed Python list for `role="assistant"`
|
||||
- `reasoning_content` for thinking models
|
||||
|
||||
Pass `tools=json.loads(request.Tools)` and (when `request.Metadata.get("enable_thinking") == "true"`) `enable_thinking=True` to `apply_chat_template`. Wrap in `try/except TypeError` because not every tokenizer template accepts those kwargs.
|
||||
|
||||
## CPU support and the SIMD/library minefield
|
||||
|
||||
vLLM publishes prebuilt CPU wheels at `https://github.com/vllm-project/vllm/releases/...`. The pin lives in `backend/python/vllm/requirements-cpu-after.txt`.
|
||||
|
||||
**Version compatibility — important:** newer vllm CPU wheels (≥ 0.15) declare `torch==2.10.0+cpu` as a hard dep, but `torch==2.10.0` only exists on the PyTorch test channel and pulls in an incompatible `torchvision`. Stay on **`vllm 0.14.1+cpu` + `torch 2.9.1+cpu`** until both upstream catch up. Bumping requires verifying torchvision/torchaudio match.
|
||||
|
||||
`requirements-cpu.txt` uses `--extra-index-url https://download.pytorch.org/whl/cpu`. `install.sh` adds `--index-strategy=unsafe-best-match` for the `cpu` profile so uv resolves transformers/vllm from PyPI while pulling torch from the PyTorch index.
|
||||
|
||||
**SIMD baseline:** the prebuilt CPU wheel is compiled with AVX-512 VNNI/BF16. On a CPU without those instructions, importing `vllm.model_executor.models.registry` SIGILLs at `_run_in_subprocess` time during model inspection. There is no runtime flag to disable it. Workarounds:
|
||||
|
||||
1. **Run on a host with the right SIMD baseline** (default — fast)
|
||||
2. **Build from source** with `FROM_SOURCE=true` env var. Plumbing exists end-to-end:
|
||||
- `install.sh` hides `requirements-cpu-after.txt`, runs `installRequirements` for the base deps, then clones vllm and `VLLM_TARGET_DEVICE=cpu uv pip install --no-deps .`
|
||||
- `backend/Dockerfile.python` declares `ARG FROM_SOURCE` + `ENV FROM_SOURCE`
|
||||
- `Makefile` `docker-build-backend` macro forwards `--build-arg FROM_SOURCE=$(FROM_SOURCE)` when set
|
||||
- Source build takes 30–50 minutes — too slow for per-PR CI but fine for local.
|
||||
|
||||
**Runtime shared libraries:** vLLM's `vllm._C` extension `dlopen`s `libnuma.so.1` at import time. If missing, the C extension silently fails and `torch.ops._C_utils.init_cpu_threads_env` is never registered → `EngineCore` crashes on `init_device` with:
|
||||
|
||||
```
|
||||
AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env'
|
||||
```
|
||||
|
||||
`backend/python/vllm/package.sh` bundles `libnuma.so.1` and `libgomp.so.1` into `${BACKEND}/lib/`, which `libbackend.sh` adds to `LD_LIBRARY_PATH` at run time. The builder stage in `backend/Dockerfile.python` installs `libnuma1`/`libgomp1` so package.sh has something to copy. Do *not* assume the production host has these — backend images are `FROM scratch`.
|
||||
|
||||
## Backend hook system (`core/config/backend_hooks.go`)
|
||||
|
||||
Per-backend defaults that used to be hardcoded in `ModelConfig.Prepare()` now live in `core/config/hooks_*.go` files and self-register via `init()`:
|
||||
|
||||
- `hooks_llamacpp.go` → GGUF metadata parsing, context size, GPU layers, jinja template
|
||||
- `hooks_vllm.go` → tool/reasoning parser auto-selection from `parser_defaults.json`
|
||||
|
||||
Hook keys:
|
||||
- `"llama-cpp"`, `"vllm"`, `"vllm-omni"`, … — backend-specific
|
||||
- `""` — runs only when `cfg.Backend` is empty (auto-detect case)
|
||||
- `"*"` — global catch-all, runs for every backend before specific hooks
|
||||
|
||||
Multiple hooks per key are supported and run in registration order. Adding a new backend default:
|
||||
|
||||
```go
|
||||
// core/config/hooks_<backend>.go
|
||||
func init() {
|
||||
RegisterBackendHook("<backend>", myDefaults)
|
||||
}
|
||||
func myDefaults(cfg *ModelConfig, modelPath string) {
|
||||
// only fill in fields the user didn't set
|
||||
}
|
||||
```
|
||||
|
||||
## The `Messages.ToProto()` fields you need to set
|
||||
|
||||
`core/schema/message.go:ToProto()` must serialize:
|
||||
- `ToolCallID` → `proto.Message.ToolCallId` (for `role="tool"` messages — links result back to the call)
|
||||
- `Reasoning` → `proto.Message.ReasoningContent`
|
||||
- `ToolCalls` → `proto.Message.ToolCalls` (JSON-encoded string)
|
||||
|
||||
These were originally not serialized and tool-calling conversations broke silently — the C++ llama.cpp backend reads them but always got empty strings. Any new field added to `schema.Message` *and* `proto.Message` needs a matching line in `ToProto()`.
|
||||
@@ -1,8 +0,0 @@
|
||||
# .air.toml
|
||||
[build]
|
||||
cmd = "make build"
|
||||
bin = "./local-ai"
|
||||
args_bin = [ "--debug" ]
|
||||
include_ext = ["go", "html", "yaml", "toml", "json", "txt", "md"]
|
||||
exclude_dir = ["pkg/grpc/proto"]
|
||||
delay = 1000
|
||||
@@ -2,6 +2,9 @@
|
||||
|
||||
cd /workspace
|
||||
|
||||
# Grab the pre-stashed backend assets to avoid build issues
|
||||
cp -r /build/backend-assets /workspace/backend-assets
|
||||
|
||||
# Ensures generated source files are present upon load
|
||||
make prepare
|
||||
|
||||
|
||||
@@ -4,14 +4,17 @@ services:
|
||||
context: ..
|
||||
dockerfile: Dockerfile
|
||||
target: devcontainer
|
||||
args:
|
||||
- FFMPEG=true
|
||||
- IMAGE_TYPE=extras
|
||||
- GO_TAGS=p2p tts
|
||||
env_file:
|
||||
- ../.env
|
||||
ports:
|
||||
- 8080:8080
|
||||
volumes:
|
||||
- localai_workspace:/workspace
|
||||
- models:/host-models
|
||||
- backends:/host-backends
|
||||
- ../models:/host-models
|
||||
- ./customization:/devcontainer-customization
|
||||
command: /bin/sh -c "while sleep 1000; do :; done"
|
||||
cap_add:
|
||||
@@ -40,9 +43,6 @@ services:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=grafana
|
||||
volumes:
|
||||
- ./grafana:/etc/grafana/provisioning/datasources
|
||||
|
||||
volumes:
|
||||
prom_data:
|
||||
localai_workspace:
|
||||
models:
|
||||
backends:
|
||||
localai_workspace:
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Reconfigure Ubuntu apt sources to point at an alternate mirror.
|
||||
#
|
||||
# Used by Dockerfiles via `RUN --mount=type=bind,source=.docker/apt-mirror.sh,...`
|
||||
# and by CI workflows on the runner to mitigate outages of the default
|
||||
# archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com pool.
|
||||
#
|
||||
# Inputs (env):
|
||||
# APT_MIRROR Replacement for archive.ubuntu.com and security.ubuntu.com
|
||||
# (e.g. "http://azure.archive.ubuntu.com" or
|
||||
# "https://mirrors.edge.kernel.org").
|
||||
# Leave empty to keep upstream. The trailing "/ubuntu/..."
|
||||
# path is preserved by the rewrite.
|
||||
# APT_PORTS_MIRROR Replacement for ports.ubuntu.com (arm64/ppc64el/...).
|
||||
# Leave empty to keep upstream.
|
||||
#
|
||||
# Both default to empty, in which case the script is a no-op.
|
||||
|
||||
set -e
|
||||
|
||||
if [ -z "${APT_MIRROR}" ] && [ -z "${APT_PORTS_MIRROR}" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Ubuntu 24.04 (noble) ships DEB822 sources at /etc/apt/sources.list.d/ubuntu.sources;
|
||||
# older releases use /etc/apt/sources.list. We rewrite whichever exists.
|
||||
for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
|
||||
[ -f "$f" ] || continue
|
||||
if [ -n "${APT_MIRROR}" ]; then
|
||||
# Use a comma delimiter so the alternation pipe in the regex
|
||||
# is not interpreted as the s/// separator.
|
||||
sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
|
||||
fi
|
||||
if [ -n "${APT_PORTS_MIRROR}" ]; then
|
||||
sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "apt-mirror: rewrote sources (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared compile logic for backend/Dockerfile.bonsai.
|
||||
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
export CCACHE_DIR=/root/.ccache
|
||||
ccache --max-size=5G || true
|
||||
ccache -z || true
|
||||
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/bonsai-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/bonsai
|
||||
|
||||
if [ -z "${BUILD_TYPE:-}" ]; then
|
||||
# Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries.
|
||||
# arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme).
|
||||
if [ "${TARGETARCH}" = "arm64" ]; then
|
||||
apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
|
||||
export CC=gcc-14 CXX=g++-14
|
||||
fi
|
||||
make bonsai-cpu-all
|
||||
else
|
||||
# GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator
|
||||
# does the compute. Keeps the GPU compile from also building the CPU variant matrix and
|
||||
# avoids the gcc-14 apt step on GPU base images such as nvidia l4t.
|
||||
make bonsai-fallback
|
||||
fi
|
||||
make bonsai-grpc
|
||||
make bonsai-rpc-server
|
||||
|
||||
ccache -s || true
|
||||
@@ -1,30 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared compile logic for backend/Dockerfile.ik-llama-cpp.
|
||||
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
export CCACHE_DIR=/root/.ccache
|
||||
ccache --max-size=5G || true
|
||||
ccache -z || true
|
||||
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/ik-llama-cpp
|
||||
|
||||
if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
|
||||
# ARM64 / ROCm: build without x86 SIMD
|
||||
make ik-llama-cpp-fallback
|
||||
else
|
||||
# ik_llama.cpp's IQK kernels require at least AVX2
|
||||
make ik-llama-cpp-avx2
|
||||
fi
|
||||
|
||||
ccache -s || true
|
||||
@@ -1,250 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Single source of truth for builder-base contents.
|
||||
#
|
||||
# Used by:
|
||||
# - backend/Dockerfile.base-grpc-builder (CI prebuilt-base source of truth)
|
||||
# - backend/Dockerfile.llama-cpp (builder-fromsource stage)
|
||||
# - backend/Dockerfile.ik-llama-cpp (builder-fromsource stage)
|
||||
# - backend/Dockerfile.turboquant (builder-fromsource stage)
|
||||
#
|
||||
# All four files invoke this script via
|
||||
# RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
|
||||
# --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
# bash /usr/local/sbin/install-base-deps
|
||||
#
|
||||
# so the prebuilt CI base image and the from-source local-dev path are
|
||||
# bit-equivalent by construction.
|
||||
#
|
||||
# Inputs (env, populated from Dockerfile ARG/ENV):
|
||||
# BUILD_TYPE ("cublas"|"l4t"|"hipblas"|"vulkan"|"sycl"|"clblas"|"")
|
||||
# CUDA_MAJOR_VERSION ("12" | "13" | "")
|
||||
# CUDA_MINOR_VERSION ("8" | "0" | "")
|
||||
# TARGETARCH ("amd64" | "arm64")
|
||||
# UBUNTU_VERSION ("2204" | "2404")
|
||||
# SKIP_DRIVERS ("false" | "true")
|
||||
# CMAKE_FROM_SOURCE ("false" | "true")
|
||||
# CMAKE_VERSION ("3.31.10")
|
||||
# GRPC_VERSION ("v1.65.0")
|
||||
# GRPC_MAKEFLAGS ("-j4 -Otarget")
|
||||
# APT_MIRROR / APT_PORTS_MIRROR (optional; consumed by /usr/local/sbin/apt-mirror)
|
||||
# AMDGPU_TARGETS (optional; only relevant for hipblas downstream)
|
||||
#
|
||||
# IMPORTANT: install logic is copied verbatim from the prior in-Dockerfile
|
||||
# RUN blocks. Do not paraphrase apt invocations / version pins / sed line
|
||||
# numbers / deb URLs — the bit-equivalence guarantee depends on it.
|
||||
|
||||
set -eux
|
||||
|
||||
# --- 0. apt mirror rewrite (no-op when APT_MIRROR / APT_PORTS_MIRROR unset) ---
|
||||
if [ -x /usr/local/sbin/apt-mirror ]; then
|
||||
APT_MIRROR="${APT_MIRROR:-}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR:-}" \
|
||||
sh /usr/local/sbin/apt-mirror
|
||||
fi
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
export MAKEFLAGS="${GRPC_MAKEFLAGS:-}"
|
||||
|
||||
# --- 1. Base apt build deps ---
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
ccache git \
|
||||
ca-certificates \
|
||||
make \
|
||||
pkg-config libcurl4-openssl-dev \
|
||||
curl unzip \
|
||||
libssl-dev wget
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# --- 2. Vulkan SDK (BUILD_TYPE=vulkan) ---
|
||||
# NB: this block intentionally installs `cmake` via apt as part of the
|
||||
# Vulkan tooling — must run before the dedicated CMake step below.
|
||||
if [ "${BUILD_TYPE:-}" = "vulkan" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils wget gpg-agent
|
||||
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
|
||||
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
|
||||
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
|
||||
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
|
||||
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
|
||||
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
|
||||
# Mesa Vulkan ICD drivers (ANV/RADV/lavapipe + Arm SoC) and their ICD
|
||||
# manifests. The LunarG SDK below only provides the loader and shader
|
||||
# tooling, not hardware drivers — without Mesa the packaged Vulkan backend
|
||||
# would ship a loader that finds no GPU. package-gpu-libs.sh bundles these
|
||||
# .so files plus their deps into the backend so it stays self-contained.
|
||||
apt-get install -y mesa-vulkan-drivers libdrm2
|
||||
if [ "amd64" = "${TARGETARCH:-}" ]; then
|
||||
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz"
|
||||
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz
|
||||
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz
|
||||
mkdir -p /opt/vulkan-sdk
|
||||
mv 1.4.335.0 /opt/vulkan-sdk/
|
||||
( cd /opt/vulkan-sdk/1.4.335.0 && \
|
||||
./vulkansdk --no-deps --maxjobs \
|
||||
vulkan-loader \
|
||||
vulkan-validationlayers \
|
||||
vulkan-extensionlayer \
|
||||
vulkan-tools \
|
||||
shaderc )
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/
|
||||
rm -rf /opt/vulkan-sdk
|
||||
fi
|
||||
if [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
mkdir vulkan
|
||||
( cd vulkan && \
|
||||
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
|
||||
tar -xvf vulkan-sdk.tar.xz && \
|
||||
rm vulkan-sdk.tar.xz && \
|
||||
cd 1.4.335.0 && \
|
||||
cp -rfv aarch64/bin/* /usr/bin/ && \
|
||||
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
|
||||
cp -rfv aarch64/include/* /usr/include/ && \
|
||||
cp -rfv aarch64/share/* /usr/share/ )
|
||||
rm -rf vulkan
|
||||
fi
|
||||
ldconfig
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 3. CUDA toolkit (BUILD_TYPE=cublas|l4t) ---
|
||||
if { [ "${BUILD_TYPE:-}" = "cublas" ] || [ "${BUILD_TYPE:-}" = "l4t" ]; } && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils
|
||||
if [ "amd64" = "${TARGETARCH:-}" ]; then
|
||||
curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb"
|
||||
fi
|
||||
if [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
|
||||
curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb"
|
||||
else
|
||||
curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb"
|
||||
fi
|
||||
fi
|
||||
dpkg -i cuda-keyring_1.1-1_all.deb
|
||||
rm -f cuda-keyring_1.1-1_all.deb
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
"cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
apt-get install -y --no-install-recommends \
|
||||
"libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcudnn9-cuda-${CUDA_MAJOR_VERSION}" \
|
||||
"cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
|
||||
fi
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 4. cuDSS / NVPL on arm64 + cublas (legacy JetPack / Tegra) ---
|
||||
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
|
||||
if [ "${BUILD_TYPE:-}" = "cublas" ] && [ "${TARGETARCH:-}" = "arm64" ]; then
|
||||
wget "https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
|
||||
dpkg -i "cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
|
||||
cp /var/cudss-local-tegra-repo-ubuntu"${UBUNTU_VERSION}"-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/
|
||||
apt-get update
|
||||
apt-get -y install cudss "cudss-cuda-${CUDA_MAJOR_VERSION}"
|
||||
wget "https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
|
||||
dpkg -i "nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
|
||||
cp /var/nvpl-local-repo-ubuntu"${UBUNTU_VERSION}"-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/
|
||||
apt-get update
|
||||
apt-get install -y nvpl
|
||||
fi
|
||||
|
||||
# --- 5. clBLAS (BUILD_TYPE=clblas) ---
|
||||
# Present in variant Dockerfiles' from-source path but not in master's
|
||||
# Dockerfile.base-grpc-builder. No CI matrix entry currently uses this,
|
||||
# but keep parity so a future BUILD_TYPE=clblas build doesn't drift.
|
||||
if [ "${BUILD_TYPE:-}" = "clblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
libclblast-dev
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 6. ROCm / HIP build deps (BUILD_TYPE=hipblas) ---
|
||||
if [ "${BUILD_TYPE:-}" = "hipblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
hipblas-dev \
|
||||
hipblaslt-dev \
|
||||
rocblas-dev
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
# I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install,
|
||||
# which results in local-ai and others not being able to locate the libraries.
|
||||
# We run ldconfig ourselves to work around this packaging deficiency.
|
||||
ldconfig
|
||||
# Log which GPU architectures have rocBLAS kernel support
|
||||
echo "rocBLAS library data architectures:"
|
||||
(ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
|
||||
echo "WARNING: No rocBLAS kernel data found"
|
||||
fi
|
||||
|
||||
echo "TARGETARCH: ${TARGETARCH:-}"
|
||||
|
||||
# --- 7. protoc (always) ---
|
||||
# The version in 22.04 is too old. We will create one as part of installing
|
||||
# the GRPC build below but that will also bring in a newer version of absl
|
||||
# which stablediffusion cannot compile with. This version of protoc is only
|
||||
# here so that we can generate the grpc code for the stablediffusion build.
|
||||
if [ "amd64" = "${TARGETARCH:-}" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc
|
||||
rm protoc.zip
|
||||
fi
|
||||
if [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc
|
||||
rm protoc.zip
|
||||
fi
|
||||
|
||||
# --- 8. CMake (apt or compiled from source) ---
|
||||
# The version in 22.04 is too old. Vulkan path above already pulled cmake
|
||||
# via apt; the from-source branch here will install over it which is fine.
|
||||
if [ "${CMAKE_FROM_SOURCE:-false}" = "true" ]; then
|
||||
curl -L -s "https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz" -o cmake.tar.gz
|
||||
tar xvf cmake.tar.gz
|
||||
( cd "cmake-${CMAKE_VERSION}" && ./configure && make && make install )
|
||||
else
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
cmake
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 9. gRPC compile + install at /opt/grpc ---
|
||||
# We install GRPC to a different prefix here so that we can copy in only
|
||||
# the build artifacts later — saves several hundred MB on the final docker
|
||||
# image size vs copying in the entire GRPC source tree and running
|
||||
# `make install` in the target container.
|
||||
#
|
||||
# The TESTONLY abseil sed patch and /opt/grpc prefix are load-bearing —
|
||||
# downstream Dockerfiles `COPY` /opt/grpc to /usr/local (or rely on the
|
||||
# prebuilt base having it at /opt/grpc).
|
||||
mkdir -p /build
|
||||
cd /build
|
||||
git clone --recurse-submodules --jobs 4 -b "${GRPC_VERSION}" --depth 1 --shallow-submodules https://github.com/grpc/grpc
|
||||
mkdir -p /build/grpc/cmake/build
|
||||
cd /build/grpc/cmake/build
|
||||
sed -i "216i\\ TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt"
|
||||
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../..
|
||||
make
|
||||
make install
|
||||
cd /
|
||||
rm -rf /build
|
||||
@@ -1,49 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared compile logic for backend/Dockerfile.llama-cpp.
|
||||
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
export CCACHE_DIR=/root/.ccache
|
||||
ccache --max-size=5G || true
|
||||
ccache -z || true
|
||||
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/llama-cpp
|
||||
if [ -z "${BUILD_TYPE:-}" ]; then
|
||||
# Pure CPU image (BUILD_TYPE empty): one build with ggml CPU_ALL_VARIANTS replaces the
|
||||
# per-microarch binaries (x86: avx/avx2/avx512/fallback; arm64: armv8.x/armv9.x). ggml
|
||||
# dlopens the best libggml-cpu-*.so at runtime by probing host CPU features.
|
||||
#
|
||||
# arm64: the CPU_ALL_VARIANTS table includes armv9.2 SME variants whose -march=...+sme is
|
||||
# rejected by the Ubuntu 24.04 default gcc-13. gcc-14 accepts it, so build the arm64
|
||||
# variants with it (the host never *selects* SME unless it has it, but every variant must
|
||||
# still compile).
|
||||
if [ "${TARGETARCH}" = "arm64" ]; then
|
||||
# The prebuilt base inherits default ports.ubuntu.com sources; honor the
|
||||
# APT_*_MIRROR build args here like the from-source path does, so this
|
||||
# apt step survives a mirror outage.
|
||||
sh /LocalAI/.docker/apt-mirror.sh || true
|
||||
apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
|
||||
export CC=gcc-14 CXX=g++-14
|
||||
fi
|
||||
make llama-cpp-cpu-all
|
||||
else
|
||||
# GPU build (cublas/hipblas/sycl/vulkan/...): the accelerator does the compute, so a
|
||||
# single fallback CPU build is enough - no per-microarch CPU variants needed. (This also
|
||||
# keeps the heavy GPU backend compile from also building the whole CPU variant matrix,
|
||||
# and avoids the gcc-14 apt step on GPU base images such as nvidia l4t.)
|
||||
make llama-cpp-fallback
|
||||
fi
|
||||
make llama-cpp-grpc
|
||||
make llama-cpp-rpc-server
|
||||
|
||||
ccache -s || true
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared compile logic for backend/Dockerfile.turboquant.
|
||||
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
export CCACHE_DIR=/root/.ccache
|
||||
ccache --max-size=5G || true
|
||||
ccache -z || true
|
||||
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/turboquant-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/turboquant
|
||||
|
||||
if [ -z "${BUILD_TYPE:-}" ]; then
|
||||
# Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries.
|
||||
# arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme).
|
||||
if [ "${TARGETARCH}" = "arm64" ]; then
|
||||
apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
|
||||
export CC=gcc-14 CXX=g++-14
|
||||
fi
|
||||
make turboquant-cpu-all
|
||||
else
|
||||
# GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator
|
||||
# does the compute. Keeps the GPU compile from also building the CPU variant matrix and
|
||||
# avoids the gcc-14 apt step on GPU base images such as nvidia l4t.
|
||||
make turboquant-fallback
|
||||
fi
|
||||
make turboquant-grpc
|
||||
make turboquant-rpc-server
|
||||
|
||||
ccache -s || true
|
||||
@@ -3,14 +3,7 @@
|
||||
.vscode
|
||||
.devcontainer
|
||||
models
|
||||
backends
|
||||
volumes
|
||||
examples/chatbot-ui/models
|
||||
backend/go/image/stablediffusion-ggml/build/
|
||||
backend/go/*/build
|
||||
backend/go/*/.cache
|
||||
backend/go/*/sources
|
||||
backend/go/*/package
|
||||
examples/rwkv/models
|
||||
examples/**/models
|
||||
Dockerfile*
|
||||
@@ -21,37 +14,4 @@ __pycache__
|
||||
|
||||
# backend virtual environments
|
||||
**/venv
|
||||
backend/python/**/source
|
||||
|
||||
# In-place llama.cpp clone + per-variant build copies. The Makefile
|
||||
# clones llama.cpp itself at the pinned LLAMA_VERSION; if a stale
|
||||
# local checkout is COPY'd into the image, the `llama.cpp:` target
|
||||
# sees the directory and skips re-cloning, so grpc-server.cpp ends
|
||||
# up compiled against whatever (likely older) commit the host had.
|
||||
backend/cpp/llama-cpp/llama.cpp
|
||||
backend/cpp/llama-cpp-*-build
|
||||
|
||||
# privacy-filter: same in-place pattern. The Makefile fetches privacy-filter.cpp
|
||||
# at the pinned commit (or symlinks a PRIVACY_FILTER_SRC checkout for local dev).
|
||||
# A stale dir/symlink COPY'd into the image makes the clone step fail (dangling
|
||||
# symlink) or compile against the wrong commit, so keep host build state out.
|
||||
backend/cpp/privacy-filter/privacy-filter.cpp
|
||||
backend/cpp/privacy-filter/build
|
||||
backend/cpp/privacy-filter/grpc-server
|
||||
backend/cpp/privacy-filter/package
|
||||
|
||||
# Rust backend build output (sources are tracked; target/ is generated)
|
||||
backend/rust/*/target
|
||||
|
||||
# Local-only artifacts that bloat the build context but the image never needs.
|
||||
# Saved image tarballs, locally-installed backends, the host-built binary, and
|
||||
# assorted tool/scratch dirs. None of these are git-tracked.
|
||||
backend-images
|
||||
local-backends
|
||||
local-ai
|
||||
.crush
|
||||
protoc
|
||||
tests
|
||||
|
||||
# Installed via npm inside the build stage; no need to ship the host copy.
|
||||
**/node_modules
|
||||
backend/python/**/source
|
||||
22
.env
22
.env
@@ -26,14 +26,24 @@
|
||||
## Disables COMPEL (Diffusers)
|
||||
# COMPEL=0
|
||||
|
||||
## Disables SD_EMBED (Diffusers)
|
||||
# SD_EMBED=0
|
||||
|
||||
## Enable/Disable single backend (useful if only one GPU is available)
|
||||
# LOCALAI_SINGLE_ACTIVE_BACKEND=true
|
||||
|
||||
# Forces shutdown of the backends if busy (only if LOCALAI_SINGLE_ACTIVE_BACKEND is set)
|
||||
# LOCALAI_FORCE_BACKEND_SHUTDOWN=true
|
||||
## Specify a build type. Available: cublas, openblas, clblas.
|
||||
## cuBLAS: This is a GPU-accelerated version of the complete standard BLAS (Basic Linear Algebra Subprograms) library. It's provided by Nvidia and is part of their CUDA toolkit.
|
||||
## OpenBLAS: This is an open-source implementation of the BLAS library that aims to provide highly optimized code for various platforms. It includes support for multi-threading and can be compiled to use hardware-specific features for additional performance. OpenBLAS can run on many kinds of hardware, including CPUs from Intel, AMD, and ARM.
|
||||
## clBLAS: This is an open-source implementation of the BLAS library that uses OpenCL, a framework for writing programs that execute across heterogeneous platforms consisting of CPUs, GPUs, and other processors. clBLAS is designed to take advantage of the parallel computing power of GPUs but can also run on any hardware that supports OpenCL. This includes hardware from different vendors like Nvidia, AMD, and Intel.
|
||||
# BUILD_TYPE=openblas
|
||||
|
||||
## Uncomment and set to true to enable rebuilding from source
|
||||
# REBUILD=true
|
||||
|
||||
## Enable go tags, available: p2p, tts
|
||||
## p2p: enable distributed inferencing
|
||||
## tts: enables text-to-speech with go-piper
|
||||
## (requires REBUILD=true)
|
||||
#
|
||||
# GO_TAGS=p2p
|
||||
|
||||
## Path where to store generated images
|
||||
# LOCALAI_IMAGE_PATH=/tmp/generated/images
|
||||
@@ -63,7 +73,7 @@
|
||||
|
||||
### Define a list of GRPC Servers for llama-cpp workers to distribute the load
|
||||
# https://github.com/ggerganov/llama.cpp/pull/6829
|
||||
# https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/README.md
|
||||
# https://github.com/ggerganov/llama.cpp/blob/master/examples/rpc/README.md
|
||||
# LLAMACPP_GRPC_SERVERS=""
|
||||
|
||||
### Enable to run parallel requests
|
||||
|
||||
1
.github/PULL_REQUEST_TEMPLATE.md
vendored
1
.github/PULL_REQUEST_TEMPLATE.md
vendored
@@ -7,7 +7,6 @@ This PR fixes #
|
||||
|
||||
**[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
|
||||
- [ ] Yes, I signed my commits.
|
||||
- [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable
|
||||
|
||||
<!--
|
||||
Thank you for contributing to LocalAI!
|
||||
|
||||
100
.github/actions/configure-apt-mirror/action.yml
vendored
100
.github/actions/configure-apt-mirror/action.yml
vendored
@@ -1,100 +0,0 @@
|
||||
name: 'Configure apt mirror'
|
||||
description: |
|
||||
Reconfigure the GitHub Actions runner's Ubuntu apt sources to use an
|
||||
alternate mirror, and emit the effective URLs as outputs so callers can
|
||||
forward them as Docker build-args.
|
||||
|
||||
Two mirror profiles depending on where the runner lives, because the
|
||||
best mirror differs by network:
|
||||
|
||||
* github-hosted runners run on Azure, so they default to the
|
||||
Azure-hosted Ubuntu mirror (lowest latency, same VPC).
|
||||
* self-hosted runners (arc-runner-set, bigger-runner, ...) typically
|
||||
cannot route to azure.archive.ubuntu.com, so they default to the
|
||||
kernel.org mirror, which is publicly reachable from anywhere.
|
||||
|
||||
Pass an empty string to either input to skip the rewrite for that
|
||||
profile and keep upstream archive.ubuntu.com / ports.ubuntu.com.
|
||||
|
||||
inputs:
|
||||
github-hosted-mirror:
|
||||
description: 'archive/security mirror URL for github-hosted runners (empty = upstream)'
|
||||
required: false
|
||||
default: 'http://azure.archive.ubuntu.com'
|
||||
github-hosted-ports-mirror:
|
||||
description: 'ports.ubuntu.com mirror URL for github-hosted runners (empty = upstream)'
|
||||
required: false
|
||||
default: 'http://azure.ports.ubuntu.com'
|
||||
self-hosted-mirror:
|
||||
description: 'archive/security mirror URL for self-hosted runners (empty = upstream)'
|
||||
required: false
|
||||
# HTTP, not HTTPS: the bare ubuntu:24.04 builder image doesn't ship
|
||||
# ca-certificates, so the very first apt-get update over TLS would
|
||||
# fail with "No system certificates available" before it can install
|
||||
# anything. apt validates package integrity via GPG signatures, so
|
||||
# plain HTTP is safe for the archive itself.
|
||||
default: 'http://mirrors.edge.kernel.org'
|
||||
self-hosted-ports-mirror:
|
||||
description: 'ports.ubuntu.com mirror URL for self-hosted runners (empty = upstream)'
|
||||
required: false
|
||||
# mirrors.edge.kernel.org does NOT carry /ubuntu-ports/ — only the
|
||||
# main /ubuntu/ archive — so arm64 builds 404 there. Leave ports
|
||||
# upstream by default. The original DDoS was on archive.ubuntu.com
|
||||
# so ports.ubuntu.com remains the path of least surprise.
|
||||
default: ''
|
||||
|
||||
outputs:
|
||||
effective-mirror:
|
||||
description: 'The mirror URL actually applied for this runner (or empty)'
|
||||
value: ${{ steps.pick.outputs.mirror }}
|
||||
effective-ports-mirror:
|
||||
description: 'The ports mirror URL actually applied for this runner (or empty)'
|
||||
value: ${{ steps.pick.outputs.ports-mirror }}
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Pick effective mirror for this runner
|
||||
id: pick
|
||||
shell: bash
|
||||
env:
|
||||
RUNNER_ENV: ${{ runner.environment }}
|
||||
GH_MIRROR: ${{ inputs.github-hosted-mirror }}
|
||||
GH_PORTS_MIRROR: ${{ inputs.github-hosted-ports-mirror }}
|
||||
SH_MIRROR: ${{ inputs.self-hosted-mirror }}
|
||||
SH_PORTS_MIRROR: ${{ inputs.self-hosted-ports-mirror }}
|
||||
run: |
|
||||
if [ "${RUNNER_ENV}" = "github-hosted" ]; then
|
||||
MIRROR="${GH_MIRROR}"
|
||||
PORTS_MIRROR="${GH_PORTS_MIRROR}"
|
||||
else
|
||||
MIRROR="${SH_MIRROR}"
|
||||
PORTS_MIRROR="${SH_PORTS_MIRROR}"
|
||||
fi
|
||||
echo "configure-apt-mirror: runner=${RUNNER_ENV} mirror='${MIRROR}' ports-mirror='${PORTS_MIRROR}'"
|
||||
echo "mirror=${MIRROR}" >> "$GITHUB_OUTPUT"
|
||||
echo "ports-mirror=${PORTS_MIRROR}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Rewrite apt sources
|
||||
if: steps.pick.outputs.mirror != '' || steps.pick.outputs.ports-mirror != ''
|
||||
shell: bash
|
||||
env:
|
||||
APT_MIRROR: ${{ steps.pick.outputs.mirror }}
|
||||
APT_PORTS_MIRROR: ${{ steps.pick.outputs.ports-mirror }}
|
||||
run: |
|
||||
set -e
|
||||
# Ubuntu 24.04 (noble) ships DEB822 sources at
|
||||
# /etc/apt/sources.list.d/ubuntu.sources; older releases use
|
||||
# /etc/apt/sources.list. Rewrite whichever exists.
|
||||
for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
|
||||
sudo test -f "$f" || continue
|
||||
if [ -n "${APT_MIRROR}" ]; then
|
||||
# Comma delimiter so the alternation pipe in the regex is not
|
||||
# interpreted as the s/// separator.
|
||||
sudo sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
|
||||
fi
|
||||
if [ -n "${APT_PORTS_MIRROR}" ]; then
|
||||
sudo sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
|
||||
fi
|
||||
done
|
||||
echo "Runner apt mirror configured (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"
|
||||
65
.github/actions/free-disk-space/action.yml
vendored
65
.github/actions/free-disk-space/action.yml
vendored
@@ -1,65 +0,0 @@
|
||||
name: 'Free disk space on hosted runners'
|
||||
description: |
|
||||
Aggressively clean GitHub-hosted ubuntu-latest runners to reclaim ~6-10 GB
|
||||
of working space before docker buildx steps. Combines jlumbroso/free-disk-space
|
||||
with explicit apt purges of large packages we never use (dotnet, ghc, mono,
|
||||
android, jdk, ...).
|
||||
|
||||
No-op on self-hosted runners; pass mode=skip to force-disable.
|
||||
|
||||
inputs:
|
||||
mode:
|
||||
description: 'hosted (default — clean) or skip (no-op)'
|
||||
required: false
|
||||
default: 'hosted'
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Free Disk Space (Ubuntu)
|
||||
if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: true
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
docker-images: true
|
||||
swap-storage: true
|
||||
|
||||
- name: Release space from worker
|
||||
if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
|
||||
shell: bash
|
||||
run: |
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
df -h
|
||||
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
|
||||
sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo apt-get remove -y '^mono-.*' || true
|
||||
sudo apt-get remove -y '^ghc-.*' || true
|
||||
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
|
||||
sudo apt-get remove -y 'php.*' || true
|
||||
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
|
||||
sudo apt-get remove -y '^google-.*' || true
|
||||
sudo apt-get remove -y azure-cli || true
|
||||
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
|
||||
sudo apt-get remove -y '^gfortran-.*' || true
|
||||
sudo apt-get remove -y microsoft-edge-stable || true
|
||||
sudo apt-get remove -y firefox || true
|
||||
sudo apt-get remove -y powershell || true
|
||||
sudo apt-get remove -y r-base-core || true
|
||||
sudo apt-get autoremove -y
|
||||
sudo apt-get clean
|
||||
sudo rm -rfv build || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf "/usr/local/share/boost" || true
|
||||
sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
|
||||
df -h
|
||||
59
.github/actions/setup-build-disk/action.yml
vendored
59
.github/actions/setup-build-disk/action.yml
vendored
@@ -1,59 +0,0 @@
|
||||
name: 'Set up build disk on hosted runners'
|
||||
description: |
|
||||
Relocate Docker's data-root to /mnt (which has ~75 GB free, vs ~20 GB
|
||||
on / after free-disk-space). Combined with the apt cleanup, gives
|
||||
~100 GB working space for buildx — enough for ROCm dev image + vLLM
|
||||
torch install + flash-attn build.
|
||||
|
||||
No-op on:
|
||||
- self-hosted runners (no /mnt expectation)
|
||||
- non-X64 runners (verify /mnt shape on ubuntu-24.04-arm separately
|
||||
before enabling there — see Task 3.2 in the migration plan)
|
||||
- mode=skip (force-disable from caller)
|
||||
|
||||
Must run after free-disk-space (which removes large packages — would
|
||||
fail mid-uninstall if Docker were stopped) and before any Docker
|
||||
operation (setup-qemu, setup-buildx, login, build) so the relocated
|
||||
data-root catches all subsequent docker activity.
|
||||
|
||||
inputs:
|
||||
mode:
|
||||
description: 'auto (default — relocate on hosted X64 only) or skip'
|
||||
required: false
|
||||
default: 'auto'
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Relocate Docker data-root to /mnt
|
||||
if: inputs.mode == 'auto' && runner.environment == 'github-hosted' && runner.arch == 'X64'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Before relocation:"
|
||||
df -h / /mnt || true
|
||||
sudo systemctl stop docker docker.socket
|
||||
sudo mkdir -p /mnt/docker-data /mnt/docker-tmp
|
||||
# buildx CLI runs as the unprivileged runner user and creates
|
||||
# config dirs under TMPDIR before binding them into the buildkit
|
||||
# container. /mnt is owned by root by default; mirror /tmp's
|
||||
# 1777 (world-writable + sticky) so non-root processes can write.
|
||||
sudo chmod 1777 /mnt/docker-tmp
|
||||
if [ -d /var/lib/docker ] && [ ! -L /var/lib/docker ]; then
|
||||
sudo rsync -a /var/lib/docker/ /mnt/docker-data/
|
||||
sudo rm -rf /var/lib/docker
|
||||
sudo ln -s /mnt/docker-data /var/lib/docker
|
||||
fi
|
||||
# daemon.json may not exist; merge data-root in or create minimal.
|
||||
if [ -f /etc/docker/daemon.json ]; then
|
||||
sudo jq '."data-root" = "/mnt/docker-data"' /etc/docker/daemon.json | sudo tee /etc/docker/daemon.json.new >/dev/null
|
||||
sudo mv /etc/docker/daemon.json.new /etc/docker/daemon.json
|
||||
else
|
||||
echo '{"data-root":"/mnt/docker-data"}' | sudo tee /etc/docker/daemon.json
|
||||
fi
|
||||
sudo systemctl start docker
|
||||
# Make TMPDIR persist for subsequent steps in the same job.
|
||||
echo "TMPDIR=/mnt/docker-tmp" >> "$GITHUB_ENV"
|
||||
echo "After relocation:"
|
||||
df -h / /mnt
|
||||
docker info | grep -i 'docker root dir' || true
|
||||
6165
.github/backend-matrix.yml
vendored
6165
.github/backend-matrix.yml
vendored
File diff suppressed because it is too large
Load Diff
27
.github/bump_deps.sh
vendored
27
.github/bump_deps.sh
vendored
@@ -1,38 +1,17 @@
|
||||
#!/bin/bash
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1
|
||||
BRANCH=$2
|
||||
VAR=$3
|
||||
FILE=$4
|
||||
|
||||
if [ -z "$FILE" ]; then
|
||||
FILE="Makefile"
|
||||
fi
|
||||
|
||||
# gh_curl follows redirects so a renamed/transferred upstream repo (GitHub
|
||||
# answers 301) still resolves, and fails on HTTP errors rather than letting an
|
||||
# error page reach sed below. `|| true` keeps a failed lookup from aborting the
|
||||
# script at exit 22 with no context — the SHA guard below reports it instead.
|
||||
LAST_COMMIT=$(gh_curl -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH" || true)
|
||||
|
||||
# Guard the sed input: anything that is not a bare 40-hex SHA (an API error
|
||||
# body, an empty response) would otherwise be spliced into the Makefile pin —
|
||||
# either corrupting it silently or blowing up sed with an unterminated
|
||||
# expression, which is how this job failed for a renamed repo.
|
||||
if ! [[ "$LAST_COMMIT" =~ ^[0-9a-f]{40}$ ]]; then
|
||||
echo "Refusing to bump $VAR: expected a 40-char commit SHA for $REPO@$BRANCH, got: $LAST_COMMIT" >&2
|
||||
exit 1
|
||||
fi
|
||||
LAST_COMMIT=$(curl -s -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")
|
||||
|
||||
# Read $VAR from Makefile (only first match)
|
||||
set +e
|
||||
CURRENT_COMMIT="$(grep -m1 "^$VAR?=" $FILE | cut -d'=' -f2)"
|
||||
CURRENT_COMMIT="$(grep -m1 "^$VAR?=" Makefile | cut -d'=' -f2)"
|
||||
set -e
|
||||
|
||||
sed -i $FILE -e "s/$VAR?=.*/$VAR?=$LAST_COMMIT/"
|
||||
sed -i Makefile -e "s/$VAR?=.*/$VAR?=$LAST_COMMIT/"
|
||||
|
||||
if [ -z "$CURRENT_COMMIT" ]; then
|
||||
echo "Could not find $VAR in Makefile."
|
||||
|
||||
13
.github/bump_docs.sh
vendored
13
.github/bump_docs.sh
vendored
@@ -1,18 +1,7 @@
|
||||
#!/bin/bash
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1
|
||||
|
||||
LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
|
||||
"https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
|
||||
|
||||
# jq prints the string "null" for a missing key, so a throttled or otherwise
|
||||
# unexpected API response would otherwise be published as the docs version.
|
||||
if [ -z "$LATEST_TAG" ] || [ "$LATEST_TAG" = "null" ]; then
|
||||
echo "Refusing to bump docs version: could not resolve the latest release tag for $REPO." >&2
|
||||
exit 1
|
||||
fi
|
||||
LATEST_TAG=$(curl -s "https://api.github.com/repos/$REPO/releases/latest" | jq -r '.tag_name')
|
||||
|
||||
cat <<< $(jq ".version = \"$LATEST_TAG\"" docs/data/version.json) > docs/data/version.json
|
||||
|
||||
58
.github/bump_vllm_metal.sh
vendored
58
.github/bump_vllm_metal.sh
vendored
@@ -1,58 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Bump the single vllm-metal pin (VLLM_METAL_VERSION) in the vLLM backend's
|
||||
# darwin (Apple Silicon) install path. The macOS/Metal build
|
||||
# (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is
|
||||
# version-locked to a specific vLLM source release. install.sh derives that vLLM
|
||||
# version at build time from vllm-metal's own installer (`vllm_v=`) at the pinned
|
||||
# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh,
|
||||
# which bumps the Linux cu130 wheel pin.
|
||||
#
|
||||
# This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the
|
||||
# darwin build can only use the exact vLLM version vllm-metal supports, so it may
|
||||
# lag the Linux pin (requirements-cublas13-after.txt) until vllm-metal catches up.
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1 # vllm-project/vllm-metal
|
||||
FILE=$2 # backend/python/vllm/install.sh
|
||||
VAR=$3 # VLLM_METAL_VERSION (used for the workflow's output file names)
|
||||
|
||||
if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
|
||||
echo "usage: $0 <repo> <install-file> <var-name>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
|
||||
# /releases/latest returns the newest one (with its cp312 wheel asset).
|
||||
LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
|
||||
"https://api.github.com/repos/$REPO/releases/latest" \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
|
||||
|
||||
# The coupled vLLM source version lives in vllm-metal's installer at that tag.
|
||||
NEW_VLLM_VERSION=$(gh_curl \
|
||||
"https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
|
||||
| grep -oE 'vllm_v="[0-9]+\.[0-9]+\.[0-9]+"' | head -1 | cut -d'"' -f2)
|
||||
|
||||
if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then
|
||||
echo "Could not resolve vllm-metal tag ($LATEST_TAG) or its vllm_v ($NEW_VLLM_VERSION)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
set +e
|
||||
CURRENT_TAG=$(grep -oE 'VLLM_METAL_VERSION="[^"]*"' "$FILE" | head -1 | cut -d'"' -f2)
|
||||
set -e
|
||||
|
||||
# Rewrite the single pin. install.sh derives VLLM_VERSION from this tag at build
|
||||
# time, so there is nothing else to touch. peter-evans/create-pull-request opens
|
||||
# no PR on a clean tree, so a no-op rewrite (already current) is safe.
|
||||
sed -i "$FILE" \
|
||||
-e "s|VLLM_METAL_VERSION=\"[^\"]*\"|VLLM_METAL_VERSION=\"$LATEST_TAG\"|"
|
||||
|
||||
if [ -z "$CURRENT_TAG" ]; then
|
||||
echo "Could not find VLLM_METAL_VERSION=\"...\" in $FILE." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "vllm-metal ${CURRENT_TAG} -> ${LATEST_TAG} (builds vLLM ${NEW_VLLM_VERSION}): https://github.com/$REPO/releases/tag/${LATEST_TAG}" >> "${VAR}_message.txt"
|
||||
echo "${LATEST_TAG}" >> "${VAR}_commit.txt"
|
||||
48
.github/bump_vllm_wheel.sh
vendored
48
.github/bump_vllm_wheel.sh
vendored
@@ -1,48 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Bump the cublas13 vLLM wheel pin in requirements-cublas13-after.txt.
|
||||
#
|
||||
# vLLM's PyPI wheel is built against CUDA 12 so the cublas13 build pulls a
|
||||
# cu130-flavoured wheel from vLLM's per-tag index at
|
||||
# https://wheels.vllm.ai/<TAG>/cu130/. That URL segment is itself version-locked
|
||||
# (no /latest/ alias upstream), so bumping vLLM means rewriting both the URL
|
||||
# segment and the version constraint atomically. bump_deps.sh handles git-sha
|
||||
# vars in Makefiles; this script handles the two-value rewrite specific to the
|
||||
# vLLM requirements file.
|
||||
set -xe
|
||||
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/gh_curl.sh"
|
||||
|
||||
REPO=$1 # vllm-project/vllm
|
||||
FILE=$2 # backend/python/vllm/requirements-cublas13-after.txt
|
||||
VAR=$3 # VLLM_VERSION (used for output file names so the workflow can read them)
|
||||
|
||||
if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
|
||||
echo "usage: $0 <repo> <requirements-file> <var-name>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# /releases/latest returns the most recent non-prerelease tag.
|
||||
LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
|
||||
"https://api.github.com/repos/$REPO/releases/latest" \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
|
||||
|
||||
# Strip leading 'v' (vLLM tags are 'v0.20.0', the URL/version use '0.20.0').
|
||||
NEW_VERSION="${LATEST_TAG#v}"
|
||||
|
||||
set +e
|
||||
CURRENT_VERSION=$(grep -oE '^vllm==[0-9]+\.[0-9]+\.[0-9]+' "$FILE" | head -1 | cut -d= -f3)
|
||||
set -e
|
||||
|
||||
# sed both lines unconditionally — peter-evans/create-pull-request opens no PR
|
||||
# when the working tree is clean, so a no-op rewrite is safe.
|
||||
sed -i "$FILE" \
|
||||
-e "s|wheels\.vllm\.ai/[^/]*/cu130|wheels.vllm.ai/$NEW_VERSION/cu130|g" \
|
||||
-e "s|^vllm==.*|vllm==$NEW_VERSION|"
|
||||
|
||||
if [ -z "$CURRENT_VERSION" ]; then
|
||||
echo "Could not find vllm==X.Y.Z in $FILE."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Changes: https://github.com/$REPO/compare/v${CURRENT_VERSION}...${LATEST_TAG}" >> "${VAR}_message.txt"
|
||||
echo "${NEW_VERSION}" >> "${VAR}_commit.txt"
|
||||
194
.github/ci/apexentries/README.md
vendored
194
.github/ci/apexentries/README.md
vendored
@@ -1,194 +0,0 @@
|
||||
# apexentries
|
||||
|
||||
Generates gallery entries for the `mudler/*-APEX-GGUF` HuggingFace repositories.
|
||||
|
||||
Each APEX repo becomes one **family**: one entry per quality rung the repo
|
||||
publishes and one per quantization rung its unsloth counterpart publishes, all
|
||||
gathered under the **base model's** entry. LocalAI's variant selector then picks
|
||||
the build that fits the hardware in front of it.
|
||||
|
||||
## The hub is the base model entry, never a generated `*-apex` parent
|
||||
|
||||
Somebody looking for `qwen3.6-35b-a3b` must find every build of those weights
|
||||
under that one name: the APEX imatrix rungs, the unsloth quant rungs and any
|
||||
speculative build. A separate `qwen3.6-35b-a3b-apex` hub competing with the base
|
||||
entry would split the family in two and leave whichever half the user did not
|
||||
search for effectively invisible.
|
||||
|
||||
So the generator resolves the hub by stripping the `-APEX`, `-MTP` and `-TQ`
|
||||
markers and looking the result up in the index, trying both the repo-derived and
|
||||
the stem-derived candidate the same way `CounterpartCandidates` does. Then:
|
||||
|
||||
- **The hub exists** (14 of the 45 repos, resolving to 10 distinct entries).
|
||||
Nothing new is emitted for the family root. A `variants:` block is spliced into
|
||||
the entry that is already there, textually, leaving its description, icon,
|
||||
tags, overrides and files untouched. The line editing is shared with the
|
||||
`variantproposals` job via `.github/ci/galleryedit`.
|
||||
- **The hub is absent** (the other 31). A new hub is emitted, named for the base
|
||||
model and never for the APEX repo. It carries one of the discovered builds as
|
||||
its own payload so it is a complete installable entry rather than a bare index,
|
||||
and that payload is what gives it an `overrides.backend`. Without a declared
|
||||
backend the verifier would skip it, so a hub carrying feature tags would escape
|
||||
the tagging check in silence.
|
||||
|
||||
Several APEX repos routinely resolve to one base model, so both paths accumulate
|
||||
by hub name rather than assuming one family per hub.
|
||||
|
||||
Two references are always filtered out of a hub's list: anything the entry
|
||||
already declares, and the hub's own name. The self reference is not merely
|
||||
redundant. An unsloth rung whose weights the gallery already ships under the base
|
||||
name resolves, through the merge, straight back to the hub, and the verifier
|
||||
reads a self reference as a variant that declares variants of its own.
|
||||
|
||||
The four hand-written `*-apex` entries (`qwen3.6-35b-a3b-apex`,
|
||||
`gemma-4-26b-a4b-it-apex`, `qwen3.5-35b-a3b-apex`,
|
||||
`nemotron-3-nano-omni-30b-a3b-reasoning-apex`) are **ordinary builds**, not hubs.
|
||||
They are referenced from their hub's variants list like any other rung, and are
|
||||
never deleted or renamed.
|
||||
|
||||
## Flags
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|------|---------|---------|
|
||||
| `-index <path>` | `gallery/index.yaml` | Gallery index to dedup against. Read only, unless `-apply` is passed. |
|
||||
| `-only <a,b,c>` | (all) | Comma-separated full repo names (`mudler/Foo-APEX-GGUF`) to restrict generation to. A name that matches nothing is reported as a warning, since it is a typo rather than an empty result. |
|
||||
| `-out <path>` | (none) | Write the entries to add to this file. Nothing is written to the gallery. |
|
||||
| `-apply` | `false` | Splice the variants into `-index` and append the new entries to it. |
|
||||
| `-verify <path>` | (none) | Verify a gallery index and exit. Ignores every other flag. |
|
||||
|
||||
Either `-out` or `-apply` is required, otherwise the run has nothing to do.
|
||||
|
||||
`-apply` splices variant lines into existing entries and **appends** new ones. It
|
||||
never re-serialises the index: it is roughly 40,000 lines, and a YAML round trip
|
||||
would reflow the whole file, drop the anchors and merge keys the gallery relies
|
||||
on, and produce a diff nobody can review. On the three-family sample the splice
|
||||
is 24 added lines across 3 hunks with zero deletions.
|
||||
|
||||
## Discovery is by filename suffix, never by repo name
|
||||
|
||||
Builds come from the files a repo actually publishes. A filename is never
|
||||
constructed from a repo name, because the two disagree:
|
||||
`mudler/gemma-4-26B-A4B-it-APEX-GGUF` ships `gemma-4-26B-A4B-APEX-*.gguf`, and
|
||||
five other repos likewise drop a suffix (`-it`, `-2603`) or a vendor prefix
|
||||
(`NVIDIA-`) that the repo name carries. Composing a URL from the repo name would
|
||||
produce a 404 for every one of them, and the 404 would only surface after the
|
||||
entry shipped.
|
||||
|
||||
The quality ladder is matched on the trailing tier marker, `-(I-)?(Quality|
|
||||
Balanced|Compact|Mini|Nano).gguf`. The `I-` prefix marks the imatrix ladder. The
|
||||
imatrix ladder is emitted when it is non-empty and the plain ladder is used only
|
||||
as a fallback, because two of the 45 repos publish no imatrix tiers at all and
|
||||
must still contribute. Eleven repos carry a fifth `I-Nano` rung, so nothing
|
||||
assumes a fixed number of rungs.
|
||||
|
||||
Every run prints, per repo, the counts that discovery accounted for. If the
|
||||
number of classified files is short of the number of `.gguf` files the repo
|
||||
publishes, the shortfall is printed as `UNCLASSIFIED`. That check is a set
|
||||
difference on counts rather than a second pass over filenames: a second matcher
|
||||
would duplicate the tier regex and the two copies would drift. The failure it
|
||||
catches is quiet. A publishing-script typo that breaks every imatrix filename in
|
||||
a repo does not produce a short ladder; it makes the imatrix ladder empty, and
|
||||
the fallback then downgrades the whole family to the plain ladder with nothing
|
||||
said. A downstream HTTP check cannot catch it either, because it validates the
|
||||
URLs that were emitted, and an undiscovered tier emits none.
|
||||
|
||||
The same reasoning applies to `UNACCOUNTED QUANT`, printed when the unsloth
|
||||
counterpart demonstrably publishes a wanted quant that produced no build. It is
|
||||
reported at discovery time because a dropped quant leaves no trace at all in the
|
||||
finished gallery file.
|
||||
|
||||
## sha256 always comes from the API
|
||||
|
||||
Every file stanza takes its `sha256` from the HuggingFace models API
|
||||
(`lfs.sha256`). A GGUF the API describes without one is a fatal error for that
|
||||
family: the repo is reported by name and the run ends non-zero. It is never
|
||||
substituted from another field, because that is exactly how a Xet hash ends up
|
||||
masquerading as a content hash.
|
||||
|
||||
## The dflash / mtp tagging rule
|
||||
|
||||
An entry is tagged `dflash` or `mtp` **if and only if** it configures the
|
||||
matching `spec_type:draft-<feature>`. Variant ranking reads tags and nothing
|
||||
else, so a tag that does not match the configuration either promotes a build
|
||||
that is no faster or hides one that genuinely is.
|
||||
|
||||
A repo name is not configuration. `mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF` ships
|
||||
weights that carry MTP heads; an entry that does not enable them is not an MTP
|
||||
entry and is not tagged as one.
|
||||
|
||||
A generated hub inherits the tags of the build it carries as its payload, rather
|
||||
than rebuilding them from the base set, so a hub whose payload configures a
|
||||
`spec_type` stays tagged consistently with the overrides copied alongside it.
|
||||
|
||||
## Reuse reporting: two categories, not one
|
||||
|
||||
Generated entries are deduped against the gallery and against the batch itself.
|
||||
The run prints the result under two separate headings, because the two cases are
|
||||
not equivalent:
|
||||
|
||||
- **URI MATCHES** mean the gallery, or an earlier entry in this batch, already
|
||||
ships exactly these weights. Pointing the hub at the existing entry is correct
|
||||
and needs no thought.
|
||||
- **NAME COLLISIONS** mean an entry already owns the name but holds different
|
||||
weights. Referencing it would point the hub at a build other than the one
|
||||
generated. Every one of these must be inspected by hand.
|
||||
|
||||
The run then prints `HUBS SPLICED`, listing every reference that will be added to
|
||||
an entry the gallery already ships along with the line it will be added at, and
|
||||
`HUBS CREATED` for the families that get a new hub. The splices are the part a
|
||||
review has to read closely, because they modify entries somebody else wrote.
|
||||
|
||||
Hubs are deliberately kept out of the merge. A new hub carries the family's top
|
||||
rung as its own payload, so URI dedup would fold the hub into that rung and the
|
||||
family would lose the very entry point this command exists to create.
|
||||
|
||||
## Workflow: sample first, then the full set
|
||||
|
||||
Never run the full generation straight into the gallery. Generate a small,
|
||||
deliberately awkward sample, have it reviewed, then run the rest.
|
||||
|
||||
```bash
|
||||
# 1. Sample three families that between them cover the awkward shapes:
|
||||
# a standard four-rung repo, one with the extra I-Nano rung AND a file stem
|
||||
# that differs from its repo name, and one whose unsloth counterpart shards
|
||||
# its quants across subdirectories.
|
||||
go run ./.github/ci/apexentries \
|
||||
-index gallery/index.yaml \
|
||||
-only mudler/Qwen3.6-35B-A3B-APEX-GGUF,mudler/gemma-4-26B-A4B-it-APEX-GGUF,mudler/Step-3.7-Flash-APEX-GGUF \
|
||||
-out /tmp/sample.yaml
|
||||
|
||||
# 2. Verify the sample against the gallery it would join, splices included. Apply
|
||||
# to a COPY, never to the real index, and check that the diff is only the
|
||||
# intended variant lines. Compare the verifier output to the gallery's own
|
||||
# baseline: what matters is that the sample adds no new problem, not that the
|
||||
# total is zero.
|
||||
cp gallery/index.yaml /tmp/index-copy.yaml
|
||||
go run ./.github/ci/apexentries -index /tmp/index-copy.yaml -only <same list> -apply
|
||||
diff -u gallery/index.yaml /tmp/index-copy.yaml # expect zero deletions
|
||||
|
||||
go run ./.github/ci/apexentries -verify gallery/index.yaml > /tmp/baseline.log 2>&1
|
||||
go run ./.github/ci/apexentries -verify /tmp/index-copy.yaml > /tmp/spliced.log 2>&1
|
||||
diff /tmp/baseline.log /tmp/spliced.log
|
||||
|
||||
# 3. Have a human review /tmp/sample.yaml and every reported name collision.
|
||||
|
||||
# 4. Only then, the full set.
|
||||
go run ./.github/ci/apexentries -index gallery/index.yaml -apply
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
go test ./.github/ci/apexentries/
|
||||
```
|
||||
|
||||
The shared line editor has its own package:
|
||||
|
||||
```bash
|
||||
go test ./.github/ci/galleryedit/
|
||||
```
|
||||
|
||||
`.github/ci/` is invisible to `go list ./...`, so these specs are not covered by
|
||||
`make lint` or the repository test run. `.github/workflows/ci-tools-tests.yaml`
|
||||
names the package explicitly; keep that workflow in step with any package added
|
||||
under `.github/ci/`.
|
||||
70
.github/ci/apexentries/discover.go
vendored
70
.github/ci/apexentries/discover.go
vendored
@@ -1,70 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// tierRE matches the tier marker APEX repos put at the end of a weight
|
||||
// filename. Discovery is by suffix because the stem is not predictable from
|
||||
// the repo name: six of the 45 repos drop a suffix ("-it", "-2603") or a
|
||||
// vendor prefix ("NVIDIA-") that the repo name carries.
|
||||
var tierRE = regexp.MustCompile(`-(I-)?(Quality|Balanced|Compact|Mini|Nano)\.gguf$`)
|
||||
|
||||
// fullPrecisionRE matches the unquantized source weights an APEX repo publishes
|
||||
// alongside its ladder, flat (-F16.gguf) or sharded across a numbered set
|
||||
// (-F16-00001-of-00010.gguf). bf16 is accepted because some repos publish that
|
||||
// instead, and the match is case-insensitive because the casing varies between
|
||||
// publishing scripts.
|
||||
//
|
||||
// These are deliberately not tiers: they are the weights the ladder is quantized
|
||||
// FROM, and generation is scoped to the ladder itself.
|
||||
var fullPrecisionRE = regexp.MustCompile(`(?i)-b?f16(-\d{5}-of-\d{5})?\.gguf$`)
|
||||
|
||||
// IsFullPrecision reports whether a weight filename is an unquantized source.
|
||||
func IsFullPrecision(name string) bool {
|
||||
return fullPrecisionRE.MatchString(name)
|
||||
}
|
||||
|
||||
// Tier is one discovered build of an APEX repo.
|
||||
type Tier struct {
|
||||
Label string
|
||||
File GGUFFile
|
||||
}
|
||||
|
||||
// DiscoverAPEXTiers splits a repo's weight files into the imatrix ladder and
|
||||
// the plain ladder. mmproj files are never tiers.
|
||||
func DiscoverAPEXTiers(files []GGUFFile) (imatrix, plain []Tier) {
|
||||
for _, f := range files {
|
||||
if strings.HasPrefix(f.Name, "mmproj") {
|
||||
continue
|
||||
}
|
||||
m := tierRE.FindStringSubmatch(f.Name)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
if m[1] != "" {
|
||||
imatrix = append(imatrix, Tier{Label: "I-" + m[2], File: f})
|
||||
continue
|
||||
}
|
||||
plain = append(plain, Tier{Label: m[2], File: f})
|
||||
}
|
||||
return imatrix, plain
|
||||
}
|
||||
|
||||
// DiscoverMMProj returns the repo's projector file, if it publishes one. The
|
||||
// name varies across repos (mmproj.gguf, mmproj-F16.gguf,
|
||||
// mmproj-step3.7-flash-f16.gguf), so match the prefix rather than a fixed name.
|
||||
func DiscoverMMProj(files []GGUFFile) (GGUFFile, bool) {
|
||||
for _, f := range files {
|
||||
if strings.HasPrefix(f.Name, "mmproj") {
|
||||
return f, true
|
||||
}
|
||||
}
|
||||
return GGUFFile{}, false
|
||||
}
|
||||
|
||||
// FileStem returns a tier's filename with its tier suffix removed.
|
||||
func FileStem(t Tier) string {
|
||||
return tierRE.ReplaceAllString(t.File.Name, "")
|
||||
}
|
||||
68
.github/ci/apexentries/discover_test.go
vendored
68
.github/ci/apexentries/discover_test.go
vendored
@@ -1,68 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("DiscoverAPEXTiers", func() {
|
||||
It("finds tiers regardless of how the stem relates to the repo name", func() {
|
||||
// This repo is mudler/gemma-4-26B-A4B-it-APEX-GGUF but its files drop "-it".
|
||||
files := []GGUFFile{
|
||||
{Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf", SHA256: "a"},
|
||||
{Name: "gemma-4-26B-A4B-APEX-I-Nano.gguf", SHA256: "b"},
|
||||
{Name: "gemma-4-26B-A4B-APEX-Quality.gguf", SHA256: "c"},
|
||||
{Name: "mmproj-F16.gguf", SHA256: "d"},
|
||||
}
|
||||
|
||||
imatrix, plain := DiscoverAPEXTiers(files)
|
||||
|
||||
Expect(labels(imatrix)).To(ConsistOf("I-Quality", "I-Nano"))
|
||||
Expect(labels(plain)).To(ConsistOf("Quality"))
|
||||
})
|
||||
|
||||
It("excludes mmproj from the tier list", func() {
|
||||
files := []GGUFFile{{Name: "mmproj.gguf", SHA256: "d"}}
|
||||
|
||||
imatrix, plain := DiscoverAPEXTiers(files)
|
||||
|
||||
Expect(imatrix).To(BeEmpty())
|
||||
Expect(plain).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("DiscoverMMProj", func() {
|
||||
It("finds an mmproj whatever its suffix", func() {
|
||||
files := []GGUFFile{
|
||||
{Name: "Model-APEX-I-Mini.gguf", SHA256: "a"},
|
||||
{Name: "mmproj-step3.7-flash-f16.gguf", SHA256: "b"},
|
||||
}
|
||||
|
||||
got, ok := DiscoverMMProj(files)
|
||||
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(got.Name).To(Equal("mmproj-step3.7-flash-f16.gguf"))
|
||||
})
|
||||
|
||||
It("reports absence when the repo ships none", func() {
|
||||
_, ok := DiscoverMMProj([]GGUFFile{{Name: "Model-APEX-Quality.gguf", SHA256: "a"}})
|
||||
|
||||
Expect(ok).To(BeFalse())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("FileStem", func() {
|
||||
It("strips the tier suffix", func() {
|
||||
t := Tier{Label: "I-Quality", File: GGUFFile{Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf"}}
|
||||
|
||||
Expect(FileStem(t)).To(Equal("gemma-4-26B-A4B-APEX"))
|
||||
})
|
||||
})
|
||||
|
||||
func labels(ts []Tier) []string {
|
||||
out := make([]string, 0, len(ts))
|
||||
for _, t := range ts {
|
||||
out = append(out, t.Label)
|
||||
}
|
||||
return out
|
||||
}
|
||||
130
.github/ci/apexentries/hf.go
vendored
130
.github/ci/apexentries/hf.go
vendored
@@ -1,130 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/pkg/httpclient"
|
||||
)
|
||||
|
||||
// ErrNoSHA256 marks a GGUF the HuggingFace API describes without an
|
||||
// lfs.sha256. Emitting an entry without a hash would ship an unverifiable
|
||||
// download, and guessing one from another field is how a Xet hash ends up
|
||||
// masquerading as a content hash, so this is fatal rather than skippable.
|
||||
var ErrNoSHA256 = errors.New("gguf file has no lfs.sha256")
|
||||
|
||||
// GGUFFile is one .gguf sibling of a HuggingFace repo.
|
||||
type GGUFFile struct {
|
||||
Name string
|
||||
Size int64
|
||||
SHA256 string
|
||||
}
|
||||
|
||||
type apiSibling struct {
|
||||
RFilename string `json:"rfilename"`
|
||||
Size int64 `json:"size"`
|
||||
LFS *struct {
|
||||
SHA256 string `json:"sha256"`
|
||||
} `json:"lfs"`
|
||||
}
|
||||
|
||||
type apiModel struct {
|
||||
Siblings []apiSibling `json:"siblings"`
|
||||
}
|
||||
|
||||
// ParseRepoFiles returns every .gguf sibling described by a models API body.
|
||||
func ParseRepoFiles(body []byte) ([]GGUFFile, error) {
|
||||
var m apiModel
|
||||
if err := json.Unmarshal(body, &m); err != nil {
|
||||
return nil, fmt.Errorf("decoding model response: %w", err)
|
||||
}
|
||||
|
||||
var out []GGUFFile
|
||||
for _, s := range m.Siblings {
|
||||
if !strings.HasSuffix(s.RFilename, ".gguf") {
|
||||
continue
|
||||
}
|
||||
if s.LFS == nil || s.LFS.SHA256 == "" {
|
||||
return nil, fmt.Errorf("%s: %w", s.RFilename, ErrNoSHA256)
|
||||
}
|
||||
out = append(out, GGUFFile{Name: s.RFilename, Size: s.Size, SHA256: s.LFS.SHA256})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// FetchOptionalRepoFiles asks the models API for a repo the caller can do
|
||||
// without, and reports separately whether the repo was merely unreadable.
|
||||
//
|
||||
// HuggingFace answers 401 Unauthorized, not 404, for a repo that does not exist
|
||||
// when the request carries no credentials. Without a token there is therefore no
|
||||
// way to tell "this repo was never published" from "this repo is private", so an
|
||||
// optional probe has to treat 401 and 403 exactly like 404: whatever the reason,
|
||||
// there is nothing here for us to read, so there is no counterpart.
|
||||
//
|
||||
// The second return value exists because that collapse is lossy in one
|
||||
// direction: 401/403 can also mean a real, gated repo whose quants we would
|
||||
// genuinely want. The caller reports those repos so a silently dropped
|
||||
// counterpart is visible to a human rather than invisible.
|
||||
func FetchOptionalRepoFiles(client *http.Client, repo string) ([]GGUFFile, bool, error) {
|
||||
files, status, err := fetchRepoFiles(client, repo)
|
||||
if err != nil && (status == http.StatusUnauthorized || status == http.StatusForbidden) {
|
||||
return nil, true, nil
|
||||
}
|
||||
return files, false, err
|
||||
}
|
||||
|
||||
// FetchRepoFiles asks the models API for one repo. A 404 yields (nil, nil) so
|
||||
// that probing for an optional counterpart repo is not an error. Every other
|
||||
// non-200, 401 and 403 included, is an error: for a repo the run REQUIRES there
|
||||
// is no benign reading of "we cannot see it".
|
||||
func FetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, error) {
|
||||
files, _, err := fetchRepoFiles(client, repo)
|
||||
return files, err
|
||||
}
|
||||
|
||||
// fetchRepoFiles does the request and returns the HTTP status alongside the
|
||||
// result, so the optional and required callers can apply different policies to
|
||||
// the same response without duplicating the request.
|
||||
func fetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, int, error) {
|
||||
url := fmt.Sprintf("https://huggingface.co/api/models/%s?blobs=true", repo)
|
||||
req, err := http.NewRequest(http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
req.Header.Set("User-Agent", "localai-apexentries/1.0")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode == http.StatusNotFound {
|
||||
return nil, resp.StatusCode, nil
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, resp.StatusCode, fmt.Errorf("%s: unexpected status %d", repo, resp.StatusCode)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, resp.StatusCode, err
|
||||
}
|
||||
files, err := ParseRepoFiles(body)
|
||||
return files, resp.StatusCode, err
|
||||
}
|
||||
|
||||
// newHTTPClient builds the client used against the HuggingFace API. It goes
|
||||
// through pkg/httpclient rather than a bare &http.Client{} because the std
|
||||
// client follows redirects and forwards custom credential headers to the
|
||||
// redirect target on a cross-host hop (GHSA-3mj3-57v2-4636). This caller sends
|
||||
// only a User-Agent today, but it talks to an external API that could start
|
||||
// redirecting, and an HF_TOKEN header here later would then leak.
|
||||
func newHTTPClient() *http.Client {
|
||||
return httpclient.NewWithTimeout(60 * time.Second)
|
||||
}
|
||||
142
.github/ci/apexentries/hf_test.go
vendored
142
.github/ci/apexentries/hf_test.go
vendored
@@ -1,142 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestApexEntries(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "apexentries")
|
||||
}
|
||||
|
||||
// stubTransport answers every request with one canned status and body, so the
|
||||
// status handling of the fetchers can be exercised without reaching the real
|
||||
// HuggingFace API.
|
||||
type stubTransport struct {
|
||||
status int
|
||||
body string
|
||||
}
|
||||
|
||||
func (t stubTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
return &http.Response{
|
||||
StatusCode: t.status,
|
||||
Body: io.NopCloser(bytes.NewBufferString(t.body)),
|
||||
Header: make(http.Header),
|
||||
Request: req,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func stubClient(status int, body string) *http.Client {
|
||||
return &http.Client{Transport: stubTransport{status: status, body: body}}
|
||||
}
|
||||
|
||||
const oneGGUFBody = `{"siblings":[{"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}}]}`
|
||||
|
||||
var _ = Describe("FetchOptionalRepoFiles", func() {
|
||||
// HuggingFace answers 401 rather than 404 for a repo that does not exist
|
||||
// when the client carries no credentials, so an optional probe cannot tell
|
||||
// "absent" from "unauthorized" and must treat both as "no counterpart".
|
||||
It("treats a 401 as an absent repo and flags it as unavailable", func() {
|
||||
files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusUnauthorized, ""), "unsloth/Nope-GGUF")
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(files).To(BeEmpty())
|
||||
Expect(unavailable).To(BeTrue())
|
||||
})
|
||||
|
||||
It("treats a 403 as an absent repo and flags it as unavailable", func() {
|
||||
files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusForbidden, ""), "unsloth/Gated-GGUF")
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(files).To(BeEmpty())
|
||||
Expect(unavailable).To(BeTrue())
|
||||
})
|
||||
|
||||
// A clean 404 is an unambiguous absence, so it must NOT be reported as
|
||||
// unavailable: the whole point of the flag is to separate the ambiguous
|
||||
// case a human may need to look at from the settled one.
|
||||
It("treats a 404 as an absent repo without flagging it as unavailable", func() {
|
||||
files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusNotFound, ""), "unsloth/Nope-GGUF")
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(files).To(BeEmpty())
|
||||
Expect(unavailable).To(BeFalse())
|
||||
})
|
||||
|
||||
It("parses a 200 body as usual", func() {
|
||||
files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusOK, oneGGUFBody), "unsloth/Real-GGUF")
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(unavailable).To(BeFalse())
|
||||
Expect(files).To(HaveLen(1))
|
||||
Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf"))
|
||||
Expect(files[0].SHA256).To(Equal("aa"))
|
||||
})
|
||||
|
||||
// Tolerating 401/403 must not widen into tolerating everything: a 500 is a
|
||||
// broken API, not evidence about whether the repo exists.
|
||||
It("still errors on a 500", func() {
|
||||
_, _, err := FetchOptionalRepoFiles(stubClient(http.StatusInternalServerError, ""), "unsloth/Real-GGUF")
|
||||
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("unexpected status 500"))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("FetchRepoFiles", func() {
|
||||
// The APEX repo itself is not optional. A 401 there means the repo the run
|
||||
// was asked to publish cannot be read, which is a real failure and must not
|
||||
// be quietly downgraded to "no files".
|
||||
It("errors on a 401 for a required repo", func() {
|
||||
_, err := FetchRepoFiles(stubClient(http.StatusUnauthorized, ""), "mudler/Model-APEX-GGUF")
|
||||
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("unexpected status 401"))
|
||||
})
|
||||
|
||||
It("errors on a 403 for a required repo", func() {
|
||||
_, err := FetchRepoFiles(stubClient(http.StatusForbidden, ""), "mudler/Model-APEX-GGUF")
|
||||
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("unexpected status 403"))
|
||||
})
|
||||
|
||||
It("still treats a 404 as an absent repo", func() {
|
||||
files, err := FetchRepoFiles(stubClient(http.StatusNotFound, ""), "mudler/Model-APEX-GGUF")
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(files).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("ParseRepoFiles", func() {
|
||||
It("returns gguf siblings with their lfs sha256", func() {
|
||||
body := []byte(`{"siblings":[
|
||||
{"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}},
|
||||
{"rfilename":"README.md"},
|
||||
{"rfilename":"mmproj.gguf","size":5,"lfs":{"sha256":"bb","size":5}}
|
||||
]}`)
|
||||
|
||||
files, err := ParseRepoFiles(body)
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(files).To(HaveLen(2))
|
||||
Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf"))
|
||||
Expect(files[0].SHA256).To(Equal("aa"))
|
||||
Expect(files[1].Name).To(Equal("mmproj.gguf"))
|
||||
})
|
||||
|
||||
It("reports a gguf that carries no lfs sha256", func() {
|
||||
body := []byte(`{"siblings":[{"rfilename":"mmproj.gguf","size":5}]}`)
|
||||
|
||||
_, err := ParseRepoFiles(body)
|
||||
|
||||
Expect(err).To(MatchError(ErrNoSHA256))
|
||||
})
|
||||
})
|
||||
143
.github/ci/apexentries/hub.go
vendored
143
.github/ci/apexentries/hub.go
vendored
@@ -1,143 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
|
||||
"github.com/mudler/LocalAI/.github/ci/galleryedit"
|
||||
)
|
||||
|
||||
// IndexText is the gallery index seen as text: the entries it declares plus the
|
||||
// exact lines each one occupies, which is what splicing a variants block into an
|
||||
// entry the gallery already ships requires.
|
||||
//
|
||||
// It is a second, narrower read of the same file LoadExisting parses. The two
|
||||
// answer different questions: LoadExisting answers "do these weights already
|
||||
// exist anywhere", this one answers "where in the file does this entry live".
|
||||
type IndexText struct {
|
||||
Lines []string
|
||||
Entries []*indexEntry
|
||||
|
||||
byName map[string]*indexEntry
|
||||
}
|
||||
|
||||
// indexEntry is one entry of the index: its name, the variants it already
|
||||
// declares, and its coordinates in the file.
|
||||
type indexEntry struct {
|
||||
Name string `yaml:"name"`
|
||||
Variants []VariantRef `yaml:"variants"`
|
||||
|
||||
Pos galleryedit.Entry `yaml:"-"`
|
||||
}
|
||||
|
||||
// LoadIndexText reads the gallery index for editing.
|
||||
func LoadIndexText(path string) (*IndexText, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ParseIndexText(string(raw))
|
||||
}
|
||||
|
||||
// ParseIndexText pairs the decoded entries with the top level list items the
|
||||
// text actually contains.
|
||||
//
|
||||
// If the two views disagree on how many entries there are then every line number
|
||||
// a splice would compute is suspect, and the failure mode is writing a variants
|
||||
// block into the wrong model. The parse refuses instead.
|
||||
func ParseIndexText(text string) (*IndexText, error) {
|
||||
var entries []*indexEntry
|
||||
if err := yaml.Unmarshal([]byte(text), &entries); err != nil {
|
||||
return nil, fmt.Errorf("decoding gallery index: %w", err)
|
||||
}
|
||||
|
||||
lines, starts := galleryedit.Scan(text)
|
||||
if len(starts) != len(entries) {
|
||||
return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number",
|
||||
len(entries), len(starts))
|
||||
}
|
||||
|
||||
ix := &IndexText{Lines: lines, Entries: entries, byName: map[string]*indexEntry{}}
|
||||
for i, e := range entries {
|
||||
if e == nil {
|
||||
return nil, fmt.Errorf("gallery index list item %d is empty; refusing to edit by line number", i)
|
||||
}
|
||||
end := len(lines)
|
||||
if i+1 < len(starts) {
|
||||
end = starts[i+1]
|
||||
}
|
||||
e.Pos = galleryedit.Entry{Name: e.Name, StartLine: starts[i], EndLine: end}
|
||||
|
||||
// First occurrence wins, matching the gallery's own resolution.
|
||||
key := strings.ToLower(e.Name)
|
||||
if _, seen := ix.byName[key]; !seen {
|
||||
ix.byName[key] = e
|
||||
}
|
||||
}
|
||||
return ix, nil
|
||||
}
|
||||
|
||||
// Find looks an entry up by name, case insensitively.
|
||||
func (ix *IndexText) Find(name string) *indexEntry {
|
||||
return ix.byName[strings.ToLower(name)]
|
||||
}
|
||||
|
||||
// ResolveHub returns the gallery name of a family's hub and whether the gallery
|
||||
// already ships an entry under it.
|
||||
//
|
||||
// The hub is the BASE model entry, never a generated *-apex parent. Somebody
|
||||
// looking for qwen3.6-35b-a3b has to find every build of those weights under
|
||||
// that one name: the APEX imatrix rungs, the unsloth quant rungs and any
|
||||
// speculative build. A separate qwen3.6-35b-a3b-apex hub competing with the base
|
||||
// entry would split the family in two and leave whichever half the user did not
|
||||
// search for invisible.
|
||||
//
|
||||
// Both candidates are tried for the same reason CounterpartCandidates tries
|
||||
// both. The repo name and the published file stem disagree for several of these
|
||||
// repos, and either one may be what the base entry was named after.
|
||||
func ResolveHub(ix *IndexText, repoBase, stem string) (name string, exists bool) {
|
||||
candidates := CounterpartCandidates(repoBase, stem)
|
||||
for _, c := range candidates {
|
||||
if n := slug(c); ix.Find(n) != nil {
|
||||
return n, true
|
||||
}
|
||||
}
|
||||
// Nothing matched, so the family needs a hub of its own under the repo
|
||||
// derived name, which is the more reliable of the two.
|
||||
return slug(candidates[0]), false
|
||||
}
|
||||
|
||||
// HubLabel is the human-cased base model name, for prose rather than lookup.
|
||||
func HubLabel(repoBase, stem string) string {
|
||||
return CounterpartCandidates(repoBase, stem)[0]
|
||||
}
|
||||
|
||||
// filterVariants drops the references a hub must not carry: itself, and anything
|
||||
// it already lists.
|
||||
//
|
||||
// The self reference is not merely redundant. A hub that names itself makes the
|
||||
// verifier resolve the reference back to the hub, see that the hub declares
|
||||
// variants, and report a variant that declares variants of its own. It arises
|
||||
// for real rather than in theory: an unsloth rung whose weights the gallery
|
||||
// already ships under the base model name resolves, through Merge, straight back
|
||||
// to the hub that is about to reference it.
|
||||
func filterVariants(hub string, already []VariantRef, want []string) []string {
|
||||
seen := map[string]bool{strings.ToLower(hub): true}
|
||||
for _, v := range already {
|
||||
seen[strings.ToLower(v.Model)] = true
|
||||
}
|
||||
|
||||
var out []string
|
||||
for _, w := range want {
|
||||
key := strings.ToLower(w)
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
out = append(out, w)
|
||||
}
|
||||
return out
|
||||
}
|
||||
795
.github/ci/apexentries/main.go
vendored
795
.github/ci/apexentries/main.go
vendored
@@ -1,795 +0,0 @@
|
||||
// Command apexentries generates gallery entries for the mudler APEX GGUF
|
||||
// repositories: one entry per imatrix tier and per unsloth quant rung, all
|
||||
// gathered under the BASE model's entry. Builds off a *-APEX-MTP-GGUF repo turn
|
||||
// speculative decoding on, because those weights retain the model's MTP heads
|
||||
// and are only worth their extra size with the heads in use.
|
||||
//
|
||||
// The base model entry is the hub. Somebody looking for qwen3.6-35b-a3b must
|
||||
// find every build of those weights under that one name, so when the gallery
|
||||
// already ships the base entry this command splices a variants block into it
|
||||
// rather than emitting a competing *-apex parent beside it. Only a family whose
|
||||
// base model the gallery does not ship at all gets a new hub entry, and that one
|
||||
// is still named for the base model.
|
||||
//
|
||||
// Builds are discovered by inspecting the filenames a repo actually publishes.
|
||||
// Repo names do not reliably predict them: mudler/gemma-4-26B-A4B-it-APEX-GGUF
|
||||
// ships gemma-4-26B-A4B-APEX-*.gguf, and six of the 45 repos drop a suffix or a
|
||||
// vendor prefix in the same way.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
|
||||
"github.com/mudler/LocalAI/.github/ci/galleryedit"
|
||||
)
|
||||
|
||||
const (
|
||||
// entryTemplate carries no backend and no parameters of its own, which is
|
||||
// why RenderChild states everything inline.
|
||||
entryTemplate = "virtual.yaml"
|
||||
unslothOwner = "unsloth"
|
||||
authorListURL = "https://huggingface.co/api/models?author=mudler&limit=300"
|
||||
)
|
||||
|
||||
// rungRank orders the quality ladder from best to smallest. The HuggingFace API
|
||||
// returns siblings alphabetically and DiscoverAPEXTiers preserves that order, so
|
||||
// an unsorted variants list reads I-Balanced, I-Compact, I-Mini, I-Nano,
|
||||
// I-Quality. Selection ignores authored order, so this is purely so the file a
|
||||
// human reviews scans in a meaningful sequence.
|
||||
var rungRank = map[string]int{
|
||||
"I-Quality": 0, "I-Balanced": 1, "I-Compact": 2, "I-Mini": 3, "I-Nano": 4,
|
||||
"Quality": 5, "Balanced": 6, "Compact": 7, "Mini": 8, "Nano": 9,
|
||||
}
|
||||
|
||||
// baseTags are the tags every generated entry carries. dflash and mtp are never
|
||||
// among them: RenderChild adds those if and only if the entry configures the
|
||||
// matching spec_type.
|
||||
var baseTags = []string{"llm", "gguf", "cpu", "gpu"}
|
||||
|
||||
// childBuild pairs a rendered entry with its position on the quality ladder, so
|
||||
// the parent's variants list can be sorted without re-parsing entry names.
|
||||
type childBuild struct {
|
||||
entry GalleryEntry
|
||||
rank int
|
||||
}
|
||||
|
||||
// family is one APEX repo's full generated output.
|
||||
type family struct {
|
||||
repo string
|
||||
repoBase string
|
||||
stem string
|
||||
hasMMProj bool
|
||||
children []childBuild
|
||||
|
||||
// skippedRepos are counterpart candidates HuggingFace would not describe.
|
||||
// Carried on the family rather than printed and forgotten so the run can
|
||||
// summarize them next to everything else a reviewer has to eyeball.
|
||||
skippedRepos []string
|
||||
census fileCensus
|
||||
unaccounted int
|
||||
}
|
||||
|
||||
// fileCensus splits the files discovery emitted nothing for into the ones a
|
||||
// reviewer must chase and the ones that are deliberately out of scope.
|
||||
//
|
||||
// Full-precision sources are the second kind: they are the unquantized weights
|
||||
// the ladder is derived FROM, not a rung of it. Folding them into the
|
||||
// unclassified total would leave a permanent benign baseline, and a permanent
|
||||
// baseline is exactly what hides the one file that ever genuinely matters.
|
||||
type fileCensus struct {
|
||||
unclassified int
|
||||
fullPrecision int
|
||||
}
|
||||
|
||||
// add accumulates one repo's census into a running total.
|
||||
func (c *fileCensus) add(o fileCensus) {
|
||||
c.unclassified += o.unclassified
|
||||
c.fullPrecision += o.fullPrecision
|
||||
}
|
||||
|
||||
// sortedChildren returns the family's builds in ladder order, best first.
|
||||
func (f *family) sortedChildren() []childBuild {
|
||||
sorted := append([]childBuild{}, f.children...)
|
||||
sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].rank < sorted[j].rank })
|
||||
return sorted
|
||||
}
|
||||
|
||||
func main() {
|
||||
verify := flag.String("verify", "", "verify a gallery index and exit")
|
||||
index := flag.String("index", "gallery/index.yaml", "gallery index to dedup against")
|
||||
only := flag.String("only", "", "comma-separated repo names to restrict generation to")
|
||||
out := flag.String("out", "", "write the entries to add to this file")
|
||||
apply := flag.Bool("apply", false, "append the entries to add to -index")
|
||||
flag.Parse()
|
||||
|
||||
if *verify != "" {
|
||||
problems := Verify(*verify)
|
||||
for _, p := range problems {
|
||||
fmt.Fprintln(os.Stderr, p)
|
||||
}
|
||||
if len(problems) > 0 {
|
||||
fmt.Fprintf(os.Stderr, "%d problem(s)\n", len(problems))
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Println("index is sound")
|
||||
return
|
||||
}
|
||||
|
||||
if err := generate(*index, *only, *out, *apply); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func generate(indexPath, only, outPath string, apply bool) error {
|
||||
if outPath == "" && !apply {
|
||||
return fmt.Errorf("nothing to do: pass -out <file> or -apply")
|
||||
}
|
||||
|
||||
client := newHTTPClient()
|
||||
|
||||
repos, err := listAPEXRepos(client)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if only != "" {
|
||||
repos = restrict(repos, only)
|
||||
}
|
||||
if len(repos) == 0 {
|
||||
return fmt.Errorf("no APEX repos selected")
|
||||
}
|
||||
fmt.Printf("repos selected: %d\n", len(repos))
|
||||
|
||||
var families []family
|
||||
var failed []string
|
||||
|
||||
for _, repo := range repos {
|
||||
f, err := buildFamily(client, repo)
|
||||
if err != nil {
|
||||
// A missing sha256 is fatal for the family rather than skippable: an
|
||||
// entry without one ships an unverifiable download. Report which repo
|
||||
// and keep going, so one bad repo does not hide the state of the rest.
|
||||
fmt.Fprintf(os.Stderr, "FAILED %s: %v\n", repo, err)
|
||||
failed = append(failed, repo)
|
||||
continue
|
||||
}
|
||||
families = append(families, *f)
|
||||
}
|
||||
|
||||
existing, err := LoadExisting(indexPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ixText, err := LoadIndexText(indexPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("existing index: %d names, %d weight URIs, %d lines\n",
|
||||
len(existing.ByName), len(existing.ByURI), len(ixText.Lines))
|
||||
|
||||
// Only the builds go through Merge. A hub is deliberately kept out of it: a
|
||||
// new hub carries the family's top rung as its own payload, so Merge's URI
|
||||
// dedup would fold the hub into that rung and the family would lose the very
|
||||
// entry point this command exists to create. Hub names are checked against
|
||||
// the index directly, by ResolveHub.
|
||||
var generated []GalleryEntry
|
||||
for _, f := range families {
|
||||
for _, c := range f.children {
|
||||
generated = append(generated, c.entry)
|
||||
}
|
||||
}
|
||||
|
||||
add, reused := Merge(existing, generated)
|
||||
reportReuse(existing, generated, reused)
|
||||
|
||||
// Variant references are resolved from `reused`, never used to decide what to
|
||||
// emit: on a within-batch name collision Merge records reused[name] = name
|
||||
// while the first entry of that name is still in `add`, so treating presence
|
||||
// in `reused` as "dropped" would silently emit nothing for it.
|
||||
added := map[string]bool{}
|
||||
for _, e := range add {
|
||||
added[e.Name] = true
|
||||
}
|
||||
|
||||
inserts, newHubs, err := planHubs(families, ixText, reused, added)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reportHubs(ixText, inserts, newHubs)
|
||||
|
||||
skipped, census, fullPrecisionRepos, unaccounted := reportSkipped(families)
|
||||
|
||||
add = append(add, newHubs...)
|
||||
|
||||
fmt.Printf("\nentries generated: %d\nentries to add: %d\nentries reused: %d\nhubs spliced: %d\nhubs created: %d\nrepos skipped: %d\nexcluded (full precision): %d files across %d repos\nunclassified: %d\nunaccounted: %d\n",
|
||||
len(generated), len(add), len(reused), len(inserts), len(newHubs), len(skipped),
|
||||
census.fullPrecision, fullPrecisionRepos, census.unclassified, unaccounted)
|
||||
|
||||
lines, err := galleryedit.Apply(ixText.Lines, inserts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := writeEntries(add, lines, outPath, apply, indexPath); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("%d repo(s) failed: %s", len(failed), strings.Join(failed, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveVariant maps a generated child name onto whatever entry actually stands
|
||||
// for it after the merge. `added` is consulted first because a within-batch name
|
||||
// collision puts a name in BOTH add and reused, and the entry that was emitted
|
||||
// is the one the parent must reference.
|
||||
func resolveVariant(name string, reused map[string]string, added map[string]bool) string {
|
||||
if added[name] {
|
||||
return name
|
||||
}
|
||||
if target, ok := reused[name]; ok {
|
||||
return target
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
// SpecTypeForRepo reports the speculative decoding mechanism a repo's builds can
|
||||
// turn on with no extra download.
|
||||
//
|
||||
// The *-APEX-MTP-GGUF repos republish the base weights with the model's own MTP
|
||||
// heads retained, so those builds are only worth their extra size if the heads
|
||||
// are actually used. Every other APEX repo drops them, and switching MTP on
|
||||
// there would name a mechanism the weights cannot serve.
|
||||
//
|
||||
// The suffix is read off the repo the FILES come from, so nothing downstream has
|
||||
// to infer a capability from an entry name.
|
||||
func SpecTypeForRepo(repo string) string {
|
||||
if strings.HasSuffix(path.Base(repo), "-APEX-MTP-GGUF") {
|
||||
return "draft-mtp"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// buildFamily discovers everything one APEX repo and its unsloth counterpart
|
||||
// publish, and renders it.
|
||||
func buildFamily(client *http.Client, repo string) (*family, error) {
|
||||
files, err := FetchRepoFiles(client, repo)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(files) == 0 {
|
||||
return nil, fmt.Errorf("no gguf files")
|
||||
}
|
||||
|
||||
imatrix, plain := DiscoverAPEXTiers(files)
|
||||
mmproj, hasMMProj := DiscoverMMProj(files)
|
||||
|
||||
census := reportUnclassified(repo, files, imatrix, plain)
|
||||
|
||||
// The imatrix ladder is preferred, but two of the 45 repos publish no
|
||||
// imatrix tiers at all and must still contribute their plain ladder.
|
||||
ladder := imatrix
|
||||
ladderKind := "imatrix"
|
||||
if len(ladder) == 0 {
|
||||
ladder = plain
|
||||
ladderKind = "plain"
|
||||
}
|
||||
if len(ladder) == 0 {
|
||||
return nil, fmt.Errorf("no tiers discovered")
|
||||
}
|
||||
|
||||
sortTiers(ladder)
|
||||
|
||||
var mm *GGUFFile
|
||||
if hasMMProj {
|
||||
mm = &mmproj
|
||||
}
|
||||
|
||||
repoBase := strings.TrimSuffix(path.Base(repo), "-GGUF")
|
||||
f := &family{repo: repo, repoBase: repoBase, hasMMProj: hasMMProj, census: census}
|
||||
|
||||
// Only the APEX ladder can carry MTP heads; the unsloth counterpart quantizes
|
||||
// the plain weights and gets nothing from this.
|
||||
specType := SpecTypeForRepo(repo)
|
||||
|
||||
for _, t := range ladder {
|
||||
f.children = append(f.children, childBuild{
|
||||
rank: rungRank[t.Label],
|
||||
entry: RenderChild(ChildInput{
|
||||
Name: slug(repoBase) + "-" + slug(t.Label),
|
||||
Repo: repo,
|
||||
Template: entryTemplate,
|
||||
SpecType: specType,
|
||||
Weights: []GGUFFile{t.File},
|
||||
MMProj: mm,
|
||||
BaseTags: baseTags,
|
||||
}),
|
||||
})
|
||||
}
|
||||
|
||||
stem := FileStem(ladder[0])
|
||||
f.stem = stem
|
||||
fmt.Printf("%s: %d %s tier(s) [%s], stem %s, mmproj %v\n",
|
||||
repo, len(ladder), ladderKind, tierLabels(ladder), stem, hasMMProj)
|
||||
|
||||
counterpart, cpFiles, skipped, err := resolveCounterpart(client, repoBase, stem)
|
||||
f.skippedRepos = skipped
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if counterpart != "" {
|
||||
builds := DiscoverUnslothQuants(cpFiles)
|
||||
|
||||
// Called here rather than inside Verify: a quant dropped at discovery
|
||||
// leaves no trace at all in the finished gallery file, so the only place
|
||||
// the shortfall is still visible is the moment of discovery.
|
||||
unaccounted := UnaccountedQuants(cpFiles, builds)
|
||||
f.unaccounted = len(unaccounted)
|
||||
for _, p := range unaccounted {
|
||||
fmt.Fprintf(os.Stderr, "UNACCOUNTED QUANT %s: %s\n", counterpart, p)
|
||||
}
|
||||
|
||||
cpMMProj, hasCPMMProj := DiscoverMMProj(cpFiles)
|
||||
var cpMM *GGUFFile
|
||||
if hasCPMMProj {
|
||||
cpMM = &cpMMProj
|
||||
}
|
||||
cpBase := strings.TrimSuffix(path.Base(counterpart), "-GGUF")
|
||||
|
||||
for i, b := range builds {
|
||||
f.children = append(f.children, childBuild{
|
||||
rank: 100 + i,
|
||||
entry: RenderChild(ChildInput{
|
||||
Name: slug(cpBase) + "-" + slug(b.Quant),
|
||||
Repo: counterpart,
|
||||
Template: entryTemplate,
|
||||
Weights: b.Files,
|
||||
MMProj: cpMM,
|
||||
BaseTags: baseTags,
|
||||
}),
|
||||
})
|
||||
}
|
||||
fmt.Printf("%s: counterpart %s, %d quant build(s) %s\n", repo, counterpart, len(builds), quantLabels(builds))
|
||||
} else {
|
||||
fmt.Printf("%s: no unsloth counterpart\n", repo)
|
||||
}
|
||||
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// planHubs decides, per family, whether the family's builds are spliced into a
|
||||
// base model entry the gallery already ships or gathered under a new hub.
|
||||
//
|
||||
// Splicing is strongly preferred and is the measured majority-adjacent case. The
|
||||
// existing entry keeps its description, icon, tags, overrides and files
|
||||
// untouched; only variant lines are added to it.
|
||||
func planHubs(families []family, ix *IndexText, reused map[string]string, added map[string]bool) ([]galleryedit.Insert, []GalleryEntry, error) {
|
||||
// Several APEX repos can resolve to one base model, so both paths accumulate
|
||||
// by hub name rather than assuming one family per hub.
|
||||
wantByHub := map[string][]string{}
|
||||
var spliceOrder []string
|
||||
|
||||
var newHubs []GalleryEntry
|
||||
hubAt := map[string]int{}
|
||||
|
||||
for i := range families {
|
||||
f := &families[i]
|
||||
|
||||
hubName, exists := ResolveHub(ix, f.repoBase, f.stem)
|
||||
want := hubVariants(f, ix, reused, added)
|
||||
|
||||
if exists {
|
||||
if _, seen := wantByHub[hubName]; !seen {
|
||||
spliceOrder = append(spliceOrder, hubName)
|
||||
}
|
||||
wantByHub[hubName] = append(wantByHub[hubName], want...)
|
||||
continue
|
||||
}
|
||||
|
||||
if at, dup := hubAt[hubName]; dup {
|
||||
for _, v := range filterVariants(hubName, newHubs[at].Variants, want) {
|
||||
newHubs[at].Variants = append(newHubs[at].Variants, VariantRef{Model: v})
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
builds := f.sortedChildren()
|
||||
if len(builds) == 0 {
|
||||
return nil, nil, fmt.Errorf("%s: no builds to hang a hub on", f.repo)
|
||||
}
|
||||
hubAt[hubName] = len(newHubs)
|
||||
newHubs = append(newHubs, renderHub(hubName, f, builds[0], filterVariants(hubName, nil, want)))
|
||||
}
|
||||
|
||||
var inserts []galleryedit.Insert
|
||||
for _, name := range spliceOrder {
|
||||
e := ix.Find(name)
|
||||
items := filterVariants(name, e.Variants, wantByHub[name])
|
||||
if len(items) == 0 {
|
||||
continue
|
||||
}
|
||||
inserts = append(inserts, galleryedit.Insert{Entry: e.Pos, Variants: items})
|
||||
}
|
||||
return inserts, newHubs, nil
|
||||
}
|
||||
|
||||
// hubVariants is a family's full build list, in ladder order, named as the hub
|
||||
// must reference them after the merge.
|
||||
func hubVariants(f *family, ix *IndexText, reused map[string]string, added map[string]bool) []string {
|
||||
var out []string
|
||||
|
||||
// A hand-written *-apex entry is an ordinary build of these weights. It is
|
||||
// never deleted, never renamed and never treated as a hub; it is simply
|
||||
// referenced like any other rung.
|
||||
if apex := slug(f.repoBase); ix.Find(apex) != nil {
|
||||
out = append(out, apex)
|
||||
}
|
||||
for _, c := range f.sortedChildren() {
|
||||
out = append(out, resolveVariant(c.entry.Name, reused, added))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// renderHub builds the hub for a family whose base model the gallery does not
|
||||
// ship at all. It is named for the BASE model, never for the APEX repo.
|
||||
//
|
||||
// It carries one of the discovered builds as its own payload so it is a complete
|
||||
// installable entry rather than a bare index pointing at other entries. That
|
||||
// payload is what supplies overrides.backend, which matters beyond installation:
|
||||
// the verifier can only judge the tagging rule for a backend it can read, so a
|
||||
// hub carrying feature tags and no backend would escape the check in silence.
|
||||
//
|
||||
// The payload's own tags are kept rather than rebuilt from baseTags, so a hub
|
||||
// whose payload configures a spec_type stays tagged for it and consistent with
|
||||
// the overrides copied alongside.
|
||||
func renderHub(name string, f *family, payload childBuild, variants []string) GalleryEntry {
|
||||
e := payload.entry
|
||||
e.Name = name
|
||||
e.Description = fmt.Sprintf(
|
||||
"%s. Quality ladder and quantization rungs published by %s and its unsloth counterpart; LocalAI picks the build that fits the hardware.",
|
||||
HubLabel(f.repoBase, f.stem), f.repo)
|
||||
|
||||
e.Tags = append([]string{}, payload.entry.Tags...)
|
||||
if f.hasMMProj && !hasTag(e.Tags, "vision") {
|
||||
e.Tags = append(e.Tags, "vision")
|
||||
}
|
||||
|
||||
e.Variants = nil
|
||||
for _, v := range variants {
|
||||
e.Variants = append(e.Variants, VariantRef{Model: v})
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func hasTag(tags []string, want string) bool {
|
||||
for _, t := range tags {
|
||||
if t == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// resolveCounterpart probes the unsloth candidates in order and returns the
|
||||
// first that publishes files.
|
||||
//
|
||||
// CounterpartCandidates is handed a BARE repo name: its cleaner does not strip
|
||||
// an owner prefix, so passing "mudler/Foo-APEX-GGUF" would yield "mudler/Foo"
|
||||
// and compose into the nonsense probe "unsloth/mudler/Foo".
|
||||
//
|
||||
// It also returns the candidates HuggingFace refused to describe. Those are
|
||||
// indistinguishable from absent without credentials, so they are skipped, but
|
||||
// they are named rather than dropped: one of them could be a real gated repo
|
||||
// whose quants belong in the gallery.
|
||||
func resolveCounterpart(client *http.Client, repoBase, stem string) (string, []GGUFFile, []string, error) {
|
||||
var unavailable []string
|
||||
for _, cand := range CounterpartCandidates(repoBase, stem) {
|
||||
repo := unslothOwner + "/" + cand + "-GGUF"
|
||||
files, unreadable, err := FetchOptionalRepoFiles(client, repo)
|
||||
if err != nil {
|
||||
return "", nil, unavailable, fmt.Errorf("probing %s: %w", repo, err)
|
||||
}
|
||||
if unreadable {
|
||||
unavailable = append(unavailable, repo)
|
||||
continue
|
||||
}
|
||||
if len(files) > 0 {
|
||||
return repo, files, unavailable, nil
|
||||
}
|
||||
}
|
||||
return "", nil, unavailable, nil
|
||||
}
|
||||
|
||||
// reportUnclassified prints the files discovery turned into nothing.
|
||||
//
|
||||
// It is a set difference on COUNTS, not a re-match of filenames: re-matching
|
||||
// would duplicate the tier regex from discover.go and the two copies would
|
||||
// drift. The likeliest trigger is a typo or case change from a publishing script
|
||||
// rather than a genuine sixth tier, and because generation falls back to the
|
||||
// plain ladder when the imatrix one is empty, a repo whose imatrix files all
|
||||
// fail to match silently downgrades the whole family instead of erroring. The
|
||||
// downstream HTTP check cannot catch that: it validates URLs that were emitted,
|
||||
// and an undiscovered tier emits none.
|
||||
// It returns the census so the run can total it.
|
||||
func reportUnclassified(repo string, files []GGUFFile, imatrix, plain []Tier) fileCensus {
|
||||
mmprojCount, fullPrecision := 0, 0
|
||||
for _, f := range files {
|
||||
// The mmproj test comes first because projectors are themselves often
|
||||
// published at f16 (mmproj-F16.gguf), and counting such a file in both
|
||||
// buckets would understate the unclassified remainder.
|
||||
if strings.HasPrefix(f.Name, "mmproj") {
|
||||
mmprojCount++
|
||||
continue
|
||||
}
|
||||
if IsFullPrecision(f.Name) {
|
||||
fullPrecision++
|
||||
}
|
||||
}
|
||||
|
||||
classified := len(imatrix) + len(plain) + mmprojCount + fullPrecision
|
||||
if classified >= len(files) {
|
||||
return fileCensus{fullPrecision: fullPrecision}
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "UNCLASSIFIED %s: %d of %d .gguf files classified, %d unaccounted for\n",
|
||||
repo, classified, len(files), len(files)-classified)
|
||||
return fileCensus{unclassified: len(files) - classified, fullPrecision: fullPrecision}
|
||||
}
|
||||
|
||||
// reportReuse splits Merge's single reused map into the two cases it conflates.
|
||||
//
|
||||
// A URI match means the gallery already ships exactly these weights, and
|
||||
// pointing the parent at the existing entry is correct. A NAME match with a
|
||||
// different URI means an unrelated entry happens to own the name, and
|
||||
// referencing it would point the parent at different weights than were
|
||||
// generated, substituting a build without saying so. Only the first is safe to
|
||||
// wave through.
|
||||
func reportReuse(existing *ExistingIndex, generated []GalleryEntry, reused map[string]string) {
|
||||
byName := map[string]GalleryEntry{}
|
||||
for _, e := range generated {
|
||||
if _, seen := byName[e.Name]; !seen {
|
||||
byName[e.Name] = e
|
||||
}
|
||||
}
|
||||
|
||||
var nameCollisions, uriMatches []string
|
||||
for name, target := range reused {
|
||||
gen := byName[name]
|
||||
uri := ""
|
||||
if len(gen.Files) > 0 {
|
||||
uri = gen.Files[0].URI
|
||||
}
|
||||
|
||||
switch {
|
||||
case hasName(existing, name):
|
||||
nameCollisions = append(nameCollisions,
|
||||
fmt.Sprintf(" %s -> gallery entry of the same name (generated uri: %s)", name, orNone(uri)))
|
||||
case target == name:
|
||||
nameCollisions = append(nameCollisions,
|
||||
fmt.Sprintf(" %s -> earlier entry of the same name in this batch (generated uri: %s)", name, orNone(uri)))
|
||||
default:
|
||||
uriMatches = append(uriMatches, fmt.Sprintf(" %s -> %s (same weights: %s)", name, target, orNone(uri)))
|
||||
}
|
||||
}
|
||||
sort.Strings(nameCollisions)
|
||||
sort.Strings(uriMatches)
|
||||
|
||||
fmt.Printf("\nNAME COLLISIONS (%d) - inspect each by hand, the target may hold different weights\n", len(nameCollisions))
|
||||
for _, l := range nameCollisions {
|
||||
fmt.Println(l)
|
||||
}
|
||||
fmt.Printf("\nURI MATCHES (%d) - the gallery or this batch already ships these exact weights\n", len(uriMatches))
|
||||
for _, l := range uriMatches {
|
||||
fmt.Println(l)
|
||||
}
|
||||
}
|
||||
|
||||
// reportHubs prints exactly what will be written where. The splices are the part
|
||||
// a human has to read: they modify entries the gallery already ships, so the
|
||||
// review needs the target, the line, and every added reference spelled out.
|
||||
func reportHubs(ix *IndexText, inserts []galleryedit.Insert, newHubs []GalleryEntry) {
|
||||
fmt.Printf("\nHUBS SPLICED (%d) - variants added to the EXISTING base model entry, nothing else touched\n", len(inserts))
|
||||
for _, in := range inserts {
|
||||
e := ix.Find(in.Entry.Name)
|
||||
fmt.Printf(" %s (line %d, %d variant(s) already declared):\n", in.Entry.Name, in.Entry.StartLine+1, len(e.Variants))
|
||||
for _, v := range in.Variants {
|
||||
fmt.Printf(" + - model: %s\n", galleryedit.QuoteName(v))
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("\nHUBS CREATED (%d) - the gallery ships no base model entry, so one is emitted for it\n", len(newHubs))
|
||||
for _, h := range newHubs {
|
||||
fmt.Printf(" %s:\n", h.Name)
|
||||
for _, v := range h.Variants {
|
||||
fmt.Printf(" - model: %s\n", v.Model)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reportSkipped names the counterpart repos HuggingFace would not describe, and
|
||||
// totals the other two silent-shortfall counters alongside them.
|
||||
//
|
||||
// A skipped repo is not the same as a clean 404. HuggingFace answers 401 for a
|
||||
// nonexistent repo to an unauthenticated client, so the overwhelmingly likely
|
||||
// reading is "there is no such counterpart", which is the normal case for the
|
||||
// community merges. But a private or gated repo answers 401 too, and that one
|
||||
// WOULD have quants worth shipping. Printing the list is what keeps that
|
||||
// possibility auditable instead of silently discarded.
|
||||
func reportSkipped(families []family) ([]string, fileCensus, int, int) {
|
||||
var skipped []string
|
||||
var census fileCensus
|
||||
fullPrecisionRepos, unaccounted := 0, 0
|
||||
for _, f := range families {
|
||||
skipped = append(skipped, f.skippedRepos...)
|
||||
census.add(f.census)
|
||||
if f.census.fullPrecision > 0 {
|
||||
fullPrecisionRepos++
|
||||
}
|
||||
unaccounted += f.unaccounted
|
||||
}
|
||||
sort.Strings(skipped)
|
||||
|
||||
fmt.Printf("\nREPOS SKIPPED AS UNAVAILABLE (%d) - HuggingFace answered 401/403, which is indistinguishable from absent without a token; check none of these is a real gated repo\n", len(skipped))
|
||||
for _, r := range skipped {
|
||||
fmt.Printf(" %s\n", r)
|
||||
}
|
||||
return skipped, census, fullPrecisionRepos, unaccounted
|
||||
}
|
||||
|
||||
func hasName(ix *ExistingIndex, name string) bool {
|
||||
_, ok := ix.ByName[name]
|
||||
return ok
|
||||
}
|
||||
|
||||
func orNone(s string) string {
|
||||
if s == "" {
|
||||
return "(no files)"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// writeEntries emits the additions.
|
||||
//
|
||||
// -apply does two things in one pass: it writes back the spliced lines, which
|
||||
// differ from the original only by the variant lines galleryedit inserted, and
|
||||
// then appends the new entries. New entries are APPENDED rather than merged into
|
||||
// the structure, for the same reason the splice is textual: a YAML round trip
|
||||
// over 40,000 lines would reflow the whole file into an unreviewable diff.
|
||||
func writeEntries(add []GalleryEntry, lines []string, outPath string, apply bool, indexPath string) error {
|
||||
if apply {
|
||||
if err := os.WriteFile(indexPath, []byte(strings.Join(lines, "\n")), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("spliced %s\n", indexPath)
|
||||
}
|
||||
|
||||
if len(add) == 0 {
|
||||
fmt.Println("nothing to append")
|
||||
return nil
|
||||
}
|
||||
|
||||
blob, err := yaml.Marshal(add)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if outPath != "" {
|
||||
if err := os.WriteFile(outPath, blob, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("wrote %d entries to %s\n", len(add), outPath)
|
||||
}
|
||||
|
||||
if apply {
|
||||
f, err := os.OpenFile(indexPath, os.O_APPEND|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := f.Write(blob); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("appended %d entries to %s\n", len(add), indexPath)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// listAPEXRepos returns the mudler repos whose name marks them as APEX builds.
|
||||
func listAPEXRepos(client *http.Client) ([]string, error) {
|
||||
req, err := http.NewRequest(http.MethodGet, authorListURL, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("User-Agent", "localai-apexentries/1.0")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("listing models: unexpected status %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var models []struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &models); err != nil {
|
||||
return nil, fmt.Errorf("decoding model list: %w", err)
|
||||
}
|
||||
|
||||
var out []string
|
||||
for _, m := range models {
|
||||
if strings.Contains(m.ID, "APEX") {
|
||||
out = append(out, m.ID)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func restrict(repos []string, only string) []string {
|
||||
want := map[string]bool{}
|
||||
for _, r := range strings.Split(only, ",") {
|
||||
if r = strings.TrimSpace(r); r != "" {
|
||||
want[r] = true
|
||||
}
|
||||
}
|
||||
|
||||
var out []string
|
||||
for _, r := range repos {
|
||||
if want[r] {
|
||||
out = append(out, r)
|
||||
delete(want, r)
|
||||
}
|
||||
}
|
||||
// A name in -only that matched nothing is a typo, not an empty result.
|
||||
for r := range want {
|
||||
fmt.Fprintf(os.Stderr, "WARNING: -only names %s, which is not an APEX repo of this author\n", r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sortTiers(tiers []Tier) {
|
||||
sort.SliceStable(tiers, func(i, j int) bool { return rungRank[tiers[i].Label] < rungRank[tiers[j].Label] })
|
||||
}
|
||||
|
||||
func tierLabels(tiers []Tier) string {
|
||||
var out []string
|
||||
for _, t := range tiers {
|
||||
out = append(out, t.Label)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
func quantLabels(builds []QuantBuild) string {
|
||||
var out []string
|
||||
for _, b := range builds {
|
||||
l := b.Quant
|
||||
if b.Sharded {
|
||||
l += fmt.Sprintf("(%d shards)", len(b.Files))
|
||||
}
|
||||
out = append(out, l)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
// slug turns a repo, tier or quant label into a gallery entry name component.
|
||||
func slug(s string) string {
|
||||
return strings.ReplaceAll(strings.ToLower(s), "_", "-")
|
||||
}
|
||||
344
.github/ci/apexentries/main_test.go
vendored
344
.github/ci/apexentries/main_test.go
vendored
@@ -1,344 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/.github/ci/galleryedit"
|
||||
)
|
||||
|
||||
func mustIndex(text string) *IndexText {
|
||||
ix, err := ParseIndexText(text)
|
||||
ExpectWithOffset(1, err).ToNot(HaveOccurred())
|
||||
return ix
|
||||
}
|
||||
|
||||
// buildOf renders a realistic child so the specs exercise the payload a hub
|
||||
// actually inherits rather than a bare name.
|
||||
func buildOf(name, repo, file string, rank int) childBuild {
|
||||
return childBuild{
|
||||
rank: rank,
|
||||
entry: RenderChild(ChildInput{
|
||||
Name: name,
|
||||
Repo: repo,
|
||||
Template: entryTemplate,
|
||||
Weights: []GGUFFile{{Name: file, SHA256: "aa"}},
|
||||
BaseTags: baseTags,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
var _ = Describe("ResolveHub", func() {
|
||||
It("picks the base model name over the APEX name, even when both are in the gallery", func() {
|
||||
// The hub is the entry a user searches for. If the *-apex entry were
|
||||
// chosen the family would be gathered under a name nobody looks up, and
|
||||
// the base entry would go on advertising only its own build.
|
||||
ix := mustIndex("- name: qwen3.6-35b-a3b\n url: u\n- name: qwen3.6-35b-a3b-apex\n url: u\n")
|
||||
|
||||
name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX", "Qwen3.6-35B-A3B-APEX")
|
||||
|
||||
Expect(name).To(Equal("qwen3.6-35b-a3b"))
|
||||
Expect(exists).To(BeTrue())
|
||||
})
|
||||
|
||||
It("falls back to the stem-derived candidate when the repo-derived one is absent", func() {
|
||||
// gemma's repo says "-it" and its published files do not, so only one of
|
||||
// the two candidates can match whatever the base entry was named after.
|
||||
ix := mustIndex("- name: gemma-4-26b-a4b\n url: u\n")
|
||||
|
||||
name, exists := ResolveHub(ix, "gemma-4-26B-A4B-it-APEX", "gemma-4-26B-A4B-APEX")
|
||||
|
||||
Expect(name).To(Equal("gemma-4-26b-a4b"))
|
||||
Expect(exists).To(BeTrue())
|
||||
})
|
||||
|
||||
It("reports the base name as absent rather than settling for the APEX entry", func() {
|
||||
ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n url: u\n")
|
||||
|
||||
name, exists := ResolveHub(ix, "Qwen3.5-35B-A3B-APEX", "Qwen3.5-35B-A3B-APEX")
|
||||
|
||||
Expect(name).To(Equal("qwen3.5-35b-a3b"))
|
||||
Expect(exists).To(BeFalse())
|
||||
})
|
||||
|
||||
It("strips the MTP and TQ markers as well as APEX", func() {
|
||||
ix := mustIndex("- name: qwen3.6-35b-a3b\n url: u\n")
|
||||
|
||||
name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX-MTP", "Qwen3.6-35B-A3B-APEX-MTP")
|
||||
|
||||
Expect(name).To(Equal("qwen3.6-35b-a3b"))
|
||||
Expect(exists).To(BeTrue())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("planHubs", func() {
|
||||
noReuse := map[string]string{}
|
||||
allAdded := func(names ...string) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
for _, n := range names {
|
||||
out[n] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
It("splices into the existing base entry instead of emitting an *-apex parent", func() {
|
||||
ix := mustIndex("- name: step-3.7-flash\n url: u\n- name: other\n url: u\n")
|
||||
fams := []family{{
|
||||
repo: "mudler/Step-3.7-Flash-APEX-GGUF",
|
||||
repoBase: "Step-3.7-Flash-APEX",
|
||||
stem: "Step-3.7-Flash-APEX",
|
||||
children: []childBuild{buildOf("step-3.7-flash-apex-i-quality", "mudler/Step-3.7-Flash-APEX-GGUF", "a.gguf", 0)},
|
||||
}}
|
||||
|
||||
inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("step-3.7-flash-apex-i-quality"))
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(newHubs).To(BeEmpty())
|
||||
Expect(inserts).To(HaveLen(1))
|
||||
Expect(inserts[0].Entry.Name).To(Equal("step-3.7-flash"))
|
||||
Expect(inserts[0].Variants).To(Equal([]string{"step-3.7-flash-apex-i-quality"}))
|
||||
})
|
||||
|
||||
It("merges into an entry that already declares variants, without repeating one", func() {
|
||||
// The gallery's qwen3.6-35b-a3b already lists its APEX build. Re-adding it
|
||||
// would put a duplicate key's worth of noise in the diff and a duplicate
|
||||
// reference in the entry.
|
||||
ix := mustIndex("- name: qwen3.6-35b-a3b\n variants:\n - model: qwen3.6-35b-a3b-apex\n url: u\n" +
|
||||
"- name: qwen3.6-35b-a3b-apex\n url: u\n")
|
||||
fams := []family{{
|
||||
repo: "mudler/Qwen3.6-35B-A3B-APEX-GGUF",
|
||||
repoBase: "Qwen3.6-35B-A3B-APEX",
|
||||
stem: "Qwen3.6-35B-A3B-APEX",
|
||||
children: []childBuild{buildOf("qwen3.6-35b-a3b-apex-i-quality", "mudler/Qwen3.6-35B-A3B-APEX-GGUF", "a.gguf", 0)},
|
||||
}}
|
||||
|
||||
inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("qwen3.6-35b-a3b-apex-i-quality"))
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(newHubs).To(BeEmpty())
|
||||
Expect(inserts[0].Variants).To(Equal([]string{"qwen3.6-35b-a3b-apex-i-quality"}))
|
||||
|
||||
out, err := galleryedit.Apply(ix.Lines, inserts)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(strings.Count(strings.Join(out, "\n"), "variants:")).To(Equal(1))
|
||||
Expect(out).To(HaveLen(len(ix.Lines) + 1))
|
||||
})
|
||||
|
||||
It("never lets the hub reference itself", func() {
|
||||
// An unsloth rung whose weights the gallery already ships under the base
|
||||
// name resolves, through Merge, straight back to the hub. The verifier
|
||||
// reads a self reference as a variant that declares variants of its own.
|
||||
ix := mustIndex("- name: step-3.7-flash\n url: u\n")
|
||||
fams := []family{{
|
||||
repo: "mudler/Step-3.7-Flash-APEX-GGUF",
|
||||
repoBase: "Step-3.7-Flash-APEX",
|
||||
stem: "Step-3.7-Flash-APEX",
|
||||
children: []childBuild{buildOf("step-3.7-flash-ud-q4-k-m", "unsloth/Step-3.7-Flash-GGUF", "a.gguf", 100)},
|
||||
}}
|
||||
|
||||
inserts, _, err := planHubs(fams, ix, map[string]string{"step-3.7-flash-ud-q4-k-m": "step-3.7-flash"}, map[string]bool{})
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(inserts).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("emits a hub named for the base model when the gallery has none", func() {
|
||||
ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n url: u\n")
|
||||
fams := []family{{
|
||||
repo: "mudler/Qwen3.5-35B-A3B-APEX-GGUF",
|
||||
repoBase: "Qwen3.5-35B-A3B-APEX",
|
||||
stem: "Qwen3.5-35B-A3B-APEX",
|
||||
hasMMProj: true,
|
||||
children: []childBuild{
|
||||
buildOf("qwen3.5-35b-a3b-apex-i-quality", "mudler/Qwen3.5-35B-A3B-APEX-GGUF", "a.gguf", 0),
|
||||
buildOf("qwen3.5-35b-a3b-ud-q6-k", "unsloth/Qwen3.5-35B-A3B-GGUF", "b.gguf", 102),
|
||||
},
|
||||
}}
|
||||
|
||||
inserts, newHubs, err := planHubs(fams, ix, noReuse,
|
||||
allAdded("qwen3.5-35b-a3b-apex-i-quality", "qwen3.5-35b-a3b-ud-q6-k"))
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(inserts).To(BeEmpty())
|
||||
Expect(newHubs).To(HaveLen(1))
|
||||
|
||||
hub := newHubs[0]
|
||||
Expect(hub.Name).To(Equal("qwen3.5-35b-a3b"))
|
||||
Expect(hub.Name).ToNot(HaveSuffix("-apex"))
|
||||
|
||||
// A hand-written *-apex entry is an ordinary build, referenced like any
|
||||
// other rung and never deleted or renamed.
|
||||
Expect(hub.Variants).To(Equal([]VariantRef{
|
||||
{Model: "qwen3.5-35b-a3b-apex"},
|
||||
{Model: "qwen3.5-35b-a3b-apex-i-quality"},
|
||||
{Model: "qwen3.5-35b-a3b-ud-q6-k"},
|
||||
}))
|
||||
|
||||
// The verifier skips entries with no declared backend, so a hub without
|
||||
// one would escape the tagging check in silence.
|
||||
Expect(hub.Overrides).To(HaveKeyWithValue("backend", "llama-cpp"))
|
||||
Expect(hub.Files).ToNot(BeEmpty())
|
||||
Expect(hub.Tags).To(ContainElement("vision"))
|
||||
})
|
||||
|
||||
It("gathers two APEX repos that share one base model under a single hub", func() {
|
||||
ix := mustIndex("- name: unrelated\n url: u\n")
|
||||
fams := []family{
|
||||
{
|
||||
repo: "mudler/Solo-APEX-GGUF",
|
||||
repoBase: "Solo-APEX",
|
||||
stem: "Solo-APEX",
|
||||
children: []childBuild{buildOf("solo-apex-i-quality", "mudler/Solo-APEX-GGUF", "a.gguf", 0)},
|
||||
},
|
||||
{
|
||||
repo: "mudler/Solo-APEX-MTP-GGUF",
|
||||
repoBase: "Solo-APEX-MTP",
|
||||
stem: "Solo-APEX-MTP",
|
||||
children: []childBuild{buildOf("solo-apex-mtp-i-quality", "mudler/Solo-APEX-MTP-GGUF", "b.gguf", 0)},
|
||||
},
|
||||
}
|
||||
|
||||
_, newHubs, err := planHubs(fams, ix, noReuse, allAdded("solo-apex-i-quality", "solo-apex-mtp-i-quality"))
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(newHubs).To(HaveLen(1))
|
||||
Expect(newHubs[0].Name).To(Equal("solo"))
|
||||
Expect(newHubs[0].Variants).To(Equal([]VariantRef{
|
||||
{Model: "solo-apex-i-quality"},
|
||||
{Model: "solo-apex-mtp-i-quality"},
|
||||
}))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("hubVariants", func() {
|
||||
It("orders builds by quality rung rather than discovery order", func() {
|
||||
// DiscoverAPEXTiers preserves input order and the HF API returns siblings
|
||||
// alphabetically, so an unsorted list reads I-Balanced, I-Compact, I-Mini,
|
||||
// I-Nano, I-Quality. Selection ignores authored order; this is for the
|
||||
// human reading the file.
|
||||
f := family{repoBase: "X-APEX", stem: "X-APEX", children: []childBuild{
|
||||
{rank: rungRank["I-Nano"], entry: GalleryEntry{Name: "x-i-nano"}},
|
||||
{rank: 100, entry: GalleryEntry{Name: "x-ud-q4-k-m"}},
|
||||
{rank: rungRank["I-Quality"], entry: GalleryEntry{Name: "x-i-quality"}},
|
||||
{rank: rungRank["I-Compact"], entry: GalleryEntry{Name: "x-i-compact"}},
|
||||
}}
|
||||
|
||||
got := hubVariants(&f, mustIndex("- name: x\n url: u\n"), map[string]string{}, map[string]bool{})
|
||||
|
||||
Expect(got).To(Equal([]string{"x-i-quality", "x-i-compact", "x-i-nano", "x-ud-q4-k-m"}))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("ParseIndexText", func() {
|
||||
It("refuses to edit by line number when the two views of the file disagree", func() {
|
||||
_, err := ParseIndexText("- name: one\n url: u\n-\n")
|
||||
Expect(err).To(MatchError(ContainSubstring("empty")))
|
||||
})
|
||||
|
||||
It("records the line range of each entry", func() {
|
||||
ix := mustIndex("- name: first\n url: u\n- name: second\n url: u\n")
|
||||
|
||||
Expect(ix.Find("FIRST").Pos.StartLine).To(Equal(0))
|
||||
Expect(ix.Find("first").Pos.EndLine).To(Equal(2))
|
||||
Expect(ix.Find("second").Pos.StartLine).To(Equal(2))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("resolveVariant", func() {
|
||||
It("keeps an entry that was emitted even when it is also in reused", func() {
|
||||
// A within-batch name collision records reused[name] = name while the
|
||||
// FIRST entry of that name is still in add. Treating presence in reused as
|
||||
// "dropped" would emit nothing for it.
|
||||
added := map[string]bool{"dup": true}
|
||||
reused := map[string]string{"dup": "dup"}
|
||||
|
||||
Expect(resolveVariant("dup", reused, added)).To(Equal("dup"))
|
||||
})
|
||||
|
||||
It("redirects a reused name at the entry that stands in for it", func() {
|
||||
added := map[string]bool{}
|
||||
reused := map[string]string{"generated": "already-in-gallery"}
|
||||
|
||||
Expect(resolveVariant("generated", reused, added)).To(Equal("already-in-gallery"))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("slug", func() {
|
||||
It("lowercases and turns quant underscores into hyphens", func() {
|
||||
Expect(slug("UD-Q4_K_M")).To(Equal("ud-q4-k-m"))
|
||||
Expect(slug("gemma-4-26B-A4B-it-APEX")).To(Equal("gemma-4-26b-a4b-it-apex"))
|
||||
Expect(slug("I-Nano")).To(Equal("i-nano"))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("sortTiers", func() {
|
||||
It("puts the imatrix ladder in descending quality order", func() {
|
||||
tiers := []Tier{
|
||||
{Label: "I-Balanced"}, {Label: "I-Compact"}, {Label: "I-Mini"},
|
||||
{Label: "I-Nano"}, {Label: "I-Quality"},
|
||||
}
|
||||
sortTiers(tiers)
|
||||
Expect(tierLabels(tiers)).To(Equal("I-Quality,I-Balanced,I-Compact,I-Mini,I-Nano"))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("restrict", func() {
|
||||
It("keeps only the named repos", func() {
|
||||
got := restrict([]string{"mudler/A-APEX-GGUF", "mudler/B-APEX-GGUF"}, "mudler/B-APEX-GGUF")
|
||||
Expect(got).To(Equal([]string{"mudler/B-APEX-GGUF"}))
|
||||
})
|
||||
|
||||
It("returns nothing when the filter matches nothing", func() {
|
||||
Expect(restrict([]string{"mudler/A-APEX-GGUF"}, "mudler/typo")).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("reportUnclassified", func() {
|
||||
// One real imatrix rung is always present so the specs measure how the
|
||||
// remaining files are bucketed, not an empty-repo edge case.
|
||||
tier := Tier{Label: "I-Quality", File: GGUFFile{Name: "Model-APEX-I-Quality.gguf"}}
|
||||
|
||||
censusOf := func(names ...string) fileCensus {
|
||||
files := []GGUFFile{tier.File}
|
||||
for _, n := range names {
|
||||
files = append(files, GGUFFile{Name: n})
|
||||
}
|
||||
return reportUnclassified("mudler/Model-APEX-GGUF", files, []Tier{tier}, nil)
|
||||
}
|
||||
|
||||
It("counts a flat full-precision source as excluded, not unclassified", func() {
|
||||
got := censusOf("Carnice-MoE-35B-A3B-F16.gguf")
|
||||
Expect(got.fullPrecision).To(Equal(1))
|
||||
Expect(got.unclassified).To(Equal(0))
|
||||
})
|
||||
|
||||
It("counts every shard of a sharded full-precision source as excluded", func() {
|
||||
got := censusOf(
|
||||
"MiniMax-M2.7-APEX-F16-00001-of-00003.gguf",
|
||||
"MiniMax-M2.7-APEX-F16-00002-of-00003.gguf",
|
||||
"MiniMax-M2.7-APEX-F16-00003-of-00003.gguf",
|
||||
)
|
||||
Expect(got.fullPrecision).To(Equal(3))
|
||||
Expect(got.unclassified).To(Equal(0))
|
||||
})
|
||||
|
||||
It("treats bf16 the same as f16, in either case", func() {
|
||||
got := censusOf("Model-APEX-BF16.gguf", "Model-APEX-bf16-00001-of-00002.gguf", "Model-APEX-f16.gguf")
|
||||
Expect(got.fullPrecision).To(Equal(3))
|
||||
Expect(got.unclassified).To(Equal(0))
|
||||
})
|
||||
|
||||
It("still reports a genuinely unknown filename as unclassified", func() {
|
||||
got := censusOf("Model-APEX-Turbo.gguf")
|
||||
Expect(got.unclassified).To(Equal(1))
|
||||
Expect(got.fullPrecision).To(Equal(0))
|
||||
})
|
||||
|
||||
It("separates the two kinds when a repo publishes both", func() {
|
||||
got := censusOf("Model-APEX-F16.gguf", "Model-APEX-Turbo.gguf")
|
||||
Expect(got.fullPrecision).To(Equal(1))
|
||||
Expect(got.unclassified).To(Equal(1))
|
||||
})
|
||||
})
|
||||
143
.github/ci/apexentries/merge.go
vendored
143
.github/ci/apexentries/merge.go
vendored
@@ -1,143 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
const (
|
||||
hfShorthandPrefix = "huggingface://"
|
||||
hfResolvePrefix = "https://huggingface.co/"
|
||||
hfResolveInfix = "/resolve/main/"
|
||||
)
|
||||
|
||||
// canonicalURI reduces the two interchangeable spellings of a HuggingFace file
|
||||
// to one key, so a generated resolve/main URI dedups against the shorthand the
|
||||
// gallery uses for the majority of its entries.
|
||||
//
|
||||
// The repo is exactly the first two path segments; everything after is the file
|
||||
// path, which may itself contain slashes because sharded quants live in a
|
||||
// subdirectory. Anything that is not recognisably one of the two forms is
|
||||
// returned unchanged rather than guessed at, so mirrors and other hosts still
|
||||
// dedup on their literal string.
|
||||
func canonicalURI(uri string) string {
|
||||
switch {
|
||||
case strings.HasPrefix(uri, hfShorthandPrefix):
|
||||
rest := strings.TrimPrefix(uri, hfShorthandPrefix)
|
||||
owner, after, ok := strings.Cut(rest, "/")
|
||||
if !ok {
|
||||
return uri
|
||||
}
|
||||
name, file, ok := strings.Cut(after, "/")
|
||||
if !ok || owner == "" || name == "" || file == "" {
|
||||
return uri
|
||||
}
|
||||
return hfShorthandPrefix + owner + "/" + name + "/" + file
|
||||
|
||||
case strings.HasPrefix(uri, hfResolvePrefix):
|
||||
rest := strings.TrimPrefix(uri, hfResolvePrefix)
|
||||
repo, file, ok := strings.Cut(rest, hfResolveInfix)
|
||||
if !ok || file == "" {
|
||||
return uri
|
||||
}
|
||||
// A repo is owner/name and nothing more; a longer prefix means this is
|
||||
// some other huggingface.co URL that must not be rewritten.
|
||||
owner, name, ok := strings.Cut(repo, "/")
|
||||
if !ok || owner == "" || name == "" || strings.Contains(name, "/") {
|
||||
return uri
|
||||
}
|
||||
return hfShorthandPrefix + repo + "/" + file
|
||||
|
||||
default:
|
||||
return uri
|
||||
}
|
||||
}
|
||||
|
||||
// ExistingIndex is the lookup built from the current gallery: entry names, and
|
||||
// which entry claims each weight URI.
|
||||
type ExistingIndex struct {
|
||||
ByName map[string]int
|
||||
ByURI map[string]string
|
||||
}
|
||||
|
||||
// LoadExisting reads the gallery index for dedup purposes only. It is
|
||||
// deliberately not used to rewrite the file: the index is 40,000 lines, and a
|
||||
// YAML round trip would reflow the whole thing into an unreviewable diff.
|
||||
func LoadExisting(path string) (*ExistingIndex, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var entries []struct {
|
||||
Name string `yaml:"name"`
|
||||
Files []struct {
|
||||
URI string `yaml:"uri"`
|
||||
} `yaml:"files"`
|
||||
}
|
||||
if err := yaml.Unmarshal(raw, &entries); err != nil {
|
||||
return nil, fmt.Errorf("parsing %s: %w", path, err)
|
||||
}
|
||||
|
||||
ix := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
|
||||
for i, e := range entries {
|
||||
ix.ByName[e.Name] = i
|
||||
for _, f := range e.Files {
|
||||
if f.URI != "" {
|
||||
ix.ByURI[canonicalURI(f.URI)] = e.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return ix, nil
|
||||
}
|
||||
|
||||
// Merge splits generated entries into those to add and those already covered.
|
||||
// reused maps a generated name to the existing entry that stands in for it, so
|
||||
// a parent can reference what is already there instead of duplicating weights.
|
||||
// Several APEX repos share one base model, so the same counterpart rungs are
|
||||
// generated more than once in a batch. The batch has to dedup against itself as
|
||||
// well as against the gallery, tracked locally because the caller may reuse the
|
||||
// ExistingIndex it passed in.
|
||||
func Merge(existing *ExistingIndex, generated []GalleryEntry) (add []GalleryEntry, reused map[string]string) {
|
||||
reused = map[string]string{}
|
||||
batchNames := map[string]string{}
|
||||
batchURIs := map[string]string{}
|
||||
|
||||
// Canonicalized into a local copy rather than in place: an ExistingIndex may
|
||||
// be hand-built or reused by the caller, so Merge must not rewrite it.
|
||||
existingURIs := make(map[string]string, len(existing.ByURI))
|
||||
for uri, owner := range existing.ByURI {
|
||||
existingURIs[canonicalURI(uri)] = owner
|
||||
}
|
||||
|
||||
for _, e := range generated {
|
||||
// Name is checked before URI: a name collision must block the add
|
||||
// whatever the weights say, since duplicate names corrupt the index.
|
||||
if _, clash := existing.ByName[e.Name]; clash {
|
||||
reused[e.Name] = e.Name
|
||||
continue
|
||||
}
|
||||
if claimant, clash := batchNames[e.Name]; clash {
|
||||
reused[e.Name] = claimant
|
||||
continue
|
||||
}
|
||||
if len(e.Files) > 0 {
|
||||
uri := canonicalURI(e.Files[0].URI)
|
||||
if owner, ok := existingURIs[uri]; ok {
|
||||
reused[e.Name] = owner
|
||||
continue
|
||||
}
|
||||
if claimant, ok := batchURIs[uri]; ok {
|
||||
reused[e.Name] = claimant
|
||||
continue
|
||||
}
|
||||
batchURIs[uri] = e.Name
|
||||
}
|
||||
batchNames[e.Name] = e.Name
|
||||
add = append(add, e)
|
||||
}
|
||||
return add, reused
|
||||
}
|
||||
183
.github/ci/apexentries/merge_test.go
vendored
183
.github/ci/apexentries/merge_test.go
vendored
@@ -1,183 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Merge", func() {
|
||||
It("drops a generated entry whose weight URI already exists and reports the existing name", func() {
|
||||
existing := &ExistingIndex{
|
||||
ByName: map[string]int{"qwen3.6-35b-a3b-apex": 0},
|
||||
ByURI: map[string]string{
|
||||
"https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf": "qwen3.6-35b-a3b-apex",
|
||||
},
|
||||
}
|
||||
gen := []GalleryEntry{{
|
||||
Name: "x-apex-i-quality",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf"}},
|
||||
}}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(BeEmpty())
|
||||
Expect(reused).To(HaveKeyWithValue("x-apex-i-quality", "qwen3.6-35b-a3b-apex"))
|
||||
})
|
||||
|
||||
It("keeps a generated entry whose weights are new", func() {
|
||||
existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
|
||||
gen := []GalleryEntry{{
|
||||
Name: "x-apex-i-mini",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Mini.gguf"}},
|
||||
}}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(HaveLen(1))
|
||||
Expect(reused).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("refuses to add an entry whose name collides with an existing one", func() {
|
||||
existing := &ExistingIndex{
|
||||
ByName: map[string]int{"x-apex-i-mini": 0},
|
||||
ByURI: map[string]string{},
|
||||
}
|
||||
gen := []GalleryEntry{{
|
||||
Name: "x-apex-i-mini",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/other.gguf"}},
|
||||
}}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(BeEmpty())
|
||||
Expect(reused).To(HaveKeyWithValue("x-apex-i-mini", "x-apex-i-mini"))
|
||||
})
|
||||
|
||||
// The gallery records most of its URIs in huggingface:// shorthand while
|
||||
// render.go only ever emits the resolve/main form, so without
|
||||
// canonicalization the majority of the file is invisible to the dedup.
|
||||
It("matches a generated https URI against the shorthand form recorded in the gallery", func() {
|
||||
existing := &ExistingIndex{
|
||||
ByName: map[string]int{"foo-gguf-q8-0": 0},
|
||||
ByURI: map[string]string{
|
||||
"huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf": "foo-gguf-q8-0",
|
||||
},
|
||||
}
|
||||
gen := []GalleryEntry{{
|
||||
Name: "foo-apex-q8-0",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf"}},
|
||||
}}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(BeEmpty())
|
||||
Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0"))
|
||||
})
|
||||
|
||||
It("matches a generated shorthand URI against the https form recorded in the gallery", func() {
|
||||
existing := &ExistingIndex{
|
||||
ByName: map[string]int{"foo-gguf-q8-0": 0},
|
||||
ByURI: map[string]string{
|
||||
"https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf": "foo-gguf-q8-0",
|
||||
},
|
||||
}
|
||||
gen := []GalleryEntry{{
|
||||
Name: "foo-apex-q8-0",
|
||||
Files: []EntryFile{{URI: "huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf"}},
|
||||
}}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(BeEmpty())
|
||||
Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0"))
|
||||
})
|
||||
|
||||
// Sharded quants live under a subdirectory, so the file path carries slashes
|
||||
// of its own and only the first two segments are the repo.
|
||||
It("matches across both forms when the file path has a subdirectory", func() {
|
||||
existing := &ExistingIndex{
|
||||
ByName: map[string]int{"model-ud-q4-k-m": 0},
|
||||
ByURI: map[string]string{
|
||||
"huggingface://unsloth/Model-GGUF/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf": "model-ud-q4-k-m",
|
||||
},
|
||||
}
|
||||
gen := []GalleryEntry{{
|
||||
Name: "model-apex-ud-q4-k-m",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Model-GGUF/resolve/main/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf"}},
|
||||
}}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(BeEmpty())
|
||||
Expect(reused).To(HaveKeyWithValue("model-apex-ud-q4-k-m", "model-ud-q4-k-m"))
|
||||
})
|
||||
|
||||
// Several APEX repos share one base model, so the same unsloth rungs are
|
||||
// generated more than once in a single batch.
|
||||
It("adds only the first of two generated entries sharing a name", func() {
|
||||
existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
|
||||
gen := []GalleryEntry{
|
||||
{
|
||||
Name: "shared-rung-q8-0",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}},
|
||||
},
|
||||
{
|
||||
Name: "shared-rung-q8-0",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Other-GGUF/resolve/main/Other-Q8_0.gguf"}},
|
||||
},
|
||||
}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(HaveLen(1))
|
||||
Expect(add[0].Files[0].URI).To(Equal("https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"))
|
||||
Expect(reused).To(HaveKeyWithValue("shared-rung-q8-0", "shared-rung-q8-0"))
|
||||
})
|
||||
|
||||
It("adds only the first of two generated entries sharing a primary URI", func() {
|
||||
existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}}
|
||||
gen := []GalleryEntry{
|
||||
{
|
||||
Name: "shared-rung-from-apex",
|
||||
Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}},
|
||||
},
|
||||
{
|
||||
Name: "shared-rung-from-apex-mtp",
|
||||
Files: []EntryFile{{URI: "huggingface://unsloth/Shared-GGUF/Shared-Q8_0.gguf"}},
|
||||
},
|
||||
}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(HaveLen(1))
|
||||
Expect(add[0].Name).To(Equal("shared-rung-from-apex"))
|
||||
Expect(reused).To(HaveKeyWithValue("shared-rung-from-apex-mtp", "shared-rung-from-apex"))
|
||||
})
|
||||
|
||||
// Anything that is not a HuggingFace URI must survive untouched, so an
|
||||
// unrecognised scheme still dedups against the very same string.
|
||||
It("leaves a URI in neither recognised form alone and still dedups it exactly", func() {
|
||||
existing := &ExistingIndex{
|
||||
ByName: map[string]int{"mirrored-model": 0},
|
||||
ByURI: map[string]string{
|
||||
"https://mirror.example.com/weights/Model-Q8_0.gguf": "mirrored-model",
|
||||
},
|
||||
}
|
||||
gen := []GalleryEntry{
|
||||
{
|
||||
Name: "mirrored-apex",
|
||||
Files: []EntryFile{{URI: "https://mirror.example.com/weights/Model-Q8_0.gguf"}},
|
||||
},
|
||||
{
|
||||
Name: "elsewhere-apex",
|
||||
Files: []EntryFile{{URI: "https://mirror.example.com/weights/Other-Q8_0.gguf"}},
|
||||
},
|
||||
}
|
||||
|
||||
add, reused := Merge(existing, gen)
|
||||
|
||||
Expect(add).To(HaveLen(1))
|
||||
Expect(add[0].Name).To(Equal("elsewhere-apex"))
|
||||
Expect(reused).To(HaveKeyWithValue("mirrored-apex", "mirrored-model"))
|
||||
})
|
||||
})
|
||||
175
.github/ci/apexentries/render.go
vendored
175
.github/ci/apexentries/render.go
vendored
@@ -1,175 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// EntryFile is one downloadable file of a gallery entry.
|
||||
type EntryFile struct {
|
||||
Filename string `yaml:"filename"`
|
||||
SHA256 string `yaml:"sha256"`
|
||||
URI string `yaml:"uri"`
|
||||
}
|
||||
|
||||
// GalleryEntry is the subset of a gallery entry this generator writes.
|
||||
//
|
||||
// Named GalleryEntry rather than Entry because the test files dot-import
|
||||
// Ginkgo, whose table DSL exports an Entry that a package-level Entry would
|
||||
// collide with. The yaml tags are what the gallery index sees, so the Go
|
||||
// identifier is free to differ.
|
||||
type GalleryEntry struct {
|
||||
Name string `yaml:"name"`
|
||||
URL string `yaml:"url"`
|
||||
Description string `yaml:"description,omitempty"`
|
||||
Tags []string `yaml:"tags,omitempty"`
|
||||
Overrides map[string]any `yaml:"overrides,omitempty"`
|
||||
Files []EntryFile `yaml:"files,omitempty"`
|
||||
Variants []VariantRef `yaml:"variants,omitempty"`
|
||||
}
|
||||
|
||||
// VariantRef mirrors the gallery's variant reference: a name and nothing else.
|
||||
type VariantRef struct {
|
||||
Model string `yaml:"model"`
|
||||
}
|
||||
|
||||
// ChildInput is everything needed to render one non-parent entry.
|
||||
type ChildInput struct {
|
||||
Name string
|
||||
Repo string
|
||||
// DraftRepo is the repo publishing the drafter, when it is not the repo
|
||||
// publishing the weights. Speculative pairings routinely cross repos, so
|
||||
// the drafter cannot be assumed to sit next to the weights. Empty means
|
||||
// same-repo, which is how the *-APEX-MTP-GGUF repos ship.
|
||||
DraftRepo string
|
||||
Template string
|
||||
Weights []GGUFFile
|
||||
MMProj *GGUFFile
|
||||
SpecType string
|
||||
DraftFile *GGUFFile
|
||||
BaseTags []string
|
||||
}
|
||||
|
||||
// specTuning is the acceptance-window tuning each spec type ships with, copied
|
||||
// from the hand-written entries that already run these two mechanisms rather
|
||||
// than invented here. The two differ because the drafters differ: self-drafted
|
||||
// MTP heads produce a short, high-confidence proposal (15+ hand-written entries
|
||||
// use 6 with a 0.75 floor), while a separate DFlash drafter is cheap enough to
|
||||
// run far ahead unconditionally (the five hand-written dflash entries use 15 and
|
||||
// set no floor).
|
||||
var specTuning = map[string][]string{
|
||||
"draft-mtp": {"spec_n_max:6", "spec_p_min:0.75"},
|
||||
"draft-dflash": {"spec_n_max:15"},
|
||||
}
|
||||
|
||||
func hfURI(repo, file string) string {
|
||||
return fmt.Sprintf("https://huggingface.co/%s/resolve/main/%s", repo, file)
|
||||
}
|
||||
|
||||
// localPath is where a downloaded file lands.
|
||||
//
|
||||
// The hand-written entries namespace by the repo's BARE name
|
||||
// (llama-cpp/models/<repo>/<file>), which is not unique. LiquidAI/LFM2.5-8B-A1B-GGUF
|
||||
// and unsloth/LFM2.5-8B-A1B-GGUF share a basename, so both claim
|
||||
// llama-cpp/models/LFM2.5-8B-A1B-GGUF/, and installing the second after the first
|
||||
// either overwrites weights whose recorded sha256 belongs to the other file or is
|
||||
// skipped as already present. Two owners publishing the same model name is the
|
||||
// normal case for quantizers, not an edge case, so the owner has to be in the path.
|
||||
//
|
||||
// The owner becomes its own path segment rather than being folded into the
|
||||
// directory name: owner/repo is unique on HuggingFace and "/" cannot occur inside
|
||||
// either half, so this is the only form that is collision-proof by construction.
|
||||
// It still reads as the hand-written convention with the owner restored, and the
|
||||
// extra depth is already present in the index for sharded builds.
|
||||
func localPath(kind, repo, file string) string {
|
||||
// path.Dir yields "." for a repo named without an owner, which path.Join
|
||||
// drops, so such a caller keeps the historical two-segment layout.
|
||||
return path.Join("llama-cpp", kind, path.Dir(repo), path.Base(repo), file)
|
||||
}
|
||||
|
||||
// RenderChild builds one child entry.
|
||||
//
|
||||
// The dflash/mtp tag is added if and only if this entry sets a spec_type,
|
||||
// because variant ranking reads tags and nothing else, and a tag that does not
|
||||
// match what the entry configures either promotes a build that is no faster or
|
||||
// hides one that is.
|
||||
func RenderChild(in ChildInput) GalleryEntry {
|
||||
e := GalleryEntry{
|
||||
Name: in.Name,
|
||||
URL: fmt.Sprintf("github:mudler/LocalAI/gallery/%s@master", in.Template),
|
||||
Tags: append([]string{}, in.BaseTags...),
|
||||
Overrides: map[string]any{},
|
||||
}
|
||||
|
||||
// gallery/virtual.yaml carries no backend, so nothing else would name an
|
||||
// engine for these entries. Matching the hand-written entries on
|
||||
// known_usecases too: LocalAI would fall back to the backend defaults, but
|
||||
// generated entries should not read differently from their neighbours.
|
||||
e.Overrides["backend"] = "llama-cpp"
|
||||
e.Overrides["known_usecases"] = []string{"chat"}
|
||||
|
||||
options := []string{"use_jinja:true"}
|
||||
|
||||
for _, w := range in.Weights {
|
||||
e.Files = append(e.Files, EntryFile{
|
||||
Filename: localPath("models", in.Repo, w.Name),
|
||||
SHA256: w.SHA256,
|
||||
URI: hfURI(in.Repo, w.Name),
|
||||
})
|
||||
}
|
||||
e.Overrides["parameters"] = map[string]any{
|
||||
"model": localPath("models", in.Repo, in.Weights[0].Name),
|
||||
}
|
||||
|
||||
if in.MMProj != nil {
|
||||
// An explicit known_usecases SUPPRESSES the backend-default fallback in
|
||||
// core/gallery/models_types.go, so a multimodal entry left at chat-only
|
||||
// never matches FilterGalleryModelsByUsecase(FLAG_VISION) or
|
||||
// FilterGalleryModelsByMultimodal and vanishes from the UI's vision and
|
||||
// multimodal filters. 19 of the 45 APEX repos ship an mmproj.
|
||||
e.Overrides["known_usecases"] = []string{"chat", "vision"}
|
||||
e.Overrides["mmproj"] = localPath("mmproj", in.Repo, in.MMProj.Name)
|
||||
e.Files = append(e.Files, EntryFile{
|
||||
Filename: localPath("mmproj", in.Repo, in.MMProj.Name),
|
||||
SHA256: in.MMProj.SHA256,
|
||||
URI: hfURI(in.Repo, in.MMProj.Name),
|
||||
})
|
||||
}
|
||||
|
||||
// A spec type is configured independently of a drafter FILE. Weights that
|
||||
// carry their own MTP heads need no second download, and requiring one left
|
||||
// the *-APEX-MTP-GGUF builds shipping the larger heads-bearing weights with
|
||||
// the heads switched off: a strictly bigger download at the same speed,
|
||||
// ranked identically to the plain rung at the same tier.
|
||||
if in.SpecType != "" {
|
||||
options = append(options, "spec_type:"+in.SpecType)
|
||||
options = append(options, specTuning[in.SpecType]...)
|
||||
// The tag is derived from the spec type this entry sets and from nothing
|
||||
// else. Variant ranking reads tags only, so a tag taken from a repo or
|
||||
// entry NAME would promote a build that is no faster whenever the name
|
||||
// and the configuration disagree.
|
||||
e.Tags = append(e.Tags, strings.TrimPrefix(in.SpecType, "draft-"))
|
||||
}
|
||||
|
||||
if in.SpecType != "" && in.DraftFile != nil {
|
||||
// Fall back to the weights repo so pairings that publish the drafter
|
||||
// alongside the weights keep working without restating the repo.
|
||||
draftRepo := in.DraftRepo
|
||||
if draftRepo == "" {
|
||||
draftRepo = in.Repo
|
||||
}
|
||||
draftPath := localPath("models", draftRepo, in.DraftFile.Name)
|
||||
|
||||
e.Overrides["draft_model"] = draftPath
|
||||
e.Overrides["flash_attention"] = "on"
|
||||
e.Files = append(e.Files, EntryFile{
|
||||
Filename: draftPath,
|
||||
SHA256: in.DraftFile.SHA256,
|
||||
URI: hfURI(draftRepo, in.DraftFile.Name),
|
||||
})
|
||||
}
|
||||
|
||||
e.Overrides["options"] = options
|
||||
return e
|
||||
}
|
||||
249
.github/ci/apexentries/render_test.go
vendored
249
.github/ci/apexentries/render_test.go
vendored
@@ -1,249 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("RenderChild", func() {
|
||||
It("tags an entry that configures draft-dflash", func() {
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "qwen3.5-9b-dflash",
|
||||
Repo: "mudler/Example-APEX-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
|
||||
SpecType: "draft-dflash",
|
||||
DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Tags).To(ContainElement("dflash"))
|
||||
Expect(e.Tags).ToNot(ContainElement("mtp"))
|
||||
Expect(e.Overrides["options"]).To(ContainElement("spec_type:draft-dflash"))
|
||||
Expect(e.Overrides["draft_model"]).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("does not tag an MTP-named repo that configures no speculation", func() {
|
||||
// mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF ships MTP-bearing weights. Weights
|
||||
// that carry the heads are not an entry that enables them, and tagging it
|
||||
// would win the feature axis without being any faster.
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "qwen3.6-35b-a3b-apex-mtp-i-quality",
|
||||
Repo: "mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Qwen3.6-35B-A3B-APEX-MTP-I-Quality.gguf", SHA256: "a"}},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Tags).ToNot(ContainElement("mtp"))
|
||||
Expect(e.Tags).ToNot(ContainElement("dflash"))
|
||||
Expect(e.Overrides).ToNot(HaveKey("draft_model"))
|
||||
})
|
||||
|
||||
It("lists every shard of a sharded build and points the model at the first", func() {
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "step-3.7-flash-ud-q4-k-m",
|
||||
Repo: "unsloth/Step-3.7-Flash-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{
|
||||
{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"},
|
||||
{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"},
|
||||
},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Files).To(HaveLen(2))
|
||||
params, ok := e.Overrides["parameters"].(map[string]any)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(params["model"]).To(HaveSuffix("00001-of-00002.gguf"))
|
||||
Expect(e.Files[0].URI).To(Equal(
|
||||
"https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf"))
|
||||
})
|
||||
|
||||
It("wires mmproj when the repo publishes one", func() {
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "example-i-mini",
|
||||
Repo: "mudler/Example-APEX-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}},
|
||||
MMProj: &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Overrides["mmproj"]).ToNot(BeNil())
|
||||
Expect(e.Files).To(HaveLen(2))
|
||||
})
|
||||
|
||||
It("names the engine and the usecases the hand-written entries name", func() {
|
||||
// gallery/virtual.yaml supplies no backend, so an entry that omits one
|
||||
// names no engine at all and cannot load.
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "example-i-mini",
|
||||
Repo: "mudler/Example-APEX-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Overrides["backend"]).To(Equal("llama-cpp"))
|
||||
Expect(e.Overrides["known_usecases"]).To(ContainElement("chat"))
|
||||
})
|
||||
|
||||
It("draws the drafter from DraftRepo when the pairing spans two repos", func() {
|
||||
// unsloth/Qwen3-4B-GGUF pairs with a drafter published separately by
|
||||
// AtomicChat, so a drafter URI built from the weights repo 404s.
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "qwen3-4b-dflash",
|
||||
Repo: "unsloth/Qwen3-4B-GGUF",
|
||||
DraftRepo: "AtomicChat/Qwen3-4B-DFlash-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Qwen3-4B-Q4_K_M.gguf", SHA256: "a"}},
|
||||
SpecType: "draft-dflash",
|
||||
DraftFile: &GGUFFile{Name: "Qwen3-4B-DFlash.Q8_0.gguf", SHA256: "b"},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Files[0].URI).To(Equal(
|
||||
"https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf"))
|
||||
Expect(e.Files[1].URI).To(Equal(
|
||||
"https://huggingface.co/AtomicChat/Qwen3-4B-DFlash-GGUF/resolve/main/Qwen3-4B-DFlash.Q8_0.gguf"))
|
||||
Expect(e.Files[1].Filename).To(Equal(
|
||||
"llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf"))
|
||||
Expect(e.Overrides["draft_model"]).To(Equal(
|
||||
"llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf"))
|
||||
})
|
||||
|
||||
It("falls back to the weights repo for the drafter when DraftRepo is empty", func() {
|
||||
// The *-APEX-MTP-GGUF repos ship the drafter alongside the weights.
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "example-apex-dflash",
|
||||
Repo: "mudler/Example-APEX-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
|
||||
SpecType: "draft-dflash",
|
||||
DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Files[1].URI).To(Equal(
|
||||
"https://huggingface.co/mudler/Example-APEX-GGUF/resolve/main/Example-DFlash.Q8_0.gguf"))
|
||||
Expect(e.Files[1].Filename).To(Equal(
|
||||
"llama-cpp/models/mudler/Example-APEX-GGUF/Example-DFlash.Q8_0.gguf"))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("RenderChild known_usecases", func() {
|
||||
It("declares vision alongside chat when the entry carries an mmproj", func() {
|
||||
// An explicit known_usecases suppresses the backend-default fallback, so a
|
||||
// chat-only multimodal entry disappears from the UI's vision filter.
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "example-i-quality",
|
||||
Repo: "mudler/Example-APEX-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
|
||||
MMProj: &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat", "vision"))
|
||||
})
|
||||
|
||||
It("leaves a text-only entry at chat", func() {
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "example-i-quality",
|
||||
Repo: "mudler/Example-APEX-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat"))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("localPath", func() {
|
||||
It("keeps two repos with the same basename but different owners apart", func() {
|
||||
// LiquidAI and unsloth both publish LFM2.5-8B-A1B-GGUF. A path built from
|
||||
// the bare repo name gives both the same local file, so installing the
|
||||
// second overwrites or skips the first and one of them then serves bytes
|
||||
// that do not match its recorded sha256.
|
||||
liquid := RenderChild(ChildInput{
|
||||
Name: "lfm2.5-8b-a1b-i-quality",
|
||||
Repo: "LiquidAI/LFM2.5-8B-A1B-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "33ab3b8c"}},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
unsloth := RenderChild(ChildInput{
|
||||
Name: "lfm2.5-8b-a1b-q8-0",
|
||||
Repo: "unsloth/LFM2.5-8B-A1B-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "ec11666b"}},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(liquid.Files[0].Filename).ToNot(Equal(unsloth.Files[0].Filename))
|
||||
Expect(unsloth.Files[0].Filename).To(Equal(
|
||||
"llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf"))
|
||||
})
|
||||
|
||||
It("namespaces the mmproj by owner too", func() {
|
||||
e := RenderChild(ChildInput{
|
||||
Name: "example-i-quality",
|
||||
Repo: "mudler/Example-APEX-GGUF",
|
||||
Template: "virtual.yaml",
|
||||
Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}},
|
||||
MMProj: &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
|
||||
Expect(e.Overrides["mmproj"]).To(Equal(
|
||||
"llama-cpp/mmproj/mudler/Example-APEX-GGUF/mmproj-F16.gguf"))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("MTP builds", func() {
|
||||
renderTier := func(repo string) GalleryEntry {
|
||||
return RenderChild(ChildInput{
|
||||
Name: "example-i-quality",
|
||||
Repo: repo,
|
||||
Template: "virtual.yaml",
|
||||
SpecType: SpecTypeForRepo(repo),
|
||||
Weights: []GGUFFile{{Name: "Example-I-Quality.gguf", SHA256: "a"}},
|
||||
BaseTags: []string{"llm", "gguf"},
|
||||
})
|
||||
}
|
||||
|
||||
It("turns MTP on for a build off an APEX-MTP repo", func() {
|
||||
// These weights retain the model's own MTP heads, so shipping them with
|
||||
// speculation off is a strictly larger download at the same speed,
|
||||
// ranked identically to the plain rung at the same tier.
|
||||
e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF")
|
||||
|
||||
Expect(e.Overrides["options"]).To(ContainElements(
|
||||
"spec_type:draft-mtp", "spec_n_max:6", "spec_p_min:0.75"))
|
||||
Expect(e.Tags).To(ContainElement("mtp"))
|
||||
})
|
||||
|
||||
It("needs no drafter file, because the heads travel with the weights", func() {
|
||||
e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF")
|
||||
|
||||
Expect(e.Overrides).ToNot(HaveKey("draft_model"))
|
||||
Expect(e.Files).To(HaveLen(1))
|
||||
})
|
||||
|
||||
It("leaves a build off a plain APEX repo alone", func() {
|
||||
e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-GGUF")
|
||||
|
||||
Expect(e.Tags).ToNot(ContainElement("mtp"))
|
||||
Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true"))
|
||||
})
|
||||
|
||||
It("leaves an unsloth counterpart rung alone", func() {
|
||||
// The counterpart quantizes the plain weights; nothing there carries heads.
|
||||
e := renderTier("unsloth/Qwen3.6-35B-A3B-GGUF")
|
||||
|
||||
Expect(e.Tags).ToNot(ContainElement("mtp"))
|
||||
Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true"))
|
||||
})
|
||||
})
|
||||
71
.github/ci/apexentries/unsloth.go
vendored
71
.github/ci/apexentries/unsloth.go
vendored
@@ -1,71 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// WantedQuants is the fixed unsloth subset this generator emits. It is a
|
||||
// deliberate subset: unsloth publishes north of 20 quants per repo, and the
|
||||
// selector needs useful fitness points rather than every rung.
|
||||
var WantedQuants = []string{"UD-Q4_K_M", "UD-Q5_K_M", "UD-Q6_K", "Q8_0"}
|
||||
|
||||
var shardRE = regexp.MustCompile(`-(\d{5})-of-(\d{5})\.gguf$`)
|
||||
|
||||
// QuantBuild is one unsloth quantization, which may be a single file or an
|
||||
// ordered set of shards.
|
||||
type QuantBuild struct {
|
||||
Quant string
|
||||
Files []GGUFFile
|
||||
Sharded bool
|
||||
}
|
||||
|
||||
// CounterpartCandidates returns the unsloth repo base names worth probing, most
|
||||
// likely first. Both derivations are needed: the repo name finds
|
||||
// unsloth/gemma-4-26B-A4B-it-GGUF, while the file stem is what matches for
|
||||
// repos whose stem is the canonical model name.
|
||||
func CounterpartCandidates(repoName, fileStem string) []string {
|
||||
clean := func(s string) string {
|
||||
s = strings.TrimSuffix(s, "-GGUF")
|
||||
s = regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "")
|
||||
s = strings.TrimSuffix(s, "-APEX")
|
||||
return regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "")
|
||||
}
|
||||
|
||||
out := []string{clean(repoName)}
|
||||
if stem := clean(fileStem); stem != out[0] {
|
||||
out = append(out, stem)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// DiscoverUnslothQuants returns the wanted quants a repo publishes, handling
|
||||
// both the flat single-file layout and the sharded layout where a quant lives
|
||||
// in its own subdirectory.
|
||||
func DiscoverUnslothQuants(files []GGUFFile) []QuantBuild {
|
||||
var out []QuantBuild
|
||||
|
||||
for _, q := range WantedQuants {
|
||||
var flat []GGUFFile
|
||||
var shards []GGUFFile
|
||||
|
||||
for _, f := range files {
|
||||
switch {
|
||||
case !strings.Contains(f.Name, "/") && strings.HasSuffix(f.Name, "-"+q+".gguf"):
|
||||
flat = append(flat, f)
|
||||
case strings.HasPrefix(f.Name, q+"/") && shardRE.MatchString(f.Name):
|
||||
shards = append(shards, f)
|
||||
}
|
||||
}
|
||||
|
||||
switch {
|
||||
case len(flat) > 0:
|
||||
out = append(out, QuantBuild{Quant: q, Files: flat})
|
||||
case len(shards) > 0:
|
||||
sort.Slice(shards, func(i, j int) bool { return shards[i].Name < shards[j].Name })
|
||||
out = append(out, QuantBuild{Quant: q, Files: shards, Sharded: true})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
75
.github/ci/apexentries/unsloth_test.go
vendored
75
.github/ci/apexentries/unsloth_test.go
vendored
@@ -1,75 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("CounterpartCandidates", func() {
|
||||
It("offers both the repo-derived and stem-derived names", func() {
|
||||
// mudler/gemma-4-26B-A4B-it-APEX-GGUF ships gemma-4-26B-A4B-APEX-*.gguf,
|
||||
// and only the repo-derived name finds unsloth/gemma-4-26B-A4B-it-GGUF.
|
||||
got := CounterpartCandidates("gemma-4-26B-A4B-it-APEX-GGUF", "gemma-4-26B-A4B-APEX")
|
||||
|
||||
Expect(got).To(Equal([]string{"gemma-4-26B-A4B-it", "gemma-4-26B-A4B"}))
|
||||
})
|
||||
|
||||
It("strips the MTP marker", func() {
|
||||
got := CounterpartCandidates("Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF", "Qwopus3.6-35B-A3B-v1-APEX-MTP")
|
||||
|
||||
Expect(got[0]).To(Equal("Qwopus3.6-35B-A3B-v1"))
|
||||
})
|
||||
|
||||
It("strips the TQ marker", func() {
|
||||
// This is the branch that folds mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF into
|
||||
// the qwen3.5-35b-a3b hub. Without it the probe is
|
||||
// unsloth/Qwen3.5-35B-A3B-TQ-GGUF, which does not exist, so the family
|
||||
// silently loses every unsloth rung.
|
||||
got := CounterpartCandidates("Qwen3.5-35B-A3B-APEX-TQ-GGUF", "Qwen3.5-35B-A3B-APEX-TQ")
|
||||
|
||||
Expect(got).To(Equal([]string{"Qwen3.5-35B-A3B"}))
|
||||
})
|
||||
|
||||
It("does not repeat a candidate when both derivations agree", func() {
|
||||
got := CounterpartCandidates("Qwen3.6-35B-A3B-APEX-GGUF", "Qwen3.6-35B-A3B-APEX")
|
||||
|
||||
Expect(got).To(Equal([]string{"Qwen3.6-35B-A3B"}))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("DiscoverUnslothQuants", func() {
|
||||
It("finds flat single-file quants", func() {
|
||||
files := []GGUFFile{
|
||||
{Name: "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", SHA256: "a"},
|
||||
{Name: "Qwen3.6-35B-A3B-UD-IQ1_M.gguf", SHA256: "b"},
|
||||
}
|
||||
|
||||
got := DiscoverUnslothQuants(files)
|
||||
|
||||
Expect(got).To(HaveLen(1))
|
||||
Expect(got[0].Quant).To(Equal("UD-Q4_K_M"))
|
||||
Expect(got[0].Sharded).To(BeFalse())
|
||||
Expect(got[0].Files).To(HaveLen(1))
|
||||
})
|
||||
|
||||
It("collects a sharded quant from its subdirectory in shard order", func() {
|
||||
files := []GGUFFile{
|
||||
{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"},
|
||||
{Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"},
|
||||
}
|
||||
|
||||
got := DiscoverUnslothQuants(files)
|
||||
|
||||
Expect(got).To(HaveLen(1))
|
||||
Expect(got[0].Quant).To(Equal("UD-Q4_K_M"))
|
||||
Expect(got[0].Sharded).To(BeTrue())
|
||||
Expect(got[0].Files).To(HaveLen(2))
|
||||
Expect(got[0].Files[0].Name).To(HaveSuffix("00001-of-00002.gguf"))
|
||||
})
|
||||
|
||||
It("ignores quants outside the wanted subset", func() {
|
||||
files := []GGUFFile{{Name: "Model-UD-IQ2_XXS.gguf", SHA256: "a"}}
|
||||
|
||||
Expect(DiscoverUnslothQuants(files)).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
312
.github/ci/apexentries/verify.go
vendored
312
.github/ci/apexentries/verify.go
vendored
@@ -1,312 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type verifyEntry struct {
|
||||
Name string `yaml:"name"`
|
||||
Tags []string `yaml:"tags"`
|
||||
Variants []VariantRef `yaml:"variants"`
|
||||
Overrides struct {
|
||||
// Backend scopes the checks that only hold for one engine. An entry that
|
||||
// declares none takes its configuration from the referenced url: template,
|
||||
// which this verifier never reads, so it cannot be judged either way.
|
||||
Backend string `yaml:"backend"`
|
||||
Options []string `yaml:"options"`
|
||||
// MMProj and DraftModel name the files that are not weights. They are
|
||||
// the only signal for it: a drafter lands in the same models/ prefix as
|
||||
// the weights, so the path alone cannot tell them apart.
|
||||
MMProj string `yaml:"mmproj"`
|
||||
DraftModel string `yaml:"draft_model"`
|
||||
} `yaml:"overrides"`
|
||||
Files []struct {
|
||||
Filename string `yaml:"filename"`
|
||||
SHA256 string `yaml:"sha256"`
|
||||
URI string `yaml:"uri"`
|
||||
} `yaml:"files"`
|
||||
}
|
||||
|
||||
// Verify checks the invariants the variants schema and the tagging rule
|
||||
// require. It returns every problem rather than the first, so one run tells the
|
||||
// author everything that needs fixing.
|
||||
func Verify(path string) []string {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return []string{fmt.Sprintf("reading %s: %v", path, err)}
|
||||
}
|
||||
|
||||
var entries []verifyEntry
|
||||
if err := yaml.Unmarshal(raw, &entries); err != nil {
|
||||
return []string{fmt.Sprintf("parsing %s: %v", path, err)}
|
||||
}
|
||||
|
||||
var problems []string
|
||||
|
||||
byName := map[string]verifyEntry{}
|
||||
for _, e := range entries {
|
||||
if _, seen := byName[e.Name]; seen {
|
||||
problems = append(problems, fmt.Sprintf("duplicate entry name: %s", e.Name))
|
||||
continue
|
||||
}
|
||||
byName[e.Name] = e
|
||||
}
|
||||
|
||||
for _, e := range entries {
|
||||
for _, v := range e.Variants {
|
||||
target, ok := byName[v.Model]
|
||||
if !ok {
|
||||
problems = append(problems, fmt.Sprintf("%s: variant %q does not exist", e.Name, v.Model))
|
||||
continue
|
||||
}
|
||||
if len(target.Variants) > 0 {
|
||||
problems = append(problems, fmt.Sprintf("%s: variant %q declares variants of its own", e.Name, v.Model))
|
||||
}
|
||||
}
|
||||
|
||||
for _, f := range e.Files {
|
||||
if requiresSHA256(f.Filename) && f.SHA256 == "" {
|
||||
problems = append(problems, fmt.Sprintf("%s: file %s has no sha256", e.Name, f.Filename))
|
||||
}
|
||||
}
|
||||
|
||||
problems = append(problems, checkWeightCount(e)...)
|
||||
problems = append(problems, checkFeatureTag(e, "dflash")...)
|
||||
problems = append(problems, checkFeatureTag(e, "mtp")...)
|
||||
}
|
||||
|
||||
problems = append(problems, checkPathCollisions(entries)...)
|
||||
|
||||
return problems
|
||||
}
|
||||
|
||||
// checkPathCollisions catches two different upstream files claiming one local
|
||||
// path. The install layer keys on the local filename, so whichever entry is
|
||||
// installed second either overwrites weights the first entry recorded a
|
||||
// different sha256 for or is skipped as already present. Either way some entry
|
||||
// afterwards serves bytes that do not match its own checksum, and nothing at
|
||||
// install time says so.
|
||||
//
|
||||
// This is an index-wide invariant rather than a per-entry one: neither entry is
|
||||
// wrong on its own and the collision exists only in their pairing. The usual
|
||||
// source is a path scheme built from the repo's BARE name, because two owners
|
||||
// publishing the same model name is routine for quantizers.
|
||||
//
|
||||
// Sharing a path is fine when the uri is the same, which is how several entries
|
||||
// legitimately reuse one projector. Files with no uri are skipped: there is
|
||||
// nothing to compare.
|
||||
func checkPathCollisions(entries []verifyEntry) []string {
|
||||
type source struct{ uri, entry string }
|
||||
|
||||
first := map[string]source{}
|
||||
reported := map[string]bool{}
|
||||
|
||||
var problems []string
|
||||
for _, e := range entries {
|
||||
for _, f := range e.Files {
|
||||
if f.Filename == "" || f.URI == "" {
|
||||
continue
|
||||
}
|
||||
prev, seen := first[f.Filename]
|
||||
if !seen {
|
||||
first[f.Filename] = source{uri: f.URI, entry: e.Name}
|
||||
continue
|
||||
}
|
||||
if prev.uri == f.URI || reported[f.Filename] {
|
||||
continue
|
||||
}
|
||||
// Reported once per path however many entries pile onto it, so one
|
||||
// heavily reused filename cannot bury the rest of the report.
|
||||
reported[f.Filename] = true
|
||||
problems = append(problems, fmt.Sprintf(
|
||||
"local path %s is claimed by two different uris: %s (%s) and %s (%s)",
|
||||
f.Filename, prev.uri, prev.entry, f.URI, e.Name))
|
||||
}
|
||||
}
|
||||
return problems
|
||||
}
|
||||
|
||||
// auxiliaryExtensions are the metadata formats an entry ships beside its
|
||||
// weights, where an unverified download is a nuisance rather than a hole.
|
||||
//
|
||||
// The exclusion is stated as a list of metadata formats on purpose. Requiring
|
||||
// the checksum only on a blessed list of weight formats would silently exempt
|
||||
// every format nobody has shipped yet, and it already exempted safetensors
|
||||
// weights, which are downloaded and loaded exactly like GGUF ones.
|
||||
var auxiliaryExtensions = []string{".json", ".txt", ".md"}
|
||||
|
||||
// requiresSHA256 reports whether an unverified download of this file would be
|
||||
// a supply-chain hole rather than a cosmetic gap.
|
||||
func requiresSHA256(filename string) bool {
|
||||
for _, ext := range auxiliaryExtensions {
|
||||
if strings.HasSuffix(filename, ext) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// checkWeightCount catches an entry carrying two whole models. The flat-match
|
||||
// branch in DiscoverUnslothQuants appends every match, so a quant label that is
|
||||
// a suffix of another one (Q8_0 of UD-Q8_0) collects both files into one build
|
||||
// while the rendered model: points at only the first. The result downloads
|
||||
// twice the bytes and serves whichever file sorted first, silently.
|
||||
//
|
||||
// Shards are exempt because a sharded build is legitimately many files.
|
||||
//
|
||||
// The collision is a property of llama-cpp quant discovery, so the check is
|
||||
// scoped to that backend. Multi-component TTS, ASR and diffusion engines ship an
|
||||
// encoder, a decoder and a vocoder as one model, and there the second GGUF is
|
||||
// the design rather than a bug.
|
||||
func checkWeightCount(e verifyEntry) []string {
|
||||
if e.Overrides.Backend != "llama-cpp" {
|
||||
return nil
|
||||
}
|
||||
|
||||
var weights []string
|
||||
for _, f := range e.Files {
|
||||
switch {
|
||||
case !strings.HasSuffix(f.Filename, ".gguf"):
|
||||
case shardRE.MatchString(f.Filename):
|
||||
case f.Filename == e.Overrides.MMProj:
|
||||
case f.Filename == e.Overrides.DraftModel:
|
||||
default:
|
||||
weights = append(weights, f.Filename)
|
||||
}
|
||||
}
|
||||
|
||||
if len(weights) > 1 {
|
||||
return []string{fmt.Sprintf("%s: more than one weight file: %s", e.Name, strings.Join(weights, ", "))}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkFeatureTag enforces the rule in both directions. A tag without the
|
||||
// configuration promotes a build that is no faster; configuration without the
|
||||
// tag leaves a genuinely faster build ranked as plain.
|
||||
//
|
||||
// It only speaks about backends whose declaration it can actually read, because
|
||||
// a rule applied where the evidence is invisible reports noise rather than bugs.
|
||||
func checkFeatureTag(e verifyEntry, feature string) []string {
|
||||
decl, configured, judgeable := featureDeclaration(e, feature)
|
||||
if !judgeable {
|
||||
return nil
|
||||
}
|
||||
|
||||
tagged := false
|
||||
for _, t := range e.Tags {
|
||||
if t == feature {
|
||||
tagged = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
switch {
|
||||
case tagged && !configured:
|
||||
return []string{fmt.Sprintf("%s: tagged %s but sets no %s", e.Name, feature, decl)}
|
||||
case configured && !tagged:
|
||||
return []string{fmt.Sprintf("%s: sets %s but is not tagged %s", e.Name, decl, feature)}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// featureDeclaration implements the per-backend table in
|
||||
// .agents/adding-gallery-models.md. It returns the declaration the backend uses
|
||||
// to configure the feature, whether the entry carries it, and whether this
|
||||
// verifier is in a position to answer at all.
|
||||
func featureDeclaration(e verifyEntry, feature string) (decl string, configured, judgeable bool) {
|
||||
switch e.Overrides.Backend {
|
||||
case "llama-cpp":
|
||||
decl = "spec_type:draft-" + feature
|
||||
for _, o := range e.Overrides.Options {
|
||||
if strings.TrimSpace(o) == decl {
|
||||
return decl, true, true
|
||||
}
|
||||
}
|
||||
return decl, false, true
|
||||
|
||||
case "ds4":
|
||||
// ds4 carries the MTP heads in the weights and turns them on with
|
||||
// mtp_path / mtp_draft. It has no dflash counterpart, so dflash is not a
|
||||
// question that can be asked of a ds4 entry.
|
||||
if feature != "mtp" {
|
||||
return "", false, false
|
||||
}
|
||||
decl = "mtp_path:"
|
||||
for _, o := range e.Overrides.Options {
|
||||
o = strings.TrimSpace(o)
|
||||
if strings.HasPrefix(o, "mtp_path:") || strings.HasPrefix(o, "mtp_draft:") {
|
||||
return decl, true, true
|
||||
}
|
||||
}
|
||||
return decl, false, true
|
||||
|
||||
default:
|
||||
// sglang configures the feature with speculative_algorithm: in the
|
||||
// referenced gallery/*.yaml, and an entry that declares no backend takes
|
||||
// its whole configuration from its url: template. Verify reads one index
|
||||
// file and follows neither, so it must not judge these in either
|
||||
// direction.
|
||||
return "", false, false
|
||||
}
|
||||
}
|
||||
|
||||
// UnaccountedQuants reports a wanted quant the repo demonstrably publishes but
|
||||
// that discovery produced no build for. The layout that triggers it today is
|
||||
// root-level shards, which match neither branch of DiscoverUnslothQuants; no
|
||||
// counterpart ships that way yet, but a batch generator must not drop a build
|
||||
// with nothing said about it.
|
||||
func UnaccountedQuants(files []GGUFFile, builds []QuantBuild) []string {
|
||||
built := map[string]bool{}
|
||||
for _, b := range builds {
|
||||
built[b.Quant] = true
|
||||
}
|
||||
|
||||
var problems []string
|
||||
for _, q := range WantedQuants {
|
||||
if built[q] {
|
||||
continue
|
||||
}
|
||||
for _, f := range files {
|
||||
if filePublishesQuant(f.Name, q) {
|
||||
problems = append(problems, fmt.Sprintf("quant %s is published upstream (%s) but produced no build", q, f.Name))
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return problems
|
||||
}
|
||||
|
||||
// filePublishesQuant reports whether an upstream file is a publication of
|
||||
// quant q. It anchors on the quant label the way DiscoverUnslothQuants does,
|
||||
// as the trailing token of the base name or as the sharding subdirectory, so
|
||||
// the diagnostic and the discovery it audits cannot disagree about what a file
|
||||
// is.
|
||||
//
|
||||
// An unanchored match would reproduce the very collision this diagnostic warns
|
||||
// about: Q8_0 is a substring of UD-Q8_0, so a repo publishing only UD-Q8_0
|
||||
// would be reported as publishing an unbuilt Q8_0, which it does not, and
|
||||
// UD-Q8_0 is not a wanted quant at all.
|
||||
func filePublishesQuant(name, q string) bool {
|
||||
if strings.HasPrefix(name, q+"/") {
|
||||
return true
|
||||
}
|
||||
|
||||
base := name[strings.LastIndex(name, "/")+1:]
|
||||
// Shard numbering sits between the quant label and the extension, so it has
|
||||
// to come off before the label can be read as the trailing token. Root-level
|
||||
// shards are the layout that matches neither branch of
|
||||
// DiscoverUnslothQuants, and so the layout this diagnostic mainly catches.
|
||||
base = shardRE.ReplaceAllString(base, ".gguf")
|
||||
|
||||
if !strings.HasSuffix(base, "-"+q+".gguf") {
|
||||
return false
|
||||
}
|
||||
// UD- is unsloth's dynamic-quant modifier, and UD-<q> is a distinct quant
|
||||
// label rather than a publication of <q>.
|
||||
return !strings.HasSuffix(base, "-UD-"+q+".gguf")
|
||||
}
|
||||
480
.github/ci/apexentries/verify_test.go
vendored
480
.github/ci/apexentries/verify_test.go
vendored
@@ -1,480 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Verify", func() {
|
||||
write := func(body string) string {
|
||||
dir := GinkgoT().TempDir()
|
||||
p := filepath.Join(dir, "index.yaml")
|
||||
Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed())
|
||||
return p
|
||||
}
|
||||
|
||||
It("passes a sound index", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: parent
|
||||
variants:
|
||||
- model: child
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- name: child
|
||||
files:
|
||||
- filename: b.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("reports a variant pointing at a missing entry", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: parent
|
||||
variants:
|
||||
- model: ghost
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(ContainElement(ContainSubstring("ghost")))
|
||||
})
|
||||
|
||||
It("reports a variant that itself declares variants", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: parent
|
||||
variants:
|
||||
- model: child
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- name: child
|
||||
variants:
|
||||
- model: grandchild
|
||||
files:
|
||||
- filename: b.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
- name: grandchild
|
||||
files:
|
||||
- filename: c.gguf
|
||||
sha256: cc
|
||||
uri: https://example.com/c.gguf
|
||||
`))).To(ContainElement(ContainSubstring("declares variants of its own")))
|
||||
})
|
||||
|
||||
It("reports duplicate entry names", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: dup
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- name: dup
|
||||
files:
|
||||
- filename: b.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
`))).To(ContainElement(ContainSubstring("duplicate entry name")))
|
||||
})
|
||||
|
||||
It("reports a file with no sha256", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: one
|
||||
files:
|
||||
- filename: a.gguf
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(ContainElement(ContainSubstring("no sha256")))
|
||||
})
|
||||
|
||||
It("reports an entry tagged dflash without a matching spec_type", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: liar
|
||||
tags:
|
||||
- dflash
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
options:
|
||||
- use_jinja:true
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(ContainElement(ContainSubstring("tagged dflash")))
|
||||
})
|
||||
|
||||
It("reports an entry configuring spec_type without the tag", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: shy
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
options:
|
||||
- spec_type:draft-mtp
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(ContainElement(ContainSubstring("not tagged mtp")))
|
||||
})
|
||||
|
||||
// ds4 carries the MTP heads in the weights and names them with mtp_path, so
|
||||
// the rule holds there in a different vocabulary rather than not at all.
|
||||
It("reports a ds4 entry configuring mtp_path without the tag", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: ds4-shy
|
||||
overrides:
|
||||
backend: ds4
|
||||
options:
|
||||
- mtp_path:model-mtp.gguf
|
||||
- mtp_draft:2
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(ContainElement(ContainSubstring("not tagged mtp")))
|
||||
})
|
||||
|
||||
It("reports a ds4 entry tagged mtp that configures no mtp_path", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: ds4-liar
|
||||
tags:
|
||||
- mtp
|
||||
overrides:
|
||||
backend: ds4
|
||||
options:
|
||||
- context_size:4096
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(ContainElement(ContainSubstring("tagged mtp")))
|
||||
})
|
||||
|
||||
It("accepts a ds4 entry that both configures mtp_path and carries the tag", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: ds4-honest
|
||||
tags:
|
||||
- mtp
|
||||
overrides:
|
||||
backend: ds4
|
||||
options:
|
||||
- mtp_path:model-mtp.gguf
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
// sglang declares speculative_algorithm in the referenced gallery/*.yaml,
|
||||
// which Verify never reads, so it may not judge such an entry either way.
|
||||
It("says nothing about an sglang entry tagged mtp", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: sglang-mtp
|
||||
tags:
|
||||
- mtp
|
||||
overrides:
|
||||
backend: sglang
|
||||
files: []
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("says nothing about the tag on an entry with no declared backend", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: templated
|
||||
tags:
|
||||
- mtp
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
// The flat-match branch in unsloth.go appends every match, so a repo
|
||||
// publishing both a plain and a UD Q8_0 renders one entry holding two full
|
||||
// models while model: points at only the first.
|
||||
It("reports an entry holding more than one non-shard weight file", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: greedy
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
options:
|
||||
- use_jinja:true
|
||||
parameters:
|
||||
model: llama-cpp/models/repo/Model-Q8_0.gguf
|
||||
files:
|
||||
- filename: llama-cpp/models/repo/Model-Q8_0.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- filename: llama-cpp/models/repo/Model-UD-Q8_0.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
`))).To(ContainElement(ContainSubstring("more than one weight file")))
|
||||
})
|
||||
|
||||
It("accepts many shards alongside an mmproj and a drafter", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: sharded
|
||||
tags:
|
||||
- mtp
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
options:
|
||||
- spec_type:draft-mtp
|
||||
mmproj: llama-cpp/mmproj/repo/mm.gguf
|
||||
draft_model: llama-cpp/models/repo/Model-draft.gguf
|
||||
files:
|
||||
- filename: llama-cpp/models/repo/Model-00001-of-00002.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- filename: llama-cpp/models/repo/Model-00002-of-00002.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
- filename: llama-cpp/mmproj/repo/mm.gguf
|
||||
sha256: cc
|
||||
uri: https://example.com/c.gguf
|
||||
- filename: llama-cpp/models/repo/Model-draft.gguf
|
||||
sha256: dd
|
||||
uri: https://example.com/d.gguf
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
// Multi-component TTS and ASR engines legitimately ship an encoder, a
|
||||
// tokenizer, a vocoder and so on as one model, so the collision the weight
|
||||
// count catches does not exist for them.
|
||||
It("accepts a multi-component non-llama-cpp entry declaring five weights", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: multi
|
||||
overrides:
|
||||
backend: qwen3-tts-cpp
|
||||
files:
|
||||
- filename: talker.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- filename: tokenizer.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
- filename: vocoder.gguf
|
||||
sha256: cc
|
||||
uri: https://example.com/c.gguf
|
||||
- filename: encoder.gguf
|
||||
sha256: dd
|
||||
uri: https://example.com/d.gguf
|
||||
- filename: vae.gguf
|
||||
sha256: ee
|
||||
uri: https://example.com/e.gguf
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("says nothing about the weight count of an entry with no declared backend", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: templated-weights
|
||||
files:
|
||||
- filename: model-Q4_K_M.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- filename: model-mmproj-f16.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("says nothing about an auxiliary metadata file carrying no sha256", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: aux
|
||||
files:
|
||||
- filename: a.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- filename: params.json
|
||||
sha256: ""
|
||||
uri: https://example.com/params.json
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
// safetensors weights are downloaded and loaded exactly like GGUF weights,
|
||||
// so an unverified one is the same supply-chain hole.
|
||||
It("reports a safetensors weight carrying no sha256", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: vae
|
||||
files:
|
||||
- filename: wan_2.1_vae.safetensors
|
||||
sha256: ""
|
||||
uri: https://example.com/vae.safetensors
|
||||
`))).To(ContainElement(ContainSubstring("no sha256")))
|
||||
})
|
||||
|
||||
It("says nothing about a txt or md file carrying no sha256", func() {
|
||||
Expect(Verify(write(`
|
||||
- name: docs
|
||||
files:
|
||||
- filename: notes.txt
|
||||
sha256: ""
|
||||
uri: https://example.com/notes.txt
|
||||
- filename: README.md
|
||||
sha256: ""
|
||||
uri: https://example.com/README.md
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("UnaccountedQuants", func() {
|
||||
// A quant published only as root-level shards matches neither branch in
|
||||
// DiscoverUnslothQuants, so without this diagnostic the build would vanish
|
||||
// from a batch run with nothing said about it.
|
||||
It("reports a wanted quant upstream publishes but discovery dropped", func() {
|
||||
files := []GGUFFile{
|
||||
{Name: "Model-UD-Q4_K_M-00001-of-00003.gguf", SHA256: "aa"},
|
||||
{Name: "Model-UD-Q4_K_M-00002-of-00003.gguf", SHA256: "bb"},
|
||||
{Name: "Model-UD-Q4_K_M-00003-of-00003.gguf", SHA256: "cc"},
|
||||
}
|
||||
|
||||
Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).
|
||||
To(ContainElement(ContainSubstring("UD-Q4_K_M")))
|
||||
})
|
||||
|
||||
It("says nothing when every published wanted quant produced a build", func() {
|
||||
files := []GGUFFile{
|
||||
{Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"},
|
||||
{Name: "UD-Q6_K/Model-UD-Q6_K-00001-of-00002.gguf", SHA256: "bb"},
|
||||
{Name: "UD-Q6_K/Model-UD-Q6_K-00002-of-00002.gguf", SHA256: "cc"},
|
||||
}
|
||||
|
||||
Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("says nothing about a wanted quant the repo does not publish at all", func() {
|
||||
files := []GGUFFile{{Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"}}
|
||||
|
||||
Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
|
||||
})
|
||||
|
||||
// UD-Q8_0 is its own quant label and is not a wanted one. Reading it as a
|
||||
// publication of Q8_0 is the substring collision this diagnostic exists to
|
||||
// warn about, and subdirectory-sharded UD quants are the normal unsloth
|
||||
// layout for large repos, so the false positive would fire on every batch.
|
||||
It("does not read a subdirectory-sharded UD-Q8_0 as a published Q8_0", func() {
|
||||
files := []GGUFFile{
|
||||
{Name: "UD-Q8_0/Model-UD-Q8_0-00001-of-00002.gguf", SHA256: "aa"},
|
||||
{Name: "UD-Q8_0/Model-UD-Q8_0-00002-of-00002.gguf", SHA256: "bb"},
|
||||
}
|
||||
|
||||
Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty())
|
||||
})
|
||||
|
||||
// A quant in its own subdirectory but not shard-numbered matches neither
|
||||
// branch of DiscoverUnslothQuants, so it is genuinely published and
|
||||
// genuinely undiscovered.
|
||||
It("reports a wanted quant published in its own subdirectory without shard numbering", func() {
|
||||
files := []GGUFFile{{Name: "Q8_0/Model-Q8_0.gguf", SHA256: "aa"}}
|
||||
|
||||
Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).
|
||||
To(ContainElement(ContainSubstring("quant Q8_0 is published upstream")))
|
||||
})
|
||||
|
||||
// builds is empty on purpose: it isolates the file-to-quant match from
|
||||
// whatever DiscoverUnslothQuants would have made of the same file.
|
||||
It("matches the flat single-file layout", func() {
|
||||
files := []GGUFFile{{Name: "Model-Q8_0.gguf", SHA256: "aa"}}
|
||||
|
||||
Expect(UnaccountedQuants(files, nil)).
|
||||
To(ConsistOf(ContainSubstring("quant Q8_0 is published upstream")))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("Verify local path collisions", func() {
|
||||
write := func(body string) string {
|
||||
dir := GinkgoT().TempDir()
|
||||
p := filepath.Join(dir, "index.yaml")
|
||||
Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed())
|
||||
return p
|
||||
}
|
||||
|
||||
It("reports one local path claimed by two different uris", func() {
|
||||
// The shape that shipped: LiquidAI and unsloth both publish
|
||||
// LFM2.5-8B-A1B-GGUF, so a path built from the bare repo name gives both
|
||||
// entries the same local file under two different checksums.
|
||||
Expect(Verify(write(`
|
||||
- name: lfm2.5-8b-a1b
|
||||
files:
|
||||
- filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf
|
||||
sha256: 33ab3b8c
|
||||
uri: https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf
|
||||
- name: lfm2.5-8b-a1b-q8-0
|
||||
files:
|
||||
- filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf
|
||||
sha256: ec11666b
|
||||
uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf
|
||||
`))).To(ContainElement(SatisfyAll(
|
||||
ContainSubstring("claimed by two different uris"),
|
||||
ContainSubstring("lfm2.5-8b-a1b-q8-0"),
|
||||
)))
|
||||
})
|
||||
|
||||
It("accepts two entries reusing one file from the same uri", func() {
|
||||
// Sibling builds of one repo legitimately share a projector.
|
||||
Expect(Verify(write(`
|
||||
- name: a
|
||||
files:
|
||||
- filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf
|
||||
sha256: cc
|
||||
uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf
|
||||
- name: b
|
||||
files:
|
||||
- filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf
|
||||
sha256: cc
|
||||
uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("reports a collision once however many entries pile onto the path", func() {
|
||||
problems := Verify(write(`
|
||||
- name: a
|
||||
files:
|
||||
- filename: shared.gguf
|
||||
sha256: aa
|
||||
uri: https://example.com/a.gguf
|
||||
- name: b
|
||||
files:
|
||||
- filename: shared.gguf
|
||||
sha256: bb
|
||||
uri: https://example.com/b.gguf
|
||||
- name: c
|
||||
files:
|
||||
- filename: shared.gguf
|
||||
sha256: cc
|
||||
uri: https://example.com/c.gguf
|
||||
`))
|
||||
|
||||
var collisions int
|
||||
for _, p := range problems {
|
||||
if strings.Contains(p, "claimed by two different uris") {
|
||||
collisions++
|
||||
}
|
||||
}
|
||||
Expect(collisions).To(Equal(1))
|
||||
})
|
||||
|
||||
It("says nothing about files that carry no uri", func() {
|
||||
// A hand-written entry may record only a checksum. There is no upstream
|
||||
// to compare, so the check cannot conclude anything either way.
|
||||
Expect(Verify(write(`
|
||||
- name: a
|
||||
files:
|
||||
- filename: shared.gguf
|
||||
sha256: aa
|
||||
- name: b
|
||||
files:
|
||||
- filename: shared.gguf
|
||||
sha256: bb
|
||||
`))).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
152
.github/ci/galleryedit/edit.go
vendored
152
.github/ci/galleryedit/edit.go
vendored
@@ -1,152 +0,0 @@
|
||||
// Package galleryedit splices variant references into the LocalAI gallery index
|
||||
// as TEXT.
|
||||
//
|
||||
// Re-serialising the index through a YAML marshaller would reflow 40,000 lines,
|
||||
// drop the anchors and merge keys the gallery relies on, and produce a diff no
|
||||
// reviewer could read, which makes a pull request worthless even when the
|
||||
// content inside it is right. Every generator that adds variants to an entry the
|
||||
// gallery already ships therefore edits lines, and they share this package so
|
||||
// that two of them cannot drift apart on where a variants block belongs.
|
||||
package galleryedit
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var (
|
||||
entryStart = regexp.MustCompile(`^-(?: |$)`)
|
||||
inlineName = regexp.MustCompile(`^- (?:&\S+ )?name:`)
|
||||
keyName = regexp.MustCompile(`^ name:`)
|
||||
keyVariants = regexp.MustCompile(`^ variants:\s*(.*)$`)
|
||||
variantItem = regexp.MustCompile(`^ - `)
|
||||
unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`)
|
||||
)
|
||||
|
||||
// Entry is the positional view of one gallery entry: what it is called and
|
||||
// which lines it occupies. Nothing about what the entry MEANS belongs here, so
|
||||
// each caller keeps its own semantic decode and only hands over the coordinates.
|
||||
type Entry struct {
|
||||
Name string
|
||||
// StartLine and EndLine bound the entry, zero based and half open.
|
||||
StartLine int
|
||||
EndLine int
|
||||
}
|
||||
|
||||
// Insert is one entry's pending variants addition. The caller owns the contents
|
||||
// of Variants: this package neither orders nor deduplicates them, because the
|
||||
// right order and the right dedup rule differ between generators.
|
||||
type Insert struct {
|
||||
Entry Entry
|
||||
Variants []string
|
||||
}
|
||||
|
||||
// Scan splits index text into lines and reports the line each top level list
|
||||
// item begins on.
|
||||
func Scan(text string) (lines []string, starts []int) {
|
||||
lines = strings.Split(text, "\n")
|
||||
for i, line := range lines {
|
||||
if entryStart.MatchString(line) {
|
||||
starts = append(starts, i)
|
||||
}
|
||||
}
|
||||
return lines, starts
|
||||
}
|
||||
|
||||
// Apply splices every insert into the index lines and returns the new text.
|
||||
func Apply(lines []string, inserts []Insert) ([]string, error) {
|
||||
type edit struct {
|
||||
at int
|
||||
remove int
|
||||
insert []string
|
||||
}
|
||||
var edits []edit
|
||||
|
||||
for _, in := range inserts {
|
||||
if len(in.Variants) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
items := make([]string, 0, len(in.Variants))
|
||||
for _, v := range in.Variants {
|
||||
items = append(items, " - model: "+QuoteName(v))
|
||||
}
|
||||
|
||||
at, remove, err := insertionPoint(lines, in.Entry)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
block := items
|
||||
if remove > 0 || !hasVariantsKey(lines, in.Entry) {
|
||||
block = append([]string{" variants:"}, items...)
|
||||
}
|
||||
edits = append(edits, edit{at: at, remove: remove, insert: block})
|
||||
}
|
||||
|
||||
// Applying from the bottom up keeps every line number computed against the
|
||||
// original text valid while earlier edits are still pending.
|
||||
sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at })
|
||||
|
||||
out := append([]string(nil), lines...)
|
||||
for _, e := range edits {
|
||||
tail := append([]string(nil), out[e.at+e.remove:]...)
|
||||
out = append(out[:e.at], append(append([]string(nil), e.insert...), tail...)...)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func hasVariantsKey(lines []string, e Entry) bool {
|
||||
for i := e.StartLine; i < e.EndLine; i++ {
|
||||
if keyVariants.MatchString(lines[i]) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// insertionPoint reports where new variant items belong, and how many existing
|
||||
// lines the insertion replaces.
|
||||
//
|
||||
// An entry with no variants key gets one right after its name, which is where
|
||||
// the hand-written families put it. An entry with an empty "variants: []" has
|
||||
// that line replaced by a block. An entry with a block gets its items appended.
|
||||
func insertionPoint(lines []string, e Entry) (at int, remove int, err error) {
|
||||
for i := e.StartLine; i < e.EndLine; i++ {
|
||||
m := keyVariants.FindStringSubmatch(lines[i])
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(m[1]) == "[]" {
|
||||
return i, 1, nil
|
||||
}
|
||||
if strings.TrimSpace(m[1]) != "" {
|
||||
return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1]))
|
||||
}
|
||||
last := i
|
||||
for j := i + 1; j < e.EndLine && variantItem.MatchString(lines[j]); j++ {
|
||||
last = j
|
||||
}
|
||||
return last + 1, 0, nil
|
||||
}
|
||||
|
||||
if inlineName.MatchString(lines[e.StartLine]) {
|
||||
return e.StartLine + 1, 0, nil
|
||||
}
|
||||
for i := e.StartLine; i < e.EndLine; i++ {
|
||||
if keyName.MatchString(lines[i]) {
|
||||
return i + 1, 0, nil
|
||||
}
|
||||
}
|
||||
return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name)
|
||||
}
|
||||
|
||||
// QuoteName quotes a variant reference when the name would otherwise change
|
||||
// meaning as bare YAML. Config-suffixed names carry a ":" and always need it.
|
||||
func QuoteName(name string) string {
|
||||
if unsafeInName.MatchString(name) {
|
||||
return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"`
|
||||
}
|
||||
return name
|
||||
}
|
||||
133
.github/ci/variantproposals/body.go
vendored
133
.github/ci/variantproposals/body.go
vendored
@@ -1,133 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// RenderBody writes the pull request body.
|
||||
//
|
||||
// The body is the product of this job, not the diff. Grouping is a judgement
|
||||
// call that has gone wrong in both directions before, so a reviewer has to be
|
||||
// able to accept or reject each family from the body alone, without opening
|
||||
// HuggingFace to work out whether two entries hold the same weights.
|
||||
func RenderBody(r *Result, ledgerPath string) string {
|
||||
var b strings.Builder
|
||||
|
||||
b.WriteString("## Proposed gallery variant groupings\n\n")
|
||||
b.WriteString("This is a proposal, not a decision. The gallery agent adds one build per model and never joins an existing family, so entries that are alternative builds of the same weights drift apart as the gallery grows. This job re-applies the grouping heuristics from the manual sweeps and asks a human to confirm.\n\n")
|
||||
b.WriteString("Each family below lists the parent, the variants, and the evidence that they are the same weights. **Reject anything whose evidence you do not believe.**\n\n")
|
||||
b.WriteString(fmt.Sprintf("To decline a family permanently, add one line to `%s` in this pull request and close it:\n\n", ledgerPath))
|
||||
b.WriteString("```yaml\npairs:\n - {parent: some-model, variant: some-model-thing, reason: \"different finetune\"}\n```\n\n")
|
||||
|
||||
b.WriteString(fmt.Sprintf("### Proposed families (%d)\n\n", len(r.Families)))
|
||||
if len(r.Families) == 0 {
|
||||
b.WriteString("None.\n\n")
|
||||
}
|
||||
for _, f := range r.Families {
|
||||
b.WriteString(fmt.Sprintf("#### `%s`\n\n", f.Parent))
|
||||
b.WriteString("| variant | signals | evidence |\n|---|---|---|\n")
|
||||
for _, p := range f.Proposals {
|
||||
b.WriteString(fmt.Sprintf("| `%s` | %s | %s |\n", p.Variant, joinSignals(p.Evidence.Signals), describeEvidence(p.Evidence)))
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
|
||||
b.WriteString(fmt.Sprintf("### Declined by the ledger (%d)\n\n", len(r.Suppressed)))
|
||||
if len(r.Suppressed) == 0 {
|
||||
b.WriteString("Nothing the heuristics found was already on the ledger.\n\n")
|
||||
} else {
|
||||
b.WriteString("Candidates the heuristics found and the ledger has already settled. They are listed so the ledger's effect stays visible rather than silently shrinking the job's output.\n\n")
|
||||
for _, s := range r.Suppressed {
|
||||
b.WriteString(fmt.Sprintf("- `%s` + `%s`: %s\n", s.A, s.B, s.Reason))
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
|
||||
if len(r.AliasSkipped) > 0 {
|
||||
b.WriteString(fmt.Sprintf("### Aliases, not variants (%d)\n\n", len(r.AliasSkipped)))
|
||||
b.WriteString("These entries install byte for byte the same payload. An alias exists so clients can send a particular name; folding it under another entry would hide that name.\n\n")
|
||||
for _, s := range r.AliasSkipped {
|
||||
b.WriteString(fmt.Sprintf("- `%s` + `%s`: %s\n", s.A, s.B, s.Reason))
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
|
||||
if len(r.Refusals) > 0 {
|
||||
b.WriteString(fmt.Sprintf("### Found but refused (%d)\n\n", len(r.Refusals)))
|
||||
b.WriteString("Candidates the heuristics found but the authoring rules would not let this job write. They need a human edit or a rule change.\n\n")
|
||||
for _, ref := range r.Refusals {
|
||||
b.WriteString(fmt.Sprintf("- %s: %s\n", codeList(ref.Members), ref.Reason))
|
||||
}
|
||||
b.WriteString("\n")
|
||||
}
|
||||
|
||||
b.WriteString("---\n\nOpened by `.github/ci/variantproposals`. Heuristics and the rejection ledger live there and in the ledger file; a wrong proposal is a bug in one of the two.\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func joinSignals(signals []Signal) string {
|
||||
if len(signals) == 0 {
|
||||
return "inferred through another member of the family"
|
||||
}
|
||||
out := make([]string, 0, len(signals))
|
||||
for _, s := range signals {
|
||||
out = append(out, "`"+string(s)+"`")
|
||||
}
|
||||
return strings.Join(out, ", ")
|
||||
}
|
||||
|
||||
func describeEvidence(e Evidence) string {
|
||||
var parts []string
|
||||
if e.SharedStem != "" {
|
||||
parts = append(parts, fmt.Sprintf("same name once quantization markers are stripped: `%s`", e.SharedStem))
|
||||
}
|
||||
if e.SharedFile != "" {
|
||||
parts = append(parts, fmt.Sprintf("same primary weight filename once quantization markers are stripped: `%s`", e.SharedFile))
|
||||
}
|
||||
if e.SharedRepo != "" {
|
||||
parts = append(parts, fmt.Sprintf("same upstream repo `%s`", e.SharedRepo))
|
||||
}
|
||||
if len(e.QuantTokens) > 0 {
|
||||
parts = append(parts, "differing quantization tokens: `"+strings.Join(e.QuantTokens, "`, `")+"`")
|
||||
}
|
||||
if len(parts) == 0 {
|
||||
return "reached this family through another member"
|
||||
}
|
||||
return strings.Join(parts, "; ")
|
||||
}
|
||||
|
||||
func codeList(names []string) string {
|
||||
out := make([]string, 0, len(names))
|
||||
for _, n := range names {
|
||||
out = append(out, "`"+n+"`")
|
||||
}
|
||||
return strings.Join(out, " + ")
|
||||
}
|
||||
|
||||
// RenderSummary is the terminal-facing digest of a run, so the workflow log
|
||||
// says what happened without anyone opening the pull request.
|
||||
func RenderSummary(r *Result) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "families proposed: %d\n", len(r.Families))
|
||||
for _, f := range r.Families {
|
||||
names := make([]string, 0, len(f.Proposals))
|
||||
for _, p := range f.Proposals {
|
||||
names = append(names, p.Variant)
|
||||
}
|
||||
fmt.Fprintf(&b, " %s <- %s\n", f.Parent, strings.Join(names, ", "))
|
||||
}
|
||||
fmt.Fprintf(&b, "declined by ledger: %d\n", len(r.Suppressed))
|
||||
for _, s := range r.Suppressed {
|
||||
fmt.Fprintf(&b, " %s\n", s)
|
||||
}
|
||||
fmt.Fprintf(&b, "aliases skipped: %d\n", len(r.AliasSkipped))
|
||||
for _, s := range r.AliasSkipped {
|
||||
fmt.Fprintf(&b, " %s\n", s)
|
||||
}
|
||||
fmt.Fprintf(&b, "refused: %d\n", len(r.Refusals))
|
||||
for _, ref := range r.Refusals {
|
||||
fmt.Fprintf(&b, " %s: %s\n", strings.Join(ref.Members, " + "), ref.Reason)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
42
.github/ci/variantproposals/edit.go
vendored
42
.github/ci/variantproposals/edit.go
vendored
@@ -1,42 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/mudler/LocalAI/.github/ci/galleryedit"
|
||||
)
|
||||
|
||||
// ApplyFamilies writes the proposed variant lists into the index text.
|
||||
//
|
||||
// The line editing itself lives in galleryedit, shared with the apexentries
|
||||
// generator. Both jobs add variants to entries the gallery already ships, and a
|
||||
// second answer to "where does a variants block go" would drift from this one;
|
||||
// see that package for why the edit is textual rather than a YAML round trip.
|
||||
func ApplyFamilies(ix *Index, families []Family) ([]string, error) {
|
||||
byName, _ := ix.ByName()
|
||||
|
||||
var inserts []galleryedit.Insert
|
||||
for _, f := range families {
|
||||
entry, ok := byName[strings.ToLower(f.Parent)]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("parent %q is not in the index", f.Parent)
|
||||
}
|
||||
|
||||
variants := make([]string, 0, len(f.Proposals))
|
||||
for _, p := range f.Proposals {
|
||||
variants = append(variants, p.Variant)
|
||||
}
|
||||
|
||||
inserts = append(inserts, galleryedit.Insert{
|
||||
Entry: galleryedit.Entry{
|
||||
Name: entry.Name,
|
||||
StartLine: entry.StartLine,
|
||||
EndLine: entry.EndLine,
|
||||
},
|
||||
Variants: variants,
|
||||
})
|
||||
}
|
||||
|
||||
return galleryedit.Apply(ix.Lines, inserts)
|
||||
}
|
||||
153
.github/ci/variantproposals/edit_test.go
vendored
153
.github/ci/variantproposals/edit_test.go
vendored
@@ -1,153 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("ApplyFamilies", func() {
|
||||
apply := func(ix *Index, families []Family) []string {
|
||||
lines, err := ApplyFamilies(ix, families)
|
||||
ExpectWithOffset(1, err).ToNot(HaveOccurred())
|
||||
return lines
|
||||
}
|
||||
|
||||
// insertedLines is what a reviewer would see in the diff. A textual editor
|
||||
// that reflowed the file would show thousands here, which is the failure
|
||||
// this whole approach exists to avoid.
|
||||
insertedLines := func(before, after []string) int {
|
||||
remaining := map[string]int{}
|
||||
for _, l := range before {
|
||||
remaining[l]++
|
||||
}
|
||||
n := 0
|
||||
for _, l := range after {
|
||||
if remaining[l] > 0 {
|
||||
remaining[l]--
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
It("adds a variants block right after the entry's name and touches nothing else", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("foo-model", "acme/repo", "foo-model-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("foo-model-q8_0", "acme/repo", "foo-model-Q8_0.gguf", "bb"),
|
||||
)
|
||||
out := apply(ix, []Family{{Parent: "foo-model", Proposals: []Proposal{{Variant: "foo-model-q8_0"}}}})
|
||||
|
||||
Expect(out[0]).To(Equal("- name: foo-model"))
|
||||
Expect(out[1]).To(Equal(" variants:"))
|
||||
Expect(out[2]).To(Equal(" - model: foo-model-q8_0"))
|
||||
Expect(len(out)).To(Equal(len(ix.Lines) + 2))
|
||||
Expect(insertedLines(ix.Lines, out)).To(Equal(2))
|
||||
})
|
||||
|
||||
It("appends to a variants block that already exists", func() {
|
||||
ix := indexOf(`- name: partial
|
||||
variants:
|
||||
- model: partial-q8_0
|
||||
url: u
|
||||
overrides:
|
||||
parameters:
|
||||
model: partial-Q4_K_M.gguf
|
||||
`, entryYAML("partial-f16", "acme/repo", "partial-f16.gguf", "cc"))
|
||||
out := apply(ix, []Family{{Parent: "partial", Proposals: []Proposal{{Variant: "partial-f16"}}}})
|
||||
|
||||
Expect(out[1]).To(Equal(" variants:"))
|
||||
Expect(out[2]).To(Equal(" - model: partial-q8_0"))
|
||||
Expect(out[3]).To(Equal(" - model: partial-f16"))
|
||||
Expect(out[4]).To(Equal(" url: u"))
|
||||
})
|
||||
|
||||
It("replaces an explicit empty list rather than leaving two variants keys", func() {
|
||||
ix := indexOf(`- name: emptied
|
||||
variants: []
|
||||
url: u
|
||||
`, entryYAML("emptied-q8_0", "acme/repo", "emptied-Q8_0.gguf", "cc"))
|
||||
out := apply(ix, []Family{{Parent: "emptied", Proposals: []Proposal{{Variant: "emptied-q8_0"}}}})
|
||||
|
||||
Expect(strings.Join(out[:4], "\n")).To(Equal("- name: emptied\n variants:\n - model: emptied-q8_0\n url: u"))
|
||||
Expect(strings.Count(strings.Join(out, "\n"), "variants:")).To(Equal(1))
|
||||
})
|
||||
|
||||
It("quotes a config-suffixed name so the reference stays a string", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("phi-2-chat", "acme/repo", "phi-2-chat-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("phi-2-chat:Q8_0", "acme/repo", "phi-2-chat-Q8_0.gguf", "bb"),
|
||||
)
|
||||
out := apply(ix, []Family{{Parent: "phi-2-chat", Proposals: []Proposal{{Variant: "phi-2-chat:Q8_0"}}}})
|
||||
Expect(out[2]).To(Equal(` - model: "phi-2-chat:Q8_0"`))
|
||||
|
||||
// The result has to still be a gallery, and the reference has to
|
||||
// resolve to the entry it names.
|
||||
reparsed, err := ParseIndex(strings.Join(out, "\n"))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(reparsed.Entries[0].Variants).To(ConsistOf(VariantRef{Model: "phi-2-chat:Q8_0"}))
|
||||
})
|
||||
|
||||
It("keeps line numbers correct when several entries are edited at once", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("alpha", "acme/repo", "alpha-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("alpha-q8_0", "acme/repo", "alpha-Q8_0.gguf", "bb"),
|
||||
entryYAML("beta", "acme/repo", "beta-Q4_K_M.gguf", "cc"),
|
||||
entryYAML("beta-q8_0", "acme/repo", "beta-Q8_0.gguf", "dd"),
|
||||
)
|
||||
out := apply(ix, []Family{
|
||||
{Parent: "alpha", Proposals: []Proposal{{Variant: "alpha-q8_0"}}},
|
||||
{Parent: "beta", Proposals: []Proposal{{Variant: "beta-q8_0"}}},
|
||||
})
|
||||
|
||||
reparsed, err := ParseIndex(strings.Join(out, "\n"))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(reparsed.Entries).To(HaveLen(4))
|
||||
Expect(reparsed.Entries[0].Variants).To(ConsistOf(VariantRef{Model: "alpha-q8_0"}))
|
||||
Expect(reparsed.Entries[2].Variants).To(ConsistOf(VariantRef{Model: "beta-q8_0"}))
|
||||
Expect(reparsed.Entries[1].Variants).To(BeEmpty())
|
||||
Expect(reparsed.Entries[3].Variants).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("fails loudly rather than editing an entry it cannot find", func() {
|
||||
ix := indexOf(entryYAML("only", "acme/repo", "only-Q4_K_M.gguf", "aa"))
|
||||
_, err := ApplyFamilies(ix, []Family{{Parent: "missing", Proposals: []Proposal{{Variant: "x"}}}})
|
||||
Expect(err).To(MatchError(ContainSubstring("not in the index")))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("ParseIndex", func() {
|
||||
It("records the anchor an entry defines and the anchor an entry merges", func() {
|
||||
ix := indexOf(`- &anc
|
||||
name: anchored
|
||||
url: u
|
||||
`, `- !!merge <<: *anc
|
||||
name: child
|
||||
`)
|
||||
Expect(ix.Entries[0].AnchorName).To(Equal("anc"))
|
||||
Expect(ix.Entries[1].MergesFrom).To(Equal("anc"))
|
||||
Expect(ix.MergeChildren("anc")).To(HaveLen(1))
|
||||
})
|
||||
|
||||
It("carries merged values into the child, so an inherited variants key is visible", func() {
|
||||
ix := indexOf(`- &anc
|
||||
name: anchored
|
||||
url: u
|
||||
variants:
|
||||
- model: something
|
||||
`, `- !!merge <<: *anc
|
||||
name: child
|
||||
`)
|
||||
Expect(ix.Entries[1].HasVariants()).To(BeTrue())
|
||||
})
|
||||
|
||||
It("refuses a list item that decodes to nothing", func() {
|
||||
// Every line number the editor works from comes from pairing decoded
|
||||
// entries with top level list items. If those two views can disagree,
|
||||
// the editor writes into the wrong entry, so the parse refuses instead.
|
||||
_, err := ParseIndex("- name: one\n url: u\n-\n")
|
||||
Expect(err).To(MatchError(ContainSubstring("empty")))
|
||||
})
|
||||
})
|
||||
281
.github/ci/variantproposals/index.go
vendored
281
.github/ci/variantproposals/index.go
vendored
@@ -1,281 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
|
||||
"github.com/mudler/LocalAI/.github/ci/galleryedit"
|
||||
)
|
||||
|
||||
// File is the subset of a gallery file entry the proposer reads.
|
||||
type File struct {
|
||||
Filename string `yaml:"filename"`
|
||||
URI string `yaml:"uri"`
|
||||
SHA256 string `yaml:"sha256"`
|
||||
}
|
||||
|
||||
// VariantRef mirrors the gallery's variant reference.
|
||||
type VariantRef struct {
|
||||
Model string `yaml:"model"`
|
||||
}
|
||||
|
||||
// GalleryEntry is one gallery entry, carrying both the semantics the heuristics need
|
||||
// and the text range the editor needs.
|
||||
//
|
||||
// The two views are kept together deliberately. The editor must not round-trip
|
||||
// the index through a YAML marshaller: the gallery is 40,000 lines and a
|
||||
// reflowed diff cannot be reviewed, which defeats the entire point of a job
|
||||
// whose output is a human decision.
|
||||
type GalleryEntry struct {
|
||||
Name string `yaml:"name"`
|
||||
URL string `yaml:"url"`
|
||||
ConfigFile map[string]any `yaml:"config_file"`
|
||||
Overrides map[string]any `yaml:"overrides"`
|
||||
Files []File `yaml:"files"`
|
||||
Variants []VariantRef `yaml:"variants"`
|
||||
|
||||
// Index is the entry's position in gallery order.
|
||||
Index int `yaml:"-"`
|
||||
// StartLine and EndLine bound the entry's lines, zero based and half open.
|
||||
StartLine int `yaml:"-"`
|
||||
EndLine int `yaml:"-"`
|
||||
// AnchorName is set when the entry defines a YAML anchor. Adding a variants
|
||||
// key to such an entry is inherited by everything that merges it, which is
|
||||
// why proposals involving anchors get special treatment.
|
||||
AnchorName string `yaml:"-"`
|
||||
// MergesFrom is the anchor this entry pulls in with "!!merge <<:".
|
||||
MergesFrom string `yaml:"-"`
|
||||
}
|
||||
|
||||
// Index is a parsed gallery index: entries plus the exact lines they came from.
|
||||
type Index struct {
|
||||
Lines []string
|
||||
Entries []*GalleryEntry
|
||||
}
|
||||
|
||||
var (
|
||||
anchorStart = regexp.MustCompile(`^- &(\S+)`)
|
||||
mergeStart = regexp.MustCompile(`^- !!merge <<: \*(\S+)`)
|
||||
)
|
||||
|
||||
// LoadIndex reads and parses a gallery index file.
|
||||
func LoadIndex(path string) (*Index, error) {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ParseIndex(string(data))
|
||||
}
|
||||
|
||||
// ParseIndex builds an Index from the raw text of a gallery index.
|
||||
//
|
||||
// The YAML decode and the textual scan are cross checked against each other: if
|
||||
// they disagree on how many entries there are, every line number the editor
|
||||
// would use is suspect, so the run fails rather than editing the wrong entry.
|
||||
func ParseIndex(text string) (*Index, error) {
|
||||
var entries []*GalleryEntry
|
||||
if err := yaml.Unmarshal([]byte(text), &entries); err != nil {
|
||||
return nil, fmt.Errorf("decoding gallery index: %w", err)
|
||||
}
|
||||
|
||||
lines, starts := galleryedit.Scan(text)
|
||||
if len(starts) != len(entries) {
|
||||
return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number", len(entries), len(starts))
|
||||
}
|
||||
|
||||
for i, e := range entries {
|
||||
if e == nil {
|
||||
return nil, fmt.Errorf("gallery index list item %d is empty; refusing to edit by line number", i)
|
||||
}
|
||||
e.Index = i
|
||||
e.StartLine = starts[i]
|
||||
if i+1 < len(starts) {
|
||||
e.EndLine = starts[i+1]
|
||||
} else {
|
||||
e.EndLine = len(lines)
|
||||
}
|
||||
if m := anchorStart.FindStringSubmatch(lines[e.StartLine]); m != nil {
|
||||
e.AnchorName = m[1]
|
||||
}
|
||||
if m := mergeStart.FindStringSubmatch(lines[e.StartLine]); m != nil {
|
||||
e.MergesFrom = m[1]
|
||||
}
|
||||
}
|
||||
|
||||
return &Index{Lines: lines, Entries: entries}, nil
|
||||
}
|
||||
|
||||
// MergeChildren lists the entries that pull in the given anchor.
|
||||
func (ix *Index) MergeChildren(anchor string) []*GalleryEntry {
|
||||
var out []*GalleryEntry
|
||||
for _, e := range ix.Entries {
|
||||
if e.MergesFrom == anchor {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ByName indexes entries by lowercased name. A name appearing twice keeps the
|
||||
// first occurrence, matching the gallery's own first-match-wins resolution, and
|
||||
// the duplicates are returned so the caller can refuse to touch them: a
|
||||
// proposal naming an ambiguous entry cannot be reviewed.
|
||||
func (ix *Index) ByName() (map[string]*GalleryEntry, map[string]int) {
|
||||
byName := make(map[string]*GalleryEntry, len(ix.Entries))
|
||||
counts := make(map[string]int, len(ix.Entries))
|
||||
for _, e := range ix.Entries {
|
||||
key := strings.ToLower(e.Name)
|
||||
counts[key]++
|
||||
if _, seen := byName[key]; !seen {
|
||||
byName[key] = e
|
||||
}
|
||||
}
|
||||
dupes := map[string]int{}
|
||||
for name, n := range counts {
|
||||
if n > 1 {
|
||||
dupes[name] = n
|
||||
}
|
||||
}
|
||||
return byName, dupes
|
||||
}
|
||||
|
||||
// Installable reports whether installing this entry would put anything on disk.
|
||||
// A variant target that installs nothing is a dead end for the selector, so it
|
||||
// is never proposed as one.
|
||||
func (e *GalleryEntry) Installable() bool {
|
||||
return e.URL != "" || len(e.ConfigFile) > 0 || len(e.Overrides) > 0 || len(e.Files) > 0
|
||||
}
|
||||
|
||||
// HasVariants reports whether the entry already offers builds of its own. Such
|
||||
// an entry cannot be a variant target: nesting is what the gallery's own
|
||||
// resolution refuses.
|
||||
func (e *GalleryEntry) HasVariants() bool {
|
||||
return len(e.Variants) > 0
|
||||
}
|
||||
|
||||
// auxiliaryFile matches the shared side files that several unrelated models
|
||||
// legitimately hand out the same copy of. Grouping on one of these is how an
|
||||
// earlier sweep linked four wan-2.1 entries to each other and Z-Image-Turbo to
|
||||
// qwen3-4b: they shared a text encoder, not weights.
|
||||
var auxiliaryFile = regexp.MustCompile(`(?i)(mmproj|vae|clip|t5|umt5|text_?encoder|tokenizer|\bae\b|^ae\.|scheduler|config)`)
|
||||
|
||||
// IsAuxiliaryFile reports whether a filename is a side file rather than the
|
||||
// model's own weights.
|
||||
func IsAuxiliaryFile(filename string) bool {
|
||||
base := filename
|
||||
if i := strings.LastIndex(base, "/"); i >= 0 {
|
||||
base = base[i+1:]
|
||||
}
|
||||
return auxiliaryFile.MatchString(base)
|
||||
}
|
||||
|
||||
// PrimaryWeightFile returns the filename of the entry's own weights, and
|
||||
// whether one could be identified unambiguously.
|
||||
//
|
||||
// The declared overrides.parameters.model wins because that is the file the
|
||||
// backend is actually pointed at. Falling back to the file list only works when
|
||||
// exactly one non-auxiliary file is present; anything else is ambiguous, and
|
||||
// guessing is precisely the failure mode this heuristic has already had.
|
||||
func (e *GalleryEntry) PrimaryWeightFile() (string, bool) {
|
||||
if params, ok := e.Overrides["parameters"].(map[string]any); ok {
|
||||
if model, ok := params["model"].(string); ok && model != "" && !IsAuxiliaryFile(model) {
|
||||
return model, true
|
||||
}
|
||||
}
|
||||
var candidates []string
|
||||
for _, f := range e.Files {
|
||||
if f.Filename == "" || IsAuxiliaryFile(f.Filename) {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, f.Filename)
|
||||
}
|
||||
if len(candidates) == 1 {
|
||||
return candidates[0], true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// SourceRepo returns the upstream repository the entry's files come from, as a
|
||||
// coarse "host + owner + repo" key.
|
||||
func (e *GalleryEntry) SourceRepo() string {
|
||||
for _, f := range e.Files {
|
||||
if f.URI == "" {
|
||||
continue
|
||||
}
|
||||
return repoKey(f.URI)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func repoKey(uri string) string {
|
||||
u := strings.ToLower(uri)
|
||||
u = strings.TrimPrefix(u, "huggingface://")
|
||||
u = strings.TrimPrefix(u, "https://huggingface.co/")
|
||||
u = strings.TrimPrefix(u, "http://huggingface.co/")
|
||||
parts := strings.Split(u, "/")
|
||||
if len(parts) >= 2 {
|
||||
return parts[0] + "/" + parts[1]
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
// SameInstallPayload reports whether two entries install byte for byte the same
|
||||
// thing.
|
||||
//
|
||||
// Entries like this are aliases, not variants. whisper-1 exists so a client
|
||||
// speaking the OpenAI API can send that name and get whisper-base; folding it
|
||||
// under whisper-base as a variant would hide the very name clients send.
|
||||
func SameInstallPayload(a, b *GalleryEntry) bool {
|
||||
if a.URL != b.URL {
|
||||
return false
|
||||
}
|
||||
if !sameYAML(a.Overrides, b.Overrides) || !sameYAML(a.ConfigFile, b.ConfigFile) {
|
||||
return false
|
||||
}
|
||||
return sameChecksums(a.Files, b.Files)
|
||||
}
|
||||
|
||||
func sameChecksums(a, b []File) bool {
|
||||
if len(a) != len(b) || len(a) == 0 {
|
||||
return false
|
||||
}
|
||||
ha := make([]string, 0, len(a))
|
||||
hb := make([]string, 0, len(b))
|
||||
for _, f := range a {
|
||||
if f.SHA256 == "" {
|
||||
return false
|
||||
}
|
||||
ha = append(ha, f.SHA256)
|
||||
}
|
||||
for _, f := range b {
|
||||
if f.SHA256 == "" {
|
||||
return false
|
||||
}
|
||||
hb = append(hb, f.SHA256)
|
||||
}
|
||||
sort.Strings(ha)
|
||||
sort.Strings(hb)
|
||||
for i := range ha {
|
||||
if ha[i] != hb[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func sameYAML(a, b any) bool {
|
||||
ba, err := yaml.Marshal(a)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
bb, err := yaml.Marshal(b)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return string(ba) == string(bb)
|
||||
}
|
||||
180
.github/ci/variantproposals/ledger.go
vendored
180
.github/ci/variantproposals/ledger.go
vendored
@@ -1,180 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// Ledger records the grouping decisions a human has already made against the
|
||||
// proposer, so a declined candidate stays declined instead of coming back every
|
||||
// night until reviewers stop reading the job's pull requests.
|
||||
//
|
||||
// It is checked in next to the gallery and is meant to be edited inside the
|
||||
// proposal pull request itself: declining a family is adding one flow-mapping
|
||||
// line under pairs or groups and closing the PR.
|
||||
type Ledger struct {
|
||||
// Tokens are name segments that mark a distinct model rather than another
|
||||
// build of the same one: finetune names, language codes, product suffixes.
|
||||
// A candidate whose two names differ by any of these is never proposed.
|
||||
Tokens []LedgerToken `yaml:"tokens"`
|
||||
// Pairs are individual candidates a human considered and declined. Order
|
||||
// does not matter: the pair is matched both ways round.
|
||||
Pairs []LedgerPair `yaml:"pairs"`
|
||||
// Groups decline every pair drawn from a set at once, for families like a
|
||||
// per-language release where listing each pair would be unreadable.
|
||||
Groups []LedgerGroup `yaml:"groups"`
|
||||
}
|
||||
|
||||
type LedgerToken struct {
|
||||
Token string `yaml:"token"`
|
||||
Reason string `yaml:"reason"`
|
||||
}
|
||||
|
||||
type LedgerPair struct {
|
||||
Parent string `yaml:"parent"`
|
||||
Variant string `yaml:"variant"`
|
||||
Reason string `yaml:"reason"`
|
||||
}
|
||||
|
||||
type LedgerGroup struct {
|
||||
Members []string `yaml:"members"`
|
||||
Reason string `yaml:"reason"`
|
||||
}
|
||||
|
||||
// LoadLedger reads a ledger file. A missing file is not an error: a gallery
|
||||
// that has declined nothing yet is a legitimate state, and failing the job over
|
||||
// it would only teach people to keep an empty file around.
|
||||
func LoadLedger(path string) (*Ledger, error) {
|
||||
data, err := os.ReadFile(path)
|
||||
if os.IsNotExist(err) {
|
||||
return &Ledger{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ParseLedger(data)
|
||||
}
|
||||
|
||||
func ParseLedger(data []byte) (*Ledger, error) {
|
||||
l := &Ledger{}
|
||||
if err := yaml.Unmarshal(data, l); err != nil {
|
||||
return nil, fmt.Errorf("parsing ledger: %w", err)
|
||||
}
|
||||
for i, t := range l.Tokens {
|
||||
if strings.TrimSpace(t.Token) == "" {
|
||||
return nil, fmt.Errorf("ledger tokens[%d] has an empty token", i)
|
||||
}
|
||||
}
|
||||
for i, p := range l.Pairs {
|
||||
if strings.TrimSpace(p.Parent) == "" || strings.TrimSpace(p.Variant) == "" {
|
||||
return nil, fmt.Errorf("ledger pairs[%d] needs both parent and variant", i)
|
||||
}
|
||||
}
|
||||
return l, nil
|
||||
}
|
||||
|
||||
// Suppression is a ledger hit: why a candidate was not proposed, in words a
|
||||
// reviewer can check against the ledger file.
|
||||
type Suppression struct {
|
||||
A string
|
||||
B string
|
||||
Reason string
|
||||
}
|
||||
|
||||
func (s Suppression) String() string {
|
||||
return fmt.Sprintf("%s + %s: %s", s.A, s.B, s.Reason)
|
||||
}
|
||||
|
||||
// Suppresses reports whether the ledger has already declined pairing these two
|
||||
// entries, and why.
|
||||
//
|
||||
// The token rule is applied to the segments the two names do not share. Two
|
||||
// builds of the same weights differ only in quantization markers, so any
|
||||
// ledgered token showing up in that difference is by construction a claim that
|
||||
// the entries are different models.
|
||||
func (l *Ledger) Suppresses(a, b string) (Suppression, bool) {
|
||||
la, lb := strings.ToLower(a), strings.ToLower(b)
|
||||
for _, p := range l.Pairs {
|
||||
lp, lv := strings.ToLower(p.Parent), strings.ToLower(p.Variant)
|
||||
if (lp == la && lv == lb) || (lp == lb && lv == la) {
|
||||
return Suppression{A: a, B: b, Reason: p.Reason}, true
|
||||
}
|
||||
}
|
||||
for _, g := range l.Groups {
|
||||
var seenA, seenB bool
|
||||
for _, m := range g.Members {
|
||||
lm := strings.ToLower(m)
|
||||
if lm == la {
|
||||
seenA = true
|
||||
}
|
||||
if lm == lb {
|
||||
seenB = true
|
||||
}
|
||||
}
|
||||
if seenA && seenB {
|
||||
return Suppression{A: a, B: b, Reason: g.Reason}, true
|
||||
}
|
||||
}
|
||||
diff := differingSegments(la, lb)
|
||||
for _, t := range l.Tokens {
|
||||
token := strings.ToLower(strings.TrimSpace(t.Token))
|
||||
if _, ok := diff[token]; ok {
|
||||
reason := t.Reason
|
||||
if reason == "" {
|
||||
reason = fmt.Sprintf("names differ by %q", token)
|
||||
}
|
||||
return Suppression{A: a, B: b, Reason: fmt.Sprintf("%s (token %q)", reason, token)}, true
|
||||
}
|
||||
}
|
||||
return Suppression{}, false
|
||||
}
|
||||
|
||||
// segments splits a name into the atoms the token rules are written against.
|
||||
func segments(name string) []string {
|
||||
fields := strings.FieldsFunc(strings.ToLower(name), func(r rune) bool {
|
||||
return r == '-' || r == '_' || r == '.' || r == ':' || r == '/'
|
||||
})
|
||||
return fields
|
||||
}
|
||||
|
||||
// differingSegments returns the set of segments present in exactly one of the
|
||||
// two names.
|
||||
func differingSegments(a, b string) map[string]struct{} {
|
||||
setA := map[string]int{}
|
||||
for _, s := range segments(a) {
|
||||
setA[s]++
|
||||
}
|
||||
setB := map[string]int{}
|
||||
for _, s := range segments(b) {
|
||||
setB[s]++
|
||||
}
|
||||
diff := map[string]struct{}{}
|
||||
for s := range setA {
|
||||
if setB[s] == 0 {
|
||||
diff[s] = struct{}{}
|
||||
}
|
||||
}
|
||||
for s := range setB {
|
||||
if setA[s] == 0 {
|
||||
diff[s] = struct{}{}
|
||||
}
|
||||
}
|
||||
return diff
|
||||
}
|
||||
|
||||
// SortedSuppressions gives the ledger's effect on one run in a stable order, so
|
||||
// the pull request body reads the same way for the same gallery.
|
||||
func SortedSuppressions(in []Suppression) []Suppression {
|
||||
out := append([]Suppression(nil), in...)
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].A != out[j].A {
|
||||
return out[i].A < out[j].A
|
||||
}
|
||||
return out[i].B < out[j].B
|
||||
})
|
||||
return out
|
||||
}
|
||||
65
.github/ci/variantproposals/main.go
vendored
65
.github/ci/variantproposals/main.go
vendored
@@ -1,65 +0,0 @@
|
||||
// Command variant-proposals looks for gallery entries that are alternative
|
||||
// builds of the same weights but are not grouped under one another, and writes
|
||||
// a proposal for a human to accept or reject.
|
||||
//
|
||||
// It never decides. Grouping has gone wrong repeatedly in both directions, so
|
||||
// the job's value is catching drift and surfacing candidates with their
|
||||
// evidence, not automating the call. The scheduled workflow feeds its output to
|
||||
// a pull request in the same shape as .github/checksum_checker.sh.
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func main() {
|
||||
index := flag.String("index", "gallery/index.yaml", "path to the gallery index")
|
||||
ledger := flag.String("ledger", "gallery/variant-exclusions.yaml", "path to the rejection ledger")
|
||||
bodyOut := flag.String("body-out", "", "write the pull request body here")
|
||||
apply := flag.Bool("apply", false, "write the proposed groupings back into the index")
|
||||
flag.Parse()
|
||||
|
||||
if err := run(*index, *ledger, *bodyOut, *apply); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "variant-proposals:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func run(indexPath, ledgerPath, bodyOut string, apply bool) error {
|
||||
ix, err := LoadIndex(indexPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ledger, err := LoadLedger(ledgerPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
result := Propose(ix, ledger)
|
||||
fmt.Print(RenderSummary(result))
|
||||
|
||||
if !result.HasProposals() {
|
||||
// An empty pull request every night is how a proposal job gets muted.
|
||||
fmt.Println("nothing to propose")
|
||||
return nil
|
||||
}
|
||||
|
||||
if bodyOut != "" {
|
||||
if err := os.WriteFile(bodyOut, []byte(RenderBody(result, ledgerPath)), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if !apply {
|
||||
return nil
|
||||
}
|
||||
|
||||
lines, err := ApplyFamilies(ix, result.Families)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(indexPath, []byte(strings.Join(lines, "\n")), 0o644)
|
||||
}
|
||||
615
.github/ci/variantproposals/propose.go
vendored
615
.github/ci/variantproposals/propose.go
vendored
@@ -1,615 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Signal names the grouping heuristic that linked two entries.
|
||||
type Signal string
|
||||
|
||||
const (
|
||||
// SignalName is "same name once quantization markers are stripped".
|
||||
SignalName Signal = "name-modulo-quant"
|
||||
// SignalConfigSuffix is the ":" convention, foo:q8_0 as a build of foo.
|
||||
SignalConfigSuffix Signal = "config-suffix"
|
||||
// SignalWeightFile is "same primary weight filename once quantization
|
||||
// markers are stripped", auxiliary files excluded.
|
||||
SignalWeightFile Signal = "weight-filename"
|
||||
)
|
||||
|
||||
// Evidence is what a reviewer needs in order to agree or disagree without
|
||||
// opening HuggingFace: what the two entries share, and what differs.
|
||||
type Evidence struct {
|
||||
Signals []Signal
|
||||
SharedStem string
|
||||
SharedFile string
|
||||
SharedRepo string
|
||||
QuantTokens []string
|
||||
}
|
||||
|
||||
// Proposal is one variant target offered to one parent.
|
||||
type Proposal struct {
|
||||
Variant string
|
||||
Evidence Evidence
|
||||
}
|
||||
|
||||
// Family is a complete proposal: one parent gaining one or more variants.
|
||||
type Family struct {
|
||||
Parent string
|
||||
Proposals []Proposal
|
||||
}
|
||||
|
||||
// Refusal is a family the heuristics found but the rules would not let through.
|
||||
// Refusals are reported rather than dropped: a candidate the job keeps refusing
|
||||
// is either a rule worth revisiting or a gallery bug worth fixing.
|
||||
type Refusal struct {
|
||||
Members []string
|
||||
Reason string
|
||||
}
|
||||
|
||||
// Result is one run of the proposer.
|
||||
type Result struct {
|
||||
Families []Family
|
||||
Refusals []Refusal
|
||||
Suppressed []Suppression
|
||||
AliasSkipped []Suppression
|
||||
}
|
||||
|
||||
// HasProposals reports whether the run found anything to open a pull request
|
||||
// about. A job that opens an empty pull request every night is a job people
|
||||
// filter out of their inbox.
|
||||
func (r *Result) HasProposals() bool {
|
||||
return len(r.Families) > 0
|
||||
}
|
||||
|
||||
// sizeToken matches a parameter-count marker: 8b, 1.7b, a3b for an active
|
||||
// expert count, e2b for the Gemma effective sizes, 8x7b for a mixture.
|
||||
//
|
||||
// This is a structural rule rather than a ledger entry because it is about the
|
||||
// shape of the token, not about any one model. Different parameter sizes were
|
||||
// mis-grouped by an earlier sweep and the failure is systematic.
|
||||
var sizeToken = regexp.MustCompile(`^(?:[0-9]+(?:\.[0-9]+)?[bm]|[ae][0-9]+(?:\.[0-9]+)?b|[0-9]+x[0-9]+(?:\.[0-9]+)?b)$`)
|
||||
|
||||
func differsByParameterSize(a, b string) (string, bool) {
|
||||
for seg := range differingSegments(a, b) {
|
||||
if sizeToken.MatchString(seg) {
|
||||
return seg, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// genericFileStem lists weight filenames too generic to be evidence of
|
||||
// anything. Two entries both shipping "model.safetensors" share a convention,
|
||||
// not a set of weights.
|
||||
var genericFileStem = map[string]struct{}{
|
||||
"model": {}, "weights": {}, "pytorch_model": {}, "diffusion_pytorch_model": {},
|
||||
"consolidated": {}, "ggml-model": {}, "model-00001-of-00002": {},
|
||||
}
|
||||
|
||||
// minFileStemLength keeps short, collision-prone filename stems from linking
|
||||
// unrelated entries.
|
||||
const minFileStemLength = 6
|
||||
|
||||
type pair struct {
|
||||
a, b int
|
||||
evidence Evidence
|
||||
}
|
||||
|
||||
// Propose runs the grouping heuristics over a gallery index and returns what it
|
||||
// would offer a human, what it refused, and what the ledger silenced.
|
||||
//
|
||||
// Nothing here touches the network or git, and the index is not modified.
|
||||
func Propose(ix *Index, ledger *Ledger) *Result {
|
||||
if ledger == nil {
|
||||
ledger = &Ledger{}
|
||||
}
|
||||
result := &Result{}
|
||||
|
||||
byName, dupes := ix.ByName()
|
||||
|
||||
// Existing relationships. A target already claimed must not be claimed
|
||||
// again, and two entries already in one family need no proposal.
|
||||
claimedBy := map[string]string{}
|
||||
familyOf := map[string]string{}
|
||||
for _, e := range ix.Entries {
|
||||
if !e.HasVariants() {
|
||||
continue
|
||||
}
|
||||
familyOf[strings.ToLower(e.Name)] = strings.ToLower(e.Name)
|
||||
for _, v := range e.Variants {
|
||||
target := strings.ToLower(v.Model)
|
||||
if _, taken := claimedBy[target]; !taken {
|
||||
claimedBy[target] = strings.ToLower(e.Name)
|
||||
}
|
||||
familyOf[target] = strings.ToLower(e.Name)
|
||||
}
|
||||
}
|
||||
|
||||
candidates := map[[2]int]*Evidence{}
|
||||
|
||||
addPair := func(i, j int, sig Signal, apply func(*Evidence)) {
|
||||
if i == j {
|
||||
return
|
||||
}
|
||||
if i > j {
|
||||
i, j = j, i
|
||||
}
|
||||
key := [2]int{i, j}
|
||||
ev, ok := candidates[key]
|
||||
if !ok {
|
||||
ev = &Evidence{}
|
||||
candidates[key] = ev
|
||||
}
|
||||
for _, s := range ev.Signals {
|
||||
if s == sig {
|
||||
apply(ev)
|
||||
return
|
||||
}
|
||||
}
|
||||
ev.Signals = append(ev.Signals, sig)
|
||||
apply(ev)
|
||||
}
|
||||
|
||||
// Signal 1 and 2: entries sharing a name stem.
|
||||
byStem := map[string][]int{}
|
||||
for _, e := range ix.Entries {
|
||||
if e.Name == "" {
|
||||
continue
|
||||
}
|
||||
byStem[NameStem(e.Name)] = append(byStem[NameStem(e.Name)], e.Index)
|
||||
}
|
||||
for stem, members := range byStem {
|
||||
if len(members) < 2 {
|
||||
continue
|
||||
}
|
||||
for i := 0; i < len(members); i++ {
|
||||
for j := i + 1; j < len(members); j++ {
|
||||
a, b := ix.Entries[members[i]], ix.Entries[members[j]]
|
||||
sig := SignalName
|
||||
if HasConfigSuffix(a.Name) || HasConfigSuffix(b.Name) {
|
||||
sig = SignalConfigSuffix
|
||||
}
|
||||
// The bare parent carries no marker in its name, so the
|
||||
// evidence would read "differs by q8_0" and say nothing about
|
||||
// what the parent is. The weight filenames fill that in.
|
||||
fa, _ := a.PrimaryWeightFile()
|
||||
fb, _ := b.PrimaryWeightFile()
|
||||
addPair(members[i], members[j], sig, func(ev *Evidence) {
|
||||
ev.SharedStem = stem
|
||||
ev.QuantTokens = quantDifference(a.Name, b.Name, fa, fb)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Signal 3: entries whose own weight file is the same file at a different
|
||||
// quantization. Auxiliary files never take part.
|
||||
byFile := map[string][]int{}
|
||||
for _, e := range ix.Entries {
|
||||
primary, ok := e.PrimaryWeightFile()
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
stem := FileStem(primary)
|
||||
if len(stem) < minFileStemLength {
|
||||
continue
|
||||
}
|
||||
if _, generic := genericFileStem[stem]; generic {
|
||||
continue
|
||||
}
|
||||
byFile[stem] = append(byFile[stem], e.Index)
|
||||
}
|
||||
for stem, members := range byFile {
|
||||
if len(members) < 2 {
|
||||
continue
|
||||
}
|
||||
for i := 0; i < len(members); i++ {
|
||||
for j := i + 1; j < len(members); j++ {
|
||||
a, b := ix.Entries[members[i]], ix.Entries[members[j]]
|
||||
// The filename alone is not evidence. Publishers reuse the
|
||||
// upstream filename for finetunes and for models that merely
|
||||
// embed the base weights: bert-embeddings, an ultravox audio
|
||||
// model and a roleplay finetune all ship a file called
|
||||
// llama-3.2-1b-instruct-q4_k_m.gguf. Requiring the same
|
||||
// upstream repository turns the signal back into what it
|
||||
// claims to be, one repo publishing one file at two
|
||||
// quantizations. Two repos holding the same weights is a fact
|
||||
// no filename proves, so it stays a human call.
|
||||
repo := a.SourceRepo()
|
||||
if repo == "" || repo != b.SourceRepo() {
|
||||
continue
|
||||
}
|
||||
fa, _ := a.PrimaryWeightFile()
|
||||
fb, _ := b.PrimaryWeightFile()
|
||||
addPair(members[i], members[j], SignalWeightFile, func(ev *Evidence) {
|
||||
ev.SharedFile = stem
|
||||
ev.SharedRepo = repo
|
||||
if len(ev.QuantTokens) == 0 {
|
||||
ev.QuantTokens = quantDifference(fa, fb)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Filter candidates. Everything dropped here is dropped for a reason a
|
||||
// reviewer can read back off the ledger or the rules.
|
||||
var kept []pair
|
||||
for key, ev := range candidates {
|
||||
a, b := ix.Entries[key[0]], ix.Entries[key[1]]
|
||||
la, lb := strings.ToLower(a.Name), strings.ToLower(b.Name)
|
||||
if la == lb {
|
||||
continue
|
||||
}
|
||||
if dupes[la] > 0 || dupes[lb] > 0 {
|
||||
result.Refusals = append(result.Refusals, Refusal{
|
||||
Members: []string{a.Name, b.Name},
|
||||
Reason: "one of these names appears more than once in the gallery, so a variant reference to it is ambiguous",
|
||||
})
|
||||
continue
|
||||
}
|
||||
if fa, fb := familyOf[la], familyOf[lb]; fa != "" && fa == fb {
|
||||
continue
|
||||
}
|
||||
if seg, differs := differsByParameterSize(la, lb); differs {
|
||||
result.Suppressed = append(result.Suppressed, Suppression{
|
||||
A: a.Name, B: b.Name, Reason: fmt.Sprintf("different parameter sizes (segment %q)", seg),
|
||||
})
|
||||
continue
|
||||
}
|
||||
if s, ok := ledger.Suppresses(a.Name, b.Name); ok {
|
||||
result.Suppressed = append(result.Suppressed, s)
|
||||
continue
|
||||
}
|
||||
if SameInstallPayload(a, b) {
|
||||
result.AliasSkipped = append(result.AliasSkipped, Suppression{
|
||||
A: a.Name, B: b.Name,
|
||||
Reason: "identical install payload; these are aliases of one build, not alternative builds",
|
||||
})
|
||||
continue
|
||||
}
|
||||
kept = append(kept, pair{a: key[0], b: key[1], evidence: *ev})
|
||||
}
|
||||
|
||||
sort.Slice(kept, func(i, j int) bool {
|
||||
if kept[i].a != kept[j].a {
|
||||
return kept[i].a < kept[j].a
|
||||
}
|
||||
return kept[i].b < kept[j].b
|
||||
})
|
||||
|
||||
// Components. A pair from either signal joins the same family, so a chain
|
||||
// of alternative builds discovered by different signals stays one family
|
||||
// rather than two overlapping ones that would double claim a target.
|
||||
parent := map[int]int{}
|
||||
var find func(int) int
|
||||
find = func(x int) int {
|
||||
if p, ok := parent[x]; ok && p != x {
|
||||
parent[x] = find(p)
|
||||
return parent[x]
|
||||
}
|
||||
if _, ok := parent[x]; !ok {
|
||||
parent[x] = x
|
||||
}
|
||||
return parent[x]
|
||||
}
|
||||
union := func(x, y int) {
|
||||
rx, ry := find(x), find(y)
|
||||
if rx != ry {
|
||||
parent[ry] = rx
|
||||
}
|
||||
}
|
||||
evidenceFor := map[[2]int]Evidence{}
|
||||
for _, p := range kept {
|
||||
union(p.a, p.b)
|
||||
evidenceFor[[2]int{p.a, p.b}] = p.evidence
|
||||
}
|
||||
|
||||
components := map[int][]int{}
|
||||
for _, p := range kept {
|
||||
for _, m := range []int{p.a, p.b} {
|
||||
root := find(m)
|
||||
if !contains(components[root], m) {
|
||||
components[root] = append(components[root], m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
roots := make([]int, 0, len(components))
|
||||
for r := range components {
|
||||
roots = append(roots, r)
|
||||
}
|
||||
sort.Ints(roots)
|
||||
|
||||
proposedTargets := map[string]string{}
|
||||
for _, root := range roots {
|
||||
members := components[root]
|
||||
sort.Ints(members)
|
||||
family, refusal := buildFamily(ix, members, evidenceFor, claimedBy, proposedTargets, byName)
|
||||
if refusal != nil {
|
||||
result.Refusals = append(result.Refusals, *refusal)
|
||||
continue
|
||||
}
|
||||
if family == nil {
|
||||
continue
|
||||
}
|
||||
for _, p := range family.Proposals {
|
||||
proposedTargets[strings.ToLower(p.Variant)] = family.Parent
|
||||
}
|
||||
result.Families = append(result.Families, *family)
|
||||
}
|
||||
|
||||
sort.Slice(result.Families, func(i, j int) bool { return result.Families[i].Parent < result.Families[j].Parent })
|
||||
result.Suppressed = SortedSuppressions(result.Suppressed)
|
||||
result.AliasSkipped = SortedSuppressions(result.AliasSkipped)
|
||||
result.Refusals = dedupeRefusals(result.Refusals)
|
||||
return result
|
||||
}
|
||||
|
||||
// dedupeRefusals collapses the same refusal reached from both orderings of a
|
||||
// pair, and sorts what is left. A reviewer reading the same complaint twice
|
||||
// learns to skim the section.
|
||||
func dedupeRefusals(in []Refusal) []Refusal {
|
||||
seen := map[string]struct{}{}
|
||||
var out []Refusal
|
||||
for _, r := range in {
|
||||
members := append([]string(nil), r.Members...)
|
||||
sort.Strings(members)
|
||||
key := strings.Join(members, "\x00") + "\x00" + r.Reason
|
||||
if _, dup := seen[key]; dup {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
out = append(out, r)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if a, b := strings.Join(out[i].Members, ","), strings.Join(out[j].Members, ","); a != b {
|
||||
return a < b
|
||||
}
|
||||
return out[i].Reason < out[j].Reason
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
func contains(xs []int, x int) bool {
|
||||
for _, v := range xs {
|
||||
if v == x {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// buildFamily turns a connected component into a proposal, or refuses it.
|
||||
func buildFamily(ix *Index, members []int, evidenceFor map[[2]int]Evidence, claimedBy map[string]string, proposedTargets map[string]string, byName map[string]*GalleryEntry) (*Family, *Refusal) {
|
||||
names := make([]string, 0, len(members))
|
||||
for _, m := range members {
|
||||
names = append(names, ix.Entries[m].Name)
|
||||
}
|
||||
|
||||
parentIdx, err := selectParent(ix, members)
|
||||
if err != nil {
|
||||
return nil, &Refusal{Members: names, Reason: err.Error()}
|
||||
}
|
||||
parentEntry := ix.Entries[parentIdx]
|
||||
parentName := strings.ToLower(parentEntry.Name)
|
||||
|
||||
// A parent that is itself somebody's variant would create a chain, which
|
||||
// the gallery's own resolution refuses to install.
|
||||
if owner, claimed := claimedBy[parentName]; claimed {
|
||||
return nil, &Refusal{Members: names, Reason: fmt.Sprintf("the natural parent %q is already a variant of %q; proposing it as a parent would nest variants", parentEntry.Name, owner)}
|
||||
}
|
||||
if owner, claimed := proposedTargets[parentName]; claimed {
|
||||
return nil, &Refusal{Members: names, Reason: fmt.Sprintf("the natural parent %q is already proposed as a variant of %q; proposing it as a parent would nest variants", parentEntry.Name, owner)}
|
||||
}
|
||||
|
||||
// Adding a variants key to an anchor is inherited by every entry that
|
||||
// merges it, silently grouping models nobody proposed. Handling that means
|
||||
// editing each merging child too, which is a larger change than this job
|
||||
// should make unsupervised, so it refuses and hands the reviewer the list.
|
||||
if parentEntry.AnchorName != "" {
|
||||
children := ix.MergeChildren(parentEntry.AnchorName)
|
||||
if len(children) > 0 {
|
||||
childNames := make([]string, 0, len(children))
|
||||
for _, c := range children {
|
||||
childNames = append(childNames, c.Name)
|
||||
}
|
||||
return nil, &Refusal{
|
||||
Members: names,
|
||||
Reason: fmt.Sprintf("the parent %q defines YAML anchor &%s, and a variants key added there is inherited by the %d entries that merge it (%s). Grouping this family by hand also means adding an explicit `variants: []` to each of those entries",
|
||||
parentEntry.Name, parentEntry.AnchorName, len(children), strings.Join(childNames, ", ")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
existing := map[string]struct{}{}
|
||||
for _, v := range parentEntry.Variants {
|
||||
existing[strings.ToLower(v.Model)] = struct{}{}
|
||||
}
|
||||
|
||||
family := &Family{Parent: parentEntry.Name}
|
||||
for _, m := range members {
|
||||
if m == parentIdx {
|
||||
continue
|
||||
}
|
||||
target := ix.Entries[m]
|
||||
lower := strings.ToLower(target.Name)
|
||||
if _, already := existing[lower]; already {
|
||||
continue
|
||||
}
|
||||
if target.HasVariants() {
|
||||
return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q already offers variants of its own, so it cannot itself be a variant target", target.Name)}
|
||||
}
|
||||
if !target.Installable() {
|
||||
return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q has no url, config_file, overrides or files, so it is not independently installable", target.Name)}
|
||||
}
|
||||
if owner, claimed := claimedBy[lower]; claimed && owner != parentName {
|
||||
return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q is already a variant of %q; a target claimed by two parents is not something the gallery resolves predictably", target.Name, owner)}
|
||||
}
|
||||
if owner, claimed := proposedTargets[lower]; claimed && owner != parentEntry.Name {
|
||||
return nil, &Refusal{Members: names, Reason: fmt.Sprintf("%q is already proposed as a variant of %q in this same run", target.Name, owner)}
|
||||
}
|
||||
family.Proposals = append(family.Proposals, Proposal{
|
||||
Variant: target.Name,
|
||||
Evidence: lookupEvidence(evidenceFor, parentIdx, m),
|
||||
})
|
||||
}
|
||||
|
||||
if len(family.Proposals) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
sort.Slice(family.Proposals, func(i, j int) bool { return family.Proposals[i].Variant < family.Proposals[j].Variant })
|
||||
return family, nil
|
||||
}
|
||||
|
||||
func lookupEvidence(evidenceFor map[[2]int]Evidence, a, b int) Evidence {
|
||||
if a > b {
|
||||
a, b = b, a
|
||||
}
|
||||
if ev, ok := evidenceFor[[2]int{a, b}]; ok {
|
||||
return ev
|
||||
}
|
||||
// The two entries reached the same family through a third one. Say so
|
||||
// rather than inventing evidence that was never observed for this pair.
|
||||
return Evidence{Signals: []Signal{SignalName}}
|
||||
}
|
||||
|
||||
// selectParent picks the entry the others should hang off.
|
||||
//
|
||||
// The bare name wins when there is one: it is the name a user types and the one
|
||||
// documentation links to. Otherwise the smallest build wins, judged by the
|
||||
// quantization token in the entry's own weight filename, so the default install
|
||||
// is the one most hosts can actually run.
|
||||
func selectParent(ix *Index, members []int) (int, error) {
|
||||
// The family's own stem: the one the most members reduce to, shortest name
|
||||
// breaking a tie. An entry named exactly that is the bare entry.
|
||||
stemCount := map[string]int{}
|
||||
for _, m := range members {
|
||||
stemCount[NameStem(ix.Entries[m].Name)]++
|
||||
}
|
||||
// Only a stem two or more members reduce to is the family's own stem. A
|
||||
// stem reached by exactly one member is just that member's name, and
|
||||
// treating it as the family stem would crown whichever name happens to be
|
||||
// shortest rather than whichever build is the base one.
|
||||
familyStem := ""
|
||||
for stem, n := range stemCount {
|
||||
if n < 2 {
|
||||
continue
|
||||
}
|
||||
if familyStem == "" || n > stemCount[familyStem] ||
|
||||
(n == stemCount[familyStem] && len(stem) < len(familyStem)) ||
|
||||
(n == stemCount[familyStem] && len(stem) == len(familyStem) && stem < familyStem) {
|
||||
familyStem = stem
|
||||
}
|
||||
}
|
||||
|
||||
var bare []int
|
||||
for _, m := range members {
|
||||
e := ix.Entries[m]
|
||||
if HasConfigSuffix(e.Name) {
|
||||
continue
|
||||
}
|
||||
if strings.ToLower(e.Name) == familyStem {
|
||||
bare = append(bare, m)
|
||||
}
|
||||
}
|
||||
if len(bare) == 1 {
|
||||
return bare[0], nil
|
||||
}
|
||||
if len(bare) > 1 {
|
||||
names := make([]string, 0, len(bare))
|
||||
for _, m := range bare {
|
||||
names = append(names, ix.Entries[m].Name)
|
||||
}
|
||||
return 0, fmt.Errorf("more than one entry is named exactly %q (%s), so which one is the base build is a judgement this job will not make", familyStem, strings.Join(names, ", "))
|
||||
}
|
||||
|
||||
// No shared stem to be named after. An entry whose name every other member
|
||||
// extends is still recognisably the base one, and this is the only handle
|
||||
// left for families whose weights carry no readable quantization token at
|
||||
// all, such as the ONNX builds.
|
||||
if prefix, ok := uniquePrefixMember(ix, members); ok {
|
||||
return prefix, nil
|
||||
}
|
||||
|
||||
best := -1
|
||||
bestWidth := 1 << 20
|
||||
for _, m := range members {
|
||||
e := ix.Entries[m]
|
||||
width := unknownWidth
|
||||
if primary, ok := e.PrimaryWeightFile(); ok {
|
||||
width = BuildWidth(primary)
|
||||
}
|
||||
// Members are visited in gallery order, so a strict comparison leaves
|
||||
// the earliest entry holding a tie and the choice is deterministic.
|
||||
if width < bestWidth {
|
||||
best, bestWidth = m, width
|
||||
}
|
||||
}
|
||||
if best < 0 {
|
||||
return 0, fmt.Errorf("no member could be identified as the smallest build")
|
||||
}
|
||||
if bestWidth == unknownWidth {
|
||||
names := make([]string, 0, len(members))
|
||||
for _, m := range members {
|
||||
names = append(names, ix.Entries[m].Name)
|
||||
}
|
||||
return 0, fmt.Errorf("no member declares a weight file whose quantization can be read (%s), so the smallest build cannot be identified", strings.Join(names, ", "))
|
||||
}
|
||||
return best, nil
|
||||
}
|
||||
|
||||
// uniquePrefixMember reports the single member whose name every other member's
|
||||
// name starts with, if there is exactly one.
|
||||
func uniquePrefixMember(ix *Index, members []int) (int, bool) {
|
||||
found := -1
|
||||
for _, m := range members {
|
||||
name := strings.ToLower(ix.Entries[m].Name)
|
||||
isPrefix := true
|
||||
for _, other := range members {
|
||||
if other == m {
|
||||
continue
|
||||
}
|
||||
if !strings.HasPrefix(strings.ToLower(ix.Entries[other].Name), name) {
|
||||
isPrefix = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if !isPrefix {
|
||||
continue
|
||||
}
|
||||
if found >= 0 {
|
||||
return 0, false
|
||||
}
|
||||
found = m
|
||||
}
|
||||
return found, found >= 0
|
||||
}
|
||||
|
||||
// quantDifference lists the quantization tokens that tell two names apart. It
|
||||
// is the compact form of the evidence: "these differ only by q4_k_m vs q8_0".
|
||||
func quantDifference(names ...string) []string {
|
||||
var out []string
|
||||
seen := map[string]struct{}{}
|
||||
for _, name := range names {
|
||||
// Filenames arrive here too, so the extension goes first and "/" counts
|
||||
// as a separator. "_" deliberately does not: it holds "q4_k_m" together.
|
||||
trimmed := weightExtension.ReplaceAllString(name, "")
|
||||
for _, seg := range strings.FieldsFunc(strings.ToLower(trimmed), func(r rune) bool { return r == '-' || r == '/' }) {
|
||||
if !IsQuantToken(seg) {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[seg]; ok {
|
||||
continue
|
||||
}
|
||||
seen[seg] = struct{}{}
|
||||
out = append(out, seg)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
429
.github/ci/variantproposals/propose_test.go
vendored
429
.github/ci/variantproposals/propose_test.go
vendored
@@ -1,429 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// entryYAML writes one gallery entry with a single weight file, which is the
|
||||
// shape almost every real entry has. Specs that need something else write the
|
||||
// YAML out by hand.
|
||||
func entryYAML(name, repo, filename, sha string) string {
|
||||
return fmt.Sprintf(`- name: %s
|
||||
url: github:mudler/LocalAI/gallery/virtual.yaml@master
|
||||
overrides:
|
||||
parameters:
|
||||
model: %s
|
||||
files:
|
||||
- filename: %s
|
||||
uri: huggingface://%s/%s
|
||||
sha256: %s
|
||||
`, name, filename, filename, repo, filename, sha)
|
||||
}
|
||||
|
||||
func indexOf(entries ...string) *Index {
|
||||
ix, err := ParseIndex(strings.Join(entries, ""))
|
||||
ExpectWithOffset(1, err).ToNot(HaveOccurred())
|
||||
return ix
|
||||
}
|
||||
|
||||
// familyNames flattens a result into "parent <- variant, variant" strings, the
|
||||
// form the specs assert against.
|
||||
func familyNames(r *Result) []string {
|
||||
out := make([]string, 0, len(r.Families))
|
||||
for _, f := range r.Families {
|
||||
names := make([]string, 0, len(f.Proposals))
|
||||
for _, p := range f.Proposals {
|
||||
names = append(names, p.Variant)
|
||||
}
|
||||
out = append(out, f.Parent+" <- "+strings.Join(names, ", "))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func refusalReasons(r *Result) string {
|
||||
var b strings.Builder
|
||||
for _, ref := range r.Refusals {
|
||||
b.WriteString(strings.Join(ref.Members, " + ") + ": " + ref.Reason + "\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func suppressionReasons(r *Result) string {
|
||||
var b strings.Builder
|
||||
for _, s := range r.Suppressed {
|
||||
b.WriteString(s.String() + "\n")
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
var _ = Describe("Propose", func() {
|
||||
Describe("the grouping signals", func() {
|
||||
It("groups entries whose names differ only by a quantization marker", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("foo-model", "acme/foo-GGUF", "foo-model-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("foo-model-q8_0", "acme/foo-GGUF", "foo-model-Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(ConsistOf("foo-model <- foo-model-q8_0"))
|
||||
Expect(r.Families[0].Proposals[0].Evidence.Signals).To(ContainElement(SignalName))
|
||||
Expect(r.Families[0].Proposals[0].Evidence.SharedStem).To(Equal("foo-model"))
|
||||
Expect(r.Families[0].Proposals[0].Evidence.QuantTokens).To(ContainElements("q4_k_m", "q8_0"))
|
||||
})
|
||||
|
||||
It("groups entries that use the colon config-suffix convention", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("bar-model", "acme/bar-GGUF", "bar-model-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("bar-model:grammar-functioncall", "acme/bar-GGUF", "bar-model-Q4_K_M-grammar.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(ConsistOf("bar-model <- bar-model:grammar-functioncall"))
|
||||
Expect(r.Families[0].Proposals[0].Evidence.Signals).To(ContainElement(SignalConfigSuffix))
|
||||
})
|
||||
|
||||
It("groups entries whose own weight file is the same file at another quantization", func() {
|
||||
// The names share no stem, so only the filename signal can link
|
||||
// these two.
|
||||
ix := indexOf(
|
||||
entryYAML("omni-cpp", "Serveurperso/Omni-GGUF", "omnivoice-base-Q8_0.gguf", "aa"),
|
||||
entryYAML("omni-cpp-hq", "Serveurperso/Omni-GGUF", "omnivoice-base-BF16.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(ConsistOf("omni-cpp <- omni-cpp-hq"))
|
||||
ev := r.Families[0].Proposals[0].Evidence
|
||||
Expect(ev.Signals).To(ConsistOf(SignalWeightFile))
|
||||
Expect(ev.SharedFile).To(Equal("omnivoice-base"))
|
||||
Expect(ev.SharedRepo).To(Equal("serveurperso/omni-gguf"))
|
||||
})
|
||||
|
||||
It("does not let a shared auxiliary file link unrelated models", func() {
|
||||
// Both entries ship the same text encoder. That is a packaging
|
||||
// convention, not evidence of shared weights: this is how an
|
||||
// earlier sweep linked four wan-2.1 entries to each other.
|
||||
ix := indexOf(`- name: wan-2.1-t2v
|
||||
url: u
|
||||
files:
|
||||
- filename: wan-2.1-t2v-Q4_K_M.gguf
|
||||
uri: huggingface://acme/wan/wan-2.1-t2v-Q4_K_M.gguf
|
||||
sha256: aa
|
||||
- filename: umt5-xxl-encoder-Q8_0.gguf
|
||||
uri: huggingface://acme/wan/umt5-xxl-encoder-Q8_0.gguf
|
||||
sha256: cc
|
||||
`, `- name: z-image-turbo
|
||||
url: u
|
||||
files:
|
||||
- filename: z-image-turbo-Q4_K_M.gguf
|
||||
uri: huggingface://acme/wan/z-image-turbo-Q4_K_M.gguf
|
||||
sha256: bb
|
||||
- filename: umt5-xxl-encoder-Q8_0.gguf
|
||||
uri: huggingface://acme/wan/umt5-xxl-encoder-Q8_0.gguf
|
||||
sha256: cc
|
||||
`)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("does not treat a shared filename in two different repos as evidence", func() {
|
||||
// A finetune republished under the base model's filename is the
|
||||
// most common way this signal misfires.
|
||||
ix := indexOf(
|
||||
entryYAML("llama-3.2-3b-instruct", "hugging-quants/Llama-3.2-3B-Instruct-GGUF", "llama-3.2-3b-instruct-q4_k_m.gguf", "aa"),
|
||||
entryYAML("llama-3.2-3b-shiro-roleplay", "someone/Shiro-GGUF", "Llama-3.2-3B-Instruct.Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("what must never be proposed", func() {
|
||||
It("does not group different parameter sizes that share a prefix", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("qwen3-tts-cpp-0.6b-base", "Serveurperso/Qwen3-TTS-GGUF", "qwen3-tts-talker-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("qwen3-tts-cpp-1.7b-base", "Serveurperso/Qwen3-TTS-GGUF", "qwen3-tts-talker-Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(suppressionReasons(r)).To(ContainSubstring("different parameter sizes"))
|
||||
})
|
||||
|
||||
It("does not group the Gemma effective sizes", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("gemma-4-e2b-it", "google/gemma-GGUF", "gemma-4-it-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("gemma-4-e4b-it", "google/gemma-GGUF", "gemma-4-it-Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(suppressionReasons(r)).To(ContainSubstring("different parameter sizes"))
|
||||
})
|
||||
|
||||
It("does not group entries with a byte-identical install payload", func() {
|
||||
// whisper-1 exists so OpenAI-compatible clients can send that name.
|
||||
// Folding it under whisper-base would hide the name they send.
|
||||
payload := ` url: github:mudler/LocalAI/gallery/whisper-base.yaml@master
|
||||
overrides:
|
||||
parameters:
|
||||
model: ggml-whisper-base.bin
|
||||
files:
|
||||
- filename: ggml-whisper-base.bin
|
||||
uri: huggingface://ggerganov/whisper.cpp/ggml-base.bin
|
||||
sha256: aa
|
||||
`
|
||||
ix := indexOf("- name: whisper-base\n"+payload, "- name: whisper-1\n"+payload)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(r.AliasSkipped).To(HaveLen(1))
|
||||
Expect(r.AliasSkipped[0].Reason).To(ContainSubstring("aliases"))
|
||||
})
|
||||
|
||||
DescribeTable("declines the categories the ledger records",
|
||||
func(nameA, nameB string, ledgerYAML string) {
|
||||
ix := indexOf(
|
||||
entryYAML(nameA, "acme/repo", "shared-weights-Q4_K_M.gguf", "aa"),
|
||||
entryYAML(nameB, "acme/repo", "shared-weights-Q8_0.gguf", "bb"),
|
||||
)
|
||||
ledger, err := ParseLedger([]byte(ledgerYAML))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
// Without the ledger these would be proposed, which is what
|
||||
// makes the ledger load bearing rather than decorative.
|
||||
Expect(familyNames(Propose(ix, nil))).ToNot(BeEmpty())
|
||||
|
||||
r := Propose(ix, ledger)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(r.Suppressed).To(HaveLen(1))
|
||||
},
|
||||
Entry("a finetune", "base-model", "base-model-abliterated",
|
||||
"tokens:\n - {token: abliterated, reason: finetune}\n"),
|
||||
Entry("a distill", "base-model", "base-model-distilled",
|
||||
"tokens:\n - {token: distilled, reason: distilled}\n"),
|
||||
Entry("English-only versus multilingual ASR", "whisper-small", "whisper-small-en",
|
||||
"pairs:\n - {parent: whisper-small, variant: whisper-small-en, reason: English-only versus multilingual}\n"),
|
||||
Entry("two products sharing a prefix", "vibevoice-cpp", "vibevoice-cpp-asr",
|
||||
"pairs:\n - {parent: vibevoice-cpp, variant: vibevoice-cpp-asr, reason: different products}\n"),
|
||||
Entry("a per-language release", "kokoros-de", "kokoros-ja",
|
||||
"groups:\n - {members: [kokoros, kokoros-de, kokoros-ja], reason: different languages}\n"),
|
||||
)
|
||||
|
||||
It("reports the ledger's reason so its effect stays visible", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("base-model", "acme/repo", "shared-weights-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("base-model-heretic", "acme/repo", "shared-weights-Q8_0.gguf", "bb"),
|
||||
)
|
||||
ledger, err := ParseLedger([]byte("tokens:\n - {token: heretic, reason: \"finetune, not a re-quantization\"}\n"))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
r := Propose(ix, ledger)
|
||||
Expect(suppressionReasons(r)).To(ContainSubstring("finetune, not a re-quantization"))
|
||||
Expect(suppressionReasons(r)).To(ContainSubstring(`token "heretic"`))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("parent selection", func() {
|
||||
It("picks the bare-named entry when one exists", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("base-model-q8_0", "acme/repo", "base-model-Q8_0.gguf", "aa"),
|
||||
entryYAML("base-model", "acme/repo", "base-model-Q4_K_M.gguf", "bb"),
|
||||
entryYAML("base-model-f16", "acme/repo", "base-model-f16.gguf", "cc"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(ConsistOf("base-model <- base-model-f16, base-model-q8_0"))
|
||||
})
|
||||
|
||||
It("picks the smallest build when no entry is bare-named", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("ced-base-f16", "acme/repo", "ced-base-f16.gguf", "aa"),
|
||||
entryYAML("ced-base-q8", "acme/repo", "ced-base-Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(ConsistOf("ced-base-q8 <- ced-base-f16"))
|
||||
})
|
||||
|
||||
It("judges the smallest build by the quantization in the model filename, not the name", func() {
|
||||
// The names carry no marker at all; only the filenames say which
|
||||
// build is which.
|
||||
ix := indexOf(
|
||||
entryYAML("thing-hq", "acme/repo", "thing-weights-BF16.gguf", "aa"),
|
||||
entryYAML("thing-lite", "acme/repo", "thing-weights-Q4_K_M.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(ConsistOf("thing-lite <- thing-hq"))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("the rules a proposal has to respect", func() {
|
||||
It("refuses to nest: a target that already offers variants of its own", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("nest-model", "acme/repo", "nest-model-Q4_K_M.gguf", "aa"),
|
||||
`- name: nest-model-q8_0
|
||||
url: u
|
||||
variants:
|
||||
- model: nest-model-q8_0-mtp
|
||||
overrides:
|
||||
parameters:
|
||||
model: nest-model-Q8_0.gguf
|
||||
files:
|
||||
- filename: nest-model-Q8_0.gguf
|
||||
uri: huggingface://acme/repo/nest-model-Q8_0.gguf
|
||||
sha256: bb
|
||||
`,
|
||||
entryYAML("nest-model-q8_0-mtp", "other/repo", "nest-model-mtp.gguf", "cc"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("already offers variants of its own"))
|
||||
})
|
||||
|
||||
It("refuses to nest: a parent that is already somebody else's variant", func() {
|
||||
ix := indexOf(
|
||||
`- name: outer
|
||||
url: u
|
||||
variants:
|
||||
- model: middle
|
||||
overrides:
|
||||
parameters:
|
||||
model: outer-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: outer-Q4_K_M.gguf
|
||||
uri: huggingface://acme/repo/outer-Q4_K_M.gguf
|
||||
sha256: aa
|
||||
`,
|
||||
entryYAML("middle", "acme/other", "middle-Q4_K_M.gguf", "bb"),
|
||||
entryYAML("middle-q8_0", "acme/other", "middle-Q8_0.gguf", "cc"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("would nest variants"))
|
||||
})
|
||||
|
||||
It("refuses to let two parents claim one target", func() {
|
||||
ix := indexOf(
|
||||
`- name: claimant
|
||||
url: u
|
||||
variants:
|
||||
- model: contested-q8_0
|
||||
overrides:
|
||||
parameters:
|
||||
model: claimant-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: claimant-Q4_K_M.gguf
|
||||
uri: huggingface://acme/repo/claimant-Q4_K_M.gguf
|
||||
sha256: aa
|
||||
`,
|
||||
entryYAML("contested", "acme/other", "contested-Q4_K_M.gguf", "bb"),
|
||||
entryYAML("contested-q8_0", "acme/other", "contested-Q8_0.gguf", "cc"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("already a variant of"))
|
||||
})
|
||||
|
||||
It("refuses a target that is not independently installable", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("stub-model", "acme/repo", "stub-model-Q4_K_M.gguf", "aa"),
|
||||
"- name: stub-model-q8_0\n description: a stanza nobody finished\n",
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("not independently installable"))
|
||||
})
|
||||
|
||||
It("refuses a family whose parent defines a merge anchor, naming the entries that would inherit", func() {
|
||||
ix := indexOf(
|
||||
`- &anchored
|
||||
name: anchored-model
|
||||
url: u
|
||||
overrides:
|
||||
parameters:
|
||||
model: anchored-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: anchored-Q4_K_M.gguf
|
||||
uri: huggingface://acme/repo/anchored-Q4_K_M.gguf
|
||||
sha256: aa
|
||||
`,
|
||||
`- !!merge <<: *anchored
|
||||
name: anchored-child
|
||||
variants: []
|
||||
overrides:
|
||||
parameters:
|
||||
model: unrelated-child-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: unrelated-child-Q4_K_M.gguf
|
||||
uri: huggingface://other/repo/unrelated-child-Q4_K_M.gguf
|
||||
sha256: cc
|
||||
`,
|
||||
entryYAML("anchored-model-q8_0", "acme/repo", "anchored-Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("defines YAML anchor &anchored"))
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("anchored-child"))
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("variants: []"))
|
||||
})
|
||||
|
||||
It("refuses an entry whose name is not unique in the gallery", func() {
|
||||
ix := indexOf(
|
||||
entryYAML("twin", "acme/repo", "twin-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("twin", "acme/repo", "twin-Q4_K_M.gguf", "aa"),
|
||||
entryYAML("twin-q8_0", "acme/repo", "twin-Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(BeEmpty())
|
||||
Expect(refusalReasons(r)).To(ContainSubstring("appears more than once"))
|
||||
})
|
||||
|
||||
It("says nothing about a pair that is already grouped", func() {
|
||||
ix := indexOf(
|
||||
`- name: settled
|
||||
url: u
|
||||
variants:
|
||||
- model: settled-q8_0
|
||||
overrides:
|
||||
parameters:
|
||||
model: settled-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: settled-Q4_K_M.gguf
|
||||
uri: huggingface://acme/repo/settled-Q4_K_M.gguf
|
||||
sha256: aa
|
||||
`,
|
||||
entryYAML("settled-q8_0", "acme/repo", "settled-Q8_0.gguf", "bb"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(r.HasProposals()).To(BeFalse())
|
||||
Expect(r.Refusals).To(BeEmpty())
|
||||
Expect(r.Suppressed).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("adds only the missing members to a family that already exists", func() {
|
||||
ix := indexOf(
|
||||
`- name: partial
|
||||
url: u
|
||||
variants:
|
||||
- model: partial-q8_0
|
||||
overrides:
|
||||
parameters:
|
||||
model: partial-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: partial-Q4_K_M.gguf
|
||||
uri: huggingface://acme/repo/partial-Q4_K_M.gguf
|
||||
sha256: aa
|
||||
`,
|
||||
entryYAML("partial-q8_0", "acme/repo", "partial-Q8_0.gguf", "bb"),
|
||||
entryYAML("partial-f16", "acme/repo", "partial-f16.gguf", "cc"),
|
||||
)
|
||||
r := Propose(ix, nil)
|
||||
Expect(familyNames(r)).To(ConsistOf("partial <- partial-f16"))
|
||||
})
|
||||
})
|
||||
|
||||
It("does not modify the index it was given", func() {
|
||||
text := entryYAML("foo-model", "acme/foo-GGUF", "foo-model-Q4_K_M.gguf", "aa") +
|
||||
entryYAML("foo-model-q8_0", "acme/foo-GGUF", "foo-model-Q8_0.gguf", "bb")
|
||||
ix, err := ParseIndex(text)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
before := strings.Join(ix.Lines, "\n")
|
||||
Propose(ix, nil)
|
||||
Expect(strings.Join(ix.Lines, "\n")).To(Equal(before))
|
||||
})
|
||||
})
|
||||
151
.github/ci/variantproposals/quant.go
vendored
151
.github/ci/variantproposals/quant.go
vendored
@@ -1,151 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Quantization and precision markers that distinguish one build of a set of
|
||||
// weights from another build of the same weights. Stripping them from a name
|
||||
// is what lets the proposer notice that two entries are the same model.
|
||||
//
|
||||
// qat and apex are in this list on a maintainer ruling: they are quantization
|
||||
// techniques applied to published weights, not separate weights. Names that use
|
||||
// "apex" to mean a finetune are handled by the rejection ledger instead, because
|
||||
// no amount of pattern matching can tell the two uses apart.
|
||||
const quantAlternation = `q[2-8](?:_[0-9a-z]+)*|pq[2-8](?:_[0-9a-z]+)*|iq[1-9][0-9a-z]*(?:_[0-9a-z]+)*|i1|` +
|
||||
`f16|f32|bf16|fp16|fp32|fp8|fp4|nvfp4|mxfp4(?:_moe)*|awq|gptq|qat|apex|gguf|ggml|[0-9]+bit|g[0-9]+`
|
||||
|
||||
// quantSegment matches a whole hyphen-delimited segment of an entry name.
|
||||
// Names separate their parts with "-" and keep quantization tokens internally
|
||||
// joined with "_", so a segment is the right unit here: "q4_k_m" arrives whole.
|
||||
var quantSegment = regexp.MustCompile(`^(?:` + quantAlternation + `)$`)
|
||||
|
||||
// quantFileSuffix matches a trailing quantization token in a weight filename.
|
||||
// Filenames mix "-", "_" and "." as separators, so unlike entry names they
|
||||
// cannot be split into segments up front without tearing "Q4_K_M" apart.
|
||||
var quantFileSuffix = regexp.MustCompile(`(?i)[-_.](?:` + quantAlternation + `)$`)
|
||||
|
||||
var weightExtension = regexp.MustCompile(`(?i)\.(gguf|ggml|safetensors|bin|pt|pth|onnx)$`)
|
||||
|
||||
// IsQuantToken reports whether a single name segment is a quantization or
|
||||
// precision marker rather than part of the model's identity.
|
||||
func IsQuantToken(segment string) bool {
|
||||
return quantSegment.MatchString(strings.ToLower(segment))
|
||||
}
|
||||
|
||||
// NameStem reduces an entry name to the identity it shares with its alternative
|
||||
// builds: the config suffix after ":" is dropped, then trailing quantization
|
||||
// segments are stripped.
|
||||
//
|
||||
// It implements the first two grouping signals together because they answer the
|
||||
// same question. "foo:q8_0" and "foo-q8_0" are both alternative builds of "foo",
|
||||
// and the caller that needs to report which convention was used can compare the
|
||||
// name against the stem itself.
|
||||
//
|
||||
// At least one segment always survives, so a name made entirely of quantization
|
||||
// tokens does not collapse to the empty stem and swallow every other such name.
|
||||
func NameStem(name string) string {
|
||||
base := strings.ToLower(strings.TrimSpace(name))
|
||||
if i := strings.Index(base, ":"); i >= 0 {
|
||||
base = base[:i]
|
||||
}
|
||||
segments := strings.Split(base, "-")
|
||||
for len(segments) > 1 && quantSegment.MatchString(segments[len(segments)-1]) {
|
||||
segments = segments[:len(segments)-1]
|
||||
}
|
||||
return strings.Join(segments, "-")
|
||||
}
|
||||
|
||||
// HasConfigSuffix reports whether a name uses the ":" convention for naming a
|
||||
// config variant of another entry.
|
||||
func HasConfigSuffix(name string) bool {
|
||||
return strings.Contains(name, ":")
|
||||
}
|
||||
|
||||
// FileStem reduces a weight filename to the identity shared by its other
|
||||
// quantizations: directories, extension and trailing quantization tokens go.
|
||||
//
|
||||
// This is the third grouping signal. It is the one that has misfired before, so
|
||||
// callers must filter auxiliary files out before handing a filename here: a
|
||||
// shared text encoder is not evidence of shared weights.
|
||||
func FileStem(filename string) string {
|
||||
base := filename
|
||||
if i := strings.LastIndex(base, "/"); i >= 0 {
|
||||
base = base[i+1:]
|
||||
}
|
||||
base = weightExtension.ReplaceAllString(base, "")
|
||||
for {
|
||||
stripped := quantFileSuffix.ReplaceAllString(base, "")
|
||||
if stripped == base {
|
||||
break
|
||||
}
|
||||
base = stripped
|
||||
}
|
||||
return strings.ToLower(base)
|
||||
}
|
||||
|
||||
// bitsPerWeight ranks quantization tokens so the smallest build of a family can
|
||||
// be identified when no bare-named entry exists to be the parent.
|
||||
//
|
||||
// The figures are nominal bits per weight, not measured file sizes. Ranking is
|
||||
// all that is asked of them, and a nominal figure is available from the name
|
||||
// alone without downloading anything.
|
||||
func bitsPerWeight(token string) (int, bool) {
|
||||
t := strings.ToLower(token)
|
||||
switch {
|
||||
case t == "i1":
|
||||
return 1, true
|
||||
case strings.HasPrefix(t, "nvfp4"), strings.HasPrefix(t, "mxfp4"), t == "fp4":
|
||||
return 4, true
|
||||
case t == "fp8":
|
||||
return 8, true
|
||||
case t == "f16", t == "bf16", t == "fp16":
|
||||
return 16, true
|
||||
case t == "f32", t == "fp32":
|
||||
return 32, true
|
||||
case t == "awq", t == "gptq":
|
||||
return 4, true
|
||||
}
|
||||
if m := regexp.MustCompile(`^p?q([1-9])`).FindStringSubmatch(t); m != nil {
|
||||
n, _ := strconv.Atoi(m[1])
|
||||
return n, true
|
||||
}
|
||||
if m := regexp.MustCompile(`^iq([1-9])`).FindStringSubmatch(t); m != nil {
|
||||
n, _ := strconv.Atoi(m[1])
|
||||
return n, true
|
||||
}
|
||||
if m := regexp.MustCompile(`^([0-9]+)bit$`).FindStringSubmatch(t); m != nil {
|
||||
n, _ := strconv.Atoi(m[1])
|
||||
return n, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// unknownWidth sorts after every recognised quantization so an entry whose
|
||||
// build cannot be read from its filename never wins the "smallest build" tie
|
||||
// break by accident.
|
||||
const unknownWidth = 1 << 10
|
||||
|
||||
// BuildWidth reports the nominal bits per weight of the build a filename holds.
|
||||
// An unreadable filename gets unknownWidth.
|
||||
func BuildWidth(filename string) int {
|
||||
base := filename
|
||||
if i := strings.LastIndex(base, "/"); i >= 0 {
|
||||
base = base[i+1:]
|
||||
}
|
||||
base = weightExtension.ReplaceAllString(base, "")
|
||||
best := unknownWidth
|
||||
for {
|
||||
m := quantFileSuffix.FindString(base)
|
||||
if m == "" {
|
||||
break
|
||||
}
|
||||
if bits, ok := bitsPerWeight(m[1:]); ok && bits < best {
|
||||
best = bits
|
||||
}
|
||||
base = base[:len(base)-len(m)]
|
||||
}
|
||||
return best
|
||||
}
|
||||
92
.github/ci/variantproposals/quant_test.go
vendored
92
.github/ci/variantproposals/quant_test.go
vendored
@@ -1,92 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("quantization markers", func() {
|
||||
DescribeTable("NameStem strips the markers that distinguish builds, not models",
|
||||
func(name, expected string) {
|
||||
Expect(NameStem(name)).To(Equal(expected))
|
||||
},
|
||||
Entry("plain q4", "foo-model-q4_k_m", "foo-model"),
|
||||
Entry("q8_0", "foo-model-q8_0", "foo-model"),
|
||||
Entry("q5_1", "foo-model-q5_1", "foo-model"),
|
||||
Entry("q2 with group size", "ternary-bonsai-8b-q2-g64", "ternary-bonsai-8b"),
|
||||
Entry("iq variant", "ideogram-4-iq4nl-ggml", "ideogram-4"),
|
||||
Entry("i1 imatrix", "orca-agent-v0.1-i1", "orca-agent-v0.1"),
|
||||
Entry("f16", "ced-base-f16", "ced-base"),
|
||||
Entry("bf16", "some-model-bf16", "some-model"),
|
||||
Entry("fp8", "some-model-fp8", "some-model"),
|
||||
Entry("nvfp4", "qwen3.6-27b-nvfp4", "qwen3.6-27b"),
|
||||
Entry("mxfp4_moe", "huihui-qwen3-vl-30b-a3b-instruct-abliterated-mxfp4_moe", "huihui-qwen3-vl-30b-a3b-instruct-abliterated"),
|
||||
Entry("pq2", "ternary-bonsai-8b-pq2", "ternary-bonsai-8b"),
|
||||
Entry("awq", "some-model-awq", "some-model"),
|
||||
Entry("gptq", "some-model-gptq", "some-model"),
|
||||
Entry("Nbit", "qwen3-8b-mlx-4bit", "qwen3-8b-mlx"),
|
||||
Entry("gguf", "some-model-gguf", "some-model"),
|
||||
Entry("ggml", "flux.1-dev-ggml", "flux.1-dev"),
|
||||
Entry("qat is a quantization technique", "gemma-3-27b-it-qat", "gemma-3-27b-it"),
|
||||
Entry("apex is a quantization technique", "qwen3.6-35b-a3b-apex", "qwen3.6-35b-a3b"),
|
||||
Entry("stacked markers", "gemma-4-e2b-it-qat-q4_0", "gemma-4-e2b-it"),
|
||||
Entry("the config suffix is dropped", "phi-2-chat:Q8_0", "phi-2-chat"),
|
||||
Entry("a non-quant config suffix is dropped too", "meta-llama-3.1-8b-instruct:grammar-functioncall", "meta-llama-3.1-8b-instruct"),
|
||||
)
|
||||
|
||||
DescribeTable("NameStem leaves alone what identifies a different model",
|
||||
func(name, expected string) {
|
||||
Expect(NameStem(name)).To(Equal(expected))
|
||||
},
|
||||
Entry("parameter size", "qwen3-tts-cpp-0.6b-base", "qwen3-tts-cpp-0.6b-base"),
|
||||
Entry("language suffix", "kokoros-de", "kokoros-de"),
|
||||
Entry("English-only ASR", "whisper-small-en", "whisper-small-en"),
|
||||
Entry("finetune", "qwen3-30b-a3b-abliterated", "qwen3-30b-a3b-abliterated"),
|
||||
Entry("product suffix", "vibevoice-cpp-asr", "vibevoice-cpp-asr"),
|
||||
)
|
||||
|
||||
It("never strips a name down to nothing", func() {
|
||||
Expect(NameStem("q4_k_m")).To(Equal("q4_k_m"))
|
||||
Expect(NameStem("f16-q8_0")).To(Equal("f16"))
|
||||
})
|
||||
|
||||
DescribeTable("FileStem reduces a weight filename to the weights it holds",
|
||||
func(filename, expected string) {
|
||||
Expect(FileStem(filename)).To(Equal(expected))
|
||||
},
|
||||
Entry("directory and extension go", "bonsai/models/Ternary-Bonsai-8B-gguf/Ternary-Bonsai-8B-Q2_0.gguf", "ternary-bonsai-8b"),
|
||||
Entry("underscored quant token stays whole", "Llama-3.2-1B-Instruct-Q4_K_M.gguf", "llama-3.2-1b-instruct"),
|
||||
Entry("dot separated quant token", "Llama-3.2-3B-Instruct.Q4_K_M.gguf", "llama-3.2-3b-instruct"),
|
||||
Entry("group size suffix", "Ternary-Bonsai-8B-Q2_0_g64.gguf", "ternary-bonsai-8b"),
|
||||
Entry("bf16", "omnivoice-cpp-hq/omnivoice-base-BF16.gguf", "omnivoice-base"),
|
||||
Entry("safetensors", "some/dir/Model-Name-fp8.safetensors", "model-name"),
|
||||
)
|
||||
|
||||
DescribeTable("BuildWidth reads the nominal width out of a filename",
|
||||
func(filename string, expected int) {
|
||||
Expect(BuildWidth(filename)).To(Equal(expected))
|
||||
},
|
||||
Entry("q4", "foo-Q4_K_M.gguf", 4),
|
||||
Entry("q8", "foo-Q8_0.gguf", 8),
|
||||
Entry("q2", "foo-Q2_0.gguf", 2),
|
||||
Entry("f16", "foo-f16.gguf", 16),
|
||||
Entry("bf16", "foo-BF16.gguf", 16),
|
||||
Entry("iq3", "foo-iq3_xxs.gguf", 3),
|
||||
Entry("nothing readable sorts last", "foo.gguf", unknownWidth),
|
||||
)
|
||||
|
||||
It("treats an auxiliary file as never being the model's own weights", func() {
|
||||
for _, f := range []string{
|
||||
"mmproj-model-f16.gguf",
|
||||
"dir/vae-BF16.gguf",
|
||||
"clip_l.safetensors",
|
||||
"umt5-xxl-encoder-Q8_0.gguf",
|
||||
"t5xxl_fp16.safetensors",
|
||||
"ae.safetensors",
|
||||
"omnivoice-tokenizer-Q8_0.gguf",
|
||||
} {
|
||||
Expect(IsAuxiliaryFile(f)).To(BeTrue(), "expected %q to be auxiliary", f)
|
||||
}
|
||||
Expect(IsAuxiliaryFile("gemma-3-27b-it-Q4_K_M.gguf")).To(BeFalse())
|
||||
})
|
||||
})
|
||||
@@ -1,13 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestVariantProposals(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "gallery variant proposals")
|
||||
}
|
||||
18
.github/dependabot.yml
vendored
18
.github/dependabot.yml
vendored
@@ -29,6 +29,10 @@ updates:
|
||||
schedule:
|
||||
# Check for updates to GitHub Actions every weekday
|
||||
interval: "weekly"
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/backend/python/autogptq"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/backend/python/bark"
|
||||
schedule:
|
||||
@@ -45,16 +49,6 @@ updates:
|
||||
directory: "/backend/python/diffusers"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
# torch and transformers are deliberately pinned in this backend (see
|
||||
# backend/python/diffusers/requirements-*.txt and issue #9979), and the
|
||||
# l4t12 variant resolves them from the Jetson pip index
|
||||
# (https://pypi.jetson-ai-lab.io/jp6/cu129/). dependabot cannot authenticate
|
||||
# against that index and fails the whole weekly update with a
|
||||
# private_source_authentication_failure. Ignore the two pinned deps we don't
|
||||
# want bumped anyway so the job stays green.
|
||||
ignore:
|
||||
- dependency-name: "torch"
|
||||
- dependency-name: "transformers"
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/backend/python/exllama"
|
||||
schedule:
|
||||
@@ -71,6 +65,10 @@ updates:
|
||||
directory: "/backend/python/openvoice"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/backend/python/parler-tts"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/backend/python/rerankers"
|
||||
schedule:
|
||||
|
||||
213
.github/gallery-agent/gallery.go
vendored
213
.github/gallery-agent/gallery.go
vendored
@@ -1,213 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/mudler/LocalAI/core/gallery/importers"
|
||||
"sigs.k8s.io/yaml"
|
||||
)
|
||||
|
||||
func formatTextContent(text string) string {
|
||||
return formatTextContentWithIndent(text, 4, 6)
|
||||
}
|
||||
|
||||
// formatTextContentWithIndent formats text content with specified base and list item indentation
|
||||
func formatTextContentWithIndent(text string, baseIndent int, listItemIndent int) string {
|
||||
var formattedLines []string
|
||||
lines := strings.Split(text, "\n")
|
||||
for _, line := range lines {
|
||||
trimmed := strings.TrimRight(line, " \t\r")
|
||||
if trimmed == "" {
|
||||
// Keep empty lines as empty (no indentation)
|
||||
formattedLines = append(formattedLines, "")
|
||||
} else {
|
||||
// Preserve relative indentation from yaml.Marshal output
|
||||
// Count existing leading spaces to preserve relative structure
|
||||
leadingSpaces := len(trimmed) - len(strings.TrimLeft(trimmed, " \t"))
|
||||
trimmedStripped := strings.TrimLeft(trimmed, " \t")
|
||||
|
||||
var totalIndent int
|
||||
if strings.HasPrefix(trimmedStripped, "-") {
|
||||
// List items: use listItemIndent (ignore existing leading spaces)
|
||||
totalIndent = listItemIndent
|
||||
} else {
|
||||
// Regular lines: use baseIndent + preserve relative indentation
|
||||
// This handles both top-level keys (leadingSpaces=0) and nested properties (leadingSpaces>0)
|
||||
totalIndent = baseIndent + leadingSpaces
|
||||
}
|
||||
|
||||
indentStr := strings.Repeat(" ", totalIndent)
|
||||
formattedLines = append(formattedLines, indentStr+trimmedStripped)
|
||||
}
|
||||
}
|
||||
formattedText := strings.Join(formattedLines, "\n")
|
||||
// Remove any trailing spaces from the formatted description
|
||||
formattedText = strings.TrimRight(formattedText, " \t")
|
||||
return formattedText
|
||||
}
|
||||
|
||||
// generateYAMLEntry generates a YAML entry for a model using the specified anchor
|
||||
func generateYAMLEntry(model ProcessedModel, quantization string) string {
|
||||
modelConfig, err := importers.DiscoverModelConfig("https://huggingface.co/"+model.ModelID, json.RawMessage(`{ "quantization": "`+quantization+`"}`))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// Extract model name from ModelID
|
||||
parts := strings.Split(model.ModelID, "/")
|
||||
modelName := model.ModelID
|
||||
if len(parts) > 0 {
|
||||
modelName = strings.ToLower(parts[len(parts)-1])
|
||||
}
|
||||
// Remove common suffixes
|
||||
modelName = strings.ReplaceAll(modelName, "-gguf", "")
|
||||
modelName = strings.ReplaceAll(modelName, "-q4_k_m", "")
|
||||
modelName = strings.ReplaceAll(modelName, "-q4_k_s", "")
|
||||
modelName = strings.ReplaceAll(modelName, "-q3_k_m", "")
|
||||
modelName = strings.ReplaceAll(modelName, "-q2_k", "")
|
||||
|
||||
description := model.ReadmeContent
|
||||
if description == "" {
|
||||
description = fmt.Sprintf("AI model: %s", modelName)
|
||||
}
|
||||
|
||||
// Clean up description to prevent YAML linting issues
|
||||
description = cleanTextContent(description)
|
||||
formattedDescription := formatTextContent(description)
|
||||
|
||||
// Strip name and description from config file since they are
|
||||
// already present at the gallery entry level and should not
|
||||
// appear under overrides.
|
||||
configFileContent := modelConfig.ConfigFile
|
||||
var cfgMap map[string]any
|
||||
if err := yaml.Unmarshal([]byte(configFileContent), &cfgMap); err == nil {
|
||||
delete(cfgMap, "name")
|
||||
delete(cfgMap, "description")
|
||||
if cleaned, err := yaml.Marshal(cfgMap); err == nil {
|
||||
configFileContent = string(cleaned)
|
||||
}
|
||||
}
|
||||
|
||||
configFile := formatTextContent(configFileContent)
|
||||
|
||||
filesYAML, _ := yaml.Marshal(modelConfig.Files)
|
||||
|
||||
// Files section: list items need 4 spaces (not 6), since files: is at 2 spaces
|
||||
files := formatTextContentWithIndent(string(filesYAML), 4, 4)
|
||||
|
||||
// Build metadata sections
|
||||
var metadataSections []string
|
||||
|
||||
// Add license if present
|
||||
if model.License != "" {
|
||||
metadataSections = append(metadataSections, fmt.Sprintf(` license: "%s"`, model.License))
|
||||
}
|
||||
|
||||
// Add tags if present
|
||||
if len(model.Tags) > 0 {
|
||||
tagsYAML, _ := yaml.Marshal(model.Tags)
|
||||
tagsFormatted := formatTextContentWithIndent(string(tagsYAML), 4, 4)
|
||||
tagsFormatted = strings.TrimRight(tagsFormatted, "\n")
|
||||
metadataSections = append(metadataSections, fmt.Sprintf(" tags:\n%s", tagsFormatted))
|
||||
}
|
||||
|
||||
// Add icon if present
|
||||
if model.Icon != "" {
|
||||
metadataSections = append(metadataSections, fmt.Sprintf(` icon: %s`, model.Icon))
|
||||
}
|
||||
|
||||
// Build the metadata block
|
||||
metadataBlock := ""
|
||||
if len(metadataSections) > 0 {
|
||||
metadataBlock = strings.Join(metadataSections, "\n") + "\n"
|
||||
}
|
||||
|
||||
yamlTemplate := ""
|
||||
yamlTemplate = `- name: "%s"
|
||||
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
|
||||
urls:
|
||||
- https://huggingface.co/%s
|
||||
description: |
|
||||
%s%s
|
||||
overrides:
|
||||
%s
|
||||
files:
|
||||
%s`
|
||||
// Trim trailing newlines from formatted sections to prevent extra blank lines
|
||||
formattedDescription = strings.TrimRight(formattedDescription, "\n")
|
||||
configFile = strings.TrimRight(configFile, "\n")
|
||||
files = strings.TrimRight(files, "\n")
|
||||
// Add newline before metadata block if present
|
||||
if metadataBlock != "" {
|
||||
metadataBlock = "\n" + strings.TrimRight(metadataBlock, "\n")
|
||||
}
|
||||
return fmt.Sprintf(yamlTemplate,
|
||||
modelName,
|
||||
model.ModelID,
|
||||
formattedDescription,
|
||||
metadataBlock,
|
||||
configFile,
|
||||
files,
|
||||
)
|
||||
}
|
||||
|
||||
// generateYAMLForModels generates YAML entries for selected models and appends to index.yaml
|
||||
func generateYAMLForModels(ctx context.Context, models []ProcessedModel, quantization string) error {
|
||||
|
||||
// Generate YAML entries for each model
|
||||
var yamlEntries []string
|
||||
for _, model := range models {
|
||||
fmt.Printf("Generating YAML entry for model: %s\n", model.ModelID)
|
||||
|
||||
// Generate YAML entry
|
||||
yamlEntry := generateYAMLEntry(model, quantization)
|
||||
yamlEntries = append(yamlEntries, yamlEntry)
|
||||
}
|
||||
|
||||
// Prepend to index.yaml (write at the top)
|
||||
if len(yamlEntries) > 0 {
|
||||
indexPath := getGalleryIndexPath()
|
||||
fmt.Printf("Prepending YAML entries to %s...\n", indexPath)
|
||||
|
||||
// Read current content
|
||||
content, err := os.ReadFile(indexPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to read %s: %w", indexPath, err)
|
||||
}
|
||||
|
||||
existingContent := string(content)
|
||||
yamlBlock := strings.Join(yamlEntries, "\n")
|
||||
|
||||
// Check if file starts with "---"
|
||||
var newContent string
|
||||
if strings.HasPrefix(existingContent, "---\n") {
|
||||
// File starts with "---", prepend new entries after it
|
||||
restOfContent := strings.TrimPrefix(existingContent, "---\n")
|
||||
// Ensure proper spacing: "---\n" + new entries + "\n" + rest of content
|
||||
newContent = "---\n" + yamlBlock + "\n" + restOfContent
|
||||
} else if strings.HasPrefix(existingContent, "---") {
|
||||
// File starts with "---" but no newline after
|
||||
restOfContent := strings.TrimPrefix(existingContent, "---")
|
||||
newContent = "---\n" + yamlBlock + "\n" + strings.TrimPrefix(restOfContent, "\n")
|
||||
} else {
|
||||
// No "---" at start, prepend new entries at the very beginning
|
||||
// Trim leading whitespace from existing content
|
||||
existingContent = strings.TrimLeft(existingContent, " \t\n\r")
|
||||
newContent = yamlBlock + "\n" + existingContent
|
||||
}
|
||||
|
||||
// Write back to file
|
||||
err = os.WriteFile(indexPath, []byte(newContent), 0644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to write %s: %w", indexPath, err)
|
||||
}
|
||||
|
||||
fmt.Printf("Successfully prepended %d models to %s\n", len(yamlEntries), indexPath)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
301
.github/gallery-agent/helpers.go
vendored
301
.github/gallery-agent/helpers.go
vendored
@@ -1,301 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
|
||||
"sigs.k8s.io/yaml"
|
||||
)
|
||||
|
||||
var galleryIndexPath = os.Getenv("GALLERY_INDEX_PATH")
|
||||
|
||||
// getGalleryIndexPath returns the gallery index file path, with a default fallback
|
||||
func getGalleryIndexPath() string {
|
||||
if galleryIndexPath != "" {
|
||||
return galleryIndexPath
|
||||
}
|
||||
return "gallery/index.yaml"
|
||||
}
|
||||
|
||||
type galleryModel struct {
|
||||
Name string `yaml:"name"`
|
||||
Urls []string `yaml:"urls"`
|
||||
}
|
||||
|
||||
// loadGalleryURLSet parses gallery/index.yaml once and returns the set of
|
||||
// HuggingFace model URLs already present in the gallery.
|
||||
func loadGalleryURLSet() (map[string]struct{}, error) {
|
||||
indexPath := getGalleryIndexPath()
|
||||
content, err := os.ReadFile(indexPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read %s: %w", indexPath, err)
|
||||
}
|
||||
|
||||
var galleryModels []galleryModel
|
||||
if err := yaml.Unmarshal(content, &galleryModels); err != nil {
|
||||
return nil, fmt.Errorf("failed to unmarshal %s: %w", indexPath, err)
|
||||
}
|
||||
|
||||
set := make(map[string]struct{}, len(galleryModels))
|
||||
for _, gm := range galleryModels {
|
||||
for _, u := range gm.Urls {
|
||||
set[u] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
// Also skip URLs already proposed in open (unmerged) gallery-agent PRs.
|
||||
// The workflow injects these via EXTRA_SKIP_URLS so we don't keep
|
||||
// re-proposing the same model every run while a PR is waiting to merge.
|
||||
for _, line := range strings.FieldsFunc(os.Getenv("EXTRA_SKIP_URLS"), func(r rune) bool {
|
||||
return r == '\n' || r == ',' || r == ' '
|
||||
}) {
|
||||
u := strings.TrimSpace(line)
|
||||
if u != "" {
|
||||
set[u] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
return set, nil
|
||||
}
|
||||
|
||||
// modelAlreadyInGallery checks whether a HuggingFace model repo is already
|
||||
// referenced in the gallery URL set.
|
||||
func modelAlreadyInGallery(set map[string]struct{}, modelID string) bool {
|
||||
_, ok := set["https://huggingface.co/"+modelID]
|
||||
return ok
|
||||
}
|
||||
|
||||
// baseModelFromTags returns the first `base_model:<repo>` value found in the
|
||||
// tag list, or "" if none is present. HuggingFace surfaces the base model
|
||||
// declared in the model card's YAML frontmatter as such a tag.
|
||||
func baseModelFromTags(tags []string) string {
|
||||
for _, t := range tags {
|
||||
if strings.HasPrefix(t, "base_model:") {
|
||||
return strings.TrimPrefix(t, "base_model:")
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// licenseFromTags returns the `license:<id>` value from the tag list, or "".
|
||||
func licenseFromTags(tags []string) string {
|
||||
for _, t := range tags {
|
||||
if strings.HasPrefix(t, "license:") {
|
||||
return strings.TrimPrefix(t, "license:")
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// curatedTags produces the gallery tag list from HuggingFace's raw tag set.
|
||||
// Always includes llm + gguf, then adds whitelisted family / capability
|
||||
// markers when they appear in the HF tag list.
|
||||
func curatedTags(hfTags []string) []string {
|
||||
whitelist := []string{
|
||||
"gpu", "cpu",
|
||||
"llama", "mistral", "mixtral", "qwen", "qwen2", "qwen3",
|
||||
"gemma", "gemma2", "gemma3", "phi", "phi3", "phi4",
|
||||
"deepseek", "yi", "falcon", "command-r",
|
||||
"vision", "multimodal", "code", "chat",
|
||||
"instruction-tuned", "reasoning", "thinking",
|
||||
}
|
||||
seen := map[string]struct{}{}
|
||||
out := []string{"llm", "gguf"}
|
||||
seen["llm"] = struct{}{}
|
||||
seen["gguf"] = struct{}{}
|
||||
|
||||
hfSet := map[string]struct{}{}
|
||||
for _, t := range hfTags {
|
||||
hfSet[strings.ToLower(t)] = struct{}{}
|
||||
}
|
||||
for _, w := range whitelist {
|
||||
if _, ok := hfSet[w]; ok {
|
||||
if _, dup := seen[w]; !dup {
|
||||
out = append(out, w)
|
||||
seen[w] = struct{}{}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// resolveReadme fetches a description-quality README for a (possibly
|
||||
// quantized) repo: if a `base_model:` tag is present, fetch the base repo's
|
||||
// README; otherwise fall back to the repo's own README.
|
||||
func resolveReadme(client *hfapi.Client, modelID string, hfTags []string) (string, error) {
|
||||
if base := baseModelFromTags(hfTags); base != "" && base != modelID {
|
||||
if content, err := client.GetReadmeContent(base, "README.md"); err == nil && strings.TrimSpace(content) != "" {
|
||||
return cleanTextContent(content), nil
|
||||
}
|
||||
}
|
||||
content, err := client.GetReadmeContent(modelID, "README.md")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return cleanTextContent(content), nil
|
||||
}
|
||||
|
||||
// extractDescription turns a raw HuggingFace README into a concise plain-text
|
||||
// description suitable for embedding in gallery/index.yaml: strips YAML
|
||||
// frontmatter, HTML tags/comments, markdown images, link URLs (keeping the
|
||||
// link text), markdown tables, and then truncates at a paragraph boundary
|
||||
// around ~1200 characters. Raw README should still be used for icon
|
||||
// extraction — call this only for the `description:` field.
|
||||
func extractDescription(readme string) string {
|
||||
s := readme
|
||||
|
||||
// Strip leading YAML frontmatter: `---\n...\n---\n` at start of file.
|
||||
if strings.HasPrefix(strings.TrimLeft(s, " \t\n"), "---") {
|
||||
trimmed := strings.TrimLeft(s, " \t\n")
|
||||
rest := strings.TrimPrefix(trimmed, "---")
|
||||
if idx := strings.Index(rest, "\n---"); idx >= 0 {
|
||||
after := rest[idx+len("\n---"):]
|
||||
after = strings.TrimPrefix(after, "\n")
|
||||
s = after
|
||||
}
|
||||
}
|
||||
|
||||
// Strip HTML comments and tags.
|
||||
s = regexp.MustCompile(`(?s)<!--.*?-->`).ReplaceAllString(s, "")
|
||||
s = regexp.MustCompile(`(?is)<[^>]+>`).ReplaceAllString(s, "")
|
||||
|
||||
// Strip markdown images entirely.
|
||||
s = regexp.MustCompile(`!\[[^\]]*\]\([^)]*\)`).ReplaceAllString(s, "")
|
||||
// Replace markdown links `[text](url)` with just `text`.
|
||||
s = regexp.MustCompile(`\[([^\]]+)\]\([^)]+\)`).ReplaceAllString(s, "$1")
|
||||
|
||||
// Drop table lines and horizontal rules, and flatten all leading
|
||||
// whitespace: generateYAMLEntry embeds this under a `description: |`
|
||||
// literal block whose indentation is set by the first non-empty line.
|
||||
// If any line has extra leading whitespace (e.g. from an indented
|
||||
// `<p align="center">` block in the original README), YAML will pick
|
||||
// that up as the block's indent and every later line at a smaller
|
||||
// indent blows the block scalar. Stripping leading whitespace here
|
||||
// guarantees uniform 4-space indentation after formatTextContent runs.
|
||||
var kept []string
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
t := strings.TrimLeft(line, " \t")
|
||||
ts := strings.TrimSpace(t)
|
||||
if strings.HasPrefix(ts, "|") {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(ts, ":--") || strings.HasPrefix(ts, "---") || strings.HasPrefix(ts, "===") {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, t)
|
||||
}
|
||||
s = strings.Join(kept, "\n")
|
||||
|
||||
// Normalise whitespace and drop any leading blank lines so the literal
|
||||
// block in YAML doesn't start with a blank first line (which would
|
||||
// break the indentation detector the same way).
|
||||
s = cleanTextContent(s)
|
||||
s = strings.TrimLeft(s, " \t\n")
|
||||
|
||||
// Truncate at a paragraph boundary around maxLen chars.
|
||||
const maxLen = 1200
|
||||
if len(s) > maxLen {
|
||||
cut := strings.LastIndex(s[:maxLen], "\n\n")
|
||||
if cut < maxLen/3 {
|
||||
cut = maxLen
|
||||
}
|
||||
s = strings.TrimRight(s[:cut], " \t\n") + "\n\n..."
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// cleanTextContent removes trailing spaces/tabs and collapses multiple empty
|
||||
// lines so README content embeds cleanly into YAML without lint noise.
|
||||
func cleanTextContent(text string) string {
|
||||
lines := strings.Split(text, "\n")
|
||||
var cleaned []string
|
||||
var prevEmpty bool
|
||||
for _, line := range lines {
|
||||
trimmed := strings.TrimRight(line, " \t\r")
|
||||
if trimmed == "" {
|
||||
if !prevEmpty {
|
||||
cleaned = append(cleaned, "")
|
||||
}
|
||||
prevEmpty = true
|
||||
} else {
|
||||
cleaned = append(cleaned, trimmed)
|
||||
prevEmpty = false
|
||||
}
|
||||
}
|
||||
return strings.TrimRight(strings.Join(cleaned, "\n"), "\n")
|
||||
}
|
||||
|
||||
// extractIconFromReadme scans README content for an image URL usable as a
|
||||
// gallery entry icon.
|
||||
func extractIconFromReadme(readmeContent string) string {
|
||||
if readmeContent == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
markdownImageRegex := regexp.MustCompile(`(?i)!\[[^\]]*\]\(([^)]+\.(png|jpg|jpeg|svg|webp|gif))\)`)
|
||||
htmlImageRegex := regexp.MustCompile(`(?i)<img[^>]+src=["']([^"']+\.(png|jpg|jpeg|svg|webp|gif))["']`)
|
||||
plainImageRegex := regexp.MustCompile(`(?i)https?://[^\s<>"']+\.(png|jpg|jpeg|svg|webp|gif)`)
|
||||
|
||||
if m := markdownImageRegex.FindStringSubmatch(readmeContent); len(m) > 1 && strings.HasPrefix(strings.ToLower(m[1]), "http") {
|
||||
return strings.TrimSpace(m[1])
|
||||
}
|
||||
if m := htmlImageRegex.FindStringSubmatch(readmeContent); len(m) > 1 && strings.HasPrefix(strings.ToLower(m[1]), "http") {
|
||||
return strings.TrimSpace(m[1])
|
||||
}
|
||||
if m := plainImageRegex.FindStringSubmatch(readmeContent); len(m) > 0 && strings.HasPrefix(strings.ToLower(m[0]), "http") {
|
||||
return strings.TrimSpace(m[0])
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// getHuggingFaceAvatarURL returns the HF avatar URL for a user, or "".
|
||||
func getHuggingFaceAvatarURL(author string) string {
|
||||
if author == "" {
|
||||
return ""
|
||||
}
|
||||
userURL := fmt.Sprintf("https://huggingface.co/api/users/%s/overview", author)
|
||||
resp, err := http.Get(userURL)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return ""
|
||||
}
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
var info map[string]any
|
||||
if err := json.Unmarshal(body, &info); err != nil {
|
||||
return ""
|
||||
}
|
||||
if v, ok := info["avatarUrl"].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
if v, ok := info["avatar"].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// extractModelIcon extracts an icon URL from the README, falling back to the
|
||||
// HuggingFace user avatar.
|
||||
func extractModelIcon(model ProcessedModel) string {
|
||||
if icon := extractIconFromReadme(model.ReadmeContent); icon != "" {
|
||||
return icon
|
||||
}
|
||||
if model.Author != "" {
|
||||
if avatar := getHuggingFaceAvatarURL(model.Author); avatar != "" {
|
||||
return avatar
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
291
.github/gallery-agent/main.go
vendored
291
.github/gallery-agent/main.go
vendored
@@ -1,291 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
|
||||
)
|
||||
|
||||
// ProcessedModelFile represents a processed model file with additional metadata
|
||||
type ProcessedModelFile struct {
|
||||
Path string `json:"path"`
|
||||
Size int64 `json:"size"`
|
||||
SHA256 string `json:"sha256"`
|
||||
IsReadme bool `json:"is_readme"`
|
||||
FileType string `json:"file_type"` // "model", "readme", "other"
|
||||
}
|
||||
|
||||
// ProcessedModel represents a processed model with all gathered metadata
|
||||
type ProcessedModel struct {
|
||||
ModelID string `json:"model_id"`
|
||||
Author string `json:"author"`
|
||||
Downloads int `json:"downloads"`
|
||||
LastModified string `json:"last_modified"`
|
||||
Files []ProcessedModelFile `json:"files"`
|
||||
PreferredModelFile *ProcessedModelFile `json:"preferred_model_file,omitempty"`
|
||||
ReadmeFile *ProcessedModelFile `json:"readme_file,omitempty"`
|
||||
ReadmeContent string `json:"readme_content,omitempty"`
|
||||
ReadmeContentPreview string `json:"readme_content_preview,omitempty"`
|
||||
QuantizationPreferences []string `json:"quantization_preferences"`
|
||||
ProcessingError string `json:"processing_error,omitempty"`
|
||||
Tags []string `json:"tags,omitempty"`
|
||||
License string `json:"license,omitempty"`
|
||||
Icon string `json:"icon,omitempty"`
|
||||
}
|
||||
|
||||
// AddedModelSummary represents a summary of models added to the gallery
|
||||
type AddedModelSummary struct {
|
||||
SearchTerm string `json:"search_term"`
|
||||
TotalFound int `json:"total_found"`
|
||||
ModelsAdded int `json:"models_added"`
|
||||
AddedModelIDs []string `json:"added_model_ids"`
|
||||
AddedModelURLs []string `json:"added_model_urls"`
|
||||
Quantization string `json:"quantization"`
|
||||
ProcessingTime string `json:"processing_time"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
startTime := time.Now()
|
||||
|
||||
// Synthetic mode for local testing
|
||||
if sm := os.Getenv("SYNTHETIC_MODE"); sm == "true" || sm == "1" {
|
||||
fmt.Println("Running in SYNTHETIC MODE - generating random test data")
|
||||
if err := runSyntheticMode(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error in synthetic mode: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
searchTerm := os.Getenv("SEARCH_TERM")
|
||||
if searchTerm == "" {
|
||||
searchTerm = "GGUF"
|
||||
}
|
||||
|
||||
limitStr := os.Getenv("LIMIT")
|
||||
if limitStr == "" {
|
||||
limitStr = "15"
|
||||
}
|
||||
limit, err := strconv.Atoi(limitStr)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error parsing LIMIT: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
quantization := os.Getenv("QUANTIZATION")
|
||||
if quantization == "" {
|
||||
quantization = "Q4_K_M"
|
||||
}
|
||||
|
||||
maxModelsStr := os.Getenv("MAX_MODELS")
|
||||
if maxModelsStr == "" {
|
||||
maxModelsStr = "1"
|
||||
}
|
||||
maxModels, err := strconv.Atoi(maxModelsStr)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error parsing MAX_MODELS: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Printf("Gallery Agent Configuration:\n")
|
||||
fmt.Printf(" Search Term: %s\n", searchTerm)
|
||||
fmt.Printf(" Limit: %d\n", limit)
|
||||
fmt.Printf(" Quantization: %s\n", quantization)
|
||||
fmt.Printf(" Max Models to Add: %d\n", maxModels)
|
||||
fmt.Printf(" Gallery Index Path: %s\n", getGalleryIndexPath())
|
||||
fmt.Println()
|
||||
|
||||
// Phase 1: load current gallery and query HuggingFace.
|
||||
gallerySet, err := loadGalleryURLSet()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error loading gallery index: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("Loaded %d existing gallery entries\n", len(gallerySet))
|
||||
|
||||
client := hfapi.NewClient()
|
||||
|
||||
fmt.Println("Searching for trending models on HuggingFace...")
|
||||
rawModels, err := client.GetTrending(searchTerm, limit)
|
||||
if err != nil {
|
||||
if errors.Is(err, hfapi.ErrRateLimited) {
|
||||
fmt.Printf("HuggingFace API is rate limited after retries, skipping this run: %v\n", err)
|
||||
writeSummary(AddedModelSummary{
|
||||
SearchTerm: searchTerm,
|
||||
TotalFound: 0,
|
||||
ModelsAdded: 0,
|
||||
Quantization: quantization,
|
||||
ProcessingTime: time.Since(startTime).String(),
|
||||
})
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "Error fetching models: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("Found %d trending models matching %q\n", len(rawModels), searchTerm)
|
||||
totalFound := len(rawModels)
|
||||
|
||||
// Phase 2: drop anything already in the gallery *before* any expensive
|
||||
// per-model work (GetModelDetails, README fetches, icon lookups).
|
||||
fresh := rawModels[:0]
|
||||
for _, m := range rawModels {
|
||||
if modelAlreadyInGallery(gallerySet, m.ModelID) {
|
||||
fmt.Printf("Skipping existing model: %s\n", m.ModelID)
|
||||
continue
|
||||
}
|
||||
fresh = append(fresh, m)
|
||||
}
|
||||
fmt.Printf("%d candidates after gallery dedup\n", len(fresh))
|
||||
|
||||
// Phase 3: HuggingFace already returned these in trendingScore order —
|
||||
// just cap to MAX_MODELS.
|
||||
if len(fresh) > maxModels {
|
||||
fresh = fresh[:maxModels]
|
||||
}
|
||||
if len(fresh) == 0 {
|
||||
fmt.Println("No new models to add to the gallery.")
|
||||
writeSummary(AddedModelSummary{
|
||||
SearchTerm: searchTerm,
|
||||
TotalFound: totalFound,
|
||||
ModelsAdded: 0,
|
||||
Quantization: quantization,
|
||||
ProcessingTime: time.Since(startTime).String(),
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Phase 4: fetch details and build ProcessedModel entries for survivors.
|
||||
var processed []ProcessedModel
|
||||
quantPrefs := []string{quantization, "Q4_K_M", "Q4_K_S", "Q3_K_M", "Q2_K", "Q8_0"}
|
||||
for _, m := range fresh {
|
||||
fmt.Printf("Processing model: %s (downloads=%d)\n", m.ModelID, m.Downloads)
|
||||
|
||||
pm := ProcessedModel{
|
||||
ModelID: m.ModelID,
|
||||
Author: m.Author,
|
||||
Downloads: m.Downloads,
|
||||
LastModified: m.LastModified,
|
||||
QuantizationPreferences: quantPrefs,
|
||||
}
|
||||
|
||||
details, err := client.GetModelDetails(m.ModelID)
|
||||
if err != nil {
|
||||
fmt.Printf(" Error getting model details: %v (skipping)\n", err)
|
||||
continue
|
||||
}
|
||||
|
||||
preferred := hfapi.FindPreferredModelFile(details.Files, quantPrefs)
|
||||
if preferred == nil {
|
||||
fmt.Printf(" No GGUF file matching %v — skipping\n", quantPrefs)
|
||||
continue
|
||||
}
|
||||
|
||||
pm.Files = make([]ProcessedModelFile, len(details.Files))
|
||||
for j, f := range details.Files {
|
||||
fileType := "other"
|
||||
if f.IsReadme {
|
||||
fileType = "readme"
|
||||
} else if f.Path == preferred.Path {
|
||||
fileType = "model"
|
||||
}
|
||||
pm.Files[j] = ProcessedModelFile{
|
||||
Path: f.Path,
|
||||
Size: f.Size,
|
||||
SHA256: f.SHA256,
|
||||
IsReadme: f.IsReadme,
|
||||
FileType: fileType,
|
||||
}
|
||||
if f.Path == preferred.Path {
|
||||
copyFile := pm.Files[j]
|
||||
pm.PreferredModelFile = ©File
|
||||
}
|
||||
if f.IsReadme {
|
||||
copyFile := pm.Files[j]
|
||||
pm.ReadmeFile = ©File
|
||||
}
|
||||
}
|
||||
|
||||
// Deterministic README resolution: follow base_model tag if set.
|
||||
// Keep the raw (HTML-bearing) README around while we extract the
|
||||
// icon, then strip it down to a plain-text description for the
|
||||
// `description:` YAML field.
|
||||
readme, err := resolveReadme(client, m.ModelID, m.Tags)
|
||||
if err != nil {
|
||||
fmt.Printf(" Warning: failed to fetch README: %v\n", err)
|
||||
}
|
||||
pm.ReadmeContent = readme
|
||||
|
||||
pm.License = licenseFromTags(m.Tags)
|
||||
pm.Tags = curatedTags(m.Tags)
|
||||
pm.Icon = extractModelIcon(pm)
|
||||
|
||||
if pm.ReadmeContent != "" {
|
||||
pm.ReadmeContent = extractDescription(pm.ReadmeContent)
|
||||
pm.ReadmeContentPreview = truncateString(pm.ReadmeContent, 200)
|
||||
}
|
||||
|
||||
fmt.Printf(" License: %s, Tags: %v, Icon: %s\n", pm.License, pm.Tags, pm.Icon)
|
||||
processed = append(processed, pm)
|
||||
}
|
||||
|
||||
if len(processed) == 0 {
|
||||
fmt.Println("No processable models after detail fetch.")
|
||||
writeSummary(AddedModelSummary{
|
||||
SearchTerm: searchTerm,
|
||||
TotalFound: totalFound,
|
||||
ModelsAdded: 0,
|
||||
Quantization: quantization,
|
||||
ProcessingTime: time.Since(startTime).String(),
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Phase 5: write YAML entries.
|
||||
var addedIDs, addedURLs []string
|
||||
for _, pm := range processed {
|
||||
addedIDs = append(addedIDs, pm.ModelID)
|
||||
addedURLs = append(addedURLs, "https://huggingface.co/"+pm.ModelID)
|
||||
}
|
||||
|
||||
fmt.Println("Generating YAML entries for selected models...")
|
||||
if err := generateYAMLForModels(context.Background(), processed, quantization); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error generating YAML entries: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
writeSummary(AddedModelSummary{
|
||||
SearchTerm: searchTerm,
|
||||
TotalFound: totalFound,
|
||||
ModelsAdded: len(addedIDs),
|
||||
AddedModelIDs: addedIDs,
|
||||
AddedModelURLs: addedURLs,
|
||||
Quantization: quantization,
|
||||
ProcessingTime: time.Since(startTime).String(),
|
||||
})
|
||||
}
|
||||
|
||||
func writeSummary(summary AddedModelSummary) {
|
||||
data, err := json.MarshalIndent(summary, "", " ")
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error marshaling summary: %v\n", err)
|
||||
return
|
||||
}
|
||||
if err := os.WriteFile("gallery-agent-summary.json", data, 0644); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error writing summary file: %v\n", err)
|
||||
return
|
||||
}
|
||||
fmt.Println("Summary written to gallery-agent-summary.json")
|
||||
}
|
||||
|
||||
func truncateString(s string, maxLen int) string {
|
||||
if len(s) <= maxLen {
|
||||
return s
|
||||
}
|
||||
return s[:maxLen] + "..."
|
||||
}
|
||||
224
.github/gallery-agent/testing.go
vendored
224
.github/gallery-agent/testing.go
vendored
@@ -1,224 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand/v2"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// runSyntheticMode generates synthetic test data and appends it to the gallery
|
||||
func runSyntheticMode() error {
|
||||
generator := NewSyntheticDataGenerator()
|
||||
|
||||
// Generate a random number of synthetic models (1-3)
|
||||
numModels := generator.rand.IntN(3) + 1
|
||||
fmt.Printf("Generating %d synthetic models for testing...\n", numModels)
|
||||
|
||||
var models []ProcessedModel
|
||||
for range numModels {
|
||||
model := generator.GenerateProcessedModel()
|
||||
models = append(models, model)
|
||||
fmt.Printf("Generated synthetic model: %s\n", model.ModelID)
|
||||
}
|
||||
|
||||
// Generate YAML entries and append to gallery/index.yaml
|
||||
fmt.Println("Generating YAML entries for synthetic models...")
|
||||
err := generateYAMLForModels(context.Background(), models, "Q4_K_M")
|
||||
if err != nil {
|
||||
return fmt.Errorf("error generating YAML entries: %w", err)
|
||||
}
|
||||
|
||||
fmt.Printf("Successfully added %d synthetic models to the gallery for testing!\n", len(models))
|
||||
return nil
|
||||
}
|
||||
|
||||
// SyntheticDataGenerator provides methods to generate synthetic test data
|
||||
type SyntheticDataGenerator struct {
|
||||
rand *rand.Rand
|
||||
}
|
||||
|
||||
// NewSyntheticDataGenerator creates a new synthetic data generator
|
||||
func NewSyntheticDataGenerator() *SyntheticDataGenerator {
|
||||
return &SyntheticDataGenerator{
|
||||
rand: rand.New(rand.NewPCG(uint64(time.Now().UnixNano()), 0)),
|
||||
}
|
||||
}
|
||||
|
||||
// GenerateProcessedModelFile creates a synthetic ProcessedModelFile
|
||||
func (g *SyntheticDataGenerator) GenerateProcessedModelFile() ProcessedModelFile {
|
||||
fileTypes := []string{"model", "readme", "other"}
|
||||
fileType := fileTypes[g.rand.IntN(len(fileTypes))]
|
||||
|
||||
var path string
|
||||
var isReadme bool
|
||||
|
||||
switch fileType {
|
||||
case "model":
|
||||
path = fmt.Sprintf("model-%s.gguf", g.randomString(8))
|
||||
isReadme = false
|
||||
case "readme":
|
||||
path = "README.md"
|
||||
isReadme = true
|
||||
default:
|
||||
path = fmt.Sprintf("file-%s.txt", g.randomString(6))
|
||||
isReadme = false
|
||||
}
|
||||
|
||||
return ProcessedModelFile{
|
||||
Path: path,
|
||||
Size: int64(g.rand.IntN(1000000000) + 1000000), // 1MB to 1GB
|
||||
SHA256: g.randomSHA256(),
|
||||
IsReadme: isReadme,
|
||||
FileType: fileType,
|
||||
}
|
||||
}
|
||||
|
||||
// GenerateProcessedModel creates a synthetic ProcessedModel
|
||||
func (g *SyntheticDataGenerator) GenerateProcessedModel() ProcessedModel {
|
||||
authors := []string{"microsoft", "meta", "google", "openai", "anthropic", "mistralai", "huggingface"}
|
||||
modelNames := []string{"llama", "gpt", "claude", "mistral", "gemma", "phi", "qwen", "codellama"}
|
||||
|
||||
author := authors[g.rand.IntN(len(authors))]
|
||||
modelName := modelNames[g.rand.IntN(len(modelNames))]
|
||||
modelID := fmt.Sprintf("%s/%s-%s", author, modelName, g.randomString(6))
|
||||
|
||||
// Generate files
|
||||
numFiles := g.rand.IntN(5) + 2 // 2-6 files
|
||||
files := make([]ProcessedModelFile, numFiles)
|
||||
|
||||
// Ensure at least one model file and one readme
|
||||
hasModelFile := false
|
||||
hasReadme := false
|
||||
|
||||
for i := range numFiles {
|
||||
files[i] = g.GenerateProcessedModelFile()
|
||||
if files[i].FileType == "model" {
|
||||
hasModelFile = true
|
||||
}
|
||||
if files[i].FileType == "readme" {
|
||||
hasReadme = true
|
||||
}
|
||||
}
|
||||
|
||||
// Add required files if missing
|
||||
if !hasModelFile {
|
||||
modelFile := g.GenerateProcessedModelFile()
|
||||
modelFile.FileType = "model"
|
||||
modelFile.Path = fmt.Sprintf("%s-Q4_K_M.gguf", modelName)
|
||||
files = append(files, modelFile)
|
||||
}
|
||||
|
||||
if !hasReadme {
|
||||
readmeFile := g.GenerateProcessedModelFile()
|
||||
readmeFile.FileType = "readme"
|
||||
readmeFile.Path = "README.md"
|
||||
readmeFile.IsReadme = true
|
||||
files = append(files, readmeFile)
|
||||
}
|
||||
|
||||
// Find preferred model file
|
||||
var preferredModelFile *ProcessedModelFile
|
||||
for i := range files {
|
||||
if files[i].FileType == "model" {
|
||||
preferredModelFile = &files[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Find readme file
|
||||
var readmeFile *ProcessedModelFile
|
||||
for i := range files {
|
||||
if files[i].FileType == "readme" {
|
||||
readmeFile = &files[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
readmeContent := g.generateReadmeContent(modelName, author)
|
||||
|
||||
// Generate sample metadata
|
||||
licenses := []string{"apache-2.0", "mit", "llama2", "gpl-3.0", "bsd", ""}
|
||||
license := licenses[g.rand.IntN(len(licenses))]
|
||||
|
||||
sampleTags := []string{"llm", "gguf", "gpu", "cpu", "text-to-text", "chat", "instruction-tuned"}
|
||||
numTags := g.rand.IntN(4) + 3 // 3-6 tags
|
||||
tags := make([]string, numTags)
|
||||
for i := range numTags {
|
||||
tags[i] = sampleTags[g.rand.IntN(len(sampleTags))]
|
||||
}
|
||||
// Remove duplicates
|
||||
tags = g.removeDuplicates(tags)
|
||||
|
||||
// Optionally include icon (50% chance)
|
||||
icon := ""
|
||||
if g.rand.IntN(2) == 0 {
|
||||
icon = fmt.Sprintf("https://cdn-avatars.huggingface.co/v1/production/uploads/%s.png", g.randomString(24))
|
||||
}
|
||||
|
||||
return ProcessedModel{
|
||||
ModelID: modelID,
|
||||
Author: author,
|
||||
Downloads: g.rand.IntN(1000000) + 1000,
|
||||
LastModified: g.randomDate(),
|
||||
Files: files,
|
||||
PreferredModelFile: preferredModelFile,
|
||||
ReadmeFile: readmeFile,
|
||||
ReadmeContent: readmeContent,
|
||||
ReadmeContentPreview: truncateString(readmeContent, 200),
|
||||
QuantizationPreferences: []string{"Q4_K_M", "Q4_K_S", "Q3_K_M", "Q2_K"},
|
||||
ProcessingError: "",
|
||||
Tags: tags,
|
||||
License: license,
|
||||
Icon: icon,
|
||||
}
|
||||
}
|
||||
|
||||
// Helper methods for synthetic data generation
|
||||
func (g *SyntheticDataGenerator) randomString(length int) string {
|
||||
const charset = "abcdefghijklmnopqrstuvwxyz0123456789"
|
||||
b := make([]byte, length)
|
||||
for i := range b {
|
||||
b[i] = charset[g.rand.IntN(len(charset))]
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func (g *SyntheticDataGenerator) randomSHA256() string {
|
||||
const charset = "0123456789abcdef"
|
||||
b := make([]byte, 64)
|
||||
for i := range b {
|
||||
b[i] = charset[g.rand.IntN(len(charset))]
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func (g *SyntheticDataGenerator) randomDate() string {
|
||||
now := time.Now()
|
||||
daysAgo := g.rand.IntN(365) // Random date within last year
|
||||
pastDate := now.AddDate(0, 0, -daysAgo)
|
||||
return pastDate.Format("2006-01-02T15:04:05.000Z")
|
||||
}
|
||||
|
||||
func (g *SyntheticDataGenerator) removeDuplicates(slice []string) []string {
|
||||
keys := make(map[string]bool)
|
||||
result := []string{}
|
||||
for _, item := range slice {
|
||||
if !keys[item] {
|
||||
keys[item] = true
|
||||
result = append(result, item)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func (g *SyntheticDataGenerator) generateReadmeContent(modelName, author string) string {
|
||||
templates := []string{
|
||||
fmt.Sprintf("# %s Model\n\nThis is a %s model developed by %s. It's designed for various natural language processing tasks including text generation, question answering, and conversation.\n\n## Features\n\n- High-quality text generation\n- Efficient inference\n- Multiple quantization options\n- Easy to use with LocalAI\n\n## Usage\n\nUse this model with LocalAI for various AI tasks.", strings.Title(modelName), modelName, author),
|
||||
fmt.Sprintf("# %s\n\nA powerful language model from %s. This model excels at understanding and generating human-like text across multiple domains.\n\n## Capabilities\n\n- Text completion\n- Code generation\n- Creative writing\n- Technical documentation\n\n## Model Details\n\n- Architecture: Transformer-based\n- Training: Large-scale supervised learning\n- Quantization: Available in multiple formats", strings.Title(modelName), author),
|
||||
fmt.Sprintf("# %s Language Model\n\nDeveloped by %s, this model represents state-of-the-art performance in natural language understanding and generation.\n\n## Key Features\n\n- Multilingual support\n- Context-aware responses\n- Efficient memory usage\n- Fast inference speed\n\n## Applications\n\n- Chatbots and virtual assistants\n- Content generation\n- Code completion\n- Educational tools", strings.Title(modelName), author),
|
||||
}
|
||||
|
||||
return templates[g.rand.IntN(len(templates))]
|
||||
}
|
||||
39
.github/gh_curl.sh
vendored
39
.github/gh_curl.sh
vendored
@@ -1,39 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Shared curl wrapper for the nightly dependency-bump scripts.
|
||||
#
|
||||
# The bump workflow fans out to ~25 parallel matrix jobs, each querying
|
||||
# api.github.com. Anonymous API calls are capped at 60/hour per source IP and
|
||||
# GitHub-hosted runners egress through shared NAT addresses, so a random handful
|
||||
# of jobs were getting rate-limited (HTTP 403 -> curl exit 22, empty response)
|
||||
# every single night. Authenticating with GITHUB_TOKEN lifts the ceiling to
|
||||
# 1000/hour; the retries absorb whatever transient blips remain.
|
||||
|
||||
# Wraps curl with GitHub auth (when a token is present) plus retry/timeout
|
||||
# hardening. Callers pass their own headers and the URL.
|
||||
gh_curl() {
|
||||
# The bump scripts run under `set -x`; without this the Authorization header
|
||||
# would be echoed into the job log on every call.
|
||||
local had_xtrace=0
|
||||
case "$-" in
|
||||
*x*) had_xtrace=1; set +x ;;
|
||||
esac
|
||||
|
||||
local args=(
|
||||
--silent --show-error --location --fail
|
||||
# --retry-all-errors so 403 rate-limit responses are retried too; plain
|
||||
# --retry only covers 408/429/5xx. curl honours Retry-After when sent.
|
||||
--retry 5 --retry-delay 3 --retry-all-errors
|
||||
--connect-timeout 15 --max-time 60
|
||||
)
|
||||
if [ -n "${GITHUB_TOKEN:-}" ]; then
|
||||
args+=(--header "Authorization: Bearer ${GITHUB_TOKEN}")
|
||||
fi
|
||||
|
||||
curl "${args[@]}" "$@"
|
||||
local rc=$?
|
||||
|
||||
if [ "$had_xtrace" -eq 1 ]; then
|
||||
set -x
|
||||
fi
|
||||
return $rc
|
||||
}
|
||||
46
.github/scripts/anchor-digest-in-cache.sh
vendored
46
.github/scripts/anchor-digest-in-cache.sh
vendored
@@ -1,46 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Anchor a backend per-arch digest in quay.io/go-skynet/ci-cache so quay's
|
||||
# garbage collector won't reap the manifest before backend_merge.yml runs.
|
||||
#
|
||||
# Context: backend_build.yml pushes by canonical digest only
|
||||
# (push-by-digest=true). Unreferenced manifests on quay can be reaped within
|
||||
# ~1-2h, but backend-merge-jobs runs only after the *entire* per-arch build
|
||||
# matrix drains (max-parallel: 8 × dozens of entries → ~2h+). Without an
|
||||
# anchoring tag, the earliest digests are gone by the time `imagetools create`
|
||||
# tries to read them, producing "manifest not found" merge failures.
|
||||
#
|
||||
# We tag the digest under our internal ci-cache image; quay does not GC tagged
|
||||
# manifests. The user-facing manifest list still references the original
|
||||
# digest in local-ai-backends. backend_merge.yml deletes the anchor tag after
|
||||
# the user-facing manifest is published — see cleanup-keepalive-tags.sh.
|
||||
#
|
||||
# Required env:
|
||||
# GITHUB_RUN_ID - current workflow run id (set automatically by GHA)
|
||||
# TAG_SUFFIX - matrix entry's tag-suffix (e.g. -gpu-nvidia-cuda-12-vllm)
|
||||
# PLATFORM_TAG - amd64 / arm64 / single (single = singleton matrix entry)
|
||||
# DIGEST - canonical content digest from build step (sha256:...)
|
||||
#
|
||||
# Optional env:
|
||||
# ANCHOR_IMAGE - target image (default: quay.io/go-skynet/ci-cache)
|
||||
# SOURCE_IMAGE - source image (default: quay.io/go-skynet/local-ai-backends)
|
||||
# GITHUB_STEP_SUMMARY - if set, an anchored-by line is appended to it
|
||||
set -euo pipefail
|
||||
|
||||
: "${GITHUB_RUN_ID:?}"
|
||||
: "${TAG_SUFFIX:?}"
|
||||
: "${PLATFORM_TAG:?}"
|
||||
: "${DIGEST:?}"
|
||||
|
||||
anchor_image="${ANCHOR_IMAGE:-quay.io/go-skynet/ci-cache}"
|
||||
source_image="${SOURCE_IMAGE:-quay.io/go-skynet/local-ai-backends}"
|
||||
|
||||
tag="keepalive-${GITHUB_RUN_ID}${TAG_SUFFIX}-${PLATFORM_TAG}"
|
||||
|
||||
docker buildx imagetools create \
|
||||
-t "${anchor_image}:${tag}" \
|
||||
"${source_image}@${DIGEST}"
|
||||
|
||||
echo "anchored ${DIGEST} as ${anchor_image}:${tag}"
|
||||
if [[ -n "${GITHUB_STEP_SUMMARY:-}" ]]; then
|
||||
echo "anchored \`${DIGEST}\` as \`${anchor_image}:${tag}\`" >> "${GITHUB_STEP_SUMMARY}"
|
||||
fi
|
||||
49
.github/scripts/cleanup-keepalive-tags.sh
vendored
49
.github/scripts/cleanup-keepalive-tags.sh
vendored
@@ -1,49 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Best-effort cleanup of the keepalive anchor tags written by
|
||||
# anchor-digest-in-cache.sh. Called from backend_merge.yml after the
|
||||
# user-facing manifest list has been published.
|
||||
#
|
||||
# Quay's docker registry v2 doesn't allow tag deletes — only digest deletes.
|
||||
# The proper delete is the quay REST API, which requires an OAuth-scoped
|
||||
# token. We try QUAY_TOKEN as a bearer token: if the secret is an OAuth app
|
||||
# token (typical for service accounts) the delete succeeds; otherwise this
|
||||
# is a soft no-op and the tag persists until manually pruned.
|
||||
#
|
||||
# Cleanup failure MUST NOT fail the merge — the merge has already produced
|
||||
# the user-facing manifest list at this point and the keepalive tags are
|
||||
# pure overhead. We always exit 0.
|
||||
#
|
||||
# Required env:
|
||||
# GITHUB_RUN_ID - current workflow run id (set automatically by GHA)
|
||||
# TAG_SUFFIX - matrix entry's tag-suffix (e.g. -gpu-nvidia-cuda-12-vllm)
|
||||
# QUAY_TOKEN - bearer token for quay's REST API
|
||||
#
|
||||
# Optional env:
|
||||
# QUAY_REPO - target repo (default: go-skynet/ci-cache)
|
||||
# PLATFORM_TAGS - space-separated list of platform-tag values to try
|
||||
# (default: "amd64 arm64 single")
|
||||
# We don't know which platform-tag(s) exist for this
|
||||
# tag-suffix without an extra API call, so we just try
|
||||
# all three and ignore 404s for the ones that don't.
|
||||
set -uo pipefail
|
||||
|
||||
: "${GITHUB_RUN_ID:?}"
|
||||
: "${TAG_SUFFIX:?}"
|
||||
: "${QUAY_TOKEN:?}"
|
||||
|
||||
quay_repo="${QUAY_REPO:-go-skynet/ci-cache}"
|
||||
platform_tags="${PLATFORM_TAGS:-amd64 arm64 single}"
|
||||
|
||||
for plat in $platform_tags; do
|
||||
tag="keepalive-${GITHUB_RUN_ID}${TAG_SUFFIX}-${plat}"
|
||||
url="https://quay.io/api/v1/repository/${quay_repo}/tag/${tag}"
|
||||
http=$(curl -sS -o /dev/null -w '%{http_code}' \
|
||||
-X DELETE -H "Authorization: Bearer ${QUAY_TOKEN}" "$url" || echo "000")
|
||||
case "$http" in
|
||||
204|200) echo "deleted $tag" ;;
|
||||
404) echo "not present: $tag" ;;
|
||||
401|403) echo "auth not OAuth-scoped (http $http) for $tag - skipping; orphan tag will persist" ;;
|
||||
*) echo "unexpected http $http deleting $tag - skipping" ;;
|
||||
esac
|
||||
done
|
||||
exit 0
|
||||
370
.github/workflows/backend.yml
vendored
370
.github/workflows/backend.yml
vendored
@@ -1,370 +0,0 @@
|
||||
---
|
||||
name: 'build backend container images'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
tags:
|
||||
- '*'
|
||||
schedule:
|
||||
# Weekly full-matrix rebuild to pick up upstream Python wheel updates
|
||||
# (torch, transformers, vllm, ...) which most backends pull unpinned.
|
||||
# The DEPS_REFRESH build-arg in backend_build.yml busts the install
|
||||
# layer cache on a new ISO week, but only fires when the build runs.
|
||||
# Path filtering on commit-driven pushes (scripts/changed-backends.js)
|
||||
# skips untouched backends, so without this cron those images would
|
||||
# drift on stale wheels indefinitely. C++/Go backends with pinned
|
||||
# deps cache-hit and finish fast.
|
||||
#
|
||||
# Schedule events have no event.ref / event.before, so the script's
|
||||
# changedFiles==null fallback emits the full matrix automatically —
|
||||
# no script changes needed.
|
||||
- cron: '0 6 * * 0' # Sundays 06:00 UTC
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: ci-backends-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
generate-matrix:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
|
||||
matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
|
||||
merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
|
||||
has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
|
||||
has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
|
||||
has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
|
||||
# Single-arch backends are sharded across SINGLEARCH_SHARDS matrix jobs to
|
||||
# stay under GitHub's 256-jobs-per-matrix limit (see changed-backends.js).
|
||||
matrix-singlearch-1: ${{ steps.set-matrix.outputs['matrix-singlearch-1'] }}
|
||||
merge-matrix-singlearch-1: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-1'] }}
|
||||
has-backends-singlearch-1: ${{ steps.set-matrix.outputs['has-backends-singlearch-1'] }}
|
||||
has-merges-singlearch-1: ${{ steps.set-matrix.outputs['has-merges-singlearch-1'] }}
|
||||
matrix-singlearch-2: ${{ steps.set-matrix.outputs['matrix-singlearch-2'] }}
|
||||
merge-matrix-singlearch-2: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-2'] }}
|
||||
has-backends-singlearch-2: ${{ steps.set-matrix.outputs['has-backends-singlearch-2'] }}
|
||||
has-merges-singlearch-2: ${{ steps.set-matrix.outputs['has-merges-singlearch-2'] }}
|
||||
matrix-singlearch-3: ${{ steps.set-matrix.outputs['matrix-singlearch-3'] }}
|
||||
merge-matrix-singlearch-3: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-3'] }}
|
||||
has-backends-singlearch-3: ${{ steps.set-matrix.outputs['has-backends-singlearch-3'] }}
|
||||
has-merges-singlearch-3: ${{ steps.set-matrix.outputs['has-merges-singlearch-3'] }}
|
||||
matrix-singlearch-4: ${{ steps.set-matrix.outputs['matrix-singlearch-4'] }}
|
||||
merge-matrix-singlearch-4: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-4'] }}
|
||||
has-backends-singlearch-4: ${{ steps.set-matrix.outputs['has-backends-singlearch-4'] }}
|
||||
has-merges-singlearch-4: ${{ steps.set-matrix.outputs['has-merges-singlearch-4'] }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
bun add js-yaml
|
||||
bun add @octokit/core
|
||||
|
||||
# Filter the backend matrix from .github/backend-matrix.yml against the
|
||||
# files changed by this push. Tag pushes set FORCE_ALL=true so the script
|
||||
# falls through to the full matrix (releases must rebuild everything).
|
||||
# The script splits the linux matrix into single-arch and multi-arch
|
||||
# groups so backend-merge-jobs can `needs:` only the multi-arch one —
|
||||
# see the comment block above the merge job for context.
|
||||
- name: Filter matrix for changed backends
|
||||
id: set-matrix
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GITHUB_EVENT_PATH: ${{ github.event_path }}
|
||||
FORCE_ALL: ${{ startsWith(github.ref, 'refs/tags/') && 'true' || 'false' }}
|
||||
run: bun run scripts/changed-backends.js
|
||||
|
||||
# Multi-arch backends — entries with a `platform-tag` set, paired with a
|
||||
# sibling entry sharing the same `tag-suffix` (one amd64 leg, one arm64
|
||||
# leg). Their digests are the inputs to backend-merge-jobs, so they're in
|
||||
# their own matrix to bound how long the merge waits before quay GCs the
|
||||
# untagged digests.
|
||||
backend-jobs-multiarch:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-multiarch'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
|
||||
|
||||
# Single-arch backends — no `platform-tag`. Heavy ones (CUDA, ROCm, Intel
|
||||
# oneAPI, vLLM/sglang) live here. Independent of the merge job: they can
|
||||
# take their full ~6h cold without blocking manifest assembly for the
|
||||
# multi-arch backends whose per-arch digests would otherwise sit untagged
|
||||
# on quay long enough to be GC'd.
|
||||
backend-jobs-singlearch-1:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-1'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-1']) }}
|
||||
|
||||
backend-jobs-singlearch-2:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-2'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-2']) }}
|
||||
|
||||
backend-jobs-singlearch-3:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-3'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-3']) }}
|
||||
|
||||
backend-jobs-singlearch-4:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-4'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-4']) }}
|
||||
|
||||
# Apply tags to per-arch digests via `imagetools create`. Split into two
|
||||
# jobs that mirror the build split so each merge waits ONLY on its
|
||||
# corresponding build matrix:
|
||||
#
|
||||
# - backend-merge-jobs-multiarch needs backend-jobs-multiarch (~2-3h)
|
||||
# - backend-merge-jobs-singlearch needs backend-jobs-singlearch (up to ~6h)
|
||||
#
|
||||
# If a single shared merge job depended on both, slow CUDA singlearch
|
||||
# builds would block multiarch merges long enough for quay's GC to reap
|
||||
# the multiarch per-arch digests (the bug fixed by PR #9746). Singletons
|
||||
# also need a merge step because backend_build.yml pushes by canonical
|
||||
# digest only — no tags are applied at build time.
|
||||
backend-merge-jobs-multiarch:
|
||||
needs: [generate-matrix, backend-jobs-multiarch]
|
||||
# !cancelled() lets the merge run even when a few build legs failed.
|
||||
# Without it, GHA's default `needs:` cascade skips the entire merge
|
||||
# matrix on a single failed/cancelled cell. We still want to publish
|
||||
# the manifest lists for tag-suffixes whose legs all succeeded.
|
||||
# Observed in v4.2.1: 2 singlearch build failures cascade-skipped all
|
||||
# ~199 singlearch merge entries.
|
||||
if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-multiarch'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
|
||||
|
||||
# One merge shard per build shard: backend-merge-jobs-singlearch-<n> needs only
|
||||
# backend-jobs-singlearch-<n>, preserving the "merge waits only on its own
|
||||
# build" property while staying under the 256-jobs-per-matrix limit.
|
||||
backend-merge-jobs-singlearch-1:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-1]
|
||||
if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-1'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-1']) }}
|
||||
|
||||
backend-merge-jobs-singlearch-2:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-2]
|
||||
if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-2'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-2']) }}
|
||||
|
||||
backend-merge-jobs-singlearch-3:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-3]
|
||||
if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-3'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-3']) }}
|
||||
|
||||
backend-merge-jobs-singlearch-4:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-4]
|
||||
if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-4'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-4']) }}
|
||||
|
||||
backend-jobs-darwin:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs.has-backends-darwin == 'true'
|
||||
uses: ./.github/workflows/backend_build_darwin.yml
|
||||
with:
|
||||
backend: ${{ matrix.backend }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
go-version: "1.25.x"
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
lang: ${{ matrix.lang || 'python' }}
|
||||
use-pip: ${{ matrix.backend == 'diffusers' }}
|
||||
runs-on: "macos-latest"
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix-darwin) }}
|
||||
288
.github/workflows/backend_build.yml
vendored
288
.github/workflows/backend_build.yml
vendored
@@ -1,288 +0,0 @@
|
||||
---
|
||||
name: 'build backend container images (reusable)'
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
base-image:
|
||||
description: 'Base image'
|
||||
required: true
|
||||
type: string
|
||||
build-type:
|
||||
description: 'Build type'
|
||||
default: ''
|
||||
type: string
|
||||
cuda-major-version:
|
||||
description: 'CUDA major version'
|
||||
default: "12"
|
||||
type: string
|
||||
cuda-minor-version:
|
||||
description: 'CUDA minor version'
|
||||
default: "1"
|
||||
type: string
|
||||
platforms:
|
||||
description: 'Platforms'
|
||||
default: ''
|
||||
type: string
|
||||
platform-tag:
|
||||
description: |
|
||||
Short tag identifying the platform leg, e.g. "amd64" or "arm64".
|
||||
Used to scope the per-arch registry cache and the digest artifact name.
|
||||
Required for split-and-merge multi-arch builds; pass "amd64" for
|
||||
single-arch amd64 builds too. Optional (default '') during the
|
||||
migration to per-arch matrix expansion; will be flipped to
|
||||
required: true in Phase 6 once all callers pass an explicit value.
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
tag-latest:
|
||||
description: 'Tag latest'
|
||||
default: ''
|
||||
type: string
|
||||
tag-suffix:
|
||||
description: 'Tag suffix'
|
||||
default: ''
|
||||
type: string
|
||||
runs-on:
|
||||
description: 'Runs on'
|
||||
required: true
|
||||
default: ''
|
||||
type: string
|
||||
backend:
|
||||
description: 'Backend to build'
|
||||
required: true
|
||||
type: string
|
||||
context:
|
||||
description: 'Build context'
|
||||
required: true
|
||||
type: string
|
||||
dockerfile:
|
||||
description: 'Build Dockerfile'
|
||||
required: true
|
||||
type: string
|
||||
skip-drivers:
|
||||
description: 'Skip drivers'
|
||||
default: 'false'
|
||||
type: string
|
||||
ubuntu-version:
|
||||
description: 'Ubuntu version'
|
||||
required: false
|
||||
default: '2204'
|
||||
type: string
|
||||
amdgpu-targets:
|
||||
description: 'AMD GPU targets for ROCm/HIP builds'
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
builder-base-image:
|
||||
description: |
|
||||
Pre-built builder base image (e.g. quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64).
|
||||
When set, the variant Dockerfile uses its `builder-prebuilt` stage which FROMs this
|
||||
image directly instead of running its own gRPC stage + apt installs. Empty for
|
||||
backends whose Dockerfile doesn't support a prebuilt base.
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
secrets:
|
||||
dockerUsername:
|
||||
required: false
|
||||
dockerPassword:
|
||||
required: false
|
||||
quayUsername:
|
||||
required: true
|
||||
quayPassword:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
backend-build:
|
||||
runs-on: ${{ inputs.runs-on }}
|
||||
env:
|
||||
quay_username: ${{ secrets.quayUsername }}
|
||||
steps:
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Configure apt mirror on runner
|
||||
id: apt_mirror
|
||||
uses: ./.github/actions/configure-apt-mirror
|
||||
|
||||
- name: Free disk space
|
||||
uses: ./.github/actions/free-disk-space
|
||||
with:
|
||||
mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
|
||||
|
||||
- name: Set up build disk
|
||||
uses: ./.github/actions/setup-build-disk
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai-backends
|
||||
localai/localai-backends
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
|
||||
- name: Docker meta for PR
|
||||
id: meta_pull_request
|
||||
if: github.event_name == 'pull_request'
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/ci-tests
|
||||
tags: |
|
||||
type=ref,event=branch,suffix=${{ github.event.number }}-${{ inputs.backend }}-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}
|
||||
type=semver,pattern={{raw}},suffix=${{ github.event.number }}-${{ inputs.backend }}-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}
|
||||
type=sha,suffix=${{ github.event.number }}-${{ inputs.backend }}-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
## End testing image
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@master
|
||||
with:
|
||||
platforms: all
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
id: buildx
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
username: ${{ secrets.dockerUsername }}
|
||||
password: ${{ secrets.dockerPassword }}
|
||||
|
||||
- name: Login to Quay.io
|
||||
if: ${{ env.quay_username != '' }}
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
# Weekly cache-buster for the per-backend `make` step. Most Python
|
||||
# backends list unpinned deps (torch, transformers, vllm, ...), so a
|
||||
# warm cache freezes upstream versions indefinitely. Rolling this
|
||||
# weekly forces a re-resolve of the install layer at most once per
|
||||
# week, picking up newer wheels without a full cold rebuild.
|
||||
- name: Compute deps refresh key
|
||||
id: deps_refresh
|
||||
run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Build and push by digest
|
||||
id: build
|
||||
uses: docker/build-push-action@v7
|
||||
if: github.event_name != 'pull_request'
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
build-args: |
|
||||
BUILD_TYPE=${{ inputs.build-type }}
|
||||
SKIP_DRIVERS=${{ inputs.skip-drivers }}
|
||||
CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
|
||||
CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
|
||||
BASE_IMAGE=${{ inputs.base-image }}
|
||||
BACKEND=${{ inputs.backend }}
|
||||
UBUNTU_VERSION=${{ inputs.ubuntu-version }}
|
||||
AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
|
||||
APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
|
||||
BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
|
||||
BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
|
||||
context: ${{ inputs.context }}
|
||||
file: ${{ inputs.dockerfile }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
|
||||
platforms: ${{ inputs.platforms }}
|
||||
outputs: |
|
||||
type=image,name=quay.io/go-skynet/local-ai-backends,push-by-digest=true,name-canonical=true,push=true
|
||||
type=image,name=localai/localai-backends,push-by-digest=true,name-canonical=true,push=true
|
||||
# Disable provenance: with mode=max (the default for push:true)
|
||||
# buildx bundles a per-registry attestation manifest into each
|
||||
# registry's manifest list, which makes the resulting list digest
|
||||
# diverge across registries. steps.build.outputs.digest then
|
||||
# only matches one of them, and the merge job's
|
||||
# `imagetools create <reg>@sha256:<digest>` lookup fails on the
|
||||
# other. Disabling provenance keeps the digest content-only and
|
||||
# identical across both registries — required for digest-based
|
||||
# cross-registry merge.
|
||||
provenance: false
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Export digest
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
mkdir -p /tmp/digests
|
||||
digest="${{ steps.build.outputs.digest }}"
|
||||
touch "/tmp/digests/${digest#sha256:}"
|
||||
|
||||
# See .github/scripts/anchor-digest-in-cache.sh for why this is needed
|
||||
# and how it interacts with backend_merge.yml's cleanup step.
|
||||
- name: Anchor digest in ci-cache so quay GC won't reap before merge
|
||||
if: github.event_name != 'pull_request'
|
||||
env:
|
||||
TAG_SUFFIX: ${{ inputs.tag-suffix }}
|
||||
PLATFORM_TAG: ${{ inputs.platform-tag || 'single' }}
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
run: .github/scripts/anchor-digest-in-cache.sh
|
||||
|
||||
# Artifact name uses a `--` separator between tag-suffix and platform-tag
|
||||
# to avoid prefix collisions during the merge job's pattern-based download.
|
||||
# Tag-suffixes are not prefix-disjoint (e.g. -gpu-nvidia-cuda-12-vllm is a
|
||||
# prefix of -gpu-nvidia-cuda-12-vllm-omni); a single `-` separator plus the
|
||||
# merge-side `digests<tag-suffix>-*` glob would let one merge over-match
|
||||
# the other backend's artifacts. The `-single` placeholder for empty
|
||||
# platform-tag (single-arch entries) keeps the artifact name non-trailing.
|
||||
- name: Upload digest artifact
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: digests${{ inputs.tag-suffix }}--${{ inputs.platform-tag || 'single' }}
|
||||
path: /tmp/digests/*
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
|
||||
- name: Build (PR)
|
||||
uses: docker/build-push-action@v7
|
||||
if: github.event_name == 'pull_request'
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
build-args: |
|
||||
BUILD_TYPE=${{ inputs.build-type }}
|
||||
SKIP_DRIVERS=${{ inputs.skip-drivers }}
|
||||
CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
|
||||
CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
|
||||
BASE_IMAGE=${{ inputs.base-image }}
|
||||
BACKEND=${{ inputs.backend }}
|
||||
UBUNTU_VERSION=${{ inputs.ubuntu-version }}
|
||||
AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
|
||||
APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
|
||||
BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
|
||||
BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
|
||||
context: ${{ inputs.context }}
|
||||
file: ${{ inputs.dockerfile }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
platforms: ${{ inputs.platforms }}
|
||||
push: ${{ env.quay_username != '' }}
|
||||
tags: ${{ steps.meta_pull_request.outputs.tags }}
|
||||
labels: ${{ steps.meta_pull_request.outputs.labels }}
|
||||
|
||||
|
||||
|
||||
- name: job summary
|
||||
run: |
|
||||
echo "Built image: ${{ steps.meta.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
|
||||
341
.github/workflows/backend_build_darwin.yml
vendored
341
.github/workflows/backend_build_darwin.yml
vendored
@@ -1,341 +0,0 @@
|
||||
---
|
||||
name: 'build darwin python backend container images (reusable)'
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
backend:
|
||||
description: 'Backend to build'
|
||||
required: true
|
||||
type: string
|
||||
build-type:
|
||||
description: 'Build type (e.g., mps)'
|
||||
default: ''
|
||||
type: string
|
||||
use-pip:
|
||||
description: 'Use pip to install dependencies'
|
||||
default: false
|
||||
type: boolean
|
||||
lang:
|
||||
description: 'Programming language (e.g. go)'
|
||||
default: 'python'
|
||||
type: string
|
||||
go-version:
|
||||
description: 'Go version to use'
|
||||
default: '1.24.x'
|
||||
type: string
|
||||
tag-suffix:
|
||||
description: 'Tag suffix for the built image'
|
||||
required: true
|
||||
type: string
|
||||
runs-on:
|
||||
description: 'Runner to use'
|
||||
default: 'macOS-14'
|
||||
type: string
|
||||
secrets:
|
||||
dockerUsername:
|
||||
required: false
|
||||
dockerPassword:
|
||||
required: false
|
||||
quayUsername:
|
||||
required: true
|
||||
quayPassword:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
darwin-backend-build:
|
||||
runs-on: ${{ inputs.runs-on }}
|
||||
strategy:
|
||||
matrix:
|
||||
go-version: ['${{ inputs.go-version }}']
|
||||
env:
|
||||
# Keep the brew Cellar stable across cache restores. Without these,
|
||||
# `brew install` would auto-update brew itself and re-link formulas,
|
||||
# mutating the very paths the cache just restored.
|
||||
HOMEBREW_NO_AUTO_UPDATE: '1'
|
||||
HOMEBREW_NO_INSTALL_CLEANUP: '1'
|
||||
HOMEBREW_NO_ANALYTICS: '1'
|
||||
steps:
|
||||
- name: Clone
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Setup Go ${{ matrix.go-version }}
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: ${{ matrix.go-version }}
|
||||
# Caches ~/go/pkg/mod and ~/Library/Caches/go-build keyed on go.sum.
|
||||
# Shared across every darwin matrix entry — first job in a run warms
|
||||
# it, the rest hit warm.
|
||||
cache: true
|
||||
|
||||
# You can test your matrix by printing the current Go version
|
||||
- name: Display Go version
|
||||
run: go version
|
||||
|
||||
# ---- Homebrew cache ----
|
||||
# macOS runners have no Docker daemon, so the BuildKit registry cache used
|
||||
# for Linux backend images (see .agents/ci-caching.md) doesn't apply here.
|
||||
# We cache the brew downloads + Cellar entries for the formulas we install
|
||||
# below. Read on every run, write only on master/tag pushes — same policy
|
||||
# as the Linux registry cache.
|
||||
- name: Restore Homebrew cache
|
||||
id: brew-cache
|
||||
uses: actions/cache/restore@v6
|
||||
with:
|
||||
path: |
|
||||
~/Library/Caches/Homebrew/downloads
|
||||
/opt/homebrew/Cellar/protobuf
|
||||
/opt/homebrew/Cellar/grpc
|
||||
/opt/homebrew/Cellar/protoc-gen-go
|
||||
/opt/homebrew/Cellar/protoc-gen-go-grpc
|
||||
/opt/homebrew/Cellar/libomp
|
||||
/opt/homebrew/Cellar/llvm
|
||||
/opt/homebrew/Cellar/ccache
|
||||
/opt/homebrew/Cellar/blake3
|
||||
/opt/homebrew/Cellar/fmt
|
||||
/opt/homebrew/Cellar/hiredis
|
||||
/opt/homebrew/Cellar/xxhash
|
||||
/opt/homebrew/Cellar/zstd
|
||||
/opt/homebrew/Cellar/nlohmann-json
|
||||
/opt/homebrew/Cellar/opus
|
||||
key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
|
||||
|
||||
- name: Dependencies
|
||||
run: |
|
||||
# ccache is always installed (used by the llama-cpp variant build) so
|
||||
# the brew cache content stays stable across every backend in the
|
||||
# matrix — they all share one cache key.
|
||||
# blake3, fmt, hiredis, xxhash, zstd are ccache's runtime dylib deps.
|
||||
# Without explicitly installing them, a brew cache-hit run restores
|
||||
# ccache's Cellar dir but skips installing those transitive deps,
|
||||
# and ccache fails at runtime with `dyld: Library not loaded`.
|
||||
# nlohmann-json is header-only and required by the ds4 backend
|
||||
# (dsml_renderer.cpp includes <nlohmann/json.hpp>); on Linux it comes
|
||||
# from the apt-installed nlohmann-json3-dev in the build image.
|
||||
# opus + pkg-config are required by the opus go backend: its
|
||||
# Makefile/package.sh call `pkg-config --cflags/--libs opus` to build
|
||||
# libopusshim.dylib and to locate libopus.dylib for bundling. brew's
|
||||
# pkg-config defaults its search path to the Homebrew prefix so the
|
||||
# opus.pc is found.
|
||||
brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd nlohmann-json opus pkg-config
|
||||
# Force-reinstall ccache so brew re-validates its full runtime-dep
|
||||
# closure on every run. This is the durable fix: when the upstream
|
||||
# ccache formula gains a new transitive dep (as it has multiple times
|
||||
# already), we don't have to chase missing dylibs one at a time.
|
||||
# The downloads cache makes the reinstall fast (~5s on a hit).
|
||||
brew reinstall ccache
|
||||
# Same pattern for grpc: its CMake config (used by the llama-cpp
|
||||
# `grpc-server` target) does find_package(absl). The cache restores
|
||||
# /opt/homebrew/Cellar/grpc so brew above no-ops the install, but
|
||||
# abseil isn't in our Cellar cache list and never gets installed
|
||||
# alongside, leaving grpc's CMake unable to resolve it. Reinstalling
|
||||
# grpc re-validates and pulls abseil in, mirroring the ccache fix.
|
||||
brew reinstall grpc
|
||||
# The brew cache restores the Cellar dirs but NOT the bin symlinks
|
||||
# at /opt/homebrew/bin/*. brew install above sees the Cellar present
|
||||
# and decides "already installed" without re-linking, so on a cache-
|
||||
# hit run the formulas aren't on PATH. Force-link them; --overwrite
|
||||
# tolerates pre-existing symlinks from earlier installs.
|
||||
brew link --overwrite protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd nlohmann-json opus pkg-config 2>/dev/null || true
|
||||
|
||||
- name: Save Homebrew cache
|
||||
if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
|
||||
uses: actions/cache/save@v6
|
||||
with:
|
||||
path: |
|
||||
~/Library/Caches/Homebrew/downloads
|
||||
/opt/homebrew/Cellar/protobuf
|
||||
/opt/homebrew/Cellar/grpc
|
||||
/opt/homebrew/Cellar/protoc-gen-go
|
||||
/opt/homebrew/Cellar/protoc-gen-go-grpc
|
||||
/opt/homebrew/Cellar/libomp
|
||||
/opt/homebrew/Cellar/llvm
|
||||
/opt/homebrew/Cellar/ccache
|
||||
/opt/homebrew/Cellar/blake3
|
||||
/opt/homebrew/Cellar/fmt
|
||||
/opt/homebrew/Cellar/hiredis
|
||||
/opt/homebrew/Cellar/xxhash
|
||||
/opt/homebrew/Cellar/zstd
|
||||
/opt/homebrew/Cellar/nlohmann-json
|
||||
/opt/homebrew/Cellar/opus
|
||||
key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
|
||||
|
||||
# ---- ccache for llama.cpp CMake builds ----
|
||||
# Three CMake variants (fallback, grpc, rpc-server) compile the same
|
||||
# llama.cpp source tree with overlapping flags — ccache dedupes object
|
||||
# files across them. Key on the pinned LLAMA_VERSION so a pin bump
|
||||
# invalidates cleanly; restore-keys fall back to the latest entry for the
|
||||
# same pin so unchanged TUs stay warm even when the cache is fresh.
|
||||
- name: Compute llama.cpp version
|
||||
if: inputs.backend == 'llama-cpp'
|
||||
id: llama-version
|
||||
run: |
|
||||
version=$(grep '^LLAMA_VERSION' backend/cpp/llama-cpp/Makefile | head -1 | cut -d= -f2 | cut -d'?' -f1 | tr -d ' ')
|
||||
echo "version=${version}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Restore ccache
|
||||
if: inputs.backend == 'llama-cpp'
|
||||
id: ccache-cache
|
||||
uses: actions/cache/restore@v6
|
||||
with:
|
||||
path: ~/Library/Caches/ccache
|
||||
key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-
|
||||
|
||||
- name: Configure ccache
|
||||
if: inputs.backend == 'llama-cpp'
|
||||
run: |
|
||||
mkdir -p "$HOME/Library/Caches/ccache"
|
||||
ccache -M 2G
|
||||
ccache -z
|
||||
# llama-cpp-darwin.sh reads CMAKE_ARGS / CCACHE_DIR from env.
|
||||
{
|
||||
echo "CMAKE_ARGS=${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache"
|
||||
echo "CCACHE_DIR=$HOME/Library/Caches/ccache"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
# ---- Python wheel cache (uv + pip) ----
|
||||
# Mirrors the Linux DEPS_REFRESH cadence (see .agents/ci-caching.md): the
|
||||
# ISO-week segment of the cache key forces at most one cold rebuild per
|
||||
# backend per week, automatically picking up newer wheels for unpinned
|
||||
# deps (torch, mlx, diffusers, …). Restore-keys fall back to the most
|
||||
# recent build of the same backend so off-week PRs still hit warm.
|
||||
- name: Compute weekly cache bucket
|
||||
if: inputs.lang == 'python'
|
||||
id: weekly
|
||||
run: echo "bucket=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Restore Python wheel cache
|
||||
if: inputs.lang == 'python'
|
||||
id: pyenv-cache
|
||||
uses: actions/cache/restore@v6
|
||||
with:
|
||||
path: |
|
||||
~/Library/Caches/pip
|
||||
~/Library/Caches/uv
|
||||
key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
|
||||
restore-keys: |
|
||||
pyenv-darwin-${{ inputs.backend }}-
|
||||
|
||||
# llama-cpp on Darwin uses a bespoke build script (scripts/build/llama-cpp-darwin.sh)
|
||||
# that compiles three CMake variants from backend/cpp/llama-cpp and bundles dylibs
|
||||
# via otool — it doesn't fit the build-darwin-go-backend / build-darwin-python-backend
|
||||
# mold. Drive it via its dedicated `backends/llama-cpp-darwin` make target instead.
|
||||
- name: Build ${{ inputs.backend }}-darwin (llama-cpp)
|
||||
if: inputs.backend == 'llama-cpp'
|
||||
run: |
|
||||
make protogen-go
|
||||
make backends/llama-cpp-darwin
|
||||
|
||||
- name: Build ds4 backend (Darwin Metal)
|
||||
if: inputs.backend == 'ds4'
|
||||
run: |
|
||||
make backends/ds4-darwin
|
||||
|
||||
# privacy-filter is a C++/ggml backend like ds4 - a single grpc-server with
|
||||
# otool dylib bundling - so it gets its own bespoke darwin script rather than
|
||||
# the generic build-darwin-go-backend path.
|
||||
- name: Build privacy-filter backend (Darwin Metal)
|
||||
if: inputs.backend == 'privacy-filter'
|
||||
run: |
|
||||
make protogen-go
|
||||
make backends/privacy-filter-darwin
|
||||
|
||||
- name: Build ${{ inputs.backend }}-darwin
|
||||
if: inputs.backend != 'llama-cpp' && inputs.backend != 'ds4' && inputs.backend != 'privacy-filter'
|
||||
run: |
|
||||
make protogen-go
|
||||
BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
|
||||
|
||||
- name: ccache stats
|
||||
if: inputs.backend == 'llama-cpp'
|
||||
run: ccache -s
|
||||
|
||||
- name: Save ccache
|
||||
if: inputs.backend == 'llama-cpp' && github.event_name != 'pull_request'
|
||||
uses: actions/cache/save@v6
|
||||
with:
|
||||
path: ~/Library/Caches/ccache
|
||||
key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
|
||||
|
||||
- name: Save Python wheel cache
|
||||
if: inputs.lang == 'python' && github.event_name != 'pull_request' && steps.pyenv-cache.outputs.cache-hit != 'true'
|
||||
uses: actions/cache/save@v6
|
||||
with:
|
||||
path: |
|
||||
~/Library/Caches/pip
|
||||
~/Library/Caches/uv
|
||||
key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
|
||||
|
||||
- name: Upload ${{ inputs.backend }}.tar
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: ${{ inputs.backend }}-tar
|
||||
path: backend-images/${{ inputs.backend }}.tar
|
||||
|
||||
darwin-backend-publish:
|
||||
needs: darwin-backend-build
|
||||
if: github.event_name != 'pull_request'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Download ${{ inputs.backend }}.tar
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: ${{ inputs.backend }}-tar
|
||||
path: .
|
||||
|
||||
- name: Install crane
|
||||
run: |
|
||||
curl -L https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar -xz
|
||||
sudo mv crane /usr/local/bin/
|
||||
|
||||
- name: Log in to DockerHub
|
||||
run: |
|
||||
echo "${{ secrets.dockerPassword }}" | crane auth login docker.io -u "${{ secrets.dockerUsername }}" --password-stdin
|
||||
|
||||
- name: Log in to quay.io
|
||||
run: |
|
||||
echo "${{ secrets.quayPassword }}" | crane auth login quay.io -u "${{ secrets.quayUsername }}" --password-stdin
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
localai/localai-backends
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=auto
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
|
||||
- name: Docker meta
|
||||
id: quaymeta
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai-backends
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=auto
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
|
||||
- name: Push Docker image (DockerHub)
|
||||
run: |
|
||||
for tag in $(echo "${{ steps.meta.outputs.tags }}" | tr ',' '\n'); do
|
||||
crane push ${{ inputs.backend }}.tar $tag
|
||||
done
|
||||
|
||||
- name: Push Docker image (Quay)
|
||||
run: |
|
||||
for tag in $(echo "${{ steps.quaymeta.outputs.tags }}" | tr ',' '\n'); do
|
||||
crane push ${{ inputs.backend }}.tar $tag
|
||||
done
|
||||
217
.github/workflows/backend_merge.yml
vendored
217
.github/workflows/backend_merge.yml
vendored
@@ -1,217 +0,0 @@
|
||||
---
|
||||
name: 'merge backend manifest list (reusable)'
|
||||
|
||||
# Reusable workflow that joins per-arch digest artifacts (uploaded by
|
||||
# backend_build.yml when called with platform-tag) into a single tagged
|
||||
# multi-arch manifest list. Called once per backend by backend.yml after
|
||||
# both per-arch build jobs succeed.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag-latest:
|
||||
description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
tag-suffix:
|
||||
description: 'Backend tag suffix (e.g. -cpu-faster-whisper). Used to compute the artifact pattern and the final tag suffix.'
|
||||
required: true
|
||||
type: string
|
||||
secrets:
|
||||
dockerUsername:
|
||||
required: false
|
||||
dockerPassword:
|
||||
required: false
|
||||
quayUsername:
|
||||
required: true
|
||||
quayPassword:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
merge:
|
||||
runs-on: ubuntu-latest
|
||||
# id-token: write is required for keyless cosign — the workflow
|
||||
# exchanges the GitHub OIDC token for a short-lived Fulcio cert that
|
||||
# signs each pushed manifest. Without this permission the runner
|
||||
# cannot mint the token, and `cosign sign` fails with "no token".
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
env:
|
||||
quay_username: ${{ secrets.quayUsername }}
|
||||
# cosign v2.4.x still gates --registry-referrers-mode=oci-1-1 behind
|
||||
# this flag. Without it, signing fails with:
|
||||
# invalid argument "oci-1-1" for "--registry-referrers-mode" flag:
|
||||
# in order to use mode "oci-1-1", you must set COSIGN_EXPERIMENTAL=1
|
||||
COSIGN_EXPERIMENTAL: '1'
|
||||
steps:
|
||||
# Sparse checkout: the merge job needs `.github/scripts/` (for the
|
||||
# keepalive cleanup script) but none of the source tree.
|
||||
- name: Checkout (.github/scripts only)
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
sparse-checkout: |
|
||||
.github/scripts
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
# `--` separator anchors the glob so we don't over-match sibling
|
||||
# backends whose tag-suffix happens to be a prefix of ours
|
||||
# (e.g. -cpu-vllm vs -cpu-vllm-omni). Must stay in sync with the
|
||||
# upload-artifact name in backend_build.yml.
|
||||
- name: Download digests
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
pattern: digests${{ inputs.tag-suffix }}--*
|
||||
merge-multiple: true
|
||||
path: /tmp/digests
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
# cosign signs each pushed manifest list with --recursive so the
|
||||
# index and every per-arch entry get an attached Sigstore bundle.
|
||||
# Recent cosign releases always emit the new bundle format, so
|
||||
# there's no extra CLI flag to opt into it.
|
||||
- name: Install cosign
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: sigstore/cosign-installer@v3
|
||||
with:
|
||||
cosign-release: 'v2.4.1'
|
||||
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
username: ${{ secrets.dockerUsername }}
|
||||
password: ${{ secrets.dockerPassword }}
|
||||
|
||||
- name: Login to Quay.io
|
||||
if: ${{ env.quay_username != '' }}
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai-backends
|
||||
localai/localai-backends
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
|
||||
# Source from ci-cache, not local-ai-backends.
|
||||
#
|
||||
# The build job pushes per-arch manifests to local-ai-backends with
|
||||
# push-by-digest=true (no tag), then anchors a tagged copy into
|
||||
# ci-cache so the manifest can be retrieved hours later when this
|
||||
# merge runs. Quay's manifest GC, however, is per-repository: the
|
||||
# anchor tag in ci-cache protects the manifest there, but the same
|
||||
# digest in local-ai-backends has no tag in *that* repo and gets
|
||||
# reaped independently. Sourcing local-ai-backends@<digest> here
|
||||
# then fails with "manifest not found" — exactly the regression
|
||||
# we hit on v4.2.2 (19/37 multiarch merges failed).
|
||||
#
|
||||
# ci-cache@<digest> resolves because we anchored it there. buildx
|
||||
# imagetools create copies the manifest into local-ai-backends
|
||||
# (cross-repo within the same registry, blobs already cross-mounted
|
||||
# from the original push so no transfer needed) and publishes the
|
||||
# manifest list with the user-facing tags. The resulting manifest
|
||||
# list is fully self-contained in local-ai-backends — child digests
|
||||
# only, no embedded references to ci-cache.
|
||||
- name: Create manifest list and push (quay)
|
||||
if: github.event_name != 'pull_request'
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '
|
||||
.tags
|
||||
| map(select(startswith("quay.io/")))
|
||||
| map("-t " + .)
|
||||
| join(" ")
|
||||
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No quay.io tags from docker/metadata-action; skipping quay merge"
|
||||
exit 0
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'quay.io/go-skynet/ci-cache@sha256:%s ' *)
|
||||
# Resolve the manifest-list digest (any tag points at it) so
|
||||
# cosign can sign by digest. Signing by tag would leave the
|
||||
# signature orphaned the next time the tag moves.
|
||||
first_tag=$(jq -cr '
|
||||
.tags | map(select(startswith("quay.io/"))) | .[0]
|
||||
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}')
|
||||
# --recursive walks the list and signs every per-arch entry
|
||||
# too — clients that resolve a tag to a platform-specific
|
||||
# manifest before checking signatures need the per-arch
|
||||
# signatures, not just the list-level one.
|
||||
cosign sign --yes --recursive \
|
||||
--registry-referrers-mode=oci-1-1 \
|
||||
"quay.io/go-skynet/local-ai-backends@${digest}"
|
||||
|
||||
- name: Create manifest list and push (dockerhub)
|
||||
if: github.event_name != 'pull_request'
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '
|
||||
.tags
|
||||
| map(select(startswith("localai/")))
|
||||
| map("-t " + .)
|
||||
| join(" ")
|
||||
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
|
||||
exit 0
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'localai/localai-backends@sha256:%s ' *)
|
||||
first_tag=$(jq -cr '
|
||||
.tags | map(select(startswith("localai/"))) | .[0]
|
||||
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}')
|
||||
cosign sign --yes --recursive \
|
||||
--registry-referrers-mode=oci-1-1 \
|
||||
"localai/localai-backends@${digest}"
|
||||
|
||||
- name: Inspect manifest
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
|
||||
docker buildx imagetools inspect "$first_tag"
|
||||
fi
|
||||
|
||||
# See .github/scripts/cleanup-keepalive-tags.sh for why this is
|
||||
# best-effort and what the failure modes are.
|
||||
- name: Cleanup keepalive tags in ci-cache
|
||||
if: github.event_name != 'pull_request' && success()
|
||||
env:
|
||||
TAG_SUFFIX: ${{ inputs.tag-suffix }}
|
||||
QUAY_TOKEN: ${{ secrets.quayPassword }}
|
||||
run: .github/scripts/cleanup-keepalive-tags.sh
|
||||
|
||||
- name: Job summary
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
|
||||
jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
|
||||
echo >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
|
||||
ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"
|
||||
294
.github/workflows/backend_pr.yml
vendored
294
.github/workflows/backend_pr.yml
vendored
@@ -1,294 +0,0 @@
|
||||
name: 'build backend container images (PR-filtered)'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
group: ci-backends-pr-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
generate-matrix:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
|
||||
matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
|
||||
merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
|
||||
has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
|
||||
has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
|
||||
has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
|
||||
# Single-arch backends are sharded across SINGLEARCH_SHARDS matrix jobs to
|
||||
# stay under GitHub's 256-jobs-per-matrix limit (see changed-backends.js).
|
||||
matrix-singlearch-1: ${{ steps.set-matrix.outputs['matrix-singlearch-1'] }}
|
||||
merge-matrix-singlearch-1: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-1'] }}
|
||||
has-backends-singlearch-1: ${{ steps.set-matrix.outputs['has-backends-singlearch-1'] }}
|
||||
has-merges-singlearch-1: ${{ steps.set-matrix.outputs['has-merges-singlearch-1'] }}
|
||||
matrix-singlearch-2: ${{ steps.set-matrix.outputs['matrix-singlearch-2'] }}
|
||||
merge-matrix-singlearch-2: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-2'] }}
|
||||
has-backends-singlearch-2: ${{ steps.set-matrix.outputs['has-backends-singlearch-2'] }}
|
||||
has-merges-singlearch-2: ${{ steps.set-matrix.outputs['has-merges-singlearch-2'] }}
|
||||
matrix-singlearch-3: ${{ steps.set-matrix.outputs['matrix-singlearch-3'] }}
|
||||
merge-matrix-singlearch-3: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-3'] }}
|
||||
has-backends-singlearch-3: ${{ steps.set-matrix.outputs['has-backends-singlearch-3'] }}
|
||||
has-merges-singlearch-3: ${{ steps.set-matrix.outputs['has-merges-singlearch-3'] }}
|
||||
matrix-singlearch-4: ${{ steps.set-matrix.outputs['matrix-singlearch-4'] }}
|
||||
merge-matrix-singlearch-4: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-4'] }}
|
||||
has-backends-singlearch-4: ${{ steps.set-matrix.outputs['has-backends-singlearch-4'] }}
|
||||
has-merges-singlearch-4: ${{ steps.set-matrix.outputs['has-merges-singlearch-4'] }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
bun add js-yaml
|
||||
bun add @octokit/core
|
||||
|
||||
# filters the matrix in backend.yml; splits into single-arch and
|
||||
# multi-arch groups so backend-merge-jobs can `needs:` only the latter
|
||||
# (matches backend.yml's structure).
|
||||
- name: Filter matrix for changed backends
|
||||
id: set-matrix
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GITHUB_EVENT_PATH: ${{ github.event_path }}
|
||||
run: bun run scripts/changed-backends.js
|
||||
|
||||
backend-jobs-multiarch:
|
||||
needs: generate-matrix
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
if: needs.generate-matrix.outputs['has-backends-multiarch'] == 'true'
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
|
||||
backend-jobs-singlearch-1:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-1'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-1']) }}
|
||||
|
||||
backend-jobs-singlearch-2:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-2'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-2']) }}
|
||||
|
||||
backend-jobs-singlearch-3:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-3'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-3']) }}
|
||||
|
||||
backend-jobs-singlearch-4:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch-4'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-4']) }}
|
||||
backend-merge-jobs-multiarch:
|
||||
needs: [generate-matrix, backend-jobs-multiarch]
|
||||
# backend_merge.yml's push-side steps are all gated on
|
||||
# github.event_name != 'pull_request', so on a PR the merge job would
|
||||
# do nothing. Skip it entirely to avoid spinning up an empty runner.
|
||||
# !cancelled() lets the merge run even when a few build legs fail —
|
||||
# see the matching note in backend.yml.
|
||||
if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-multiarch'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
|
||||
|
||||
backend-merge-jobs-singlearch-1:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-1]
|
||||
if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-1'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-1']) }}
|
||||
|
||||
backend-merge-jobs-singlearch-2:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-2]
|
||||
if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-2'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-2']) }}
|
||||
|
||||
backend-merge-jobs-singlearch-3:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-3]
|
||||
if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-3'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-3']) }}
|
||||
|
||||
backend-merge-jobs-singlearch-4:
|
||||
needs: [generate-matrix, backend-jobs-singlearch-4]
|
||||
if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-4'] == 'true' }}
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-4']) }}
|
||||
backend-jobs-darwin:
|
||||
needs: generate-matrix
|
||||
uses: ./.github/workflows/backend_build_darwin.yml
|
||||
if: needs.generate-matrix.outputs.has-backends-darwin == 'true'
|
||||
with:
|
||||
backend: ${{ matrix.backend }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
go-version: "1.25.x"
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
lang: ${{ matrix.lang || 'python' }}
|
||||
use-pip: ${{ matrix.backend == 'diffusers' }}
|
||||
runs-on: "macos-latest"
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix-darwin) }}
|
||||
161
.github/workflows/base-images.yml
vendored
161
.github/workflows/base-images.yml
vendored
@@ -1,161 +0,0 @@
|
||||
---
|
||||
name: 'build base-grpc images'
|
||||
|
||||
# Builds + pushes pre-compiled builder base images that downstream
|
||||
# llama-cpp / ik-llama-cpp / turboquant variant Dockerfiles will FROM
|
||||
# (PR 2). Each base contains apt deps + protoc + cmake + gRPC at
|
||||
# /opt/grpc + (conditionally) CUDA / ROCm / Vulkan toolchains.
|
||||
#
|
||||
# Triggers:
|
||||
# - schedule (Saturdays 05:00 UTC) - picks up Ubuntu/CUDA/ROCm
|
||||
# security updates and re-runs ahead of the backend.yml weekly
|
||||
# cron (Sundays 06:00 UTC).
|
||||
# - workflow_dispatch - manual one-off rebuild.
|
||||
# - push to master that touches Dockerfile.base-grpc-builder or
|
||||
# this workflow itself - keeps bases in sync with their inputs.
|
||||
#
|
||||
# Bootstrap (one-time after this PR merges):
|
||||
# gh workflow run base-images.yml --ref master
|
||||
# Wait ~30 min for all 9 matrix variants to push to
|
||||
# quay.io/go-skynet/ci-cache:base-grpc-* before merging PR 2.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 5 * * 6'
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'backend/Dockerfile.base-grpc-builder'
|
||||
- '.github/workflows/base-images.yml'
|
||||
# The install logic and apt-mirror helper are bind-mounted into
|
||||
# Dockerfile.base-grpc-builder at build time — changes to either
|
||||
# affect the produced base images and must trigger a rebuild.
|
||||
- '.docker/install-base-deps.sh'
|
||||
- '.docker/apt-mirror.sh'
|
||||
|
||||
concurrency:
|
||||
group: ci-base-images-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
build:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- tag: 'base-grpc-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: ''
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: ''
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-cuda-12-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'cublas'
|
||||
cuda-major-version: '12'
|
||||
cuda-minor-version: '8'
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-cuda-13-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:22.04'
|
||||
build-type: 'cublas'
|
||||
cuda-major-version: '13'
|
||||
cuda-minor-version: '0'
|
||||
ubuntu-version: '2204'
|
||||
- tag: 'base-grpc-cuda-13-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'cublas'
|
||||
cuda-major-version: '13'
|
||||
cuda-minor-version: '0'
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-rocm-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'rocm/dev-ubuntu-24.04:7.2.1'
|
||||
build-type: 'hipblas'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-vulkan-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'vulkan'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-vulkan-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'vulkan'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-intel-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04'
|
||||
build-type: 'sycl'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
# Legacy JetPack r36.4.0 base for older Jetson devices (CUDA 12).
|
||||
# Distinct from base-grpc-cuda-13-arm64 (Ubuntu 24.04 + CUDA 13 sbsa)
|
||||
# which targets newer Jetsons. Some matrix entries
|
||||
# (-nvidia-l4t-arm64-llama-cpp / -turboquant) still build against
|
||||
# the JetPack image, so we need a matching base.
|
||||
- tag: 'base-grpc-l4t-cuda-12-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'nvcr.io/nvidia/l4t-jetpack:r36.4.0'
|
||||
build-type: 'l4t'
|
||||
cuda-major-version: '12'
|
||||
cuda-minor-version: '0'
|
||||
ubuntu-version: '2204'
|
||||
# JetPack r36.4.0 already ships CUDA preinstalled at /usr/local/cuda;
|
||||
# apt-installing cuda-nvcc-12-0 from the public repos fails because
|
||||
# those packages aren't published for the JetPack apt feed. Match
|
||||
# the original l4t matrix entry which set skip-drivers: 'true'.
|
||||
skip-drivers: 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
submodules: false
|
||||
- name: Free disk space
|
||||
uses: ./.github/actions/free-disk-space
|
||||
- name: Set up build disk
|
||||
uses: ./.github/actions/setup-build-disk
|
||||
- uses: docker/setup-qemu-action@master
|
||||
with:
|
||||
platforms: all
|
||||
- uses: docker/setup-buildx-action@master
|
||||
- uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
- uses: docker/build-push-action@v7
|
||||
with:
|
||||
context: .
|
||||
file: ./backend/Dockerfile.base-grpc-builder
|
||||
build-args: |
|
||||
BASE_IMAGE=${{ matrix.base-image }}
|
||||
BUILD_TYPE=${{ matrix.build-type }}
|
||||
CUDA_MAJOR_VERSION=${{ matrix.cuda-major-version }}
|
||||
CUDA_MINOR_VERSION=${{ matrix.cuda-minor-version }}
|
||||
UBUNTU_VERSION=${{ matrix.ubuntu-version }}
|
||||
SKIP_DRIVERS=${{ matrix.skip-drivers || 'false' }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }},mode=max,ignore-error=true
|
||||
provenance: false
|
||||
tags: quay.io/go-skynet/ci-cache:${{ matrix.tag }}
|
||||
push: true
|
||||
77
.github/workflows/build-test.yaml
vendored
77
.github/workflows/build-test.yaml
vendored
@@ -1,77 +0,0 @@
|
||||
name: Build test
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
pull_request:
|
||||
|
||||
# Supersede an in-flight run when a PR gets a new push. Keyed on the PR number
|
||||
# so every push to the same PR shares a group; on a master push the key falls
|
||||
# back to github.sha (unique per commit) and cancel-in-progress is false, so
|
||||
# master runs never cancel each other -- each commit is built on its own.
|
||||
concurrency:
|
||||
group: ci-build-test-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
build-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: Set up Go
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: 1.25
|
||||
- name: Run GoReleaser
|
||||
run: |
|
||||
make dev-dist
|
||||
launcher-build-darwin:
|
||||
runs-on: macos-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: Set up Go
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: 1.25
|
||||
- name: Build launcher for macOS ARM64
|
||||
run: |
|
||||
make build-launcher-darwin
|
||||
ls -liah dist
|
||||
- name: Upload macOS launcher artifacts
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: launcher-macos
|
||||
path: dist/
|
||||
retention-days: 30
|
||||
|
||||
launcher-build-linux:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: Configure apt mirror on runner
|
||||
uses: ./.github/actions/configure-apt-mirror
|
||||
- name: Set up Go
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: 1.25
|
||||
- name: Build launcher for Linux
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install golang gcc libgl1-mesa-dev xorg-dev libxkbcommon-dev
|
||||
make build-launcher-linux
|
||||
- name: Upload Linux launcher artifacts
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: launcher-linux
|
||||
path: local-ai-launcher-linux.tar.xz
|
||||
retention-days: 30
|
||||
48
.github/workflows/bump-inference-defaults.yml
vendored
48
.github/workflows/bump-inference-defaults.yml
vendored
@@ -1,48 +0,0 @@
|
||||
name: Bump inference defaults
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Run daily at 06:00 UTC
|
||||
- cron: '0 6 * * *'
|
||||
workflow_dispatch: # Allow manual trigger
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
bump:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: go.mod
|
||||
|
||||
- name: Re-fetch inference defaults
|
||||
run: make generate-force
|
||||
|
||||
- name: Check for changes
|
||||
id: diff
|
||||
run: |
|
||||
if git diff --quiet core/config/inference_defaults.json; then
|
||||
echo "changed=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "changed=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Create Pull Request
|
||||
if: steps.diff.outputs.changed == 'true'
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
commit-message: "chore: bump inference defaults from unsloth"
|
||||
title: "chore: bump inference defaults from unsloth"
|
||||
body: |
|
||||
Auto-generated update of `core/config/inference_defaults.json` from
|
||||
[unsloth's inference_defaults.json](https://github.com/unslothai/unsloth/blob/main/studio/backend/assets/configs/inference_defaults.json).
|
||||
|
||||
This PR was created automatically by the `bump-inference-defaults` workflow.
|
||||
branch: chore/bump-inference-defaults
|
||||
delete-branch: true
|
||||
labels: automated
|
||||
195
.github/workflows/bump_deps.yaml
vendored
195
.github/workflows/bump_deps.yaml
vendored
@@ -1,136 +1,39 @@
|
||||
name: Bump Backend dependencies
|
||||
name: Bump dependencies
|
||||
on:
|
||||
schedule:
|
||||
- cron: 0 20 * * *
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
bump-backends:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
bump:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- repository: "ggml-org/llama.cpp"
|
||||
variable: "LLAMA_VERSION"
|
||||
variable: "CPPLLAMA_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/cpp/llama-cpp/Makefile"
|
||||
- repository: "ikawrakow/ik_llama.cpp"
|
||||
variable: "IK_LLAMA_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/cpp/ik-llama-cpp/Makefile"
|
||||
- repository: "TheTom/llama-cpp-turboquant"
|
||||
variable: "TURBOQUANT_VERSION"
|
||||
branch: "feature/turboquant-kv-cache"
|
||||
file: "backend/cpp/turboquant/Makefile"
|
||||
- repository: "PrismML-Eng/llama.cpp"
|
||||
variable: "BONSAI_VERSION"
|
||||
branch: "prism"
|
||||
file: "backend/cpp/bonsai/Makefile"
|
||||
- repository: "antirez/ds4"
|
||||
variable: "DS4_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/cpp/ds4/Makefile"
|
||||
- repository: "meituan-longcat/LongCat-Video"
|
||||
variable: "LONGCAT_VIDEO_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/python/longcat-video/Makefile"
|
||||
- repository: "localai-org/privacy-filter.cpp"
|
||||
variable: "PRIVACY_FILTER_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/cpp/privacy-filter/Makefile"
|
||||
- repository: "ggml-org/whisper.cpp"
|
||||
- repository: "ggerganov/whisper.cpp"
|
||||
variable: "WHISPER_CPP_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/whisper/Makefile"
|
||||
- repository: "CrispStrobe/CrispASR"
|
||||
variable: "CRISPASR_VERSION"
|
||||
- repository: "PABannier/bark.cpp"
|
||||
variable: "BARKCPP_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/crispasr/Makefile"
|
||||
- repository: "mudler/parakeet.cpp"
|
||||
variable: "PARAKEET_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/parakeet-cpp/Makefile"
|
||||
- repository: "mudler/vllm.cpp"
|
||||
variable: "VLLM_CPP_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/vllm-cpp/Makefile"
|
||||
- repository: "localai-org/moss-transcribe.cpp"
|
||||
variable: "MOSS_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/moss-transcribe-cpp/Makefile"
|
||||
- repository: "localai-org/ced.cpp"
|
||||
variable: "CED_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/ced/Makefile"
|
||||
- repository: "localai-org/voice-detect.cpp"
|
||||
variable: "VOICEDETECT_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/voice-detect/Makefile"
|
||||
- repository: "mudler/face-detect.cpp"
|
||||
variable: "FACEDETECT_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/face-detect/Makefile"
|
||||
- repository: "mudler/depth-anything.cpp"
|
||||
variable: "DEPTHANYTHING_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/depth-anything-cpp/Makefile"
|
||||
- repository: "leejet/stable-diffusion.cpp"
|
||||
variable: "STABLEDIFFUSION_GGML_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/stablediffusion-ggml/Makefile"
|
||||
- repository: "mudler/go-stable-diffusion"
|
||||
variable: "STABLEDIFFUSION_VERSION"
|
||||
branch: "master"
|
||||
- repository: "mudler/go-piper"
|
||||
variable: "PIPER_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/piper/Makefile"
|
||||
- repository: "antirez/voxtral.c"
|
||||
variable: "VOXTRAL_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/voxtral/Makefile"
|
||||
- repository: "ace-step/acestep.cpp"
|
||||
variable: "ACESTEP_CPP_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/acestep-cpp/Makefile"
|
||||
- repository: "PABannier/sam3.cpp"
|
||||
variable: "SAM3_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/sam3-cpp/Makefile"
|
||||
- repository: "localai-org/rf-detr.cpp"
|
||||
variable: "RFDETR_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/rfdetr-cpp/Makefile"
|
||||
- repository: "mudler/locate-anything.cpp"
|
||||
variable: "LOCATEANYTHING_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/locate-anything-cpp/Makefile"
|
||||
- repository: "ServeurpersoCom/qwentts.cpp"
|
||||
variable: "QWEN3TTS_CPP_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/qwen3-tts-cpp/Makefile"
|
||||
- repository: "ServeurpersoCom/omnivoice.cpp"
|
||||
variable: "OMNIVOICE_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/omnivoice-cpp/Makefile"
|
||||
- repository: "localai-org/vibevoice.cpp"
|
||||
variable: "VIBEVOICE_CPP_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/vibevoice-cpp/Makefile"
|
||||
- repository: "mudler/magpie-tts.cpp"
|
||||
variable: "MAGPIETTS_CPP_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/magpie-tts-cpp/Makefile"
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/checkout@v4
|
||||
- name: Bump dependencies 🔧
|
||||
id: bump
|
||||
env:
|
||||
# This job fans out to ~25 parallel matrix entries, all querying
|
||||
# api.github.com from runner IPs that share the 60/hour anonymous
|
||||
# rate limit. Authenticating raises it to 1000/hour, which is what
|
||||
# kept a random handful of these red every night.
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_deps.sh ${{ matrix.repository }} ${{ matrix.branch }} ${{ matrix.variable }} ${{ matrix.file }}
|
||||
bash .github/bump_deps.sh ${{ matrix.repository }} ${{ matrix.branch }} ${{ matrix.variable }}
|
||||
{
|
||||
echo 'message<<EOF'
|
||||
cat "${{ matrix.variable }}_message.txt"
|
||||
@@ -144,7 +47,7 @@ jobs:
|
||||
rm -rfv ${{ matrix.variable }}_message.txt
|
||||
rm -rfv ${{ matrix.variable }}_commit.txt
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
uses: peter-evans/create-pull-request@v7
|
||||
with:
|
||||
token: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
push-to-fork: ci-forks/LocalAI
|
||||
@@ -154,77 +57,5 @@ jobs:
|
||||
body: ${{ steps.bump.outputs.message }}
|
||||
signoff: true
|
||||
|
||||
bump-vllm-wheel:
|
||||
# vLLM's cu130 wheel comes from a per-tag index URL (no /latest/ alias),
|
||||
# so the cublas13 requirements file pins both a URL segment and a version
|
||||
# constraint. bump_deps.sh handles git-sha-in-Makefile only — this job
|
||||
# rewrites both values atomically when a new vLLM stable tag ships.
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bump vLLM cu130 wheel pin 🔧
|
||||
id: bump
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_vllm_wheel.sh vllm-project/vllm backend/python/vllm/requirements-cublas13-after.txt VLLM_VERSION
|
||||
{
|
||||
echo 'message<<EOF'
|
||||
cat "VLLM_VERSION_message.txt"
|
||||
echo EOF
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
{
|
||||
echo 'commit<<EOF'
|
||||
cat "VLLM_VERSION_commit.txt"
|
||||
echo EOF
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
rm -rfv VLLM_VERSION_message.txt VLLM_VERSION_commit.txt
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
token: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
push-to-fork: ci-forks/LocalAI
|
||||
commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel'
|
||||
title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`'
|
||||
branch: "update/VLLM_VERSION"
|
||||
body: ${{ steps.bump.outputs.message }}
|
||||
signoff: true
|
||||
|
||||
bump-vllm-metal:
|
||||
# The darwin (Apple Silicon) vLLM build installs vllm-metal, which is locked
|
||||
# to a specific vLLM source release. install.sh pins both VLLM_METAL_VERSION
|
||||
# (the wheel release) and VLLM_VERSION (the vLLM it builds against); this job
|
||||
# tracks vllm-project/vllm-metal and rewrites both atomically. Separate from
|
||||
# bump-vllm-wheel because darwin follows vllm-metal, not vllm/vllm latest.
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bump vllm-metal pin 🔧
|
||||
id: bump
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_vllm_metal.sh vllm-project/vllm-metal backend/python/vllm/install.sh VLLM_METAL_VERSION
|
||||
{
|
||||
echo 'message<<EOF'
|
||||
cat "VLLM_METAL_VERSION_message.txt"
|
||||
echo EOF
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
{
|
||||
echo 'commit<<EOF'
|
||||
cat "VLLM_METAL_VERSION_commit.txt"
|
||||
echo EOF
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
rm -rfv VLLM_METAL_VERSION_message.txt VLLM_METAL_VERSION_commit.txt
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
token: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
push-to-fork: ci-forks/LocalAI
|
||||
commit-message: ':arrow_up: Update vllm-project/vllm-metal (darwin)'
|
||||
title: 'chore: :arrow_up: Update vllm-metal (darwin) to `${{ steps.bump.outputs.commit }}`'
|
||||
branch: "update/VLLM_METAL_VERSION"
|
||||
body: ${{ steps.bump.outputs.message }}
|
||||
signoff: true
|
||||
|
||||
|
||||
13
.github/workflows/bump_docs.yaml
vendored
13
.github/workflows/bump_docs.yaml
vendored
@@ -1,11 +1,10 @@
|
||||
name: Bump Documentation
|
||||
name: Bump dependencies
|
||||
on:
|
||||
schedule:
|
||||
- cron: 0 20 * * *
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
bump-docs:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
bump:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -13,16 +12,12 @@ jobs:
|
||||
- repository: "mudler/LocalAI"
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/checkout@v4
|
||||
- name: Bump dependencies 🔧
|
||||
env:
|
||||
# Authenticated API calls get 1000 req/hour instead of the 60/hour
|
||||
# anonymous cap that is shared across every job on the runner's IP.
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
bash .github/bump_docs.sh ${{ matrix.repository }}
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
uses: peter-evans/create-pull-request@v7
|
||||
with:
|
||||
token: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
push-to-fork: ci-forks/LocalAI
|
||||
|
||||
18
.github/workflows/checksum_checker.yaml
vendored
18
.github/workflows/checksum_checker.yaml
vendored
@@ -5,16 +5,22 @@ on:
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
checksum_check:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: arc-runner-set
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Configure apt mirror on runner
|
||||
uses: ./.github/actions/configure-apt-mirror
|
||||
- name: Force Install GIT latest
|
||||
run: |
|
||||
sudo apt-get update \
|
||||
&& sudo apt-get install -y software-properties-common \
|
||||
&& sudo apt-get update \
|
||||
&& sudo add-apt-repository -y ppa:git-core/ppa \
|
||||
&& sudo apt-get update \
|
||||
&& sudo apt-get install -y git
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y pip wget
|
||||
sudo pip install --upgrade pip
|
||||
pip install huggingface_hub
|
||||
- name: 'Setup yq'
|
||||
uses: dcarbone/install-yq-action@v1.3.1
|
||||
@@ -30,7 +36,7 @@ jobs:
|
||||
sudo chmod 777 /hf_cache
|
||||
bash .github/checksum_checker.sh gallery/index.yaml
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
uses: peter-evans/create-pull-request@v7
|
||||
with:
|
||||
token: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
push-to-fork: ci-forks/LocalAI
|
||||
|
||||
39
.github/workflows/ci-tools-tests.yaml
vendored
39
.github/workflows/ci-tools-tests.yaml
vendored
@@ -1,39 +0,0 @@
|
||||
---
|
||||
# The packages under .github/ci/ are invisible to `go list ./...`, so neither
|
||||
# `make lint` nor the repository test run ever touches them. Their specs are
|
||||
# dead weight until a workflow names each package explicitly.
|
||||
name: 'CI tool tests'
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- '.github/ci/**'
|
||||
- '.github/workflows/ci-tools-tests.yaml'
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
paths:
|
||||
- '.github/ci/**'
|
||||
jobs:
|
||||
ci-tools:
|
||||
name: 'Test the .github/ci generators'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: go.mod
|
||||
cache: false
|
||||
|
||||
# The discovery heuristics are the risky part of these tools. A regression
|
||||
# produces confident, wrong gallery entries, which is worse than no tool.
|
||||
- name: 'Test the APEX entry generator'
|
||||
run: go test ./.github/ci/apexentries/
|
||||
|
||||
- name: 'Test the variant proposer'
|
||||
run: go test ./.github/ci/variantproposals/
|
||||
|
||||
# Shared by both generators above. Its behaviour is exercised through their
|
||||
# specs; this step exists so a break in the shared package fails under its
|
||||
# own name rather than as a puzzling failure in whichever caller ran first.
|
||||
- name: 'Test the shared gallery editor'
|
||||
run: go test ./.github/ci/galleryedit/
|
||||
@@ -9,18 +9,18 @@ permissions:
|
||||
|
||||
jobs:
|
||||
dependabot:
|
||||
if: github.repository == 'mudler/LocalAI' && github.actor == 'dependabot[bot]'
|
||||
runs-on: ubuntu-latest
|
||||
if: ${{ github.actor == 'dependabot[bot]' }}
|
||||
steps:
|
||||
- name: Dependabot metadata
|
||||
id: metadata
|
||||
uses: dependabot/fetch-metadata@v2.5.0
|
||||
uses: dependabot/fetch-metadata@v2.3.0
|
||||
with:
|
||||
github-token: "${{ secrets.GITHUB_TOKEN }}"
|
||||
skip-commit-verification: true
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Approve a PR if not already approved
|
||||
run: |
|
||||
11
.github/workflows/deploy-explorer.yaml
vendored
11
.github/workflows/deploy-explorer.yaml
vendored
@@ -12,11 +12,10 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
build-linux:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Clone
|
||||
uses: actions/checkout@v7
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
- uses: actions/setup-go@v5
|
||||
@@ -32,9 +31,9 @@ jobs:
|
||||
make protogen-go
|
||||
- name: Build api
|
||||
run: |
|
||||
CGO_ENABLED=0 make build
|
||||
CGO_ENABLED=0 make build-api
|
||||
- name: rm
|
||||
uses: appleboy/ssh-action@v1.2.5
|
||||
uses: appleboy/ssh-action@v1.2.1
|
||||
with:
|
||||
host: ${{ secrets.EXPLORER_SSH_HOST }}
|
||||
username: ${{ secrets.EXPLORER_SSH_USERNAME }}
|
||||
@@ -43,7 +42,7 @@ jobs:
|
||||
script: |
|
||||
sudo rm -rf local-ai/ || true
|
||||
- name: copy file via ssh
|
||||
uses: appleboy/scp-action@v1.0.0
|
||||
uses: appleboy/scp-action@v0.1.7
|
||||
with:
|
||||
host: ${{ secrets.EXPLORER_SSH_HOST }}
|
||||
username: ${{ secrets.EXPLORER_SSH_USERNAME }}
|
||||
@@ -54,7 +53,7 @@ jobs:
|
||||
rm: true
|
||||
target: ./local-ai
|
||||
- name: restarting
|
||||
uses: appleboy/ssh-action@v1.2.5
|
||||
uses: appleboy/ssh-action@v1.2.1
|
||||
with:
|
||||
host: ${{ secrets.EXPLORER_SSH_HOST }}
|
||||
username: ${{ secrets.EXPLORER_SSH_USERNAME }}
|
||||
|
||||
214
.github/workflows/gallery-agent.yaml
vendored
214
.github/workflows/gallery-agent.yaml
vendored
@@ -1,214 +0,0 @@
|
||||
name: Gallery Agent
|
||||
on:
|
||||
|
||||
schedule:
|
||||
- cron: '0 */12 * * *' # Run every 4 hours
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
search_term:
|
||||
description: 'Search term for models'
|
||||
required: false
|
||||
default: 'GGUF'
|
||||
type: string
|
||||
limit:
|
||||
description: 'Maximum number of models to process'
|
||||
required: false
|
||||
default: '15'
|
||||
type: string
|
||||
quantization:
|
||||
description: 'Preferred quantization format'
|
||||
required: false
|
||||
default: 'Q4_K_M'
|
||||
type: string
|
||||
max_models:
|
||||
description: 'Maximum number of models to add to the gallery'
|
||||
required: false
|
||||
default: '1'
|
||||
type: string
|
||||
jobs:
|
||||
gallery-agent:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Set up Go
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.21'
|
||||
- name: Proto Dependencies
|
||||
run: |
|
||||
# Install protoc
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
|
||||
rm protoc.zip
|
||||
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
|
||||
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
|
||||
PATH="$PATH:$HOME/go/bin" make protogen-go
|
||||
- name: Process gallery-agent PR commands
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
SEARCH: 'gallery agent in:title'
|
||||
run: |
|
||||
# Walk gallery-agent PRs and act on maintainer comments:
|
||||
# /gallery-agent blacklist → label `gallery-agent/blacklisted` + close (never repropose)
|
||||
# /gallery-agent recreate → close without label (next run may repropose)
|
||||
# Only comments from OWNER / MEMBER / COLLABORATOR are honored so
|
||||
# random users can't drive the bot.
|
||||
#
|
||||
# We scan both open PRs AND recently-closed PRs that don't already
|
||||
# carry the blacklist label. This covers the common flow where a
|
||||
# maintainer writes /gallery-agent blacklist and immediately clicks
|
||||
# Close — without this, the next scheduled run wouldn't see the
|
||||
# command (PR is already closed) and would repropose the model.
|
||||
gh label create gallery-agent/blacklisted \
|
||||
--repo "$REPO" --color ededed \
|
||||
--description "gallery-agent must not repropose this model" 2>/dev/null || true
|
||||
|
||||
prs_open=$(gh pr list --repo "$REPO" --state open --search "$SEARCH" \
|
||||
--json number --jq '.[].number')
|
||||
# Closed PRs from the last 14 days that don't yet have the blacklist label.
|
||||
# Bounded window keeps the scan cheap while covering late-applied commands.
|
||||
since=$(date -u -d '14 days ago' +%Y-%m-%d)
|
||||
prs_closed=$(gh pr list --repo "$REPO" --state closed \
|
||||
--search "$SEARCH closed:>=$since -label:gallery-agent/blacklisted" \
|
||||
--json number --jq '.[].number')
|
||||
prs=$(printf '%s\n%s\n' "$prs_open" "$prs_closed" | sort -u | sed '/^$/d')
|
||||
for pr in $prs; do
|
||||
state=$(gh pr view "$pr" --repo "$REPO" --json state --jq '.state')
|
||||
cmds=$(gh pr view "$pr" --repo "$REPO" --json comments \
|
||||
--jq '.comments[] | select(.authorAssociation=="OWNER" or .authorAssociation=="MEMBER" or .authorAssociation=="COLLABORATOR") | .body')
|
||||
if echo "$cmds" | grep -qE '(^|[[:space:]])/gallery-agent[[:space:]]+blacklist([[:space:]]|$)'; then
|
||||
echo "PR #$pr: blacklist command found (state=$state)"
|
||||
gh pr edit "$pr" --repo "$REPO" --add-label gallery-agent/blacklisted || true
|
||||
if [ "$state" = "OPEN" ]; then
|
||||
gh pr close "$pr" --repo "$REPO" --comment "Blacklisted via \`/gallery-agent blacklist\`. This model will not be reproposed." || true
|
||||
fi
|
||||
elif [ "$state" = "OPEN" ] && echo "$cmds" | grep -qE '(^|[[:space:]])/gallery-agent[[:space:]]+recreate([[:space:]]|$)'; then
|
||||
echo "PR #$pr: recreate command found"
|
||||
gh pr close "$pr" --repo "$REPO" --comment "Closed via \`/gallery-agent recreate\`. The next scheduled run will propose this model again." || true
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Collect skip URLs for the gallery agent
|
||||
id: open_prs
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
SEARCH: 'gallery agent in:title'
|
||||
run: |
|
||||
# Skip set =
|
||||
# URLs from any open gallery-agent PR (avoid duplicate PRs for the same model while one is pending)
|
||||
# + URLs from closed PRs carrying the `gallery-agent/blacklisted` label (hard blacklist)
|
||||
# Plain-closed PRs without the label are ignored — closing a PR is
|
||||
# not by itself a "never propose again" signal; maintainers must
|
||||
# opt in via the /gallery-agent blacklist comment command.
|
||||
urls_open=$(gh pr list --repo "$REPO" --state open --search "$SEARCH" \
|
||||
--json body --jq '[.[].body] | join("\n")' \
|
||||
| grep -oE 'https://huggingface\.co/[^ )]+' || true)
|
||||
urls_blacklist=$(gh pr list --repo "$REPO" --state closed --search "$SEARCH" \
|
||||
--label gallery-agent/blacklisted \
|
||||
--json body --jq '[.[].body] | join("\n")' \
|
||||
| grep -oE 'https://huggingface\.co/[^ )]+' || true)
|
||||
urls=$(printf '%s\n%s\n' "$urls_open" "$urls_blacklist" | sort -u | sed '/^$/d')
|
||||
echo "Skip URLs:"
|
||||
echo "$urls"
|
||||
{
|
||||
echo "urls<<EOF"
|
||||
echo "$urls"
|
||||
echo "EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Run gallery agent
|
||||
env:
|
||||
SEARCH_TERM: ${{ github.event.inputs.search_term || 'GGUF' }}
|
||||
LIMIT: ${{ github.event.inputs.limit || '15' }}
|
||||
QUANTIZATION: ${{ github.event.inputs.quantization || 'Q4_K_M' }}
|
||||
MAX_MODELS: ${{ github.event.inputs.max_models || '1' }}
|
||||
EXTRA_SKIP_URLS: ${{ steps.open_prs.outputs.urls }}
|
||||
run: |
|
||||
export GALLERY_INDEX_PATH=$PWD/gallery/index.yaml
|
||||
go run ./.github/gallery-agent
|
||||
|
||||
- name: Check for changes
|
||||
id: check_changes
|
||||
run: |
|
||||
if git diff --quiet gallery/index.yaml; then
|
||||
echo "changes=false" >> $GITHUB_OUTPUT
|
||||
echo "No changes detected in gallery/index.yaml"
|
||||
else
|
||||
echo "changes=true" >> $GITHUB_OUTPUT
|
||||
echo "Changes detected in gallery/index.yaml"
|
||||
git diff gallery/index.yaml
|
||||
fi
|
||||
|
||||
- name: Read gallery agent summary
|
||||
id: read_summary
|
||||
if: steps.check_changes.outputs.changes == 'true'
|
||||
run: |
|
||||
if [ -f "./gallery-agent-summary.json" ]; then
|
||||
echo "summary_exists=true" >> $GITHUB_OUTPUT
|
||||
# Extract summary data using jq
|
||||
echo "search_term=$(jq -r '.search_term' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
|
||||
echo "total_found=$(jq -r '.total_found' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
|
||||
echo "models_added=$(jq -r '.models_added' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
|
||||
echo "quantization=$(jq -r '.quantization' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
|
||||
echo "processing_time=$(jq -r '.processing_time' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
|
||||
|
||||
# Create a formatted list of added models with URLs
|
||||
added_models=$(jq -r 'range(0; .added_model_ids | length) as $i | "- [\(.added_model_ids[$i])](\(.added_model_urls[$i]))"' ./gallery-agent-summary.json | tr '\n' '\n')
|
||||
echo "added_models<<EOF" >> $GITHUB_OUTPUT
|
||||
echo "$added_models" >> $GITHUB_OUTPUT
|
||||
echo "EOF" >> $GITHUB_OUTPUT
|
||||
rm -f ./gallery-agent-summary.json
|
||||
else
|
||||
echo "summary_exists=false" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Create Pull Request
|
||||
if: steps.check_changes.outputs.changes == 'true'
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
token: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
push-to-fork: ci-forks/LocalAI
|
||||
commit-message: 'chore(model gallery): :robot: add new models via gallery agent'
|
||||
title: 'chore(model gallery): :robot: add ${{ steps.read_summary.outputs.models_added || 0 }} new models via gallery agent'
|
||||
# Branch has to be unique so PRs are not overriding each other
|
||||
branch-suffix: timestamp
|
||||
body: |
|
||||
This PR was automatically created by the gallery agent workflow.
|
||||
|
||||
**Summary:**
|
||||
- **Search Term:** ${{ steps.read_summary.outputs.search_term || github.event.inputs.search_term || 'GGUF' }}
|
||||
- **Models Found:** ${{ steps.read_summary.outputs.total_found || 'N/A' }}
|
||||
- **Models Added:** ${{ steps.read_summary.outputs.models_added || '0' }}
|
||||
- **Quantization:** ${{ steps.read_summary.outputs.quantization || github.event.inputs.quantization || 'Q4_K_M' }}
|
||||
- **Processing Time:** ${{ steps.read_summary.outputs.processing_time || 'N/A' }}
|
||||
|
||||
**Added Models:**
|
||||
${{ steps.read_summary.outputs.added_models || '- No models added' }}
|
||||
|
||||
### Bot commands
|
||||
|
||||
Maintainers (owner / member / collaborator) can control this PR
|
||||
by leaving a comment with one of:
|
||||
|
||||
- `/gallery-agent recreate` — close this PR; the next scheduled
|
||||
run will propose this model again (useful if the entry needs
|
||||
to be regenerated with fresh metadata).
|
||||
- `/gallery-agent blacklist` — close this PR and permanently
|
||||
prevent the gallery agent from ever reproposing this model.
|
||||
|
||||
Plain "Close" (without a command) is treated as a no-op: the
|
||||
model may be reproposed by a future run.
|
||||
|
||||
**Workflow Details:**
|
||||
- Triggered by: `${{ github.event_name }}`
|
||||
- Run ID: `${{ github.run_id }}`
|
||||
- Commit: `${{ github.sha }}`
|
||||
signoff: true
|
||||
delete-branch: true
|
||||
54
.github/workflows/gallery_variant_proposals.yaml
vendored
54
.github/workflows/gallery_variant_proposals.yaml
vendored
@@ -1,54 +0,0 @@
|
||||
name: Propose gallery variant groupings
|
||||
on:
|
||||
schedule:
|
||||
- cron: 0 4 * * 1
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
variant_proposals:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: go.mod
|
||||
cache: false
|
||||
|
||||
# The heuristics are the risky part of this job. A regression in them
|
||||
# produces confident, wrong proposals, which is worse than no job at all.
|
||||
- name: Test the proposer
|
||||
run: go test ./.github/ci/variantproposals/
|
||||
|
||||
- name: Propose groupings 🔧
|
||||
id: propose
|
||||
run: |
|
||||
rm -f /tmp/variant-proposals-body.md
|
||||
go run ./.github/ci/variantproposals \
|
||||
-index gallery/index.yaml \
|
||||
-ledger gallery/variant-exclusions.yaml \
|
||||
-body-out /tmp/variant-proposals-body.md \
|
||||
-apply
|
||||
if [ -s /tmp/variant-proposals-body.md ]; then
|
||||
echo "have_proposals=true" >> "$GITHUB_OUTPUT"
|
||||
{
|
||||
echo 'body<<VARIANT_PROPOSAL_BODY_EOF'
|
||||
cat /tmp/variant-proposals-body.md
|
||||
echo VARIANT_PROPOSAL_BODY_EOF
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "have_proposals=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# No body file means the proposer found nothing. Opening an empty pull
|
||||
# request every run is how a proposal job gets muted by its reviewers.
|
||||
- name: Create Pull Request
|
||||
if: steps.propose.outputs.have_proposals == 'true'
|
||||
uses: peter-evans/create-pull-request@v8
|
||||
with:
|
||||
token: ${{ secrets.UPDATE_BOT_TOKEN }}
|
||||
push-to-fork: ci-forks/LocalAI
|
||||
commit-message: 'chore(model-gallery): propose variant groupings'
|
||||
title: 'chore(model-gallery): propose variant groupings for review'
|
||||
branch: "propose/variant-groupings"
|
||||
body: ${{ steps.propose.outputs.body }}
|
||||
signoff: true
|
||||
95
.github/workflows/generate_grpc_cache.yaml
vendored
Normal file
95
.github/workflows/generate_grpc_cache.yaml
vendored
Normal file
@@ -0,0 +1,95 @@
|
||||
name: 'generate and publish GRPC docker caches'
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
schedule:
|
||||
# daily at midnight
|
||||
- cron: '0 0 * * *'
|
||||
|
||||
concurrency:
|
||||
group: grpc-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
generate_caches:
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- grpc-base-image: ubuntu:22.04
|
||||
runs-on: 'arc-runner-set'
|
||||
platforms: 'linux/amd64,linux/arm64'
|
||||
runs-on: ${{matrix.runs-on}}
|
||||
steps:
|
||||
- name: Release space from worker
|
||||
if: matrix.runs-on == 'ubuntu-latest'
|
||||
run: |
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
df -h
|
||||
echo
|
||||
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
|
||||
sudo apt-get remove --auto-remove android-sdk-platform-tools || true
|
||||
sudo apt-get purge --auto-remove android-sdk-platform-tools || true
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo apt-get remove -y '^mono-.*' || true
|
||||
sudo apt-get remove -y '^ghc-.*' || true
|
||||
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
|
||||
sudo apt-get remove -y 'php.*' || true
|
||||
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
|
||||
sudo apt-get remove -y '^google-.*' || true
|
||||
sudo apt-get remove -y azure-cli || true
|
||||
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
|
||||
sudo apt-get remove -y '^gfortran-.*' || true
|
||||
sudo apt-get remove -y microsoft-edge-stable || true
|
||||
sudo apt-get remove -y firefox || true
|
||||
sudo apt-get remove -y powershell || true
|
||||
sudo apt-get remove -y r-base-core || true
|
||||
sudo apt-get autoremove -y
|
||||
sudo apt-get clean
|
||||
echo
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
sudo rm -rfv build || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf "/usr/local/share/boost" || true
|
||||
sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
|
||||
df -h
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@master
|
||||
with:
|
||||
platforms: all
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
id: buildx
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Cache GRPC
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
# The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
|
||||
# This means that even the MAKEFLAGS have to be an EXACT match.
|
||||
# If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
|
||||
build-args: |
|
||||
GRPC_BASE_IMAGE=${{ matrix.grpc-base-image }}
|
||||
GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
|
||||
GRPC_VERSION=v1.65.0
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
cache-to: type=gha,ignore-error=true
|
||||
cache-from: type=gha
|
||||
target: grpc
|
||||
platforms: ${{ matrix.platforms }}
|
||||
push: false
|
||||
19
.github/workflows/generate_intel_image.yaml
vendored
19
.github/workflows/generate_intel_image.yaml
vendored
@@ -7,17 +7,16 @@ on:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: intel-cache-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: intel-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
generate_caches:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- base-image: intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04
|
||||
runs-on: 'arc-runner-set'
|
||||
- base-image: intel/oneapi-basekit:2025.0.0-0-devel-ubuntu22.04
|
||||
runs-on: 'ubuntu-latest'
|
||||
platforms: 'linux/amd64'
|
||||
runs-on: ${{matrix.runs-on}}
|
||||
steps:
|
||||
@@ -27,14 +26,14 @@ jobs:
|
||||
platforms: all
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
|
||||
- name: Login to quay
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
@@ -44,17 +43,17 @@ jobs:
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Cache Intel images
|
||||
uses: docker/build-push-action@v7
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
build-args: |
|
||||
BASE_IMAGE=${{ matrix.base-image }}
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
tags: quay.io/go-skynet/intel-oneapi-base:24.04
|
||||
tags: quay.io/go-skynet/intel-oneapi-base:latest
|
||||
push: true
|
||||
target: intel
|
||||
platforms: ${{ matrix.platforms }}
|
||||
|
||||
75
.github/workflows/gh-pages.yml
vendored
75
.github/workflows/gh-pages.yml
vendored
@@ -1,75 +0,0 @@
|
||||
name: Deploy docs to GitHub Pages
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
paths:
|
||||
- 'docs/**'
|
||||
- 'gallery/**'
|
||||
- 'images/**'
|
||||
- '.github/ci/modelslist.go'
|
||||
- '.github/workflows/gh-pages.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
concurrency:
|
||||
group: pages
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
HUGO_VERSION: "0.146.3"
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0 # needed for enableGitInfo
|
||||
submodules: true
|
||||
|
||||
- name: Setup Go
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.22'
|
||||
cache: false
|
||||
|
||||
- name: Setup Hugo
|
||||
uses: peaceiris/actions-hugo@v3
|
||||
with:
|
||||
hugo-version: ${{ env.HUGO_VERSION }}
|
||||
extended: true
|
||||
|
||||
- name: Setup Pages
|
||||
id: pages
|
||||
uses: actions/configure-pages@v6
|
||||
|
||||
- name: Generate gallery
|
||||
run: go run ./.github/ci/modelslist.go ./gallery/index.yaml > docs/static/gallery.html
|
||||
|
||||
- name: Build site
|
||||
working-directory: docs
|
||||
run: |
|
||||
mkdir -p layouts/_default
|
||||
hugo --minify --baseURL "${{ steps.pages.outputs.base_url }}/"
|
||||
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v5
|
||||
with:
|
||||
path: docs/public
|
||||
|
||||
deploy:
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
steps:
|
||||
- name: Deploy to GitHub Pages
|
||||
id: deployment
|
||||
uses: actions/deploy-pages@v5
|
||||
241
.github/workflows/image-pr.yml
vendored
241
.github/workflows/image-pr.yml
vendored
@@ -1,103 +1,140 @@
|
||||
---
|
||||
name: 'build container images tests'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
image-build:
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
# Pushing with all jobs in parallel
|
||||
# eats the bandwidth of all the nodes
|
||||
max-parallel: ${{ github.event_name != 'pull_request' && 4 || 8 }}
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "8"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-gpu-nvidia-cuda-12'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: "ubuntu:24.04"
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "13"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-gpu-nvidia-cuda-13'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: "ubuntu:24.04"
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'hipblas'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-hipblas'
|
||||
base-image: "rocm/dev-ubuntu-24.04:7.2.1"
|
||||
runs-on: 'ubuntu-latest'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'sycl'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
|
||||
tag-suffix: 'sycl'
|
||||
runs-on: 'ubuntu-latest'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/amd64'
|
||||
platform-tag: 'amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-vulkan-core'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: "ubuntu:24.04"
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/arm64'
|
||||
platform-tag: 'arm64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-vulkan-core'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: "ubuntu:24.04"
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "13"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/arm64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-nvidia-l4t-arm64-cuda-13'
|
||||
base-image: "ubuntu:24.04"
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
ubuntu-version: '2404'
|
||||
|
||||
name: 'build container images tests'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
extras-image-build:
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
ffmpeg: ${{ matrix.ffmpeg }}
|
||||
image-type: ${{ matrix.image-type }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
grpc-base-image: ${{ matrix.grpc-base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
# Pushing with all jobs in parallel
|
||||
# eats the bandwidth of all the nodes
|
||||
max-parallel: ${{ github.event_name != 'pull_request' && 4 || 8 }}
|
||||
matrix:
|
||||
include:
|
||||
# This is basically covered by the AIO test
|
||||
# - build-type: ''
|
||||
# platforms: 'linux/amd64'
|
||||
# tag-latest: 'false'
|
||||
# tag-suffix: '-ffmpeg'
|
||||
# ffmpeg: 'true'
|
||||
# image-type: 'extras'
|
||||
# runs-on: 'arc-runner-set'
|
||||
# base-image: "ubuntu:22.04"
|
||||
# makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-cublas-cuda12-ffmpeg'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
# - build-type: 'hipblas'
|
||||
# platforms: 'linux/amd64'
|
||||
# tag-latest: 'false'
|
||||
# tag-suffix: '-hipblas'
|
||||
# ffmpeg: 'false'
|
||||
# image-type: 'extras'
|
||||
# base-image: "rocm/dev-ubuntu-22.04:6.1"
|
||||
# grpc-base-image: "ubuntu:22.04"
|
||||
# runs-on: 'arc-runner-set'
|
||||
# makeflags: "--jobs=3 --output-sync=target"
|
||||
# - build-type: 'sycl_f16'
|
||||
# platforms: 'linux/amd64'
|
||||
# tag-latest: 'false'
|
||||
# base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
# grpc-base-image: "ubuntu:22.04"
|
||||
# tag-suffix: 'sycl-f16-ffmpeg'
|
||||
# ffmpeg: 'true'
|
||||
# image-type: 'extras'
|
||||
# runs-on: 'arc-runner-set'
|
||||
# makeflags: "--jobs=3 --output-sync=target"
|
||||
# core-image-build:
|
||||
# uses: ./.github/workflows/image_build.yml
|
||||
# with:
|
||||
# tag-latest: ${{ matrix.tag-latest }}
|
||||
# tag-suffix: ${{ matrix.tag-suffix }}
|
||||
# ffmpeg: ${{ matrix.ffmpeg }}
|
||||
# image-type: ${{ matrix.image-type }}
|
||||
# build-type: ${{ matrix.build-type }}
|
||||
# cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
# cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
# platforms: ${{ matrix.platforms }}
|
||||
# runs-on: ${{ matrix.runs-on }}
|
||||
# base-image: ${{ matrix.base-image }}
|
||||
# grpc-base-image: ${{ matrix.grpc-base-image }}
|
||||
# makeflags: ${{ matrix.makeflags }}
|
||||
# secrets:
|
||||
# dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
# dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
# quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
# quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
# strategy:
|
||||
# matrix:
|
||||
# include:
|
||||
# - build-type: ''
|
||||
# platforms: 'linux/amd64'
|
||||
# tag-latest: 'false'
|
||||
# tag-suffix: '-ffmpeg-core'
|
||||
# ffmpeg: 'true'
|
||||
# image-type: 'core'
|
||||
# runs-on: 'ubuntu-latest'
|
||||
# base-image: "ubuntu:22.04"
|
||||
# makeflags: "--jobs=4 --output-sync=target"
|
||||
# - build-type: 'sycl_f16'
|
||||
# platforms: 'linux/amd64'
|
||||
# tag-latest: 'false'
|
||||
# base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
# grpc-base-image: "ubuntu:22.04"
|
||||
# tag-suffix: 'sycl-f16-ffmpeg-core'
|
||||
# ffmpeg: 'true'
|
||||
# image-type: 'core'
|
||||
# runs-on: 'arc-runner-set'
|
||||
# makeflags: "--jobs=3 --output-sync=target"
|
||||
# - build-type: 'cublas'
|
||||
# cuda-major-version: "12"
|
||||
# cuda-minor-version: "0"
|
||||
# platforms: 'linux/amd64'
|
||||
# tag-latest: 'false'
|
||||
# tag-suffix: '-cublas-cuda12-ffmpeg-core'
|
||||
# ffmpeg: 'true'
|
||||
# image-type: 'core'
|
||||
# runs-on: 'ubuntu-latest'
|
||||
# base-image: "ubuntu:22.04"
|
||||
# makeflags: "--jobs=4 --output-sync=target"
|
||||
# - build-type: 'vulkan'
|
||||
# platforms: 'linux/amd64'
|
||||
# tag-latest: 'false'
|
||||
# tag-suffix: '-vulkan-ffmpeg-core'
|
||||
# ffmpeg: 'true'
|
||||
# image-type: 'core'
|
||||
# runs-on: 'ubuntu-latest'
|
||||
# base-image: "ubuntu:22.04"
|
||||
# makeflags: "--jobs=4 --output-sync=target"
|
||||
|
||||
709
.github/workflows/image.yml
vendored
709
.github/workflows/image.yml
vendored
@@ -1,315 +1,404 @@
|
||||
---
|
||||
name: 'build container images'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
tags:
|
||||
- '*'
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
hipblas-jobs:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
ubuntu-codename: ${{ matrix.ubuntu-codename }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- build-type: 'hipblas'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-hipblas'
|
||||
base-image: "rocm/dev-ubuntu-24.04:7.2.1"
|
||||
runs-on: 'ubuntu-latest'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
name: 'build container images'
|
||||
|
||||
core-image-build:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
ubuntu-codename: ${{ matrix.ubuntu-codename }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
#max-parallel: ${{ github.event_name != 'pull_request' && 2 || 4 }}
|
||||
matrix:
|
||||
include:
|
||||
- build-type: ''
|
||||
platforms: 'linux/amd64'
|
||||
platform-tag: 'amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
base-image: "ubuntu:24.04"
|
||||
runs-on: 'ubuntu-latest'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: ''
|
||||
platforms: 'linux/arm64'
|
||||
platform-tag: 'arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
base-image: "ubuntu:24.04"
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "8"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-nvidia-cuda-12'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: "ubuntu:24.04"
|
||||
skip-drivers: 'false'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "13"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-nvidia-cuda-13'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: "ubuntu:24.04"
|
||||
skip-drivers: 'false'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/amd64'
|
||||
platform-tag: 'amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-vulkan'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: "ubuntu:24.04"
|
||||
skip-drivers: 'false'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/arm64'
|
||||
platform-tag: 'arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-vulkan'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: "ubuntu:24.04"
|
||||
skip-drivers: 'false'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'intel'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"
|
||||
tag-suffix: '-gpu-intel'
|
||||
runs-on: 'ubuntu-latest'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
tags:
|
||||
- '*'
|
||||
|
||||
core-image-merge:
|
||||
# !cancelled(): without it, GHA's default `needs:` cascade skips the
|
||||
# merge whenever any matrix cell of the parent build fails or is
|
||||
# cancelled. Same fix as backend.yml's merge jobs — we still want to
|
||||
# publish the manifest list for tag-suffixes whose legs all succeeded.
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: core-image-build
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
concurrency:
|
||||
group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
gpu-vulkan-image-merge:
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: core-image-build
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-vulkan'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
jobs:
|
||||
hipblas-jobs:
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
ffmpeg: ${{ matrix.ffmpeg }}
|
||||
image-type: ${{ matrix.image-type }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
grpc-base-image: ${{ matrix.grpc-base-image }}
|
||||
aio: ${{ matrix.aio }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
latest-image: ${{ matrix.latest-image }}
|
||||
latest-image-aio: ${{ matrix.latest-image-aio }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
# Pushing with all jobs in parallel
|
||||
# eats the bandwidth of all the nodes
|
||||
max-parallel: 2
|
||||
matrix:
|
||||
include:
|
||||
- build-type: 'hipblas'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-hipblas-ffmpeg'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'extras'
|
||||
aio: "-aio-gpu-hipblas"
|
||||
base-image: "rocm/dev-ubuntu-22.04:6.1"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
latest-image: 'latest-gpu-hipblas'
|
||||
latest-image-aio: 'latest-aio-gpu-hipblas'
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'hipblas'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-hipblas'
|
||||
ffmpeg: 'false'
|
||||
image-type: 'extras'
|
||||
base-image: "rocm/dev-ubuntu-22.04:6.1"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'hipblas'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-hipblas-ffmpeg-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
base-image: "rocm/dev-ubuntu-22.04:6.1"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'hipblas'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-hipblas-core'
|
||||
ffmpeg: 'false'
|
||||
image-type: 'core'
|
||||
base-image: "rocm/dev-ubuntu-22.04:6.1"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
self-hosted-jobs:
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
ffmpeg: ${{ matrix.ffmpeg }}
|
||||
image-type: ${{ matrix.image-type }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
grpc-base-image: ${{ matrix.grpc-base-image }}
|
||||
aio: ${{ matrix.aio }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
latest-image: ${{ matrix.latest-image }}
|
||||
latest-image-aio: ${{ matrix.latest-image-aio }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
# Pushing with all jobs in parallel
|
||||
# eats the bandwidth of all the nodes
|
||||
max-parallel: ${{ github.event_name != 'pull_request' && 5 || 8 }}
|
||||
matrix:
|
||||
include:
|
||||
# Extra images
|
||||
- build-type: ''
|
||||
#platforms: 'linux/amd64,linux/arm64'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
ffmpeg: ''
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: ''
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-ffmpeg'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "11"
|
||||
cuda-minor-version: "7"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-cublas-cuda11'
|
||||
ffmpeg: ''
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-cublas-cuda12'
|
||||
ffmpeg: ''
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "11"
|
||||
cuda-minor-version: "7"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-cublas-cuda11-ffmpeg'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
aio: "-aio-gpu-nvidia-cuda-11"
|
||||
latest-image: 'latest-gpu-nvidia-cuda-11'
|
||||
latest-image-aio: 'latest-aio-gpu-nvidia-cuda-11'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-cublas-cuda12-ffmpeg'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
aio: "-aio-gpu-nvidia-cuda-12"
|
||||
latest-image: 'latest-gpu-nvidia-cuda-12'
|
||||
latest-image-aio: 'latest-aio-gpu-nvidia-cuda-12'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: ''
|
||||
#platforms: 'linux/amd64,linux/arm64'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
ffmpeg: ''
|
||||
image-type: 'extras'
|
||||
base-image: "ubuntu:22.04"
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'sycl_f16'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
tag-suffix: '-sycl-f16-ffmpeg'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
aio: "-aio-gpu-intel-f16"
|
||||
latest-image: 'latest-gpu-intel-f16'
|
||||
latest-image-aio: 'latest-aio-gpu-intel-f16'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'sycl_f32'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
tag-suffix: '-sycl-f32-ffmpeg'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'extras'
|
||||
runs-on: 'arc-runner-set'
|
||||
aio: "-aio-gpu-intel-f32"
|
||||
latest-image: 'latest-gpu-intel-f32'
|
||||
latest-image-aio: 'latest-aio-gpu-intel-f32'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
# Core images
|
||||
- build-type: 'sycl_f16'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
tag-suffix: '-sycl-f16-core'
|
||||
ffmpeg: 'false'
|
||||
image-type: 'core'
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'sycl_f32'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
tag-suffix: '-sycl-f32-core'
|
||||
ffmpeg: 'false'
|
||||
image-type: 'core'
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'sycl_f16'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
tag-suffix: '-sycl-f16-ffmpeg-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
- build-type: 'sycl_f32'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
|
||||
grpc-base-image: "ubuntu:22.04"
|
||||
tag-suffix: '-sycl-f32-ffmpeg-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
|
||||
# Single-arch server-image merges. Same conceptual fix as the backend
|
||||
# singletons in PR #9781: image_build.yml pushes by canonical digest
|
||||
# only, so without a downstream merge step there's no tag for consumers
|
||||
# (no :latest-gpu-nvidia-cuda-12, no :v<X>-gpu-nvidia-cuda-12, etc.).
|
||||
# Each merge job needs only its parent build matrix and is filtered by
|
||||
# tag-suffix in image_merge.yml's artifact-download pattern.
|
||||
gpu-nvidia-cuda-12-image-merge:
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: core-image-build
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-nvidia-cuda-12'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
gpu-nvidia-cuda-13-image-merge:
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: core-image-build
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-nvidia-cuda-13'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
gpu-intel-image-merge:
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: core-image-build
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-intel'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
gpu-hipblas-image-merge:
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: hipblas-jobs
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-hipblas'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
nvidia-l4t-arm64-image-merge:
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: gh-runner
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-nvidia-l4t-arm64'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
nvidia-l4t-arm64-cuda-13-image-merge:
|
||||
if: ${{ !cancelled() && github.repository == 'mudler/LocalAI' }}
|
||||
needs: gh-runner
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-nvidia-l4t-arm64-cuda-13'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
gh-runner:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
ubuntu-codename: ${{ matrix.ubuntu-codename }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-nvidia-l4t-arm64'
|
||||
base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'true'
|
||||
ubuntu-version: "2204"
|
||||
ubuntu-codename: 'jammy'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "13"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-nvidia-l4t-arm64-cuda-13'
|
||||
base-image: "ubuntu:24.04"
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
|
||||
core-image-build:
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
ffmpeg: ${{ matrix.ffmpeg }}
|
||||
image-type: ${{ matrix.image-type }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
aio: ${{ matrix.aio }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
grpc-base-image: ${{ matrix.grpc-base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
latest-image: ${{ matrix.latest-image }}
|
||||
latest-image-aio: ${{ matrix.latest-image-aio }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
max-parallel: ${{ github.event_name != 'pull_request' && 2 || 4 }}
|
||||
matrix:
|
||||
include:
|
||||
- build-type: ''
|
||||
platforms: 'linux/amd64,linux/arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-ffmpeg-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
base-image: "ubuntu:22.04"
|
||||
runs-on: 'arc-runner-set'
|
||||
aio: "-aio-cpu"
|
||||
latest-image: 'latest-cpu'
|
||||
latest-image-aio: 'latest-aio-cpu'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "11"
|
||||
cuda-minor-version: "7"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-cublas-cuda11-core'
|
||||
ffmpeg: ''
|
||||
image-type: 'core'
|
||||
base-image: "ubuntu:22.04"
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-cublas-cuda12-core'
|
||||
ffmpeg: ''
|
||||
image-type: 'core'
|
||||
base-image: "ubuntu:22.04"
|
||||
runs-on: 'arc-runner-set'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "11"
|
||||
cuda-minor-version: "7"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-cublas-cuda11-ffmpeg-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-cublas-cuda12-ffmpeg-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
skip-drivers: 'false'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-vulkan-ffmpeg-core'
|
||||
latest-image: 'latest-vulkan-ffmpeg-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
runs-on: 'arc-runner-set'
|
||||
base-image: "ubuntu:22.04"
|
||||
skip-drivers: 'false'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
gh-runner:
|
||||
uses: ./.github/workflows/image_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
ffmpeg: ${{ matrix.ffmpeg }}
|
||||
image-type: ${{ matrix.image-type }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
aio: ${{ matrix.aio }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
grpc-base-image: ${{ matrix.grpc-base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
latest-image: ${{ matrix.latest-image }}
|
||||
latest-image-aio: ${{ matrix.latest-image-aio }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "0"
|
||||
platforms: 'linux/arm64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-nvidia-l4t-arm64-core'
|
||||
latest-image: 'latest-nvidia-l4t-arm64-core'
|
||||
ffmpeg: 'true'
|
||||
image-type: 'core'
|
||||
base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'true'
|
||||
315
.github/workflows/image_build.yml
vendored
315
.github/workflows/image_build.yml
vendored
@@ -8,6 +8,11 @@ on:
|
||||
description: 'Base image'
|
||||
required: true
|
||||
type: string
|
||||
grpc-base-image:
|
||||
description: 'GRPC Base image, must be a compatible image with base-image'
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
build-type:
|
||||
description: 'Build type'
|
||||
default: ''
|
||||
@@ -18,33 +23,40 @@ on:
|
||||
type: string
|
||||
cuda-minor-version:
|
||||
description: 'CUDA minor version'
|
||||
default: "9"
|
||||
default: "4"
|
||||
type: string
|
||||
platforms:
|
||||
description: 'Platforms'
|
||||
default: ''
|
||||
type: string
|
||||
platform-tag:
|
||||
description: |
|
||||
Short tag identifying the platform leg, e.g. "amd64" or "arm64".
|
||||
Used to scope the per-arch registry cache and the digest artifact name.
|
||||
Optional during the migration; will be flipped to required: true once
|
||||
every caller passes an explicit value.
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
tag-latest:
|
||||
description: 'Tag latest'
|
||||
default: ''
|
||||
type: string
|
||||
latest-image:
|
||||
description: 'Tag latest'
|
||||
default: ''
|
||||
type: string
|
||||
latest-image-aio:
|
||||
description: 'Tag latest'
|
||||
default: ''
|
||||
type: string
|
||||
tag-suffix:
|
||||
description: 'Tag suffix'
|
||||
default: ''
|
||||
type: string
|
||||
ffmpeg:
|
||||
description: 'FFMPEG'
|
||||
default: ''
|
||||
type: string
|
||||
skip-drivers:
|
||||
description: 'Skip drivers by default'
|
||||
default: 'false'
|
||||
type: string
|
||||
image-type:
|
||||
description: 'Image type'
|
||||
default: ''
|
||||
type: string
|
||||
runs-on:
|
||||
description: 'Runs on'
|
||||
required: true
|
||||
@@ -55,15 +67,10 @@ on:
|
||||
required: false
|
||||
default: '--jobs=4 --output-sync=target'
|
||||
type: string
|
||||
ubuntu-version:
|
||||
description: 'Ubuntu version'
|
||||
aio:
|
||||
description: 'AIO Image Name'
|
||||
required: false
|
||||
default: '2204'
|
||||
type: string
|
||||
ubuntu-codename:
|
||||
description: 'Ubuntu codename'
|
||||
required: false
|
||||
default: 'noble'
|
||||
default: ''
|
||||
type: string
|
||||
secrets:
|
||||
dockerUsername:
|
||||
@@ -78,26 +85,63 @@ jobs:
|
||||
reusable_image-build:
|
||||
runs-on: ${{ inputs.runs-on }}
|
||||
steps:
|
||||
|
||||
- name: Force Install GIT latest
|
||||
run: |
|
||||
sudo apt-get update \
|
||||
&& sudo apt-get install -y software-properties-common \
|
||||
&& sudo apt-get update \
|
||||
&& sudo add-apt-repository -y ppa:git-core/ppa \
|
||||
&& sudo apt-get update \
|
||||
&& sudo apt-get install -y git
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v7
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Configure apt mirror on runner
|
||||
id: apt_mirror
|
||||
uses: ./.github/actions/configure-apt-mirror
|
||||
|
||||
- name: Free disk space
|
||||
uses: ./.github/actions/free-disk-space
|
||||
with:
|
||||
mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
|
||||
|
||||
- name: Set up build disk
|
||||
uses: ./.github/actions/setup-build-disk
|
||||
- name: Release space from worker
|
||||
if: inputs.runs-on == 'ubuntu-latest'
|
||||
run: |
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
df -h
|
||||
echo
|
||||
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
|
||||
sudo apt-get remove --auto-remove android-sdk-platform-tools || true
|
||||
sudo apt-get purge --auto-remove android-sdk-platform-tools || true
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo apt-get remove -y '^mono-.*' || true
|
||||
sudo apt-get remove -y '^ghc-.*' || true
|
||||
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
|
||||
sudo apt-get remove -y 'php.*' || true
|
||||
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
|
||||
sudo apt-get remove -y '^google-.*' || true
|
||||
sudo apt-get remove -y azure-cli || true
|
||||
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
|
||||
sudo apt-get remove -y '^gfortran-.*' || true
|
||||
sudo apt-get remove -y microsoft-edge-stable || true
|
||||
sudo apt-get remove -y firefox || true
|
||||
sudo apt-get remove -y powershell || true
|
||||
sudo apt-get remove -y r-base-core || true
|
||||
sudo apt-get autoremove -y
|
||||
sudo apt-get clean
|
||||
echo
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
sudo rm -rfv build || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf "/usr/local/share/boost" || true
|
||||
sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
|
||||
df -h
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/metadata-action@v6
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai
|
||||
@@ -106,24 +150,50 @@ jobs:
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
type=raw,value={{branch}}-{{date 'X'}}-{{sha}},enable={{is_default_branch}}
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
- name: Docker meta for PR
|
||||
id: meta_pull_request
|
||||
if: github.event_name == 'pull_request'
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/ci-tests
|
||||
tags: |
|
||||
type=ref,event=branch,suffix=localai${{ github.event.number }}-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}
|
||||
type=semver,pattern={{raw}},suffix=localai${{ github.event.number }}-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}
|
||||
type=sha,suffix=localai${{ github.event.number }}-${{ inputs.build-type }}-${{ inputs.cuda-major-version }}-${{ inputs.cuda-minor-version }}
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }}
|
||||
- name: Docker meta for PR
|
||||
id: meta_pull_request
|
||||
if: github.event_name == 'pull_request'
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: |
|
||||
ttl.sh/localai-ci-pr-${{ github.event.number }}
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }}
|
||||
- name: Docker meta AIO (quay.io)
|
||||
if: inputs.aio != ''
|
||||
id: meta_aio
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.aio }}
|
||||
|
||||
- name: Docker meta AIO (dockerhub)
|
||||
if: inputs.aio != ''
|
||||
id: meta_aio_dockerhub
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: |
|
||||
localai/localai
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
flavor: |
|
||||
suffix=${{ inputs.aio }}
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@master
|
||||
with:
|
||||
@@ -135,107 +205,142 @@ jobs:
|
||||
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.dockerUsername }}
|
||||
password: ${{ secrets.dockerPassword }}
|
||||
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
- name: Build and push by digest
|
||||
id: build
|
||||
uses: docker/build-push-action@v7
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v6
|
||||
if: github.event_name != 'pull_request'
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
# The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
|
||||
# This means that even the MAKEFLAGS have to be an EXACT match.
|
||||
# If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
|
||||
# This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
|
||||
build-args: |
|
||||
BUILD_TYPE=${{ inputs.build-type }}
|
||||
CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
|
||||
CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
|
||||
FFMPEG=${{ inputs.ffmpeg }}
|
||||
IMAGE_TYPE=${{ inputs.image-type }}
|
||||
BASE_IMAGE=${{ inputs.base-image }}
|
||||
GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
|
||||
GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
|
||||
GRPC_VERSION=v1.65.0
|
||||
MAKEFLAGS=${{ inputs.makeflags }}
|
||||
SKIP_DRIVERS=${{ inputs.skip-drivers }}
|
||||
UBUNTU_VERSION=${{ inputs.ubuntu-version }}
|
||||
UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
|
||||
APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
|
||||
cache-from: type=gha
|
||||
platforms: ${{ inputs.platforms }}
|
||||
outputs: |
|
||||
type=image,name=quay.io/go-skynet/local-ai,push-by-digest=true,name-canonical=true,push=true
|
||||
type=image,name=localai/localai,push-by-digest=true,name-canonical=true,push=true
|
||||
# See backend_build.yml for the rationale — provenance=mode=max
|
||||
# diverges the manifest-list digest per registry, breaking the
|
||||
# downstream imagetools create lookup.
|
||||
provenance: false
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Export digest
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
mkdir -p /tmp/digests
|
||||
digest="${{ steps.build.outputs.digest }}"
|
||||
touch "/tmp/digests/${digest#sha256:}"
|
||||
|
||||
# See .github/scripts/anchor-digest-in-cache.sh for why this is needed
|
||||
# and how it interacts with image_merge.yml's cleanup step. Mirrors the
|
||||
# same anchor in backend_build.yml — quay's per-repo manifest GC reaps
|
||||
# untagged manifests in local-ai before the merge runs.
|
||||
- name: Anchor digest in ci-cache so quay GC won't reap before merge
|
||||
if: github.event_name != 'pull_request'
|
||||
env:
|
||||
TAG_SUFFIX: ${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}
|
||||
PLATFORM_TAG: ${{ inputs.platform-tag || 'single' }}
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
SOURCE_IMAGE: quay.io/go-skynet/local-ai
|
||||
run: .github/scripts/anchor-digest-in-cache.sh
|
||||
|
||||
- name: Upload digest artifact
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
# `--` separator + 'single' placeholder for empty platform-tag —
|
||||
# same pattern as backend_build.yml. Prevents prefix collisions
|
||||
# in the merge-side glob (e.g. -nvidia-l4t-arm64 is a prefix of
|
||||
# -nvidia-l4t-arm64-cuda-13).
|
||||
name: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}--${{ inputs.platform-tag || 'single' }}
|
||||
path: /tmp/digests/*
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
### Start testing image
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v7
|
||||
uses: docker/build-push-action@v6
|
||||
if: github.event_name == 'pull_request'
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
# The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.
|
||||
# This means that even the MAKEFLAGS have to be an EXACT match.
|
||||
# If the build-args are not an EXACT match, it will result in a cache miss, which will require GRPC to be built from scratch.
|
||||
# This is why some build args like GRPC_VERSION and MAKEFLAGS are hardcoded
|
||||
build-args: |
|
||||
BUILD_TYPE=${{ inputs.build-type }}
|
||||
CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
|
||||
CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
|
||||
FFMPEG=${{ inputs.ffmpeg }}
|
||||
IMAGE_TYPE=${{ inputs.image-type }}
|
||||
BASE_IMAGE=${{ inputs.base-image }}
|
||||
GRPC_BASE_IMAGE=${{ inputs.grpc-base-image || inputs.base-image }}
|
||||
GRPC_MAKEFLAGS=--jobs=4 --output-sync=target
|
||||
GRPC_VERSION=v1.65.0
|
||||
MAKEFLAGS=${{ inputs.makeflags }}
|
||||
SKIP_DRIVERS=${{ inputs.skip-drivers }}
|
||||
UBUNTU_VERSION=${{ inputs.ubuntu-version }}
|
||||
UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
|
||||
APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
cache-from: type=gha
|
||||
platforms: ${{ inputs.platforms }}
|
||||
#push: true
|
||||
push: true
|
||||
tags: ${{ steps.meta_pull_request.outputs.tags }}
|
||||
labels: ${{ steps.meta_pull_request.outputs.labels }}
|
||||
- name: Testing image
|
||||
if: github.event_name == 'pull_request'
|
||||
run: |
|
||||
echo "Image is available at ttl.sh/localai-ci-pr-${{ github.event.number }}:${{ steps.meta_pull_request.outputs.version }}" >> $GITHUB_STEP_SUMMARY
|
||||
## End testing image
|
||||
- name: Build and push AIO image
|
||||
if: inputs.aio != ''
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
build-args: |
|
||||
BASE_IMAGE=quay.io/go-skynet/local-ai:${{ steps.meta.outputs.version }}
|
||||
MAKEFLAGS=${{ inputs.makeflags }}
|
||||
context: .
|
||||
file: ./Dockerfile.aio
|
||||
platforms: ${{ inputs.platforms }}
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.meta_aio.outputs.tags }}
|
||||
labels: ${{ steps.meta_aio.outputs.labels }}
|
||||
|
||||
- name: Build and push AIO image (dockerhub)
|
||||
if: inputs.aio != ''
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
build-args: |
|
||||
BASE_IMAGE=localai/localai:${{ steps.meta.outputs.version }}
|
||||
MAKEFLAGS=${{ inputs.makeflags }}
|
||||
context: .
|
||||
file: ./Dockerfile.aio
|
||||
platforms: ${{ inputs.platforms }}
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.meta_aio_dockerhub.outputs.tags }}
|
||||
labels: ${{ steps.meta_aio_dockerhub.outputs.labels }}
|
||||
|
||||
- name: Cleanup
|
||||
run: |
|
||||
docker builder prune -f
|
||||
docker system prune --force --volumes --all
|
||||
|
||||
- name: Latest tag
|
||||
# run this on branches, when it is a tag and there is a latest-image defined
|
||||
if: github.event_name != 'pull_request' && inputs.latest-image != '' && github.ref_type == 'tag'
|
||||
run: |
|
||||
docker pull localai/localai:${{ steps.meta.outputs.version }}
|
||||
docker tag localai/localai:${{ steps.meta.outputs.version }} localai/localai:${{ inputs.latest-image }}
|
||||
docker push localai/localai:${{ inputs.latest-image }}
|
||||
docker pull quay.io/go-skynet/local-ai:${{ steps.meta.outputs.version }}
|
||||
docker tag quay.io/go-skynet/local-ai:${{ steps.meta.outputs.version }} quay.io/go-skynet/local-ai:${{ inputs.latest-image }}
|
||||
docker push quay.io/go-skynet/local-ai:${{ inputs.latest-image }}
|
||||
- name: Latest AIO tag
|
||||
# run this on branches, when it is a tag and there is a latest-image defined
|
||||
if: github.event_name != 'pull_request' && inputs.latest-image-aio != '' && github.ref_type == 'tag'
|
||||
run: |
|
||||
docker pull localai/localai:${{ steps.meta_aio_dockerhub.outputs.version }}
|
||||
docker tag localai/localai:${{ steps.meta_aio_dockerhub.outputs.version }} localai/localai:${{ inputs.latest-image-aio }}
|
||||
docker push localai/localai:${{ inputs.latest-image-aio }}
|
||||
docker pull quay.io/go-skynet/local-ai:${{ steps.meta_aio.outputs.version }}
|
||||
docker tag quay.io/go-skynet/local-ai:${{ steps.meta_aio.outputs.version }} quay.io/go-skynet/local-ai:${{ inputs.latest-image-aio }}
|
||||
docker push quay.io/go-skynet/local-ai:${{ inputs.latest-image-aio }}
|
||||
|
||||
- name: job summary
|
||||
run: |
|
||||
echo "Built image: ${{ steps.meta.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
- name: job summary(AIO)
|
||||
if: inputs.aio != ''
|
||||
run: |
|
||||
echo "Built image: ${{ steps.meta_aio.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
146
.github/workflows/image_merge.yml
vendored
146
.github/workflows/image_merge.yml
vendored
@@ -1,146 +0,0 @@
|
||||
---
|
||||
name: 'merge LocalAI image manifest list (reusable)'
|
||||
|
||||
# Reusable workflow that joins per-arch digest artifacts (uploaded by
|
||||
# image_build.yml when called with platform-tag) into a single tagged
|
||||
# multi-arch manifest list.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag-latest:
|
||||
description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
tag-suffix:
|
||||
description: 'Image tag suffix (empty for core image). Used in artifact pattern with a -core placeholder for empty.'
|
||||
required: true
|
||||
type: string
|
||||
secrets:
|
||||
dockerUsername:
|
||||
required: false
|
||||
dockerPassword:
|
||||
required: false
|
||||
quayUsername:
|
||||
required: true
|
||||
quayPassword:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
merge:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
quay_username: ${{ secrets.quayUsername }}
|
||||
steps:
|
||||
# Sparse checkout: needed for .github/scripts/ (the keepalive cleanup
|
||||
# script). Skips the rest of the source tree.
|
||||
- name: Checkout (.github/scripts only)
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
sparse-checkout: |
|
||||
.github/scripts
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
- name: Download digests
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
# `--` separator anchors the glob so we don't over-match sibling
|
||||
# tag-suffixes (e.g. -nvidia-l4t-arm64 vs -nvidia-l4t-arm64-cuda-13).
|
||||
# Must stay in sync with image_build.yml's upload-artifact name.
|
||||
pattern: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}--*
|
||||
merge-multiple: true
|
||||
path: /tmp/digests
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
username: ${{ secrets.dockerUsername }}
|
||||
password: ${{ secrets.dockerPassword }}
|
||||
|
||||
- name: Login to Quay.io
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai
|
||||
localai/localai
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
type=raw,value={{branch}}-{{date 'X'}}-{{sha}},enable={{is_default_branch}}
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
|
||||
# Source from ci-cache, not local-ai. See backend_merge.yml for the
|
||||
# detailed rationale — quay's manifest GC is per-repository, so the
|
||||
# untagged digest in local-ai gets reaped while the same content lives
|
||||
# tagged under ci-cache (anchored by image_build.yml). buildx imagetools
|
||||
# create copies the manifest into local-ai (blobs already cross-mounted)
|
||||
# and publishes the manifest list with user-facing tags. End state in
|
||||
# local-ai is self-contained; no embedded reference to ci-cache.
|
||||
- name: Create manifest list and push (quay)
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '.tags | map(select(startswith("quay.io/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No quay.io tags from docker/metadata-action; skipping quay merge"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'quay.io/go-skynet/ci-cache@sha256:%s ' *)
|
||||
fi
|
||||
|
||||
- name: Create manifest list and push (dockerhub)
|
||||
if: github.event_name != 'pull_request'
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '.tags | map(select(startswith("localai/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'localai/localai@sha256:%s ' *)
|
||||
fi
|
||||
|
||||
- name: Inspect manifest
|
||||
run: |
|
||||
set -euo pipefail
|
||||
first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
|
||||
docker buildx imagetools inspect "$first_tag"
|
||||
fi
|
||||
|
||||
# See .github/scripts/cleanup-keepalive-tags.sh for the best-effort
|
||||
# semantics — fails soft when the registry credential isn't OAuth-scoped.
|
||||
- name: Cleanup keepalive tags in ci-cache
|
||||
if: github.event_name != 'pull_request' && success()
|
||||
env:
|
||||
TAG_SUFFIX: ${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}
|
||||
QUAY_TOKEN: ${{ secrets.quayPassword }}
|
||||
run: .github/scripts/cleanup-keepalive-tags.sh
|
||||
|
||||
- name: Job summary
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
|
||||
jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
|
||||
echo >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
|
||||
ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -9,4 +9,4 @@ jobs:
|
||||
pull-requests: write
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/labeler@v6
|
||||
- uses: actions/labeler@v5
|
||||
68
.github/workflows/lint.yml
vendored
68
.github/workflows/lint.yml
vendored
@@ -1,68 +0,0 @@
|
||||
---
|
||||
name: 'lint'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths-ignore:
|
||||
- 'docs/**'
|
||||
- 'examples/**'
|
||||
- 'README.md'
|
||||
- '**/*.md'
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: ci-lint-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
golangci-lint:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
# Full history so golangci-lint's new-from-merge-base can reach
|
||||
# origin/master and compute the diff against it.
|
||||
fetch-depth: 0
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.26.x'
|
||||
cache: false
|
||||
- name: install golangci-lint
|
||||
run: |
|
||||
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh \
|
||||
| sh -s -- -b "$(go env GOPATH)/bin" v2.11.4
|
||||
- name: generate grpc proto sources
|
||||
# pkg/grpc/proto/*.go is generated, not checked in. Several packages
|
||||
# import it, so without this step typecheck fails project-wide.
|
||||
run: make protogen-go
|
||||
- name: stub react-ui dist for go:embed
|
||||
# core/http/app.go has //go:embed react-ui/dist/*; the glob needs at
|
||||
# least one non-hidden entry to satisfy typecheck. We don't run
|
||||
# `make react-ui` here because lint doesn't need the real bundle.
|
||||
run: |
|
||||
mkdir -p core/http/react-ui/dist
|
||||
touch core/http/react-ui/dist/index.html
|
||||
- name: lint
|
||||
run: make lint
|
||||
|
||||
build-scripts:
|
||||
# The image packaging scripts encode invariants that only surface inside a
|
||||
# container build (a missing transitive dep, a partial cuDNN family). Their
|
||||
# shell tests need nothing but bash + gcc + ldd, so run them on every PR
|
||||
# rather than waiting on a multi-GB cross-arch backend image build.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: run packaging script tests
|
||||
run: make test-build-scripts
|
||||
|
||||
# The backend matrix path filter fails silently: a miss emits an empty
|
||||
# matrix, every job goes green, and the change reaches no image (#10946).
|
||||
# Its tests need only node, so they ride along with this job.
|
||||
- uses: actions/setup-node@v7
|
||||
with:
|
||||
node-version: '20'
|
||||
- name: run CI script tests
|
||||
run: make test-ci-scripts
|
||||
@@ -6,15 +6,14 @@ permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
packages: read
|
||||
issues: write # for Homebrew/actions/post-comment
|
||||
actions: write # to dispatch publish workflow
|
||||
|
||||
jobs:
|
||||
dependabot:
|
||||
if: github.repository == 'mudler/LocalAI' && github.actor == 'localai-bot' && contains(github.event.pull_request.title, 'chore:')
|
||||
runs-on: ubuntu-latest
|
||||
if: ${{ github.actor == 'localai-bot' }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Approve a PR if not already approved
|
||||
run: |
|
||||
@@ -1,29 +1,24 @@
|
||||
name: Notifications for new models
|
||||
on:
|
||||
pull_request_target:
|
||||
pull_request:
|
||||
types:
|
||||
- closed
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
|
||||
jobs:
|
||||
notify-discord:
|
||||
if: github.repository == 'mudler/LocalAI' && (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model'))
|
||||
if: ${{ (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model')) }}
|
||||
env:
|
||||
MODEL_NAME: gemma-3-12b-it-qat
|
||||
MODEL_NAME: hermes-2-theta-llama-3-8b
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0 # needed to checkout all branches for this Action to work
|
||||
ref: ${{ github.event.pull_request.head.sha }} # Checkout the PR head to get the actual changes
|
||||
- uses: mudler/localai-github-action@v1
|
||||
with:
|
||||
model: 'gemma-3-12b-it-qat' # Any from models.localai.io, or from huggingface.com with: "huggingface://<repository>/file"
|
||||
model: 'hermes-2-theta-llama-3-8b' # Any from models.localai.io, or from huggingface.com with: "huggingface://<repository>/file"
|
||||
# Check the PR diff using the current branch and the base branch of the PR
|
||||
- uses: GrantBirki/git-diff-action@v2.8.1
|
||||
- uses: GrantBirki/git-diff-action@v2.8.0
|
||||
id: git-diff-action
|
||||
with:
|
||||
json_diff_file_output: diff.json
|
||||
@@ -84,28 +79,27 @@ jobs:
|
||||
args: ${{ steps.summarize.outputs.message }}
|
||||
- name: Setup tmate session if fails
|
||||
if: ${{ failure() }}
|
||||
uses: mxschmitt/action-tmate@v3.23
|
||||
uses: mxschmitt/action-tmate@v3.19
|
||||
with:
|
||||
detached: true
|
||||
connect-timeout-seconds: 180
|
||||
limit-access-to-actor: true
|
||||
notify-twitter:
|
||||
if: github.repository == 'mudler/LocalAI' && (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model'))
|
||||
if: ${{ (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model')) }}
|
||||
env:
|
||||
MODEL_NAME: gemma-3-12b-it-qat
|
||||
MODEL_NAME: hermes-2-theta-llama-3-8b
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0 # needed to checkout all branches for this Action to work
|
||||
ref: ${{ github.event.pull_request.head.sha }} # Checkout the PR head to get the actual changes
|
||||
- name: Start LocalAI
|
||||
run: |
|
||||
echo "Starting LocalAI..."
|
||||
docker run -e -ti -d --name local-ai -p 8080:8080 localai/localai:master run --debug $MODEL_NAME
|
||||
docker run -e -ti -d --name local-ai -p 8080:8080 localai/localai:master-ffmpeg-core run --debug $MODEL_NAME
|
||||
until [ "`docker inspect -f {{.State.Health.Status}} local-ai`" == "healthy" ]; do echo "Waiting for container to be ready"; docker logs --tail 10 local-ai; sleep 2; done
|
||||
# Check the PR diff using the current branch and the base branch of the PR
|
||||
- uses: GrantBirki/git-diff-action@v2.8.1
|
||||
- uses: GrantBirki/git-diff-action@v2.8.0
|
||||
id: git-diff-action
|
||||
with:
|
||||
json_diff_file_output: diff.json
|
||||
@@ -167,7 +161,7 @@ jobs:
|
||||
TWITTER_ACCESS_TOKEN_SECRET: ${{ secrets.TWITTER_ACCESS_TOKEN_SECRET }}
|
||||
- name: Setup tmate session if fails
|
||||
if: ${{ failure() }}
|
||||
uses: mxschmitt/action-tmate@v3.23
|
||||
uses: mxschmitt/action-tmate@v3.19
|
||||
with:
|
||||
detached: true
|
||||
connect-timeout-seconds: 180
|
||||
6
.github/workflows/notify-releases.yaml
vendored
6
.github/workflows/notify-releases.yaml
vendored
@@ -6,17 +6,15 @@ on:
|
||||
|
||||
jobs:
|
||||
notify-discord:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
RELEASE_BODY: ${{ github.event.release.body }}
|
||||
RELEASE_TITLE: ${{ github.event.release.name }}
|
||||
RELEASE_TAG_NAME: ${{ github.event.release.tag_name }}
|
||||
MODEL_NAME: gemma-3-12b-it-qat
|
||||
steps:
|
||||
- uses: mudler/localai-github-action@v1
|
||||
with:
|
||||
model: 'gemma-3-12b-it-qat' # Any from models.localai.io, or from huggingface.com with: "huggingface://<repository>/file"
|
||||
model: 'hermes-2-theta-llama-3-8b' # Any from models.localai.io, or from huggingface.com with: "huggingface://<repository>/file"
|
||||
- name: Summarize
|
||||
id: summarize
|
||||
run: |
|
||||
@@ -62,4 +60,4 @@ jobs:
|
||||
DISCORD_AVATAR: "https://avatars.githubusercontent.com/u/139863280?v=4"
|
||||
uses: Ilshidur/action-discord@master
|
||||
with:
|
||||
args: ${{ steps.summarize.outputs.message }}
|
||||
args: ${{ steps.summarize.outputs.message }}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user